What is log aggregation?
Centralizing log data from every system in one place
Log aggregation is the practice of collecting log events from every host, container, service, and cloud resource and storing them in a central system where they can be searched and analyzed.
Without aggregation, logs live on individual hosts (or inside containers that die). Investigating an issue means SSHing into 50 machines and running grep — slow, incomplete, and often impossible once containers have been garbage-collected.
With aggregation, you have one URL to query every log from every source for the past 7, 30, or 90 days. The MTTR difference during an incident is measured in hours.
Log aggregation is a non-negotiable requirement for cloud-native, containerized, or microservices-based systems. Even simpler stacks benefit significantly.
The components of a log aggregation pipeline
Four stages of production logging
Shippers / agents: lightweight processes that run on every host (or as a Kubernetes DaemonSet). Tail files, read container stdout, or receive syslog. Common choices: Filebeat, Fluent Bit, Vector, Fluentd, OpenTelemetry Collector.
Transport / buffer: optional buffer between shippers and ingest to absorb spikes. Kafka or Redis Streams are common. Not needed at low volume.
Ingest and storage: receives log events, indexes them, retains them for the configured period. Elasticsearch, OpenSearch, Loki, ClickHouse, or managed platforms (Atatus, Datadog, Splunk).
Query and UI: searching, filtering, and visualizing logs. Kibana, Grafana, Splunk Web, or managed vendor UI.
Also typical: alerting (fire on query thresholds), access control (RBAC per project / team), and retention policies (hot / warm / cold tiers).
Popular log aggregation stacks
The main architectural choices in 2026
ELK (Elasticsearch, Logstash, Kibana): the long-standing open-source default. Powerful but operationally heavy — Elasticsearch clusters need tuning. Elastic also offers Elastic Cloud (managed).
Grafana Loki: logs database designed for cost efficiency (indexes labels, not full text). Pairs with Grafana for the UI, Promtail or Fluent Bit for shipping.
ClickHouse: analytical database increasingly used for logs. SQL queries, columnar storage, excellent compression. Operated via Altinity Cloud or managed providers.
Managed platforms (Atatus, Datadog, Splunk, New Relic): no cluster to run; flat per-host or usage pricing. Fastest to adopt; variable cost at scale.
Cloud native: AWS CloudWatch Logs, Azure Log Analytics, GCP Logging. Works best for workloads already on that cloud.
Setup checklist
From "nothing centralized" to "logs flowing"
Install a shipper on every host or as a Kubernetes DaemonSet. One per node, auto-restarts on failure.
Standardize log format: emit JSON structured logs where you control the application. For logs you cannot change (nginx access, syslog), configure the shipper to parse.
Add consistent fields: service, env, host, trace_id, timestamp. Enforce via logging library wrappers.
Ship to one destination initially — the central ingest. Multi-destination (e.g., SIEM and APM) can wait.
Set retention tiers: hot (7 days, fast search), warm (30 days, cheaper), cold (90+ days, archive).
Mask PII at the shipper: email, SSN, credit card. Compliance depends on this.
Monitor the pipeline itself: are shippers running, is ingest keeping up, are queries fast? Logging the loggers is a thing.
Common log aggregation pitfalls
What goes wrong
No sampling: shipping 1 TB/day of DEBUG logs. Costs explode, signal is lost. Sample INFO and below.
Field explosion: logs with unbounded cardinality fields (every unique URL as a field name) blow up Elasticsearch indexes. Keep high-cardinality values in message, not indexed fields.
Secrets in logs: API keys, tokens, passwords leaking into centralized storage. Audit log output in CI.
No correlation with traces: logs without trace_id are 10x less useful than logs with it. Hook the logger to your OTel/APM context.
Underpowered search: raw logs in S3 with Athena is cheap but slow. For incident response you need sub-second query — pay for the hot tier.
No access control: production logs can contain PII, credentials, and sensitive business data. RBAC per team.
Log aggregation and the broader observability stack
Where logs fit alongside traces and metrics
Logs are one of the three pillars of observability (alongside metrics and traces). A mature stack has all three in one platform, correlated by trace_id.
If you only have one, make it logs — forensic detail is irreplaceable. But logs alone leave you blind to latency distributions (metrics) and request paths (traces).
Modern unified platforms (Atatus, Datadog, New Relic, Grafana Cloud) collapse the three; separate stacks (ELK + Prometheus + Jaeger) require manual stitching that slows every incident.
OpenTelemetry's log signal (OTel Logs) standardizes the correlation keys and the OTLP wire protocol across all three signals.
Key Takeaways
- Log aggregation = centralized storage of logs from every host, container, service, and cloud resource.
- Pipeline: shipper → transport → storage → query/UI → alerting.
- Stacks: ELK, Loki + Grafana, ClickHouse, managed (Atatus/Datadog/Splunk), native cloud (CloudWatch).
- Install shippers everywhere, standardize JSON log format, enforce consistent fields, mask PII.
- Retention tiers: hot (7 days), warm (30 days), cold (90+). Match to query frequency.
- Correlate with traces via trace_id — unified observability platforms make this one UI.