The Three Phases of Log Pipeline Maturity
Most organisations go through the same evolution: 1) grep everything manually, 2) ship to a centralised store, 3) build structured pipelines with alerting and cost controls. We help clients jump directly to phase 3 — but understanding what goes wrong in phase 2 is instructive.
Anti-Pattern: The Single Fat Index
The most expensive mistake in log management is dumping all logs into one index. You lose the ability to set per-application retention policies, tuning becomes impossible, and a single noisy service can crowd out others.
A single monolithic index will eventually hit the 2-billion-document soft limit and cause shard imbalance. Always use data streams with per-service index templates.
Pattern: Structured Logging at the Source
Parsing unstructured text in Logstash is expensive. The cheapest parse is no parse — emit JSON at the application level. Work with development teams to standardise on a shared log schema (timestamp, level, service, trace_id, message) and you'll spend far less on ingest processing.
{
"timestamp": "2026-04-22T14:32:01.123Z",
"level": "ERROR",
"service": "payment-service",
"trace_id": "7f3b2a1c-...",
"message": "Charge failed",
"error_code": "CARD_DECLINED",
"user_id": "u_8921"
}Pattern: Backpressure and Buffering
Ingest spikes are guaranteed — a deployment, a traffic surge, a batch job. Your pipeline must absorb them without dropping logs. Kafka or a persistent queue in front of Logstash provides the buffer; configure dead-letter queues for malformed events.