Backpressure in Action

Think of a log pipeline like a highway. When traffic flows freely, data cruises from the source through to the destination. But when the exit slows — a network blip, a struggling endpoint — data starts queuing on the road itself. Eventually the backup reaches the on-ramp and new data can’t even enter.

That’s backpressure. And it’s exactly what you want.

Drag the slider to control how long the light stays green and watch traffic back up:

FLOWING — TRAFFIC MOVING FREELY

Light Timing

Flowing — traffic moving freely

What’s the alternative?

Without backpressure, there’s nothing to stop data from piling up. An unbounded queue just keeps growing — and growing — until the process runs out of memory and crashes. All queued data is lost.

▶ Start Waiting…

Bounded channels trade a little throughput for a lot of safety. Instead of an OOM crash, you get a controlled slowdown — and data that can be retried survives intact.

Backpressure and data loss

When traffic backs up all the way to the on-ramp, new cars can’t enter the highway. In a log pipeline, this is the moment backpressure reaches the source — and what happens next depends entirely on the source type.

Pick your input type and watch what it feels as pressure builds:

Input	Under pressure	Data lost?
File	Reader pauses — logs stay safe on disk	No. Catches up from checkpoint on recovery
UDP / Syslog	OS socket buffer fills and overflows	Yes. Dropped packets are gone forever
OTLP	Returns 429 to senders, who back off	No. Senders retry with exponential backoff

File inputs are the safest — the filesystem is a durable, practically infinite buffer. UDP is inherently lossy under pressure because there’s no flow control. OTLP pushes the problem back to the sender, keeping both sides honest.

The key insight: bounded channels turn an OOM crash into a recoverable slowdown. Data that can be retried (files, OTLP) survives intact. Data that can’t (UDP) drops cleanly instead of taking the whole pipeline down with it.

How the cascade works

Every input gets two dedicated OS threads connected by a bounded channel:

┌──────────┐  bounded(4)  ┌──────────┐  pipeline(16)  ┌─────────────┐
│ I/O      │─────────────▶│ CPU      │───────────────▶│ Output Pool │
│ Worker   │              │ Worker   │                │ (MRU dispatch)│
│ poll()   │              │ scan()   │                │ try_send()   │
│ accum()  │              │ sql()    │                │ per worker(1)│
└──────────┘              └──────────┘                └─────────────┘
 OS thread                 OS thread                   async tasks

Each arrow is a bounded channel with a fixed capacity. When the downstream channel fills:

1. Output pool full → workers retry with exponential backoff, their per-worker channels fill
2. Pipeline channel full (16) → CPU worker blocks on blocking_send
3. I/O–CPU channel full (4) → I/O worker blocks, poll() stops being called
4. Source feels the pressure → the input-specific behavior from the table above kicks in

When the bottleneck clears, everything drains in order — file readers catch up from checkpointed offsets, and OTLP senders retry their held batches.