Backpressure

A producer sends 1,000 messages per second. A consumer handles 900. The difference has to go somewhere.

It goes into a buffer. The buffer grows by 100 per second, forever, until memory runs out — and long before that, every message in it is waiting behind a queue of messages that have already timed out.

The arithmetic is brutal

Buffers do not solve it

The instinct on seeing a full buffer is to make it bigger. That converts a fast failure into a slow one.

The four responses

When you cannot keep up, there are only four things to do.

1. Slow the producer down. The best option when available. A bounded queue that blocks on put() does this automatically; TCP does it with its receive window; reactive streams do it by having the consumer request n items.

2. Drop. Shed load deliberately. A 429 or 503 returned in one millisecond is far better than a timeout after thirty seconds. Drop the newest if old work is still valuable, or the oldest if freshness matters — for live telemetry, dropping the oldest is almost always right.

3. Scale. Add consumers. Correct, and it takes seconds to minutes, so you still need one of the others for the interim.

4. Degrade. Serve something cheaper: a cached response, fewer results, no personalisation.

Where it has to be end to end

Backpressure only works if it propagates all the way to the source.

An HTTP handler that accepts a request, queues it, and returns 202 has removed its backpressure — the client has no idea the system is struggling and will happily send more. The queue absorbs it, then the queue fills, and the failure appears somewhere with no context.

The chain has to hold: client → load balancer → service → queue → worker → database. One unbounded buffer anywhere in that chain breaks the whole mechanism, because it will happily accept everything and hide the signal.

What to do concretely

Bound every queue. An unbounded queue is a memory leak with a scheduling policy. Pick a size from your latency budget: if you promise 200 ms and process 100/s, a queue longer than 20 is already broken.

Set timeouts, and check the deadline before working. If a request has been queued longer than the client’s timeout, drop it — the answer is worthless and the work is pure loss. This one change often recovers a system in overload.

Add a circuit breaker so a struggling dependency stops receiving traffic rather than accumulating timeouts.

Alert on queue depth, not just throughput. Throughput looks fine right up until it does not; queue depth is the leading indicator.