Message Queues

A user uploads a video. The request must transcode it, generate thumbnails, update the search index, and send an email.

Do it synchronously and the user waits ninety seconds, and any one failure fails the whole upload. Put a queue in the middle and the request returns immediately.

What the queue actually buys

Time decoupling. Producer and consumer need not be up simultaneously.

Rate decoupling. A burst is absorbed instead of dropped.

Failure isolation. The email service being down does not break uploads.

Fan-out. One event, many independent consumers.

Queue or topic

Queue — one consumer per message. Add consumers to go faster. Right when the message is a task.

Topic / pub-sub — every subscriber gets a copy. Right when the message is a fact that several parties care about. Adding a fifth consumer requires no change to the producer.

Kafka blurs these usefully: consumers in the same group share messages like a queue, and different groups each get everything like a topic.

The delivery guarantee

Three options, and only two of them exist.

At-most-once. Fire and forget. Fast, and messages are lost on failure.

At-least-once. Retry until acknowledged. Nothing is lost, and duplicates happen — a consumer that processes a message and dies before acknowledging will see it again.

Exactly-once is not achievable in the general case. A consumer and a broker cannot atomically agree that work was done, because the acknowledgement itself can be lost.

Ordering

Global ordering across a queue requires a single consumer, which removes the parallelism you built it for.

The workable compromise is ordering within a partition key. Kafka guarantees order within a partition; SQS FIFO within a message group. Choose a key such that things that must be ordered share one — user_id or order_id, usually — and accept that unrelated messages may interleave.

Log or broker

Traditional broker (RabbitMQ, SQS). The message is deleted when acknowledged. Rich routing, per-message acknowledgement, and the queue is meant to be empty.

Log (Kafka, Pulsar, Kinesis). Messages are appended and retained for a fixed period; consumers track their own offset. Replayable — a new consumer can read from the beginning, and a buggy one can be fixed and rewound.

That replay ability is the real difference, and it is why analytics and event sourcing use logs. The cost is that ordering is per-partition and consumers must manage offsets.

Two things to get right

The outbox pattern. Writing to the database and publishing a message are two operations, and the process can die between them.

// Both in one transaction. A separate process reads the outbox and publishes.
await db.transaction(async (tx) => {
  await tx.orders.insert(order);
  await tx.outbox.insert({topic: 'order.created', payload: order});
});

Without this you eventually get orders with no event, or events for orders that were rolled back. It is the single most valuable pattern in this article.

Monitor queue depth and consumer lag, not throughput. A growing queue means consumers cannot keep up, and it is the earliest warning you get — see backpressure for what to do about it.