9. Queues, Events, and Asynchronous Work
Use a queue when work can be delayed, producers and consumers need independent scaling, spikes must be buffered, or processing should happen outside the request path.
Use event streaming when multiple consumers need a durable event history, replay, partitioned ordering, or continuous processing.
Queue checklist
- Delivery guarantees and visibility timeout.
- Retry policy, exponential backoff, and maximum attempts.
- Dead-letter queue and operational replay procedure.
- Idempotent consumer behavior.
- Message size limits and schema evolution.
- Ordering requirements and partition key.
- Queue depth, oldest-message age, and processing rate.
Transactional outbox pattern Write the business change and an outbox event in the same database transaction. A publisher later delivers the event to the broker. Consumers should still be idempotent because publication and delivery can be retried.
Saga pattern Use a sequence of local transactions for a multi-service business workflow. If a later step fails, execute compensating actions where appropriate. Compensation is a business operation, not always a literal rollback.
Backpressure When consumers cannot keep up, bound queues, slow producers, reject excess work, or shed nonessential load. An unbounded queue turns overload into increasing latency and eventual resource exhaustion.
Use synchronous requests when the caller requires an immediate result and the work is bounded. Use asynchronous processing when delayed completion is acceptable and the system can expose status or completion notifications.
- Queues buffer load but do not create unlimited capacity.
- Monitor queue age and processing lag, not only queue length.
- Make consumers idempotent and define a dead-letter recovery process.
- Use outbox and saga patterns for appropriate cross-service workflows.