Queues, Messaging, Event-Driven Architecture, and Streams
Build asynchronous workflows with durable messaging, delivery semantics, ordering, and replayable event streams.
Queues, Topics, Brokers, and Consumers
A queue typically distributes each message to one consumer in a competing-consumer group, while a topic can fan out events to multiple subscribers. Product semantics vary, so understand the actual broker's retention, acknowledgment, ordering, and redelivery guarantees.
A broker buffers work and separates producer availability from consumer capacity. It does not guarantee that a consumer's database write and message acknowledgment happen atomically. If the consumer commits a database change and crashes before acknowledging, the message may be delivered again.
Use durable messages for work that must survive process failure. Define message schemas, ownership, retry policy, poison-message handling, and operational alerts. Monitor queue depth and age; a queue can be technically healthy while business work is delayed beyond usefulness.
- Understand queue versus topic semantics in the chosen broker.
- Design consumers for duplicate delivery.
- Queue age can matter more than queue depth for user outcomes.
Why should a message consumer usually be idempotent?
At-Most-Once, At-Least-Once, and Effectively-Once Processing
At-most-once delivery avoids duplicates by risking loss. At-least-once delivery retries until acknowledgment or policy limits, so duplicates are expected. Exactly-once claims are usually scoped to a particular broker transaction or processing boundary; they do not automatically cover arbitrary external side effects.
A common practical design is at-least-once delivery plus an idempotent consumer. Give each event a stable ID and record processed IDs in the same transaction as the business update. On duplicate delivery, the consumer detects the prior completion and avoids applying the effect twice.
For external actions such as sending money or calling a third-party API, use that provider's idempotency support where available and reconcile uncertain outcomes. A local “processed” flag written separately from the business update can itself race or fail, so place deduplication and state changes in one atomic boundary whenever possible.
async function handleEvent(event) {
await db.transaction(async (tx) => {
const alreadyProcessed = await tx.processedEvents.exists(event.id);
if (alreadyProcessed) return;
await applyBusinessChange(tx, event);
await tx.processedEvents.insert({ eventId: event.id });
});
// Acknowledge the message only after the transaction commits.
}- Assume duplicate delivery unless the full boundary proves otherwise.
- Deduplication and the business mutation should be atomic.
- Exactly-once guarantees must be scoped precisely.
Transactional Outbox, Sagas, and Event Sourcing
The dual-write problem occurs when an application updates a database and publishes a message separately: one operation can succeed while the other fails. The transactional outbox writes the business change and an outbox record in the same database transaction; a relay later publishes the record. Consumers still need idempotency.
A saga coordinates a business workflow across services using local transactions and compensating actions. For example, a travel booking may reserve a flight, reserve a hotel, and then confirm both; if the hotel reservation fails, the system releases the flight reservation. Compensation is a new business action, not a magical rollback of history.
Event sourcing stores state changes as an event log and derives current state by replay or projection. It supports auditability and alternate projections but introduces schema evolution, replay cost, event correction, and operational complexity. It is not required merely because a system publishes events.
- Use an outbox to avoid losing the link between a database commit and event publication.
- Sagas need explicit compensation and reconciliation.
- Event sourcing is a data-model choice with significant lifecycle costs.