Topics, Partitions, Offsets, and Consumer Groups

A partitioned event log preserves record order within a partition. A consumer group assigns partitions among its members; adding more consumers than partitions does not increase parallelism for that topic beyond available partitions. Partition keys determine both distribution and the ordering boundary.

Offsets identify positions in a partition. Committing an offset before side effects are durable can lose work after a crash; committing after side effects can cause duplicate processing. Coordinate offsets with the sink or use idempotent consumers. Increasing partition count can change key-to-partition mapping and should be evaluated against ordering assumptions.

Monitor lag by partition and its age, not only group-wide totals. A single hot partition may be far behind while the average looks healthy.

Key Takeaways
  • Ordering is normally per partition, not global.
  • Offset timing affects loss and duplicate behavior.
  • Partition keys control locality, ordering, and skew.