Event Streaming and Real-Time Stream Processing
Learn partitioned event logs, consumer groups, event-time windows, schema evolution, replay, and lag management.
Topics, Partitions, Offsets, and Consumer Groups
A partitioned event log preserves record order within a partition. A consumer group assigns partitions among its members; adding more consumers than partitions does not increase parallelism for that topic beyond available partitions. Partition keys determine both distribution and the ordering boundary.
Offsets identify positions in a partition. Committing an offset before side effects are durable can lose work after a crash; committing after side effects can cause duplicate processing. Coordinate offsets with the sink or use idempotent consumers. Increasing partition count can change key-to-partition mapping and should be evaluated against ordering assumptions.
Monitor lag by partition and its age, not only group-wide totals. A single hot partition may be far behind while the average looks healthy.
- Ordering is normally per partition, not global.
- Offset timing affects loss and duplicate behavior.
- Partition keys control locality, ordering, and skew.
Event Time, Processing Time, Windows, and Watermarks
Processing time is when a pipeline handles an event; event time is when the event occurred. Offline devices, network delays, and replay can make these values diverge. Tumbling, sliding, and session windows group events for aggregation.
A watermark estimates how far event-time processing has progressed and defines a lateness policy; it does not prove that no later event will arrive. Late events may be dropped, routed for correction, or used to update prior aggregates. The right choice depends on whether the output is exploratory analytics, billing, fraud detection, or a user-facing counter.
Replay, clock skew, duplicate events, and out-of-order delivery should be tested explicitly. Define when results become final and how corrections propagate to dashboards and downstream consumers.
- Event time and processing time are different.
- Watermarks are progress estimates, not guarantees.
- Define late-event correction semantics.
Schema Evolution, Replay, and Backpressure
Events are long-lived contracts because producers and consumers deploy independently. Additive optional fields are usually easier to roll out than changing a field's meaning. Schema registries can enforce compatibility, but policies must match actual consumer behavior.
Replay can rebuild projections or repair bugs, but may overload databases or repeat external notifications. Separate replay from live side effects, rate-limit backfills, and use versioned projections when possible. Consumers should be idempotent and observable.
When producers outpace consumers, lag grows. Scale consumers only if partition parallelism and downstream capacity permit it. Monitor oldest-event age, processing rate, failures, and skew; queue depth alone does not show whether business deadlines are being missed.
- Event schemas need compatibility policies.
- Replay requires side-effect controls.
- Scaling consumers cannot overcome every downstream bottleneck.