Event Time, Processing Time, Windows, and Watermarks
Processing time is when a pipeline handles an event; event time is when the event occurred. Offline devices, network delays, and replay can make these values diverge. Tumbling, sliding, and session windows group events for aggregation.
A watermark estimates how far event-time processing has progressed and defines a lateness policy; it does not prove that no later event will arrive. Late events may be dropped, routed for correction, or used to update prior aggregates. The right choice depends on whether the output is exploratory analytics, billing, fraud detection, or a user-facing counter.
Replay, clock skew, duplicate events, and out-of-order delivery should be tested explicitly. Define when results become final and how corrections propagate to dashboards and downstream consumers.
- Event time and processing time are different.
- Watermarks are progress estimates, not guarantees.
- Define late-event correction semantics.