Latency Percentiles, Queueing, and Backpressure

Average latency can hide severe user experiences. p95 latency is the value below which 95% of measured requests complete; p99 highlights the slowest tail. Tail latency can be driven by garbage collection, noisy neighbors, queueing, slow dependencies, or a small number of overloaded partitions.

As utilization approaches capacity, queueing delay can rise sharply. A service that accepts work faster than it can process it accumulates a queue, increases memory usage, and eventually times out or crashes. Backpressure limits admission, slows producers, or rejects excess work before the system becomes unstable.

Use bounded queues, concurrency limits, load shedding, and explicit overload responses. A queue is not infinite storage. Track queue depth, oldest-message age, processing rate, and rejection rate to understand whether the system is recovering or falling further behind.

Key Takeaways
  • Track p95/p99, not only averages.
  • Bound queues and concurrency to prevent runaway resource use.
  • Backpressure protects the system by making overload visible and controlled.