Q33: How would you design a metrics and monitoring system?
Agents collect counters, gauges, histograms, and service health signals, then send them to a time-series backend. Store labels carefully because high-cardinality dimensions can explode storage and query cost. Build dashboards and alerts around user-visible SLIs and SLOs, not only CPU. Use aggregation/downsampling and retention tiers, and ensure the monitoring system itself has independent health checks and alert delivery.
Key Takeaways
- Histograms and percentiles reveal latency distributions.
- Control label cardinality and alert on actionable symptoms.