Design Problems/Metrics/Monitoring System (like Datadog)

Metrics/Monitoring System (like Datadog)

Hard
Time-seriesAggregationAlerting

High-Level Architecture

Rendering diagram...

Design a metrics collection, storage, and monitoring system similar to Datadog or Prometheus. The system ingests millions of time-series data points per second from thousands of services, stores them efficiently, supports real-time dashboards, and triggers alerts when anomalies are detected.

Clarifying Questions

  • How many metrics per second? (Assume 10M data points/sec)
  • Retention period? (Raw: 7 days, 1-min aggregates: 30 days, 1-hour: 1 year)
  • What alerting latency is acceptable? (< 30 seconds from metric emission to alert)
  • Custom metrics or predefined? (Both — users define custom metrics)
  • Dashboard query latency? (< 2 seconds for 24h window, < 5s for 7d)