System Design Interview: Designing a Metrics Monitoring and Alerting System
Think Prometheus or Datadog: millions of time series, cheap writes, fast range queries, and alerts that fire reliably.
Requirements
- Collect metrics like CPU, latency percentiles and request counts from 100k hosts every 10 seconds.
- Dashboards query the last hour to the last year.
- Alert rules ("p99 latency > 500 ms for 5 minutes") notify on-call within a minute.
Rough scale: 100k hosts x 500 series x one point per 10 s = 5M points per second.
Data model
A series is a metric name plus labels: http_latency_ms{service="api", region="us-east"}. A point is (timestamp, float). Watch cardinality: putting user IDs in labels explodes the number of series.
Ingestion
- Pull (Prometheus scrapes
/metrics) or push (agents send to collectors). Push is simpler behind NAT; pull makes it easy to tell when a target is down. - Collectors write into a queue (Kafka) to absorb spikes, then into the storage layer.
Storage
A time-series database stores each series in compressed blocks: delta-of-delta timestamps and XOR-compressed values (the Gorilla paper) get points down to about 1.4 bytes each.
- Recent data (last few hours) stays in memory plus a write-ahead log.
- Older blocks are flushed to disk/object storage and downsampled: 10 s resolution for 2 weeks, 1 min for 3 months, 1 h for a year.
- Shard by series ID hash; an inverted index maps label values to series IDs for queries.
Alerting
An alert evaluator runs each rule on a schedule against the query layer, keeps per-rule state (pending, firing, resolved) and sends to a notification service that handles deduplication, grouping and silences before paging, emailing or posting to chat.
Trade-offs to mention
- Running the evaluator close to the data reduces delay but couples it to storage health; also alert when metrics stop arriving.
- Percentiles can't be averaged across hosts; store histograms instead.
Disclosure: I'm on the Linkjob AI team (AI help for technical interviews).