# System Design Interview: Designing a Metrics Monitoring and Alerting System Think Prometheus or Datadog: millions of time series, cheap writes, fast range queries, and alerts that fire reliably. ## Requirements - Collect metrics like CPU, latency percentiles and request counts from 100k hosts every 10 seconds. - Dashboards query the last hour to the last year. - Alert rules ("p99 latency > 500 ms for 5 minutes") notify on-call within a minute. Rough scale: 100k hosts x 500 series x one point per 10 s = 5M points per second. ## Data model A series is a metric name plus labels: `http_latency_ms{service="api", region="us-east"}`. A point is `(timestamp, float)`. Watch **cardinality**: putting user IDs in labels explodes the number of series. ## Ingestion - **Pull** (Prometheus scrapes `/metrics`) or **push** (agents send to collectors). Push is simpler behind NAT; pull makes it easy to tell when a target is down. - Collectors write into a queue (Kafka) to absorb spikes, then into the storage layer. ## Storage A time-series database stores each series in compressed blocks: delta-of-delta timestamps and XOR-compressed values (the Gorilla paper) get points down to about 1.4 bytes each. - Recent data (last few hours) stays in memory plus a write-ahead log. - Older blocks are flushed to disk/object storage and **downsampled**: 10 s resolution for 2 weeks, 1 min for 3 months, 1 h for a year. - Shard by series ID hash; an inverted index maps label values to series IDs for queries. ## Alerting An alert evaluator runs each rule on a schedule against the query layer, keeps per-rule state (pending, firing, resolved) and sends to a notification service that handles **deduplication, grouping and silences** before paging, emailing or posting to chat. ## Trade-offs to mention - Running the evaluator close to the data reduces delay but couples it to storage health; also alert when metrics *stop arriving*. - Percentiles can't be averaged across hosts; store histograms instead. --- *Disclosure: I'm on the [Linkjob AI](https://www.linkjob.ai/) team (AI help for technical interviews).*