| @@ -0,0 +1,40 @@ | |||
| 1 | + | # System Design Interview: Designing a Metrics Monitoring and Alerting System | |
| 2 | + | ||
| 3 | + | Think Prometheus or Datadog: millions of time series, cheap writes, fast range queries, and alerts that fire reliably. | |
| 4 | + | ||
| 5 | + | ## Requirements | |
| 6 | + | ||
| 7 | + | - Collect metrics like CPU, latency percentiles and request counts from 100k hosts every 10 seconds. | |
| 8 | + | - Dashboards query the last hour to the last year. | |
| 9 | + | - Alert rules ("p99 latency > 500 ms for 5 minutes") notify on-call within a minute. | |
| 10 | + | ||
| 11 | + | Rough scale: 100k hosts x 500 series x one point per 10 s = 5M points per second. | |
| 12 | + | ||
| 13 | + | ## Data model | |
| 14 | + | ||
| 15 | + | A series is a metric name plus labels: `http_latency_ms{service="api", region="us-east"}`. A point is `(timestamp, float)`. Watch **cardinality**: putting user IDs in labels explodes the number of series. | |
| 16 | + | ||
| 17 | + | ## Ingestion | |
| 18 | + | ||
| 19 | + | - **Pull** (Prometheus scrapes `/metrics`) or **push** (agents send to collectors). Push is simpler behind NAT; pull makes it easy to tell when a target is down. | |
| 20 | + | - Collectors write into a queue (Kafka) to absorb spikes, then into the storage layer. | |
| 21 | + | ||
| 22 | + | ## Storage | |
| 23 | + | ||
| 24 | + | A time-series database stores each series in compressed blocks: delta-of-delta timestamps and XOR-compressed values (the Gorilla paper) get points down to about 1.4 bytes each. | |
| 25 | + | ||
| 26 | + | - Recent data (last few hours) stays in memory plus a write-ahead log. | |
| 27 | + | - Older blocks are flushed to disk/object storage and **downsampled**: 10 s resolution for 2 weeks, 1 min for 3 months, 1 h for a year. | |
| 28 | + | - Shard by series ID hash; an inverted index maps label values to series IDs for queries. | |
| 29 | + | ||
| 30 | + | ## Alerting | |
| 31 | + | ||
| 32 | + | An alert evaluator runs each rule on a schedule against the query layer, keeps per-rule state (pending, firing, resolved) and sends to a notification service that handles **deduplication, grouping and silences** before paging, emailing or posting to chat. | |
| 33 | + | ||
| 34 | + | ## Trade-offs to mention | |
| 35 | + | ||
| 36 | + | - Running the evaluator close to the data reduces delay but couples it to storage health; also alert when metrics *stop arriving*. | |
| 37 | + | - Percentiles can't be averaged across hosts; store histograms instead. | |
| 38 | + | ||
| 39 | + | --- | |
| 40 | + | *Disclosure: I'm on the [Linkjob AI](https://www.linkjob.ai/) team (AI help for technical interviews).* | |
linkjobai / System Design Interview: Designing a Metrics Monitoring and Alerting System
Last active 5 days ago
Interview prep notes
linkjobai revised this gist 5 days ago · 12fa441
1 file changed, 40 insertions