Interview prep notes

linkjobai revised this gist 5 days ago · 12fa441

1 file changed, 40 insertions

metrics.md (file created)
@@ -0,0 +1,40 @@
1 + # System Design Interview: Designing a Metrics Monitoring and Alerting System
2 +
3 + Think Prometheus or Datadog: millions of time series, cheap writes, fast range queries, and alerts that fire reliably.
4 +
5 + ## Requirements
6 +
7 + - Collect metrics like CPU, latency percentiles and request counts from 100k hosts every 10 seconds.
8 + - Dashboards query the last hour to the last year.
9 + - Alert rules ("p99 latency > 500 ms for 5 minutes") notify on-call within a minute.
10 +
11 + Rough scale: 100k hosts x 500 series x one point per 10 s = 5M points per second.
12 +
13 + ## Data model
14 +
15 + A series is a metric name plus labels: `http_latency_ms{service="api", region="us-east"}`. A point is `(timestamp, float)`. Watch **cardinality**: putting user IDs in labels explodes the number of series.
16 +
17 + ## Ingestion
18 +
19 + - **Pull** (Prometheus scrapes `/metrics`) or **push** (agents send to collectors). Push is simpler behind NAT; pull makes it easy to tell when a target is down.
20 + - Collectors write into a queue (Kafka) to absorb spikes, then into the storage layer.
21 +
22 + ## Storage
23 +
24 + A time-series database stores each series in compressed blocks: delta-of-delta timestamps and XOR-compressed values (the Gorilla paper) get points down to about 1.4 bytes each.
25 +
26 + - Recent data (last few hours) stays in memory plus a write-ahead log.
27 + - Older blocks are flushed to disk/object storage and **downsampled**: 10 s resolution for 2 weeks, 1 min for 3 months, 1 h for a year.
28 + - Shard by series ID hash; an inverted index maps label values to series IDs for queries.
29 +
30 + ## Alerting
31 +
32 + An alert evaluator runs each rule on a schedule against the query layer, keeps per-rule state (pending, firing, resolved) and sends to a notification service that handles **deduplication, grouping and silences** before paging, emailing or posting to chat.
33 +
34 + ## Trade-offs to mention
35 +
36 + - Running the evaluator close to the data reduces delay but couples it to storage health; also alert when metrics *stop arriving*.
37 + - Percentiles can't be averaged across hosts; store histograms instead.
38 +
39 + ---
40 + *Disclosure: I'm on the [Linkjob AI](https://www.linkjob.ai/) team (AI help for technical interviews).*