Interview prep notes

Revision 12fa441ecc8da23780ce6f9d71a5673a80ca0f5b

metrics.md Raw

System Design Interview: Designing a Metrics Monitoring and Alerting System

Think Prometheus or Datadog: millions of time series, cheap writes, fast range queries, and alerts that fire reliably.

Requirements

  • Collect metrics like CPU, latency percentiles and request counts from 100k hosts every 10 seconds.
  • Dashboards query the last hour to the last year.
  • Alert rules ("p99 latency > 500 ms for 5 minutes") notify on-call within a minute.

Rough scale: 100k hosts x 500 series x one point per 10 s = 5M points per second.

Data model

A series is a metric name plus labels: http_latency_ms{service="api", region="us-east"}. A point is (timestamp, float). Watch cardinality: putting user IDs in labels explodes the number of series.

Ingestion

  • Pull (Prometheus scrapes /metrics) or push (agents send to collectors). Push is simpler behind NAT; pull makes it easy to tell when a target is down.
  • Collectors write into a queue (Kafka) to absorb spikes, then into the storage layer.

Storage

A time-series database stores each series in compressed blocks: delta-of-delta timestamps and XOR-compressed values (the Gorilla paper) get points down to about 1.4 bytes each.

  • Recent data (last few hours) stays in memory plus a write-ahead log.
  • Older blocks are flushed to disk/object storage and downsampled: 10 s resolution for 2 weeks, 1 min for 3 months, 1 h for a year.
  • Shard by series ID hash; an inverted index maps label values to series IDs for queries.

Alerting

An alert evaluator runs each rule on a schedule against the query layer, keeps per-rule state (pending, firing, resolved) and sends to a notification service that handles deduplication, grouping and silences before paging, emailing or posting to chat.

Trade-offs to mention

  • Running the evaluator close to the data reduces delay but couples it to storage health; also alert when metrics stop arriving.
  • Percentiles can't be averaged across hosts; store histograms instead.

Disclosure: I'm on the Linkjob AI team (AI help for technical interviews).