System Design

9) Observability & operations — detailed guide

Golden signals, metrics/logs/traces, SLO-based alerting, dashboards, safe rollouts, incident response, and autoscaling — with a Facebook-like Home Feed example in every section.

A) Golden signals (RED/USE): latency, traffic, errors, saturation

Two useful lenses

  • RED (user-facing endpoints): Requests, Errors, Duration (latency).
  • USE (infrastructure/dependencies): Utilization, Saturation, Errors.

What each signal means (and common pitfalls)

  • Latency: track p50/p95/p99 server-side and end-to-end. Avoid averages; retries inflate tails.
  • Traffic: RPS/QPS, concurrency (in-flight), bytes in/out. Separate reads vs writes.
  • Errors: 5xx, timeouts, dependency errors, and business failures (e.g., “no feed candidates”).
  • Saturation: CPU, memory/GC, thread pools, DB pool, queue depth, disk IO. CPU alone lies.
Golden signals — template (how to answer fast)
Template
• RED for endpoints: request rate, error rate, duration (p95/p99). • USE for dependencies: utilization, saturation, errors (DB, cache, queues). • Always include: tail latency + saturation (p95 can degrade before errors). • Pitfall checks: retries, timeouts, cardinality explosions.
Facebook-like Home Feed — golden signals we publish
Example
RED (endpoints): • GET /feed: RPS, p95/p99, 5xx + timeout rate, “empty feed” rate • POST /like/comment: RPS, p95, error rate USE (dependencies): • Redis: hit rate, latency, CPU, evictions • DB: pool usage, slow queries, replica lag • Kafka: producer errors, consumer lag, DLQ rate • Ranking service: p95, error rate, breaker opens
↑ Back to top

B) Metrics, logs & traces (sampling, exemplars, cardinality)

Metrics (what to standardize)

  • Prefer histograms for latency, not averages.
  • Standard dimensions: service, route, status_class, region, env, dependency.
  • Cardinality guardrails: never label by userId/requestId; sample or aggregate those.

Logs (what “good” looks like)

  • Structured JSON logs with: trace_id, span_id, request_id, route, tenant, outcome, latency, dependency errors.
  • Redact PII at source; never log secrets/tokens; sample INFO if needed, but keep ERROR.

Traces (and sampling strategy)

  • Propagate W3C TraceContext across HTTP + async boundaries (queues/streams).
  • Use head-based sampling at ingress (cheap) + tail-based “keep slow/error” in collector.
  • Add exemplars so a bad p95/p99 point links to a representative trace.
Signals — template (what to say)
Template
• Metrics: RED/USE, histograms for latency, avoid high-card labels. • Logs: structured JSON, trace/span IDs, PII redaction. • Traces: W3C propagation across all boundaries; head + tail sampling. • Exemplars: click from a metric spike to the exact trace.
Facebook-like Home Feed — tracing & logging practice
Example
• Every request gets request_id + trace_id. Route is normalized ("/feed/:userId" becomes "/feed/{userId}"). • Tail sampling keeps: errors, timeouts, and top 1% slow traces. • Async: Kafka messages carry traceparent; consumers create child spans. • Logs redact PII; “user_id” appears only as a hashed surrogate when needed.
↑ Back to top

C) SLOs, error budgets, and alerts (page on burn, not noise)

Define SLIs/SLOs clearly

  • Availability SLI: good / total from the same population (avoid mixing sources).
  • Latency SLI: e.g., “p95 < 200ms” for GET /feed; define success (exclude cancels?) explicitly.
  • Error budget = 1 − SLO (99.9% → 0.1% budget).

Burn-rate alerting (multi-window)

  • Fast page: detects sharp outages; Slow ticket: catches gradual regressions.
  • Pair windows (e.g., 10m and 1h) to reduce flapping.
Burn rate — template math
Template
error_budget_fraction = 1 - SLO

burn_rate ≈ error_rate / error_budget_fraction

Typical approach:
• Page: 10m burn_rate high AND 1h burn_rate high
• Ticket: 6h burn_rate moderate
SLO alerting — template (simple, interview-safe)
Template
• Define a small set of user-journey SLOs (not 200 micro-SLOs). • Alert on error budget burn with multi-window rules (page vs ticket). • Use guardrails: if burn is high, freeze risky releases and enable brownout flags. • Keep on-call humane: reduce noise; focus on user impact.
Facebook-like Home Feed — SLOs & paging rules
Example
SLOs: • GET /feed availability: 99.9% monthly • GET /feed latency: p95 ≤ 200ms • “Freshness”: 99% of feed events processed within 5s Alerts: • Page when fast+slow burn indicate we’ll consume 1 day of budget quickly. • Ticket when burn is moderate over 6–24h (slow regression). • If budget burn is high: auto rollback canary + enable brownout (serve cached feed).
↑ Back to top

D) Dashboards & drill-down flows (metrics → traces → logs)

Dashboard structure (simple and effective)

  • Top: SLO status + error budget burn + p95/p99 + error rate.
  • Middle: dependency panels (Redis/DB/Kafka) with saturation + latency.
  • Bottom: “Drill-down” links: exemplar trace links, log queries, runbook links.
Dashboards — template (what to build)
Template
• One “service overview” dashboard: SLO + golden signals + top dependencies. • One “dependency deep dive” dashboard: DB/Redis/Kafka with saturation + latency. • One-click drill-down: click from metric series to exemplar trace, then to logs. • Add runbook links beside the panels (“what to do if this spikes”).
Facebook-like Home Feed — drill-down flow
Example
If GET /feed p99 spikes: 1) Overview shows burn rate + which region/route is impacted. 2) Dependency panel shows Redis latency up + hit rate down. 3) Click exemplar trace → see slow spans in “Redis GET candidates”. 4) Click logs for that trace_id → confirm timeouts and circuit breaker opens. 5) Runbook suggests: enable cached feed + scale Redis / reduce query fanout.
↑ Back to top

E) Rollouts: canary, blue/green, feature flags (ship safely)

Progressive delivery (canary)

  • Rollout steps: 1% → 5% → 25% → 100% with automated gates.
  • Gate on: p95 latency delta, error delta, saturation delta, SLO burn delta.
  • Shadow traffic for read-only diff testing (where safe).

Feature flags (operational superpower)

  • Kill switch, % rollout, per-tenant allowlist; default-off for risky features.
  • Audit changes; short TTL configs; don’t let flags live forever (cleanup checklist).
Rollouts — template (how to describe)
Template
• Canary with automated gates: latency/error/saturation deltas + SLO burn. • Blue/green when you want instant full rollback at the edge. • Flags: kill switch + % rollout + tenant allowlist; audit trail; cleanup plan. • Always have a safe fallback mode (cached/stale) for user-facing reads.
Facebook-like Home Feed — safe rollout of new ranking model
Example
• Canary new model to 1% of users in one region. • Gate: if p95 ↑ > 10% or error rate ↑ > 0.05% → auto rollback. • Feature flag to disable “expensive embedding enrichment” instantly without redeploy. • Shadow compare: old vs new ranking output for offline diffing.
↑ Back to top

F) Incident response, runbooks, postmortems (operate like a pro)

During an incident

  • Roles: IC, Comms, Scribe, SMEs (DB/Cache/App).
  • Flow: declare severity → stabilize (rollback/brownout/rate-limit) → diagnose → fix → verify.
  • Hygiene: IC controls changes; update cadence 10–15 minutes.

Runbooks (what they should contain)

  • Trigger thresholds, scope checks, decision tree, links/commands, rollback steps, owner/escalation.
  • Verification after each action (“did p95 improve?”).

Postmortems (blameless, actionable)

  • Timeline, impact, SLIs, contributing factors, what worked/failed, actions with owners and due dates.
Incidents — template (interview-ready)
Template
• Declare severity + assign roles (IC/comms/scribe/SMEs). • Stabilize: rollback, rate-limit, brownout/bailout, increase capacity. • Diagnose with drill-down: metrics → traces → logs. • Postmortem: blameless, actions tracked like features with owners + due dates.
Facebook-like Home Feed — incident scenario
Example
Symptom: feed p99 spikes + 5xx increases. • Stabilize: enable cached feed brownout; open breaker to ranking-service; rollback canary. • Diagnose: Redis hit rate down + slow spans in “fetch candidates”. • Fix: scale Redis, reduce key cardinality, adjust TTLs. • PM: add guardrail alert on hit-rate collapse + add runbook step for fast brownout.
↑ Back to top

G) Capacity planning & autoscaling (scale on the right signals)

Capacity planning basics

  • Concurrency ≈ RPS × latency (Little’s Law intuition).
  • Keep utilization at peak ~60–70% so you have headroom for spikes and failures.
  • Model dependencies: DB pool size, cache capacity, partition counts, connection limits.

Autoscaling strategies

  • APIs: HPA on CPU + custom metrics (RPS/pod, p95). Add stabilization windows.
  • Consumers: scale on lag + processing time; cap in-flight per partition; pre-warm for known spikes.
  • Stateful: scale reads via replicas; be careful with connection pools and cache warmup.
  • Pitfalls: CPU-only scaling, no max caps, scaling the wrong tier.
Autoscaling — template (what to say)
Template
• Plan with concurrency and headroom; validate against peak traffic. • Autoscale APIs on RPS/pod + p95 (plus CPU), with stabilization to avoid thrash. • Autoscale consumers on lag + processing time; cap in-flight per partition. • Align limits: connection pools, queue depths, and upstream timeouts.
Facebook-like Home Feed — scaling plan
Example
• Feed API scales on RPS/pod and p95 latency; cooldown prevents flapping. • Ranking consumers scale on Kafka lag; cap in-flight to protect DB and Redis. • DB read replicas scale for feed reads; connection pools are bounded and coordinated with pod count.
↑ Back to top

Practical snippets (quote/adapt)

PromQL-ish — quick templates
Template
# Error rate (5xx)
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))

# Latency (p95) from histogram
histogram_quantile(
  0.95,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, route)
)

# Saturation: DB pool usage ratio
max(db_pool_in_use) / max(db_pool_capacity)

# Kafka consumer lag (conceptual)
max(kafka_consumer_group_lag{group="feed-ranker"})
Alert hygiene — template checklist
Template
• Every alert has: owner, severity, runbook link, and “what good looks like”. • Page only on user impact (SLO burn). Everything else → ticket or dashboard. • Avoid flapping: multi-window burn, for: durations, and stabilization windows.
Facebook-like Home Feed — quick runbook links
Example
• “Feed p99 spike” runbook: - Check SLO burn dashboard - Check Redis hit rate + latency - Open exemplar trace (slow span) - Enable cached feed brownout if needed - Rollback latest canary if regression is suspected
↑ Back to top

Interview one-liners

  • “Page on SLO burn, not raw metrics.”
  • “Metrics detect, traces explain, logs provide detail.”
  • “Exemplars link a bad p99 to the exact trace.”
  • “Canary + automated gates + instant rollback via flags.”
  • “Runbooks are decision trees; postmortems are blameless with tracked actions.”
  • “Autoscale on queue depth, lag, RPS/pod, and p95 — not just CPU.”
One clean closing line (template)
Template
“I design observability so we can detect in minutes, mitigate in one action, and root-cause with metrics→traces→logs.”
Facebook-like Home Feed — 20-second observability answer
Example
“We publish RED for endpoints and USE for dependencies, alert on error-budget burn, and keep dashboards drillable: metrics to exemplars to traces and logs. We ship with canaries gated on SLO deltas, run incidents with clear roles, and scale on meaningful signals like lag and RPS/pod.”
↑ Back to top