System Design

14) Testing & quality at scale — detailed guide

A practical playbook for unit/integration/contract tests, load & chaos, data quality, shadow/replay, and CI/CD gates—written as reusable templates with a single consistent example.

Example context
Facebook-like Home Feed
Read path
GET /feed → fetch candidate postIds → hydrate posts/media → rank → rules → cursor
Write path
POST /post → validate → store → async fanout/index/counters/notifications
Engagement
POST /like | /comment → idempotent write → counters → ranking features update
Common risk areas
  • celebrity fanout storms (write amplification)
  • cache stampedes and thundering herd
  • ranking regressions (latency + relevance)
  • data drift / schema drift in events/features

A) Unit, integration, and contract testing

Build confidence with a predictable pyramid: fast units, real integrations, and CDC contracts so teams can ship independently.

What you’re trying to prove

Goal
  • Correctness: business logic is right (units) and your wiring/config is right (integration).
  • Compatibility: producers/consumers don’t break each other (contract tests).
  • Safety: idempotency + retries + pagination behavior are stable under real delivery patterns.
Template — the testing pyramid that actually works
Template
Recommended distribution (guideline): • Unit tests: 70–85% (pure + deterministic, run on every PR) • Integration tests: 10–25% (real deps, run on PR + nightly) • Contract tests: 5–15% (CDC: consumer expectations verified by provider) • E2E tests: 1–5% (golden paths only; do not gate everything on E2E) Rules that prevent pain: 1) Favor determinism over coverage vanity. 2) Freeze time (clock), randomness (seed), and external IO (stubs) in unit tests. 3) Use real dependencies in integration tests (Testcontainers/embedded) to catch config/protocol bugs. 4) Contracts must cover semantics (constraints, error shapes, authz behaviors), not just JSON shape. 5) Every flaky test is either fixed quickly or quarantined with a tracked ticket.
Example — what to test for a Facebook-like Home Feed
Example
Unit (fast, pure): • Ranking rules: "blocked users removed", "no duplicate posts", "pinned items first" • Cursor rules: stable ordering, cursor round-trip, monotonic pagination • Idempotency helper: same key + same request => no duplicate side effects Integration (real deps): • /feed reads: candidate fetch + hydration with Redis/DB wired correctly • Caches: stampede protections (single-flight / request coalescing) behave as expected • Event consumption: PostCreated -> feed fanout queue -> timeline store update Contract (CDC): • Feed API schema: error shapes, cursor format, pagination semantics • Post API schema: validation errors, authz errors, idempotency behavior Thin E2E (golden): • POST /post then GET /feed shows new post within SLA • POST /like updates counters and does not double-count on retries
Common pitfalls
  • Mocking everything → tests are green but prod is red
  • Contracts check types but miss semantic constraints (e.g., authorization rules)
  • E2E gates become flaky and slow; people stop trusting CI
  • Pagination/cursor tests ignore stable ordering → duplicates or missing results
Mini checklist — service endpoints
Checklist
• Happy path + boundaries + error paths
• Idempotency: same key twice => one effect
• Retries: safe + bounded; no retry storms
• Pagination: stable sort + cursor round-trip + duplicates prevented
• AuthZ matrix: allow/deny by role and ownership
• Observability: logs/traces/metrics asserted for key operations
Interview line
“We run a pragmatic pyramid: units for logic, integrations with real deps, and CDC contracts so producers/consumers can deploy independently. E2E is thin and focused on golden paths.”
↑ Back to top

B) Load tests, stress tests, chaos, and failure injection

Prove the system meets SLOs under expected load, fails gracefully past the knee, and recovers with guardrails.

Types (what each one proves)

  • Load (steady): meets SLO at expected peak for a sustained window.
  • Stress (ramp): find the knee; identify the failure mode (timeouts? queue overflow? DB pool?).
  • Spike: sudden burst; verifies autoscaling, caching, and rate limiting.
  • Soak: long run; catches memory/FD leaks, GC issues, slow saturation.
Template — load test plan (SLO-gated)
Template
1) Model realistic traffic: • Mix: read-heavy vs write-heavy • Think time and concurrency (not just RPS) • Hot/cold key distribution (celebs are hot keys) 2) Define pass/fail gates (explicit): • p95 latency <= target • error rate <= threshold • cache hit >= target • DB pool saturation <= target • queue lag <= threshold 3) Measure the whole system: • app: p95/p99, CPU, memory, GC, thread/conn counts • deps: DB CPU, locks, pool usage, cache hit rate • queues: depth, lag, retry/DLQ rates 4) Run failure modes (chaos): • add latency to a dependency • inject 5xx • kill a pod / drop an AZ • observe: circuit breakers + brownout + fallback working
Example — Home Feed load + chaos (what we assert)
Example
Steady load (peak hour): • Gate: p95 <= 250ms, 5xx <= 0.5%, cache hit >= 85%, DB pool < 80% for 30 minutes • Observe: hydration latency, ranking CPU, Redis/DB p95, queue lag for fanout Stress ramp (find the knee): • Increase concurrency until: - cache hit drops, DB pool saturates, or p99 explodes • Confirm failure mode is “graceful”: - requests shed (429) or partial response (brownout) instead of meltdown Chaos: • Inject +300ms latency to media service: - expect timeouts + fallback thumbnails + no cascading failure • Kill 1 pod during canary: - expect retry with jitter and stable p95 after stabilization window • Simulate celebrity spike: - ensure write fanout uses batching/queues; no synchronous storms
Common pitfalls
  • Testing only GETs (writes cause the worst storms in feeds)
  • RPS-only tests (you need concurrency + think time)
  • One-IP synthetic load (CDN/WAF behavior differs from real users)
  • Chaos without guardrails (no abort switch, no blast radius limits)
Interview line
“We gate on SLOs with steady load, then run chaos (dep latency + AZ loss). Past the knee we degrade gracefully via rate limits and brownout instead of cascading failures.”
↑ Back to top

C) Data quality tests (schema, freshness, completeness)

Treat datasets like APIs: they need SLAs, owners, and automated checks.
Template — dataset SLAs + automated checks
Template
Define SLAs per dataset/event stream: • Schema compatibility: backward/forward rules + CI check • Freshness: p95 watermark lag <= X seconds • Completeness: counts/uniques within expected bounds • Accuracy/invariants: business rules hold (“no negative counts”, “debits+credits=0”) • Duplicates: <= threshold (idempotency keys) • Privacy: PII scanning + redaction checks Implementation: • CI: schema compat check against registry • Runtime: monitor lag, counts, and anomaly detection • Reconciliation: compare source-of-truth DB vs derived stream/table • Alerting: page the owner on SLA breach (like an API SLO)
Example — Home Feed data quality (events & features)
Example
Events to protect: • PostCreated, LikeAdded, CommentAdded, FollowEdgeChanged • RankingFeaturesUpdated (derived) Checks: • Schema: new fields optional with defaults; no breaking rename without upcaster • Freshness: events visible to ranker <= 60s (p95) • Completeness: post count in DB vs PostCreated stream within ±1% • Duplicates: LikeAdded dup rate <= 0.1% (idempotency key enforced) • Drift: feature distribution shift alerts (CTR, dwell time proxies)
Common pitfalls
  • Calling missing data ‘eventual consistency’ with no bound
  • Measuring freshness as average only (p95/p99 matters)
  • Silent schema drift breaks consumers weeks later
  • No ownership → nobody fixes the pipeline when it breaks
Interview line
“Every dataset has SLAs: schema compatibility, freshness, and completeness. Violations alert owners just like an API SLO.”
↑ Back to top

D) Shadow traffic, replay tests, and synthetic users

Catch regressions safely by exercising the new version with real-like traffic—without impacting users.
Template — shadow + replay + synthetic (how to do it safely)
Template
Shadow traffic (mirroring): • Mirror a small % of prod requests to v2 (responses discarded) • Scrub secrets + remove side effects (no writes, or writes go to sandbox) • Diff: status codes, key fields, latency deltas with tolerances • Guardrails: sampling, kill switch, per-endpoint allowlist Replay: • Re-run historical traffic/events against a new stack • Preserve per-key ordering where required • Throttle to realistic rates; mask PII; validate idempotency • Great for migrations, index rebuilds, ranking changes Synthetic users: • Deterministic bots run golden journeys 24/7 • Attach traces; validate SLIs; rotate creds; use sandbox tenants • Include negative paths (invalid cursor, auth failure) to catch UX issues
Example — Home Feed shadow/replay/synthetic
Example
Shadow: • Mirror 5% of GET /feed to v2 ranker (no user impact) • Diff: top-N postIds overlap, latency p95 delta, error shapes • Guardrail: block any writes; feature flags to disable expensive models Replay: • Nightly replay T-1 traffic to validate a new ranking model + cache config • Verify: no cache stampede, p99 stays within budget, relevance metrics stable Synthetic: • Bot: create post → wait → fetch feed → like → fetch feed → ensure counters consistent • Alerts: cursor failures, spike in 5xx, ranking latency regressions
Common pitfalls
  • Mirroring writes to prod stores (accidental side effects)
  • Comparing floats/timestamps without tolerances (false diffs)
  • Replay without clock isolation or ordering guarantees
  • Bots using real user tenants (pollution + privacy risk)
Interview line
“We shadow a slice of prod to v2 with no side effects, diff correctness/latency, and run nightly replays plus always-on synthetic users for golden journeys.”
↑ Back to top

E) Quality gates in CI/CD (the end-to-end pipeline)

A strong pipeline is layered: fast feedback first, then realism, then safe rollout gates with auto-rollback.
Template — CI/CD gates (ordered for speed + signal)
Template
CI (fast): • lint/static analysis → unit tests → contract tests → integration tests (real deps) → security scans Pre-prod (realistic): • load test at projected peak + headroom • chaos: dependency latency + one-AZ loss • data smoke tests on canary datasets/streams Prod rollout: • canary 1% → 5% → 25% with auto-rollback • rollback gated on: - p95/p99 latency delta - error rate delta - SLO burn rate Post-deploy: • compare dashboards (RED/USE), top queries, cache hit rates, cost deltas • run synthetic journeys continually Test hygiene: • quarantine flaky tests; deflake as tracked work • version golden fixtures; freeze clocks; deterministic seeds
Example — Home Feed rollout gates (what we watch)
Example
Pre-prod: • Peak load for 30 min: p95 <= 250ms; error <= 0.5%; cache hit >= 85%; DB pool < 80% • Chaos: add +300ms to media service; verify fallbacks and no cascade Canary: • Start 1% traffic to v2 ranker • Auto-rollback if: - p95 increases > 20% - 5xx increases > 0.3% - burn-rate crosses threshold for 5 minutes Post: • Compare ranking latency distribution, cache stampede rate, and feed engagement metrics
Interview line
“Our pipeline is layered: fast CI first, realism in pre-prod, then canary with auto-rollback on SLO burn-rate. Synthetic users keep validating the feed 24/7.”
↑ Back to top

Handy patterns & snippets

Small building blocks you can reuse everywhere.
Exponential backoff with full jitter (budget-aware)
Snippet
for attempt in 1..3:
  maxDelay = min(500ms, 50ms * 2^(attempt-1))
  sleep(rand(0, maxDelay))
  do_request()
Contract test essentials (CDC)
Checklist
• Required fields + semantic constraints (not just types)
• Consistent error schema: {code, message, details[]}
• Pagination: stable ordering + cursor round-trip
• Idempotency: same key => same outcome, no duplicate effect
• AuthZ semantics: allow/deny matrix by role/ownership
Load test recipe (API)
Checklist
• Mix: 80% reads (GET /feed), 15% engagement (POST /like), 5% writes (POST /post)
• Concurrency + think time: p50 200ms, p95 1s
• Hot keys: include celebrities + cold users
• Assertions:
  - p95/p99 latency
  - <0.5% 5xx
  - Redis hit >85%
  - DB pool <80%
  - consumer lag <30s
Data quality gates (stream)
Checklist
• Watermark delay p95 < 120s
• Duplicate rate < 0.1%
• Record counts within ±1% of source
• Schema compatibility (backward + forward) passes in CI
↑ Back to top

Quick one-liners (interview-ready)

Short lines that signal senior-level testing thinking.
  • “Pyramid: units fast, integrations with real deps, CDC for cross-team safety, thin E2E for golden paths.”
  • “Load gates on SLOs; stress finds the knee; chaos proves brownout + breakers.”
  • “Datasets have SLAs—schema, freshness, completeness—just like APIs.”
  • “Shadow a slice of prod to v2 and diff correctness/latency before cutover.”
  • “Synthetic users run golden journeys 24/7 with traces attached.”
↑ Back to top