8) Reliability & resilience — detailed guide

A) Health checks & graceful degradation (brownout / bailout)

Health checks (what each one means)

• Liveness: “Is the process fundamentally broken?” Keep it cheap (event loop alive, threads not deadlocked, heap not exploding). If it fails → restart.
• Readiness: “Can this instance serve traffic right now?” Check critical deps only (config/secrets loaded, DB reachable, cache reachable). If it fails → remove from load balancer.
• Startup (K8s): delay readiness until warmup/migrations complete (avoid early 5xx).
• Pitfalls: deep dependency checks causing cascades; liveness killing pods during long GC.

Graceful degradation

• Brownout: dim expensive features (turn off heavy ranking features, reduce fanout, fewer candidates).
• Bailout / Safe mode: serve minimal “core-only” response (cached feed, static shell, limited interactions).
• Mechanisms: feature flags, kill switches, cached fallbacks, smaller payloads, stale reads with a banner.

B) Retries + jitter, circuit breakers, bulkheads (stop cascades)

Retries (safe rules)

• Retry only idempotent calls (GET/PUT) or calls with an idempotency key.
• Use exponential backoff + full jitter to avoid synchronized retry storms.
• Add a retry budget per request (e.g., don’t burn a 2s SLO on 10 retries).
• Classify errors: retryable (timeouts/5xx/throttling) vs non-retryable (4xx validation).

base=50ms; cap=500ms; attempts=3
for i in 1..attempts:
  sleep = rand(0, min(cap, base * 2^(i-1)))
  wait(sleep)
  try request
# Keep total wait bounded by your latency SLO.

Circuit breakers (fail fast)

• Closed: normal traffic.
• Open: fail fast when error/timeout/latency crosses threshold.
• Half-open: allow small probes; close if healthy.
• Trip on both error rate and p95 latency.

Bulkheads (isolation)

• Separate pools per dependency (DB pool vs Redis pool), per-tenant concurrency, separate queues for critical vs best-effort work.
• Prevent “one bad downstream” from consuming all threads/connections.

C) Rate limiting, throttling, quotas (protect the system)

Token bucket (most common)

• Burst up to B, refill at rate r tokens/sec.
• Deny with 429 and Retry-After; optionally return rate-limit headers.
• Place it at edge/gateway and also near fragile deps (defense-in-depth).

Leaky bucket / shaping

• Leaky bucket smooths traffic into fragile dependencies.
• Great for DB write shaping, email provider, search, third-party APIs.

Quotas and cost-based limiting

• Per user / per API key / per tenant / per org; reset daily/monthly.
• Weighted tokens: read=1, write=3, fanout=10, “expensive ranking”=5.
• Pitfall: only IP-based limits (NAT/office traffic); prefer authenticated identity.

D) Backpressure (sync services + streams)

Sync services (request/response)

• Bound everything: request queue, thread pool, DB pool, upstream timeouts.
• Use watermarks: when queue/in-flight exceeds threshold → return 429/503 early.
• Adaptive concurrency: tune max in-flight from latency (like “AIMD” behavior).
• Load shed: drop optional enrichments first; reduce payload size.

Streams (Kafka/Kinesis)

• Limit in-flight per partition; pause/resume consumption; autoscale on lag.
• Retry with DLQ, not infinite redelivery loops.
• Beware “commit before side effects”: commit offsets only after durable success.

E) Disaster recovery (RTO/RPO), backups, restore drills

Key terms

• RTO: time to restore service (e.g., 15 minutes).
• RPO: max acceptable data loss window (e.g., 60 seconds).

Backups & restore readiness

• Encrypted, immutable storage (WORM) + cross-region copies.
• Regular integrity checks and “can we actually restore?” drills.
• Practice table-level restore and whole-region restore.

Standby patterns

• Cold: cheapest, slowest recovery.
• Pilot light: minimal services running, scale up during DR.
• Warm: partially running, faster failover.
• Hot: active-active or hot standby; most expensive.

F) Multi-AZ vs multi-region (topology choices)

Multi-AZ (within one region)

• Protects from AZ/rack failure; low latency; simpler operations.
• Does not protect from full regional outage or provider-wide issues.

Multi-region

• Protects from regional outage + improves geo latency.
• Active-passive: simpler invariants (single writer); failover via DNS/GSLB.
• Active-active: multi-writer; needs conflict handling (CRDT/LWW), idempotency, global IDs.

G) Metrics, SLOs & guardrails (what to monitor + what to automate)

Golden signals

• Latency: p50/p95/p99 (and tail latency).
• Traffic: RPS, QPS, bytes in/out.
• Errors: 4xx/5xx, timeouts, dependency errors.
• Saturation: CPU, memory, thread pools, connection pools, queue depth.

Protection signals (early warning)

• Retry rate, breaker opens/flaps, throttles (429), shed rate (503).
• Queue depth/consumer lag, DLQ rate, replica lag, cache hit ratio drop.
• Availability of dependencies (DB/Redis/Kafka), slow query rate.

Guardrails & rollouts

• Canary rollout (1% → 5% → 25% → 100%) with automatic rollback on SLO breach.
• Error budget burn alerts (fast burn vs slow burn).
• Feature flags for brownout/bailout with audit trails.

H) Sound bites you can reuse

• “Readiness isolates bad instances; brownout trims optional work; bailout protects core SLO.”
• “Retries are budgeted and jittered; breakers and bulkheads prevent cascades.”
• “Token buckets at the edge for fairness; leaky buckets to shape traffic into fragile deps.”
• “Backpressure is explicit: bounded queues, early 429/503, autoscale on lag.”
• “DR is tested restores + scripted failover; RTO/RPO are contractual.”
• “Multi-AZ by default; multi-region when the business needs it.”