System Design

5) Caching — detailed guide

Where to cache, what to cache, how to invalidate, and how to avoid stampedes—so you hit SLOs without privacy leaks or “green cache, red prod.”

Example context
Facebook-like Home Feed
Read path
GET /feed → fetch candidate postIds → hydrate posts/media → rank → rules → cursor
Write path
POST /post → validate → store → async fanout/index/counters/notifications
Engagement
POST /like | /comment → idempotent write → counters → ranking features update
Caching goals for this system
  • Reduce p95/p99 latency for GET /feed (hydration is expensive)
  • Protect DB and downstream services from read amplification
  • Avoid cache stampedes for hot users and hot posts
  • Keep privacy boundaries (don’t leak personalized feed to wrong user)
  • Lower origin egress for media via CDN hit ratio

A) Where to cache

Think in layers: client → edge → gateway/proxy → service → database. Each layer has different risk and payoff.
Template — choose cache layers safely
Template
1) Start with the edge (CDN) for bytes: • static assets + media • biggest win: egress + tail latency 2) Add a service cache (Redis/Memcached) for hot objects/lists: • reduces DB QPS and read amplification 3) Add in-proc cache only for micro-hot data: • small, bounded (Caffeine/Guava) • protects p95 but must be conservative 4) Use client cache for static/semi-static: • Cache-Control + ETag • content-hash filenames for perfect caching Rule of thumb: • The closer to user, the bigger the byte savings • The closer to DB, the bigger the QPS savings • The more personalized the data, the higher the risk
Example — Home Feed cache layers
Example
Client: • App caches static JS/CSS bundles (hashed) and some images CDN/Edge: • All media (images/video segments) cached with long TTL + versioned URLs • Origin shield enabled so spikes don’t hit origin Gateway/Proxy: • Coalesce duplicate requests for same user feed rebuild (if supported) Service (Redis): • feed:head:<user> (top N postIds) short TTL • post:<id> and author/profile fragments TTL minutes • negative cache for missing post IDs briefly Database: • rely on buffer pool, but do not treat DB cache as your “app cache”
Common pitfalls
  • Caching personalized responses at CDN without correct Vary → privacy leak
  • Skipping CDN and trying to “Redis your way” out of media egress
  • Using DB query cache as a scaling plan (often unpredictable and invalidates broadly)
  • Letting in-proc cache grow unbounded (OOM or GC pain)
Interview line
I use CDN for bytes, Redis for hot objects/lists, and small in-proc caches for micro-hot reads—personalized responses are carefully scoped to avoid leaks.
↑ Back to top

B) What to cache

Cache what is reused, expensive, and safe to serve slightly stale—with a clear invalidation story.
Template — pick cache candidates
Template
Pick items that are: • High reuse (many reads per write) • Expensive to compute or fetch • Small enough to store • Safe if briefly stale Typical categories: 1) Objects by ID: user, post, profile 2) Lists / heads: top-N feed, trending IDs 3) Derived results: permissions, feature flags, exchange rates 4) Rendered fragments: precomposed JSON/HTML (high risk: vary rules) 5) Negative cache: 404s briefly to prevent hot-miss storms Avoid caching: • Anything that is user-specific unless key includes identity + policy • Highly volatile correctness-critical values (unless very short TTL / write-through)
Example — Home Feed: what to cache
Example
Cache objects: • post:<postId> (text + metadata) TTL 5–15m • user:<userId>:profile TTL 15–60m Cache list heads: • feed:head:<userId> TTL 30–120s (soft TTL), hard TTL 10m • comments:top:<postId> TTL 30–120s Derived: • permissions:<viewerId>:<ownerId> TTL short + version token Negative cache: • post:missing:<postId> TTL 30–60s (prevents repeated DB misses)
Common pitfalls
  • Caching query results with unbounded keys (key explosion)
  • Caching giant payloads (serialization + memory blowup)
  • Caching authorization decisions without including relevant policy version
  • No negative caching → hot-miss floods when IDs are probed
Interview line
I cache objects by ID and the feed head; I keep TTLs aligned to freshness needs and I negative-cache misses briefly to stop hot-miss storms.
↑ Back to top

C) Invalidation (TTL, event busting, versioned keys)

Cache invalidation is hard—so use a small set of proven patterns and keep them consistent.
Template — invalidation patterns that scale
Template
1) TTL as the default: • simplest, works for semi-static and lists • jitter TTL ±10–20% to avoid synchronized expiry 2) Explicit busting (event-based): • on write, emit “entity changed” event • consumers delete or update cache keys • use Outbox/CDC so you don’t miss events 3) Versioned keys / tokens: • key contains version token (v7) • bump token to instantly invalidate many keys • old generations expire naturally via TTL Recommended combo: • TTL + event busting for objects • soft/hard TTL + refresh-ahead for hot lists
Example — Home Feed invalidation story
Example
Post updated: • DB write succeeds • Outbox publishes PostUpdated(postId) • Cache consumer deletes post:<postId> and related fragments • Feed head remains valid (it contains IDs), hydration pulls fresh post on next read Feed head: • Use soft TTL 60s and refresh-ahead • Keep hard TTL 10m as safety net • If ranking logic changes: bump feed version token and rebuild gradually
Common pitfalls
  • Dual-writes without outbox/CDC (cache and DB drift)
  • No TTL on versioned keys (old generations never leave)
  • Aggressive global purges during peak (self-inflicted outage)
  • Treating TTL as correctness guarantee (it’s a freshness knob, not truth)
Interview line
TTL is my default; correctness-sensitive objects also use event-based busting via outbox/CDC, and hot lists use soft/hard TTL with refresh-ahead.
↑ Back to top

D) Consistency patterns

Choose the pattern per data type: correctness vs latency vs complexity.
Template — map data → cache pattern
Template
Cache-aside (most common): • Read: cache → miss → DB → set • Write: DB → invalidate/update cache Read-through: • cache layer loads DB on miss (good for coalescing) Write-through: • write cache + DB synchronously (good for counters if correctness matters) Write-behind / write-back: • write cache first, DB async (fast but risky unless cache is durable and replayable) Refresh-ahead: • refresh hot keys before they expire to protect p95/p99 Rule: • Use cache-aside unless you have a strong reason not to.
Example — Home Feed: which pattern where
Example
Cache-aside: • post:<id>, user:<id>:profile Read-through + coalescing: • feed:head:<userId> (many concurrent reads) Write-through: • like_count and comment_count (if you need fast reads + strong-ish correctness) Refresh-ahead: • top feed head for active users (prevents p99 spikes when TTL expires)
Common pitfalls
  • Write-behind on correctness-critical data without durability (data loss risk)
  • Write-through everywhere (hurts write latency and availability)
  • Refresh-ahead without bounds (background refresh becomes a thundering herd)
Interview line
Cache-aside for safety, write-through for some counters, and refresh-ahead for hot lists—pattern depends on correctness and tail-latency needs.
↑ Back to top

E) Eviction & sizing

Eviction policy + memory sizing decides whether your cache is stable or constantly thrashing.
Template — eviction & sizing plan
Template
1) Pick eviction/admission: • LRU: good for recency • LFU/TinyLFU: protects from one-hit wonders • segmented policies (ARC/2Q): resist scans 2) Measure working set: • plot hit ratio vs memory (curve) • aim for where extra memory gives diminishing returns 3) Headroom: • keep 20–40% free memory to avoid sudden evictions and latency spikes 4) Hybrid caches: • small in-proc cache for micro-hot reads • distributed cache for shared reuse 5) Watch cardinality: • list caches can explode key count; add bucketing/versioning and TTLs
Example — Home Feed cache sizing intuition
Example
Data shape: • post objects are reused across many users (good cache reuse) • feed heads are per-user (high cardinality; keep TTL short) Policy: • Use TinyLFU/LFU-like admission for post objects • Keep per-user feed head small (top N IDs only) and short-lived • Reserve memory for shared objects; cap per-user keys to avoid key explosion
Common pitfalls
  • Caching per-user large blobs (memory gets eaten by cardinality)
  • LRU without admission control (one-hit wonders push out hot keys)
  • No headroom (eviction storms at peak)
  • Oversized values (serialization + network overhead dominates)
Interview line
For mixed workloads I prefer TinyLFU-style admission with LRU eviction, keep headroom, and cap high-cardinality per-user caches.
↑ Back to top

F) Stampede protection

A cache miss can amplify into an outage. Stampede protection is non-negotiable for hot keys.
Template — stop the thundering herd
Template
1) Request coalescing (single-flight): • one request rebuilds the value • others wait or serve stale 2) Locks with TTL: • Redis SET NX PX=... to acquire lock • use fencing tokens for safety 3) Jitter TTL: • randomize expiry times 4) Soft/hard TTL: • serve stale within hard TTL while refreshing in background 5) Backpressure: • cap concurrent rebuilds • shed optional work (degrade gracefully)
Example — Home Feed stampede prevention
Example
Problem: • many clients open app at the same time → feed head expires → stampede Solution: • soft TTL 60s: serve stale + trigger background refresh • hard TTL 10m: never hammer DB during refresh failures • single-flight per user: only one refresh builds the feed head • cap refresh concurrency globally to protect downstream services
Snippet — simple Redis lock for refresh (conceptual)
Snippet
// Acquire lock (10s). Only the winner rebuilds.
SET lock:feed:head:<userId> <token> NX PX 10000

// Winner rebuilds then writes cache
SET feed:head:<userId> <value> PX <ttl_ms>

// Release lock (verify token to avoid deleting someone else's lock)
if GET lock == token then DEL lock
Common pitfalls
  • No coalescing (every request rebuilds on miss)
  • Locks without TTL (deadlocks)
  • Locks without fencing tokens (stale owner deletes new lock)
  • No backpressure (refresh job floods DB under load)
Interview line
Hot keys use single-flight and soft/hard TTL. We would rather serve slightly stale than stampede the DB.
↑ Back to top

G) Metrics & alarms

If cache health isn’t visible, you’ll learn about it from an outage or a bill.
Template — cache observability checklist
Template
Core: • hit ratio (overall + per namespace) • byte hit ratio (bytes served from cache vs origin) • latency: cache get/set p50/p95/p99 Stability: • evictions/sec, memory used, fragmentation • key cardinality per namespace • error rates (timeouts, connection pool saturation) Downstream impact: • DB QPS and p95 latency • upstream request latency when cache degrades Alarms: • hit ratio drop > 10% for 5m • eviction surge • cache latency p95 jumps • connection pool > 80% or timeouts spike
Example — Home Feed: alarms that matter
Example
Key alarms: • feed:head hit ratio drop → likely TTL misconfig or stampede • post:<id> miss spike → invalidation bug or cache flush • Redis p95 latency ↑ → network saturation or CPU pressure • Origin egress ↑ + CDN hit ratio ↓ → cache key/headers regression
Interview line
We alert on hit ratio drops, eviction storms, and cache latency, plus downstream DB QPS—cache metrics must correlate to user SLOs.
↑ Back to top

H) Practical examples (copy/paste patterns)

These are the most reusable interview patterns across many systems.
Example — Post object cache (safe cache-aside)
Example
Key: post:<postId>:v1 TTL: 5–15 minutes + jitter Invalidate: PostUpdated event deletes post:<postId>:* Notes: keep payload small; store compressed if large; avoid over-fetching
Example — Feed head cache (hot list with soft/hard TTL)
Example
Key: feed:head:<userId>:v7 Value: [postId1, postId2, ... postIdN] Soft TTL: 60–120s (serve stale and refresh) Hard TTL: 10m (safety) Stampede: single-flight per user + global refresh cap
Example — CDN media caching (huge egress saver)
Example
URL: /media/<contentHash>/w=640.webp Headers: Cache-Control: public, max-age=31536000, immutable Notes: • versioned URLs so no purges needed • origin shielding on • signed URLs only when private access is required
↑ Back to top

I) Key design & HTTP headers

Cache keys define correctness. If the key doesn’t capture identity/policy, you risk leaks.
Template — safe key structure
Template
Key format: • <namespace>:<id>[:subscope][:version] Include in key when relevant: • user identity (viewerId) for personalized data • locale / device class if response differs • policy version for authorization/visibility • experiment bucket for A/B variants
Example — Home Feed keys (good vs dangerous)
Example
Good: • feed:head:<userId>:v7 • post:<postId>:v1 • perms:<viewerId>:<ownerId>:policyV3 Dangerous: • feed:head:v7 (no user identity → leaks) • profile:<userId> (no version/policy; invalidation pain) • search:<q> without filters (key misses important dimensions)
Snippet — HTTP caching header patterns
Snippet
Static assets (hashed filenames):
Cache-Control: public, max-age=31536000, immutable

Shared dynamic (CDN can cache):
Cache-Control: public, s-maxage=60, stale-while-revalidate=120
Vary: Accept-Encoding

Personalized:
Cache-Control: private, no-store
# or bypass edge and cache in service with user-scoped keys
Interview line
I design cache keys like API contracts: keys include identity and policy versions to prevent leaks, and version tokens make global invalidation safe.
↑ Back to top

J) One-liners to use in the interview

  • “I cache in layers: CDN for bytes, Redis for shared hot objects/lists, and small in-proc cache for micro-hot paths.”
  • “Invalidation is TTL by default, event-based for correctness, and version tokens for fast mass purges—with jitter to avoid herds.”
  • “Hot lists use soft/hard TTL + refresh-ahead; single-flight prevents stampedes.”
  • “Cache keys include identity and policy version—correctness and privacy come first.”
↑ Back to top