1) Clarify goals & constraints
Say
“Before I design, I’ll confirm goals, scale, and constraints so I optimize for the right thing.”
Checklist
Template- Users & use cases: Who are the primary users? Real-time UX or batch? Mobile/web/3P API?Clarify the “happy path” first (core journey), then edge cases (privacy, abuse, cold start, failures). Ask what must be real-time vs can be async.
- Scale numbers: DAU/MAU, QPS (reads/writes), concurrency, expected growth rate.Ask for read/write ratio, burstiness (peak multiplier), geo distribution, and whether traffic is “spiky” (events) or “smooth” (B2B).
- SLOs: Availability target (e.g., 99.9% monthly), latency budgets (p95/p99 per critical endpoint).Identify the few endpoints that matter most (e.g., feed read, post write). Agree on p95/p99 and error budgets early to drive trade-offs.
- Data guarantees: Durability requirements, read-your-writes vs eventual consistency.Decide what must be strong (money/privacy/auth) and what can be bounded-stale (counts/feeds/search). Define acceptable staleness windows.
- Retention & deletion: Data retention windows, legal holds, regional residency.Call out RTBF (Right-to-be-Forgotten) scope: DB, cache, search, analytics, backups. Confirm TTLs and compliance constraints.
- Topology: Single region vs multi-region, active-active vs active-passive.Tie topology to user latency + availability needs. Start simple when allowed; evolve when the business requires cross-region resilience or global latency.
- Budget & team reality: Managed services ok? Open-source + ops? Time to MVP?Pick the simplest architecture that meets requirements. Explicitly state what you’ll defer (multi-region, advanced ranking, complex search).
Facebook-like Home Feed — clarified goals & constraints
Example- Users & use cases: Personalized Home Feed for logged-in users (scroll, refresh, infinite pagination). Users can post, like, comment, follow/unfollow.Core UX: feed load + scroll. Secondary: posting and interactions. Ranking can be async/approx; privacy/blocklist must be correct.
- Scale numbers: ~500M MAU, ~200M DAU. Avg feed opens 8×/day → 1.6B reads/day.Avg ≈ 18,519 RPS; peak (×6) ≈ 111,111 RPS. Posts: ~20M/day → avg ~231 RPS; peak ~1,157 RPS.
- SLOs: Feed read p95 ≤ 200ms (server) / p99 ≤ 400ms; Post create p95 ≤ 300ms for “acknowledged”.We optimize for read availability and tail latency. Degrade gracefully: fall back to recency feed if ranking pipeline is unhealthy.
- Data guarantees: Post durability once acknowledged; privacy/block rules apply immediately; counters (likes/comments) can be eventual (seconds).Read-your-writes for author immediately after posting; followers see within a bounded staleness window (e.g., ≤ 5 seconds).
- Retention & deletion: Posts stored long-term; raw logs 30 days; derived features 90 days; RTBF supported across storage/search/analytics.Media stored in object store with lifecycle policies; deletion events fan out to caches, search indexes, and analytics sinks.
- Topology: Start single-region write + Multi-AZ; global CDN for media; add read replicas/regions later.Feed is personalized (hard to cache at CDN), so we focus on efficient read path + regional affinity; multi-region primarily for DR.
- Budget & team reality: Use managed queue/stream + managed KV/doc store where possible; ship MVP with recency ranking first.Phase 1: recency + basic relevance. Phase 2: ML ranking, advanced recommendations, stronger multi-region strategy.
2) Back-of-the-envelope sizing
Say
“I’ll sanity-check capacity to choose storage, cache size, and network limits.”
Checklist
Template- Throughput: RPS and fan-out (e.g., feed fan-out to 200 followers).Compute average vs peak, then determine whether you can do work synchronously or need queues/workers. Identify hot-key skew.
- Bandwidth: avg/peak MB/s; where to compress or push to CDN.Separate metadata bytes (API) from media bytes (CDN/object store). Compression, pagination, and prefetching matter.
- Storage: daily ingest, 30/90/365-day totals; hot vs warm vs cold tiers.Estimate record sizes + indexes + replication factor. Consider compaction/write amplification for LSM stores.
- Latency math: p95/p99 budgets across hops (LB → API → service → DB → cache).Assign a latency budget per hop and keep retry/timeout budget within the overall SLO.
- Little’s Law: L = λ × W to reason about in-flight work/queue depth.Use it to size thread pools, connection pools, and queue workers based on peak λ and target W.
Facebook-like Home Feed — quick sizing
Example- Throughput: Feed reads peak ≈ 111,111 RPS; post writes peak ≈ 1,157 RPS.Fanout pressure is dominated by follower distribution. Use hybrid: fan-out for normal authors; fan-in (pull/merge) for celebrities.
- Bandwidth: Feed payload ~25KB compressed (no media bytes) → peak API egress ≈ 2.6 GB/s.Media served via CDN/object store (images/video). Keep feed response mostly metadata + URLs; use pagination (cursor) and gzip/brotli.
- Storage: Posts ~20M/day. If metadata ~2KB/post → ~40GB/day raw (before indexes/replication).If naive fan-out-on-write for all authors: avg followers ~300 ⇒ feed entries/day enormous (≈ 358 GB/day at ~64B/entry). This motivates hybrid fanout + caching.
- Latency math: p95 budget 200ms server-side: Edge/LB 10ms + Gateway 10ms + FeedReadSvc 20ms + Redis 5ms + FeedStore 20ms + Post hydration 40ms + buffer 95ms.Keep retries bounded: timeouts per hop smaller than parent; hedged reads for tail only after a threshold; fall back to cached/recency on ranking issues.
- Little’s Law: At peak λ≈111,111 RPS and W≈0.12s (120ms) ⇒ L≈ 13,333 in-flight.That drives max in-flight requests, pool sizes, and concurrency limits. If one feed-read instance safely handles 400 RPS at target utilization, you’d need ~280 instances for peak (plus headroom/AZ).
3) API & data model first
Say
“I’ll define the contract and access patterns before picking components.”
Checklist
Template- APIs: REST/gRPC; define idempotent write semantics and pagination.Prefer cursor-based pagination for feeds. Make writes idempotent with keys (client-generated request IDs).
- Critical entities: e.g., User, Photo, Follow, FeedEntry.List the minimum entities and which service owns them. Clarify ownership boundaries to prevent cross-service coupling.
- Access patterns: top-N reads, hot writes; secondary indexes required?Write down the top 3 reads and top 3 writes and optimize the store and indexes for those—not for hypothetical queries.
- Indexes/keys: Partition keys to avoid hot partitions; composite indexes for frequent queries.Use time-bucketing and/or salting for hot keys. Ensure pagination is stable (sort key monotonic).
- Schema evolution: Avro/Protobuf/JSON with backward-compatible changes.Add optional fields with defaults; avoid breaking changes; use versioned envelopes for events; validate compatibility in CI.
Facebook-like Home Feed — API & model
Example- APIs:
GET /feed?cursor=&limit=30,POST /posts,POST /follow,POST /like.Feed uses cursor (viewerId + lastSeenScore/ts). Writes use idempotency keys (clientRequestId) to handle retries and duplicate submissions. - Critical entities: User, FollowEdge, Post, Reaction, FeedEntry (read model).Keep Post as source of truth. FeedEntry is a derived/indexed representation optimized for fast reads.
- Access patterns: (1) Read top-N feed items for viewer, (2) hydrate posts by IDs in batch, (3) write post and publish event, (4) update follow edges.Avoid N+1: fetch feedEntry IDs then batch-get posts (and optionally batch-get author profiles).
- Indexes/keys: FeedEntry partition by
viewerId, sort by(rankScore, createdAt)or(timeBucket, createdAt).Hybrid fanout: normal authors fan out postId into followers’ partitions; celebrity authors stored once and merged at read time (fan-in). - Schema evolution: Events like
PostCreated,FollowChanged,ReactionAddedin Avro/Proto with registry compatibility checks.Consumers are tolerant readers. Only additive changes while multiple versions coexist during rollout.
4) High-level architecture
Say
“Here’s the request path and the async path; I’ll highlight where we cache and queue.”
Checklist
Template- Clients → Edge: DNS, CDN/Anycast, TLS termination.Edge handles TLS, caching for static/media, and global routing. Personalized APIs usually bypass CDN cache but still benefit from Anycast/TLS offload.
- API Gateway: AuthN/AuthZ, rate-limit, routing, request shaping.Centralize identity, quotas, and request validation. Attach correlation IDs and enforce timeouts.
- Services: split read/write paths; separate feed-write from feed-read.Optimize read path for low latency; optimize write path for durability and async work dispatch.
- Storage: relational for accounts; doc/KV for feed; object store for images.Pick stores by access patterns: transactions and integrity → SQL; large-scale indexed reads → KV/doc/wide-column; large blobs → object store.
- Caching: CDN for media; Redis/Memcached for hot keys & list heads.Cache the “head” of the feed and recently accessed pages. Use request coalescing and jittered TTLs to avoid stampedes.
- Async: Kafka/SQS/PubSub for fan-out, recompute, indexing.Use streams/queues to decouple writes from heavy fanout/indexing/ranking. Consumers scale independently.
- Batch/streaming: ETL to OLAP for analytics, ranking models.Analytics/ML pipelines are separate from OLTP. They produce features/scores that feed back into ranking with controlled staleness.
Facebook-like Home Feed — architecture (mapped to the template)
Example- Clients → Edge: Mobile/Web → Anycast Edge; media via CDN; API over TLS (HTTP/2).CDN caches images/videos aggressively. Feed API responses are personalized, so caching is limited, but edge still improves latency and protects origin.
- API Gateway: Validate JWT/session, enforce per-user/app token bucket, attach trace IDs, route to FeedRead or PostWrite.Gateway also enforces request budgets (timeouts), blocks abusive clients, and provides consistent error schemas.
- Services: FeedReadSvc (low-latency) and PostWriteSvc (durable + async). Separate ReactionSvc (likes/comments) from feed rendering.Read path avoids heavy joins; write path emits events and returns quickly after durability.
- Storage: PostStore (durable metadata/text), FollowStore (graph edges), FeedStore (viewer partitions / feed index), ObjectStore (media).PostStore can be SQL or a scalable document store; FeedStore often a KV/wide-column optimized for time-ordered scans per viewer.
- Caching: Redis caches feed head pages and hydration results (post summaries, author snippets); CDN caches media.Cache-aside for reads; write-through for author read-your-writes (author sees post immediately).
- Async: PostCreated → Stream (Kafka/PubSub). Fanout workers update followers’ FeedStore (normal authors). For celebrities, store separately and merge at read time.Async also updates search indexes and feature/ranking signals. Backpressure is applied via consumer lag and rate limits.
- Batch/streaming: Events → feature pipelines → rankScore updates; fallback to recency if ranking pipeline unhealthy.ETL into OLAP (analytics) and online features (ranking). Ranking staleness is bounded and treated as a resilience concern.
5) Scale & resilience plan
Say
“I’ll show sharding/replication, failover, and how we handle pressure and retries.”
Checklist
Template- Sharding: hash on userId; virtual nodes for smooth reshard; time-bucket partitions.Design for skew: separate hot partitions (celebrities), use time buckets, and keep shard moves safe and incremental.
- Replication: sync for money/strong consistency; async for read scale; state RPO/RTO and lag budgets.Call out what you can tolerate being stale and for how long. Always mention failover and fencing to prevent split-brain writes.
- Failover: multi-AZ by default; multi-region read replicas; promotion plan.Have a runbook: health signals, who declares incident, promotion steps, DNS/GSLB, rollback strategy.
- Backpressure: queue depth thresholds; shed non-critical work; pool caps to protect DB.Bound everything: threads, queues, pools. Prefer early 429/503 with Retry-After over cascading failures.
- Retries: exponential backoff + jitter; retry budget (e.g., 500ms); idempotency keys.Retry only idempotent ops or with idempotency keys. Keep retry budget inside the endpoint SLO; trip breakers on error/latency.
- Rate limiting: token bucket at edge per IP/user/app; use Retry-After.Use quotas per tenant/user/app; weight expensive endpoints; shape traffic to fragile dependencies with leaky-bucket patterns.
Facebook-like Home Feed — scaling plan
Example- Sharding: FeedStore sharded by viewerId (many virtual shards). Celebrity authors handled via separate “celebrity stream” and merged on read.This prevents hot keys from exploding write fanout. Time buckets keep partitions manageable and enable fast truncation/TTL.
- Replication: Post durability uses stronger writes; FeedStore can be async replicated with bounded lag.Target bounded staleness for feed entries (seconds). Use fencing tokens for leader promotion in strong-write stores.
- Failover: Multi-AZ for all tiers; optional cross-region warm standby for DR (RTO minutes, RPO seconds–minutes depending on store).Start with active-passive DR. Promote read-only region to primary using scripted runbook + DNS/GSLB.
- Backpressure: Fanout workers autoscale on lag; if lag spikes, brownout: temporarily reduce ranking complexity and serve recency feed.Keep DB protected with pool caps and circuit breakers. Shed optional enrichments first.
- Retries: Full jitter retries only for safe reads; post creation uses idempotency key (clientRequestId) to prevent duplicate posts.Breakers trip on error rate + p95 latency for downstreams (FeedStore/PostStore). Retries are capped to fit p95 targets.
- Rate limiting: Edge token bucket per user/device; stricter on write endpoints and abuse-prone actions (follow/like spam).Use 429 + Retry-After; add behavioral signals and step-up checks for suspicious spikes.
6) Consistency/availability trade-offs & data flows
Say
“For each core flow, I’ll pick a stance on CAP and explain user impact.”
Checklist
Template- Profile updates / money: prefer C over A → transactional DB; read-your-writes via leader reads.Anything security-sensitive or financial favors correctness. Use transactions, explicit locks, and strong read-after-write semantics where needed.
- Feeds / counts / likes: prefer A over C → eventual consistency (seconds).Users tolerate slight staleness if UX is consistent. Keep staleness bounded and communicate (e.g., optimistic UI).
- Search/index: async pipelines; staleness window target (e.g., < 60s).Indexes are typically derived. Define SLA for indexing lag and fall back gracefully when index is behind.
- Techniques: read-repair; CRDTs for counters/sets; session stickiness or read-my-writes flags; fencing tokens.Pick techniques based on conflict risk. Mention how you prevent split-brain writes and how clients get monotonic reads where it matters.
Facebook-like Home Feed — CAP stance mapped to the template
Example- Profile updates / money: Strong(er). Privacy settings, blocks, and account security must apply immediately and correctly.On feed read: enforce privacy/block filters at read time (author visibility), and invalidate cached entries when privacy changes.
- Feeds / counts / likes: Eventual. Like/comment counters and “seen” signals can lag by seconds.Use optimistic UI and reconcile later. Keep counters in a scalable store (or CRDT-like merges) and accept bounded staleness.
- Search/index: Async. Search indexing for posts/users targets staleness < 60s under normal conditions.If index lag is high, degrade: show partial results or fall back to recency-only feed generation without search-enriched candidates.
- Techniques: Author read-your-writes (write-through cache or leader read); followers see within bounded window via async fanout; fencing tokens on promotions.If ranking pipeline is unhealthy, fall back to recency feed to preserve availability and latency SLOs.
7) Security, privacy, and compliance
Say
“Security left-to-right: identity → permissions → secrets → encryption → data lifecycle.”
Checklist
Template- AuthN/AuthZ: OAuth2/OIDC for users; mTLS/JWT for service-to-service; least-privilege.Validate issuer/audience/expiry; prefer short-lived tokens; enforce authorization centrally and at service boundaries.
- Secrets: KMS/Vault; rotation; no secrets in images or repos.Use dynamic credentials, short TTLs, and audit access. Avoid “secret zero” by using platform identity (IAM/workload identity).
- Encryption: TLS 1.2+ in transit; KMS-managed at rest; envelope encryption for PII.Encrypt sensitive fields at application level where needed; ensure key usage is auditable and rotation-friendly.
- Privacy: minimization, retention & RTBF path; audit logging for access.Redact PII from logs/traces; implement deletion events that fan out to caches/search/analytics/backups per policy.
- Abuse: WAF, bot detection, CAPTCHA, velocity limits, anomaly scoring.Rate-limit writes, detect automation, and apply step-up auth for risky actions. Track abuse metrics and feedback loops.
- Compliance: cross-border data? regional stores for residency.Have DSAR and breach runbooks. Keep retention schedules documented and enforced via TTL/lifecycle policies.
Facebook-like Home Feed — security & privacy
Example- AuthN/AuthZ: Session cookies for web; short-lived tokens for mobile; service identity via mTLS; strict audience checks on JWTs.Authorization checks for visibility happen on feed read (viewer ↔ author relationship, blocks, privacy settings).
- Secrets: KMS/Vault; dynamic DB creds for services; automated cert rotation (≤ 90 days).No secrets in client apps; all sensitive tokens are HTTP-only cookies or stored securely.
- Encryption: TLS everywhere; media URLs are signed; PII fields encrypted with envelope encryption (DEK per record).All decrypt operations audited; keys rotated via re-wrap of encrypted DEKs.
- Privacy: RTBF triggers deletion event to PostStore, FeedStore, caches, search, analytics; backups expire per policy.PII redaction in logs; access logs for sensitive operations with trace IDs.
- Abuse: Edge token buckets; spam detection on follows/likes; WAF rules; device reputation; rate-limit suspicious bursts.Use gradual friction: throttles → challenges → blocks; keep false-positive appeals path.
- Compliance: Region pinning for data residency where required; documented retention for logs and derived features.DSAR APIs and runbooks; breach response plan with ownership and timelines.
8) Observability, rollouts, testing
Say
“I’ll make it observable, ship it safely, and prove it holds at peak.”
Checklist
Template- Metrics: RED for user-facing; USE for infra. Per-endpoint p50/p95/p99; error budgets.Treat SLOs as product requirements. Track saturation (queues, pools) to catch incidents before user-facing failures.
- Logs & traces: structured logs with correlation/trace IDs; sampling (baseline + 100% on errors).Ensure logs are safe (no secrets/PII). Tracing is essential for tail latency and fanout debugging.
- Dashboards & alerts: golden signals, queue depth, cache hit ratio, DB replica lag.Alert on burn-rate and saturation, not just averages. Include DLQ size and consumer lag for async pipelines.
- Rollouts: canary (1%→5%→25%→100%), feature flags, instant rollback.Automate rollback based on SLO regression. Keep brownout flags ready to shed expensive work.
- Testing: contract tests; load tests to peak ×1.5; chaos (kill a dependency, inject 200ms latency).Validate failure modes: breakers, retries with jitter, queue backpressure, and graceful degradation.
Facebook-like Home Feed — observability plan
Example- Metrics: FeedRead p95/p99, error rate, saturation; ranking lag; cache hit for feed head; fanout consumer lag.SLOs: feed p95 ≤ 200ms; alert on error-budget burn and tail latency spikes during peak windows.
- Logs & traces: Trace feed read end-to-end (gateway → feed store → hydration). Sample 100% of slow requests and errors.Trace IDs are attached to PostCreated events to debug fanout delays and missing feed entries.
- Dashboards & alerts: Consumer lag > 30s, DLQ growth, replica lag, breaker-open rate, throttles (429s).If ranking lag breaches threshold, flip feature flag to recency feed and page owners.
- Rollouts: Canary new ranking model to 1% → 5% → 25% gated on latency and engagement deltas; rollback on SLO breach.Feature flags allow immediate brownout (disable expensive enrichments) without redeploy.
- Testing: Load test feed at peak, chaos inject 200–500ms to FeedStore and drop one AZ; verify fallback to cache/recency.Contract tests for event schemas (PostCreated/FollowChanged) to keep producers/consumers deployable independently.
9) Explicit trade-offs & alternatives
Say
“Here are the options I considered, what I chose, and when I’d switch.”
Checklist
Template- Fan-out on write vs fan-in on read (feeds): choose fan-out for small/medium creators; mix models for heavy skew.Pure fanout explodes writes for celebrities; pure fan-in increases read-time latency and compute. Hybrid is common.
- DB choice: doc/KV for feed + relational for accounts/payments; consider wide-column for huge time-series partitions.Match store to access patterns and failure modes. Keep transactional boundaries clear and avoid cross-shard transactions.
- Multi-region: start single-region write + global CDN + async read replicas; evolve to active-active/CRDTs when global write latency matters.Multi-region adds complexity. Justify it with business needs (latency, regulatory, DR).
- Consistency: eventual for counts/feeds, strong for money/profile; session consistency for UX without full strong.Be explicit about user-visible effects and how you mitigate them (optimistic UI, bounded staleness, read-your-writes).
Facebook-like Home Feed — trade-offs recap
Example- Fan-out vs fan-in: Hybrid: fan-out for normal users, fan-in merge for celebrities.Keeps read latency low for most users while preventing write amplification from extreme fanout.
- DB choice: Strong store for posts/privacy; scalable feed index store for viewer partitions; object store + CDN for media.Feed index is derived; posts are source of truth. Hydration is batched to avoid N+1 calls.
- Multi-region: Single write region + Multi-AZ initially; global edge + CDN; warm standby DR.Evolve to multi-region reads/active-active only if business needs it (global write latency, regulatory).
- Consistency: Strong for privacy/block/durability; eventual for counters/ranking/search with bounded staleness.Protect UX with read-your-writes for author and fallback to recency during ranking outages.