System Design

15) Cost & efficiency — detailed guide

Practical ways to reduce cloud spend without hurting SLOs: egress awareness, storage tiers & lifecycle, caching, commitments (reserved/spot), and tracking dollars per request and per tenant.

Example context
Facebook-like Home Feed
Read path
GET /feed → fetch candidate postIds → hydrate posts/media → rank → rules → cursor
Write path
POST /post → validate → store → async fanout/index/counters/notifications
Engagement
POST /like | /comment → idempotent write → counters → ranking features update
Cost hot spots to watch
  • media egress (images/videos), origin vs CDN
  • feed hydration fan-out (many reads per request)
  • ranking compute (models/features), especially p99
  • storage growth for events/logs/features (retention)

A) Egress awareness, storage tiers, and compression

Most cloud bills are “bytes × distance × where.” Reduce bytes, reduce distance, and avoid expensive paths.

Egress: the silent killer

Where cost explodes
  • Know your egress paths: public internet (often expensive), cross-region, cross-AZ, and origin → CDN vs CDN → user.
  • Push bytes to the edge: CDN caching + origin shielding so origin egress stays tiny.
  • Prefer private connectivity: private endpoints/peering/backbone when it reduces public egress.
Template — the egress playbook
Template
1) Inventory major byte flows: • Users ⇄ CDN ⇄ Origin • Service-to-service (cross-AZ? cross-region?) • Data pipelines exporting to analytics/partners 2) Make “origin egress” a KPI: • Track: origin GB/day, CDN hit ratio, cacheable % by route 3) Reduce bytes: • Text: Brotli/gzip • Media: correct formats + responsive sizes • APIs: avoid over-fetching; paginate; compress JSON 4) Reduce distance: • Keep compute + storage in the same region • Avoid cross-region reads and chatty calls across regions
Example — Home Feed egress & compression wins
Example
Home Feed byte flows: • Biggest cost: images/videos in feed • Goal: CDN hit ratio > 95% and avoid origin fetches Tactics: • Media served from CDN with versioned URLs (immutable cache) • Signed URLs only when required; otherwise long max-age • Brotli for JSON feed responses; keep payload lean (only fields needed for feed) • Avoid “hydrating too much” (don’t fetch full post for every candidate)

Storage tiers (object/block/db)

  • Object tiers: hot → warm (IA) → cold (archive). Cheaper as retrieval latency/fees go up.
  • DB/storage: premium for hot OLTP; cheaper for warm replicas; snapshots/archives for cold retention.
  • Lifecycle rules: transition by time + prefix/tags; auto-expire where allowed.

Compression & format hygiene

  • HTTP: Brotli for text (HTML/JS/CSS/JSON), gzip fallback; don’t recompress already-compressed media.
  • Images: WebP/AVIF, responsive sizes, edge transforms.
  • Analytics: Parquet/ORC + ZSTD/Snappy; avoid CSV/JSON for large datasets.
Common pitfalls
  • Ignoring origin egress (you see CDN traffic but miss the expensive origin misses)
  • No versioned URLs → low cacheability → unnecessary origin fetches
  • Compressing images/video again (wastes CPU for little benefit)
  • Over-fetching feed payloads and “hydrating everything” per request
Takeaway
Most savings come from cutting bytes and origin egress: cache at CDN, keep payloads lean, and use the right media formats.
↑ Back to top

B) Hot/Warm/Cold data, TTL, and archival

Retention is a product decision. Automate it with lifecycle policies so storage doesn’t grow forever.
Template — a retention & lifecycle plan
Template
1) Classify data: • user content, derived content (thumbnails), events, logs, traces, ML features, backups 2) For each class define: • business usefulness window (e.g., “feed relevance uses last 30 days”) • legal/compliance retention (if any) • access pattern (hot vs warm vs cold) 3) Encode lifecycle (automated): • hot (fast) → warm (cheaper) → cold (archive) → delete • include reporting: bytes by tier, object count, last access 4) Make queries cheaper: • partition by time • TTL indexes / table expiration • prune partitions and avoid scanning old data
Example — Home Feed data lifecycle
Example
Hot: • feed cache heads, recent posts metadata, recent engagement counters (days-weeks) Warm: • older post metadata, analytics aggregates (months) Cold: • raw events/logs needed for audits (years, if required) • old media variants not used (archive or delete) Practical policy: • logs: hot 14d → warm 90d → delete 365d • events: hot 30d → warm 180d → archive 2y (if required) • thumbnails: keep only active sizes; regenerate on demand if cheaper than storing all variants
Common pitfalls
  • No TTL → storage grows until the bill forces a fire drill
  • Keeping all media variants forever (store explosion)
  • OLTP DB used for OLAP analytics queries (expensive reads + performance risk)
  • Retention rules exist in docs but not enforced by automation
Takeaway
Automated TTL + lifecycle policies are the easiest long-term cost win. Don’t rely on humans to delete data.
↑ Back to top

C) Caching to cut cost (not just latency)

Caching reduces DB reads, compute, and egress. But only if cacheability and keys are correct.
Template — caching with a cost lens
Template
1) Decide what to cache: • expensive reads (hydration), feed heads, feature lookups, media • cache negative results briefly (404s) to stop repeated misses 2) Protect origin: • CDN + origin shielding • request coalescing (single-flight) for hot keys • jitter TTLs to avoid synchronized expirations 3) Track cache ROI: • Cache Hit Ratio (H) • Saved origin egress = total egress × (1 − H) • Saved compute/DB reads vs cache infra cost 4) Make cache safe: • versioned keys • explicit invalidation strategy (events) where needed • bounds: max size, eviction policy, per-tenant quotas
Example — Home Feed caching strategy
Example
CDN: • Media is cache-first with long TTL + versioned URLs • Origin shielding to prevent spikes from reaching origin Redis: • Cache feed “head” (first N postIds) per user for short TTL • Cache hydration results (post metadata) with TTL and invalidation on PostUpdated Anti-stampede: • single-flight on user feed rebuild • refresh-ahead for hot users • jitter TTL to prevent thundering herd
Common pitfalls
  • Bad cache keys (missing query params or vary headers) → low hit ratio
  • No stampede protection → cache makes meltdowns worse
  • Caching huge objects without limits → eviction thrash
  • Invalidation without a plan → stale content complaints and manual flushes
Takeaway
Fix cacheability and keys before scaling infra. A higher hit ratio reduces egress + DB + CPU at once.
↑ Back to top

D) Reserved vs Spot capacity and commitments

Right-size first, then cover a stable baseline with commitments, and use on-demand/spot for burst.
Template — commitment strategy that avoids regret
Template
1) Right-size before you commit: • measure CPU/mem/IO utilization and p95 latency • remove waste (oversized nodes, idle clusters, orphaned volumes) 2) Establish a baseline: • minimum steady capacity needed to meet SLOs 3) Cover baseline with discounts: • reserved instances / savings plans / committed use • keep flexibility (instance families/regions) if possible 4) Handle spikes: • autoscale on-demand • use spot/preemptible for interruptible workers (batch, async fanout, indexing) 5) Resilience for spot: • checkpointing, queue-based work, quick drain, retry with jitter
Example — Home Feed: what runs on spot vs reserved
Example
Reserved/baseline: • core API fleet (GET /feed) baseline to meet SLOs • primary DB and critical caches sized for peak-ish steady load Spot/preemptible: • async fanout workers • media processing/transcoding • re-indexing jobs (search / ranking feature backfills) • offline analytics batch Guardrails: • queue depth alarms and max retry budgets • graceful degradation if async capacity drops
Common pitfalls
  • Buying commitments before right-sizing (locks in waste)
  • Putting stateful critical DB on spot without strong safeguards
  • Autoscaling without stabilization → flapping and unpredictable bills
  • No cost anomaly alerts (you learn in the invoice)
Takeaway
Commitments are great for stable baselines. Put interruptible work on spot, but keep user-facing SLO paths on reliable capacity.
↑ Back to top

E) Measuring $/request and $/tenant

If you can’t measure unit economics, you can’t optimize cost safely.
Template — cost allocation + unit economics
Template
1) Tag everything: • env, service, team, and (if multi-tenant) tenant_id / account_id 2) Define unit metrics: • requests, bytes_out, CPU_ms, DB_reads, cache_miss, queue_jobs 3) Compute: • $/request = total_service_cost / successful_requests • $/tenant = tenant_share_of_shared_cost + tenant_dedicated_cost • egress cost = GB_out × $/GB (separate CDN vs origin) 4) Publish dashboards: • $/request over time and by route • $/tenant/day top offenders • origin egress trend + CDN hit ratio • cost anomalies alerting
Example — Home Feed cost events you should capture
Example
Per request (GET /feed): • tenant_id (or user segment) • bytes_out • hydration_calls_count • cache_hit/miss • ranker_cpu_ms • db_rows_read • served_media_from_cache? (yes/no) Why: • lets you see which routes/users drive cost • catches regressions (e.g., new ranking model doubles CPU or bytes_out)
Snippet — rough $/tenant from request telemetry
Snippet
-- Example: approximate $/tenant for API egress + compute (monthly)
SELECT tenant_id,
       SUM(bytes_out) / 1e9 * 0.09      AS origin_egress_usd,  -- adjust pricing
       SUM(req_cpu_ms) / 1000 / 3600 
         * 0.12                         AS compute_usd,        -- e.g., $0.12 per vCPU-hour
       COUNT(*)                         AS requests,
       (SUM(bytes_out)/1e9*0.09 + SUM(req_cpu_ms)/1000/3600*0.12) / NULLIF(COUNT(*),0)
                                       AS usd_per_request
FROM cost_events
WHERE ts >= DATE_TRUNC('month', NOW())
GROUP BY tenant_id
ORDER BY origin_egress_usd DESC;
Common pitfalls
  • No tagging → everything is ‘shared’ and nobody owns the bill
  • Only tracking totals (you need per-route and per-tenant breakdowns)
  • Ignoring cost regressions in CI/CD (you find out after the invoice)
  • Not separating CDN egress vs origin egress (wrong optimization focus)
Takeaway
Treat cost as a product KPI: publish $/request and origin egress weekly so regressions are caught early.
↑ Back to top

F) Worked example (simple but useful)

This is the kind of math interviewers love: show you can reason about cost drivers.
Template — egress savings math
Template
Given: • Total egress (GB/day) • CDN hit ratio H • Origin egress price ($/GB) Compute: • Origin GB/day = Total × (1 − H) • Origin $/month ≈ Origin GB/day × $/GB × 30 Compare to “no CDN” cost to estimate savings.
Example — Home Feed: CDN hit ratio savings
Example
Scenario: • Total egress to users: 1 TB/day (≈ 1024 GB/day) • CDN hit ratio H = 95% • Origin egress price: $0.09/GB Origin egress: • 1024 × (1 − 0.95) = 51.2 GB/day Origin egress cost: • 51.2 × 0.09 ≈ $4.61/day → ≈ $138/mo No CDN origin cost: • 1024 × 0.09 ≈ $92.16/day → ≈ $2,765/mo Estimated monthly savings: • ≈ $2,627/mo (before CDN costs) Lesson: • pushing media to CDN and keeping hit ratio high dominates savings.
Takeaway
Optimize the biggest multipliers first: bytes (media), hit ratio (CDN), and origin misses. Then tackle compute and storage.
↑ Back to top

Handy formulas & snippets

A few quick tools you can reuse in interviews and design docs.
Hit ratio & origin egress
Snippet
Origin_Egress_GB = Total_Egress_GB * (1 - Hit_Ratio)
Savings = (Total_Egress_GB - Origin_Egress_GB) * $/GB
Compute cost (instances)
Snippet
Compute$/mo = instance_count * $/hr * 730
Serverless cost (conceptual)
Snippet
$ = invocations * price_per_million / 1e6
  + GB-seconds * price_per_GB-second
  + egress
Lifecycle policy checklist
Checklist
• Hot → warm → cold → delete is automated
• Policy is tag/prefix-driven (no manual steps)
• Reports: bytes by tier, last access, object counts
• Restore drills for cold archives (prove you can retrieve)
• Partitions + TTL indexes reduce scan costs
↑ Back to top

Interview one-liners

Short, confident lines to close your answer.
  • “Egress dominates—push bytes to CDN, maximize hit ratio, and keep origin misses tiny.”
  • “Automate retention with hot/warm/cold lifecycle policies; storage growth is a product decision.”
  • “Caching is cost optimization: fix cache keys, prevent stampedes, and measure ROI.”
  • “Right-size first; cover baseline with commitments; use spot for interruptible workers.”
  • “We publish $/request and $/tenant—cost is a KPI, not a surprise invoice.”
↑ Back to top