6) Messaging & streaming — detailed guide

• Model: work items are consumed and removed; focus is “do this task”.
  Great for background jobs, retries, worker pools, priorities.
• Semantics: usually at-least-once; duplicates possible.
  Use visibility timeouts + retry policies; DLQ for poison messages.
• Ordering: best-effort; strict FIFO reduces throughput (single shard).
  If you need strict order, scope it tightly (per entity).

• Model: append-only partitions; consumers track offsets; retention by time/size.
  Great for event sourcing, audit, replay, multiple independent consumers.
• Ordering: guaranteed only within a partition.
  Never promise global ordering across partitions.
• Guarantees: at-least-once by default; “exactly-once” is effectively-once via idempotency + atomic commits.
  Kafka transactions can help, but downstream must still be idempotent.

• At-least-once: duplicates can happen → consumers must be idempotent.
• At-most-once: may drop messages → use only for non-critical signals.
• Effectively-once: idempotency + atomic commit of “processed offset” with side effect.
  Typically implemented via unique constraints/UPSERTs, transactions, or outbox pattern.

• Pick a key that needs ordering: order_id, payment_id, post_id, account_id.
  If you pick the wrong key, you’ll get out-of-order behavior where it matters.
• Hot keys: “celebrity” users / viral posts create partition hotspots.
  Mitigate with dedicated partitions, salting, or multi-stage fanout.
• More partitions: more parallelism, but more overhead (rebalance, files, memory).
  Keep partitions aligned to scale targets and operational capacity.

• Rule: in a group, only one active consumer per partition.
• Scale: add partitions and consumers; handle rebalances with fast startup/state restore.
  Commit offsets frequently, but not so often you add overhead—measure.

• Pull systems: Kafka lets consumers control rate (pause/resume).
  Prefer bounded in-flight buffers to avoid OOM and long GC pauses.
• Levers: prefetch limits, max in-flight per partition, batch sizes, max.poll.records.
  Tune to keep latency stable and avoid long processing stalls that cause rebalances.
• Autoscale signal: consumer lag + processing time (not CPU alone).
  Lag growth is the best early indicator you’re falling behind.

• Use a stable key: natural ID (payment_id) or client-supplied idempotency_key.
• Protect side effects: DB unique constraint + UPSERT, or a dedupe table keyed by the idempotency key.
  Return the previous result if the request is duplicated.
• TTL for dedupe records: keep long enough to cover retry/replay windows.
  Example: 24h–7d depending on business and retention needs.

• Retry: exponential backoff + jitter; cap attempts.
  Prevent retry storms from taking down dependencies.
• DLQ: after N tries, move the message with error snapshot + headers.
  DLQ must be searchable and have a safe replay tool (with throttling).

• Choreography: services react to events (simple, but implicit).
  Harder to see the whole workflow; duplicates/timeouts need careful handling.
• Orchestration: a workflow engine drives steps and compensations (explicit, observable).
  Temporal/Step Functions: clear state machine, retries, timeouts, audits.

In one DB transaction:
  1) write business row (e.g., post/comment)
  2) write outbox row (event payload + outbox_id)

A relay publishes:
  - message key = outbox_id (or entity_id for ordering)
  - marks outbox row as sent (idempotent)

• Stream DB WAL/binlog with Debezium/DMS; consumers build search indexes, caches, analytics.
  Best for legacy producers you can’t modify.
• Watch schema evolution, ordering, and PII filtering in the stream.
  Use schema registry and field-level policies.

• Continuous low-latency pipelines (Flink/Kafka Streams/Beam/Spark SS).
  Stateful, checkpointed, replayable.
• Use for real-time metrics, fraud signals, materialized views, near-real-time ETL.
  Be explicit about state size and checkpoint cost.

• Periodic jobs over large datasets.
  Great for heavy joins, recomputations, backfills, re-indexing.

• Processing vs event-time: event-time handles late events correctly.
  Feeds and mobile clients produce late events frequently.
• Windows: tumbling, sliding, session.
  Choose based on how users behave (sessions) vs fixed intervals (tumbling).
• Watermarks: define how long you wait for late data.
  Allowed lateness is a latency vs correctness trade-off.

• Write-side: PostCreated, PostUpdated, FollowCreated, BlockUpdated
  These events drive fanout, indexing, and cache invalidation.
• Engagement: LikeCreated, CommentCreated, Impression
  Used for ranking features, counters, notifications, and analytics.
• Read-side signals: FeedOpened, FeedItemViewed
  Power experimentation, observability, and ranking training data.

• Queues: background work like media processing, notification sends, search re-index retries.
  Work distribution, retries, DLQ — tasks are “do once (idempotent)”.
• Streams: durable event log for fanout, ranking features, audit, replay.
  Multiple consumers (ranking, notifications, analytics, search) read independently.

• Ordering requirement: per post_id (update/delete must follow create).
  Key topic by post_id for post lifecycle; key engagement by post_id or viewer_id depending on consumer needs.
• Hot key: viral post becomes a hotspot.
  Use multi-stage fanout (partition by post_id but scale with subtopics/shards for engagement).

Assume:
  DAU: 200,000,000
  Feed opens/day: 1,600,000,000  (~18,519 avg RPS)
  Peak multiplier: 6x  => ~111,111 peak RPS

If each feed-open produces ~10 events (signals/metrics):
  Events/day: 16,000,000,000
  Avg events RPS: ~185,185 RPS

Design implications:
  - partition enough for peak
  - consumers autoscale on lag
  - bounded buffers + pause/resume

• Idempotency: use event_id or outbox_id; sinks do UPSERTs.
  Duplicates are expected in at-least-once pipelines.
• Poison messages: schema mismatch / corrupt payload → DLQ with replay tooling.
  Replays must be throttled to avoid re-triggering an incident.
• Privacy: filter/redact PII in streams; enforce access controls at read time as a safety net.
  Treat event streams as production data with governance.

• Producer: publish latency, retries, batch sizes, acks.
  Spikes often signal broker health or network issues.
• Broker: partition skew, leader load, disk usage, ISR health.
  Skew causes one partition to melt while others idle.
• Consumer: lag by partition, processing time, error/retry/DLQ rates, checkpoint age.
  Lag SLOs should be per topic and per consumer group.
• Pipeline: end-to-end latency, % late events, watermark delay.
  Streaming correctness depends on event-time and late data handling.

• “99% of events processed within 5s of arrival.”
• “DLQ < 0.1% over 24 hours.”
• “Consumer lag per partition stays under 30s at p99.”

• “Queues distribute work; streams are durable logs for many consumers.”
• “Ordering is per partition; we key by the entity that needs order and handle hot keys explicitly.”
• “We design for at-least-once and make consumers idempotent; effectively-once comes from idempotency + atomic commits.”
• “Outbox keeps DB and broker consistent; CDC feeds search/analytics without app changes.”
• “Lag and end-to-end latency are the two metrics I watch first.”