Design a Notification System — Cheat Sheet

A notification system delivers messages to users across multiple channels (push, email, SMS) reliably and at scale. The key challenges are multi-channel fan-out, per-user rate limiting to prevent spam, delivery tracking, and resilient retry logic so that transient provider failures never cause permanent message loss.

• ✓Send notifications via three channels: push (FCM/APNs), email (SendGrid/SES), and SMS (Twilio)
• ✓Clients submit a notification event; the system chooses the right channel(s) per user preference
• ✓Template engine — notifications are defined as templates with variable substitution (e.g. "Hi {{name}}, your order {{orderId}} has shipped")
• ✓Priority levels — CRITICAL (OTP, security alerts) vs MARKETING (promotions) with separate processing queues
• ✓Per-user rate limiting — no more than N notifications per channel per time window
• ✓Delivery tracking — record whether each notification was sent, delivered, opened, or failed

• ✓Notifications / day: 10M ≈ 116 / sec (peak ~500 / sec)
• ✓Push (60%): 6M / day ≈ 70 / sec
• ✓Email (30%): 3M / day ≈ 35 / sec
• ✓SMS (10%): 1M / day ≈ 12 / sec
• ✓Delivery record size: ~200 bytes per notification
• ✓Storage / day: 10M × 200B = 2 GB / day

• ✓Notification API — REST API that accepts notification events from upstream services (order service, auth service, marketing platform). Validates the payload, resolves the template, applies rate limiting, and enqueues to the appropriate Kafka topic.
• ✓Template Service — Stores notification templates in a database with variable placeholders. Renders a template by substituting variables from the event payload. Caches rendered templates for repeated sends (e.g. bulk marketing campaigns).
• ✓Priority Kafka Topics — Two Kafka topics per channel: notifications-push-critical, notifications-push-marketing, notifications-email-critical, notifications-email-marketing, notifications-sms-critical, notifications-sms-marketing. Critical topics have more partitions and dedicated consumers for lower latency.
• ✓Channel Workers (Push / Email / SMS) — Kafka consumers per channel that read from their topic, call the third-party provider API (FCM, SendGrid, Twilio), and update the delivery record. Each worker maintains a circuit breaker per provider.
• ✓Rate Limiter — Redis-backed sliding window rate limiter. Enforces per-user, per-channel limits (e.g. max 10 marketing push notifications per day). Invoked by the Notification API before enqueuing; over-limit events are dropped or deferred.
• ✓Delivery Tracker — Stores the lifecycle of every notification: QUEUED → SENT → DELIVERED / FAILED. Receives webhook callbacks from providers (e.g. FCM delivery receipts, SendGrid events). Exposes a query API for dashboards and debugging.

• ✓Single notification queue vs per-channel, per-priority queues → Per-channel, per-priority Kafka topics: A single queue means a marketing email backlog can delay OTP delivery. Separate topics give CRITICAL notifications dedicated consumer groups and more partitions, guaranteeing their latency SLA is met independently.
• ✓Synchronous delivery vs async via Kafka → Async via Kafka: Calling FCM/SendGrid/Twilio synchronously in the API request would tie API latency to provider latency. Kafka decouples ingest from delivery — the API responds in <10ms, and delivery happens asynchronously at provider speed.
• ✓Fixed retry interval vs exponential backoff with jitter → Exponential backoff with jitter: Fixed interval retries cause a thundering herd — all failed messages retry simultaneously after a provider outage, often overwhelming the newly-recovered provider. Exponential backoff with jitter spreads retries over time.
• ✓Token bucket vs sliding window for rate limiting → Sliding window (sorted set in Redis): Token bucket allows a burst at the window boundary (N tokens at minute 0:59, another N at 1:00 = 2N in ~2 seconds). Sliding window distributes the limit smoothly across the window with no boundary bursting.
• ✓Push-only vs multi-channel fan-out → Multi-channel with user preference routing: Delivery depends on the device being online, notification permissions, and carrier reliability. Routing to the best channel per user (and falling back on failure) maximises the probability of the message being seen.

• ✓Start with the three channels (push, email, SMS) and immediately introduce the separate priority queues. This shows you understand that a marketing blast should never delay an OTP.
• ✓The rate-limiting question almost always comes up. Know the difference between token bucket and sliding window and why sliding window avoids boundary bursts.
• ✓Retry logic is a key deep-dive. State "exponential backoff with jitter" — and explain that jitter prevents the thundering herd after a provider outage.
• ✓Mention idempotency keys before the interviewer asks. Channel workers must not double-deliver on retry — check the delivery tracker or pass a provider idempotency key.
• ✓Template rendering: note that templates are rendered once by the API before enqueuing, not inside the worker. This avoids hitting the template database for every retry.
• ✓Bring up circuit breakers around third-party providers. Resilience4j is a good Java reference. This shows production engineering thinking.