Notification System Design Interview: Multi-Channel Delivery, Fanout, and Idempotency at Scale
A large consumer platform routinely sends billions of notifications a day across push, SMS, email, and in-app. Uber's real-time push platform has reported peaks above 250,000 messages per second and 1.5M concurrent streaming connections, and LinkedIn's Air Traffic Controller processes over 1 billion notification decisions daily. At that volume, a single bug in the dedup or retry path can double-send to hundreds of millions of people, so correctness matters as much as throughput.
A notification system takes an event, such as "your driver is arriving" or "someone commented on your post," and turns it into one or more messages delivered across the right channels to the right users. The hard part is not sending one push; it is doing this reliably for billions of events a day while respecting user preferences, deduplicating retries, rate limiting per user and per channel, and tracking delivery status end to end. The standard shape is an ingestion API that validates and enqueues requests, a fanout stage that expands an event into per-user per-channel jobs, durable queues that decouple producers from delivery, and channel-specific workers that talk to APNs, FCM, an SMS aggregator like Twilio, and an email provider like SES or SendGrid. Idempotency keys and a dedup store prevent double sends, retries use exponential backoff with jitter, and messages that exhaust retries land in a dead-letter queue for inspection. Priority lanes keep a two-factor code from waiting behind a marketing blast. Templates and user preferences are looked up per message so content is localized and channels the user has muted are skipped.
Asked at: Asked at Uber, Amazon, Meta, LinkedIn, Airbnb, DoorDash, Stripe, and most companies that run large consumer or transactional messaging. It shows up both as a standalone "design a notification service" prompt and as a subsystem inside larger designs like a ride-hailing or social feed system.
Why this question is asked
Interviewers like this problem because it looks simple and is not. A weak candidate designs a single service that loops over users and calls APNs. A strong candidate sees the queueing, the fanout amplification, the idempotency requirement under at-least-once delivery, the per-channel failure modes, and the tension between transactional and promotional traffic. It tests whether you can decompose a pipeline, reason about delivery guarantees, handle third-party rate limits and outages you do not control, and design for observability so you can prove a message was delivered. It also surfaces product judgment: preferences, quiet hours, and aggregation are what separate a spammy system from one people tolerate.
Requirements
Always clarify these in the first 5 minutes of the interview. Do not start drawing boxes until both lists are agreed.
Functional requirements
- Accept notification requests from many internal services through a single API, keyed by user, template, and channel intent.
- Support at least four channels: mobile push (APNs for iOS, FCM for Android and web), SMS, email, and in-app or web inbox.
- Fan out a single logical event to multiple recipients and multiple channels per recipient.
- Apply user notification preferences, including per-category opt-outs, channel muting, and quiet hours, before sending.
- Render content from versioned templates with per-user variables and localization.
- Deduplicate requests so a retried or replayed event does not send the same notification twice.
- Rate limit per user, per channel, and per campaign to avoid flooding a recipient or tripping provider limits.
- Track delivery status per message (queued, sent, delivered, failed, opened) and expose it for querying.
- Separate transactional traffic (OTP, security alerts, order updates) from promotional traffic with independent priority lanes.
Non-functional requirements
- High availability: the ingestion path should stay up even when a downstream provider is degraded.
- Durability: an accepted request must not be lost, so persist before acknowledging.
- Low latency for transactional messages, targeting single-digit seconds from event to device for high-priority pushes.
- Horizontal scalability to billions of messages per day with no single bottleneck in fanout or delivery.
- At-least-once delivery with idempotent consumers so retries are safe.
- Isolation between channels and between priority classes so one slow provider does not stall the rest.
- Observability: per-message tracing, per-channel success and latency metrics, and DLQ visibility.
- Cost control, since SMS and email have real per-message cost and push does not.
Back-of-envelope scale estimates
Show your math. Pulling numbers from thin air signals you have not thought about the load.
Daily notification volume
~2 billion messages/day (assumed target)
Assume 200M active users receiving an average of 10 notifications a day across all channels. 200M x 10 = 2B messages/day. This is the working figure for the rest of the estimates; a smaller product scales the same design down.
Average send rate
~23,000 messages/sec average
Derived: 2B / 86,400 seconds ≈ 23,148 per second averaged over the day. Real traffic is bursty, so provision for several times this at peak.
Peak send rate
~115,000 messages/sec (5x average, estimate)
Derived: peaks from promotional blasts and time-zone clustering commonly run 5x average, so 23K x 5 ≈ 115K/sec. For comparison, Uber's push platform has published peaks above 250K messages/sec, so this order of magnitude is realistic.
Fanout amplification
1 event to 3-5 delivery jobs
A single event often expands to push plus email plus in-app for one user, and a broadcast expands to millions of users. Plan for the queue and worker tier to carry several times the inbound event rate.
Dedup store size
~120 GB hot, ~200M keys/day (estimate)
Derived: if we keep an idempotency key for each of 2B daily messages for a 24-hour window, and each entry is roughly 60 bytes of key plus metadata, that is about 120 GB. Kept in a TTL'd store like Redis or a wide-column table with a 1-day expiry.
Delivery status writes
~46,000 writes/sec (2x message rate)
Derived: each message produces at least a "sent" and a terminal "delivered/failed" status update, so 23K x 2 ≈ 46K status writes/sec average. Provider webhooks (email opens, SMS receipts) add more, which is why status is stored in a write-optimized store.
High-level architecture
A producing service (orders, chat, marketing) calls the Notification API with a request that names the user or audience, a template id, a category, and optional data variables. The API validates the request, attaches or reads an idempotency key, checks it against a dedup store, and if it is new, persists the request and enqueues it. Behind the API sits a fanout stage: for a single-user event it looks up that user's devices and channel preferences, and for a broadcast it expands an audience into per-user jobs, often reading from a precomputed subscriber list. Each fanned-out job is a per-user, per-channel unit of work placed on a durable message queue such as Kafka or a managed queue, partitioned by user id so all work for one user stays ordered on one partition. Channel workers consume from priority-specific topics: a transactional consumer group drains a high-priority topic, a bulk consumer group drains promotional topics, so an OTP never queues behind a newsletter. A worker loads the template, renders it with user variables and locale, re-checks preferences and rate limits, and calls the channel provider: APNs over HTTP/2 for iOS, FCM for Android and web, an SMS aggregator, or an email provider. The provider response drives status: success marks the message sent, a transient failure triggers a retry with exponential backoff and jitter, and a permanent failure or exhausted retries routes the message to a dead-letter queue. Providers also send asynchronous receipts (APNs feedback, email opens, SMS delivery reports) through a webhook ingestion service that updates the same delivery-status store. A separate scheduler handles future-dated and quiet-hours sends by buffering jobs until their fire time.
In a real interview, sketch this on the whiteboard before diving into any single box.
Core components
Walk through each service. The interviewer wants to hear what each one owns, not just the names.
Notification API / ingestion gateway
The single entry point that all internal producers call. It authenticates the caller, validates the payload against the template schema, assigns or reads an idempotency key, checks the dedup store, and persists then enqueues accepted requests. It returns fast, doing no delivery work inline, so producers are decoupled from provider latency.
Fanout service
Expands one logical request into concrete per-user, per-channel delivery jobs. For a targeted event it resolves the user's registered devices and channels; for a broadcast it streams an audience list and shards it. This is where amplification happens, so it is designed to run in parallel and to backpressure onto the queue rather than hold state in memory.
Durable message queue with priority topics
Kafka or an equivalent durable log sits between fanout and delivery, partitioned by user id for ordering and grouped into separate topics per priority class. Decoupling here means a provider outage backs up a topic instead of failing user requests, and consumer groups scale independently per channel.
Channel workers
Stateless consumers, one pool per channel, that render templates and call the provider. Push workers speak APNs HTTP/2 and FCM, SMS workers call an aggregator, email workers call SES or SendGrid. Each pool tunes its own concurrency, batching, and rate limits to match the provider it owns.
Preference and template service
Stores per-user channel preferences, category opt-outs, quiet hours, and locale, plus versioned message templates. Workers query it right before sending so a preference change or template fix takes effect immediately rather than being baked in at enqueue time.
Dedup and idempotency store
A fast TTL'd store, typically Redis or a wide-column table, keyed by idempotency key or a content hash. A SET-if-not-exists on the key gates whether a message is sent, which makes at-least-once queue delivery and producer retries safe against double sends.
Delivery tracking and webhook ingestion
A write-optimized store records each message's lifecycle (queued, sent, delivered, failed, opened) and a webhook service ingests asynchronous receipts from providers to close the loop. This feeds status APIs, metrics, and the dead-letter queue tooling.
Data model
Pick the right store per table. Justify each choice with the access pattern, not by reflex.
notification_requestrequest_id (PK)idempotency_key (unique)producer_servicetemplate_idcategorycreated_atOne row per accepted logical request before fanout. The unique index on idempotency_key is the first dedup gate. Kept short-term; archived or TTL'd after the delivery window closes.
delivery_jobjob_id (PK)request_id (FK)user_idchannelprioritystatusattempt_countnext_retry_atOne row per user-per-channel unit of work. Partitioned or sharded by user_id. status moves queued to sent to delivered/failed. attempt_count and next_retry_at drive backoff.
device_tokentoken_id (PK)user_idplatform (ios/android/web)push_tokenlast_seen_atis_validMaps users to APNs/FCM tokens. Tokens rotate and expire, so provider feedback (410 gone, unregistered) flips is_valid so workers stop sending to dead tokens.
user_preferenceuser_id (PK)categorychannelenabledquiet_hourslocalePer-category, per-channel toggles plus quiet hours and locale. Read on the hot path by workers. Composite key on (user_id, category, channel) so a lookup is a single point read.
templatetemplate_idversion (PK together)channellocalebodyvariablesVersioned templates per channel and locale. Workers pin a version at render time. Keeping versions lets you roll back a bad template without redeploying.
dedup_keydedup_key (PK)user_idcreated_atttlTTL'd store (Redis/wide-column) written with SET NX at send time. Prevents duplicate delivery of the same content to the same user inside the window even when the queue redelivers.
Deep dives
These are the conversations the interviewer is steering you toward. Practice each one until you can talk through it without notes.
Fanout: single-recipient vs broadcast amplification
Fanout is where one event becomes many messages, and the two cases behave very differently. A targeted event ("your order shipped") expands to a handful of jobs: this user's iOS push, their email, and their in-app inbox entry. A broadcast ("flash sale for everyone in this city") can expand to tens of millions of jobs from one request, and doing that synchronously would time out the API. The pattern is to accept the request fast, then fan out asynchronously by streaming the audience from a precomputed subscriber list or a query, sharding it into batches, and writing per-user jobs onto the queue partitioned by user id. Partitioning by user id keeps all of a user's notifications ordered and lets you enforce per-user rate limits on a single partition. The risk is amplification overwhelming a downstream provider, so fanout writes to the queue and lets consumer groups pull at a rate the provider can absorb, rather than pushing. This is the fan-out/fan-in shape: expand, process in parallel, and collapse results into delivery status.
Idempotency and deduplication under at-least-once delivery
Queues give at-least-once delivery, and producers retry on timeouts, so the same message will sometimes be processed more than once. Without a guard, a network blip during an OTP send can text a user their code twice, or a marketing replay can email millions of people again. The fix is an idempotency key carried end to end. The producer supplies a deterministic key (or the API derives one from a content hash of user + template + a business event id), and the API does a SET-if-not-exists against a TTL'd dedup store before enqueuing. A second layer runs in the worker just before the provider call, because a job can be redelivered after the API check but before the send completes. The dedup store needs a TTL that covers the maximum retry window, commonly 24 hours, so it does not grow unbounded. The subtlety is picking the key granularity: too coarse and you suppress legitimately distinct notifications, too fine and duplicates slip through. Tying the key to the originating business event, not to a per-attempt id, is the usual answer. This is the idempotent-consumer pattern applied across two boundaries.
Retries, exponential backoff, and dead-letter queues
Provider calls fail in two ways that must be treated differently. Transient failures (a 429 rate limit, a 503, a TCP timeout, an APNs TooManyRequests) should be retried; permanent failures (an invalid token, an unsubscribed email, a malformed payload) should not, because retrying wastes budget and can look like abuse. Workers classify the provider response, and for retryable errors they reschedule the job with exponential backoff plus jitter: roughly 1s, 2s, 4s, 8s with a random spread so a provider outage does not create a synchronized retry storm when it recovers. After a bounded number of attempts, the job goes to a dead-letter queue instead of retrying forever. The DLQ is not a graveyard; it is an operational tool. Messages there are inspected, and depending on the failure they are fixed and replayed (a transient provider incident) or dropped (a permanently dead token). Keeping the DLQ per channel and per failure reason makes triage fast. For transactional messages you often keep the retry window short and aggressive, since a security alert delivered an hour late is nearly useless.
Priority lanes: transactional vs promotional
The single most important isolation in a notification system is separating traffic that a user is waiting for from traffic they merely tolerate. An OTP, a password reset, or a fraud alert is transactional: latency-sensitive, low volume, high value. A newsletter or a re-engagement nudge is promotional: high volume, bursty, latency-tolerant. If they share a queue, a 10-million-message marketing blast will bury a two-factor code behind an hour of backlog. The design gives each priority class its own topics and its own consumer pool, with the transactional pool provisioned to drain quickly and never starved. Some systems add a third "critical" lane for security messages that bypasses aggregation and quiet hours entirely. Rate limiting is also priority-aware: promotional sends are shaped and throttled to smooth peaks, while transactional sends are allowed to burst. Uber's platform explicitly uses priority queues so time-critical pushes are not delayed by bulk traffic, and this is the property interviewers look for when they ask how you would keep an OTP fast during a big campaign.
Channel providers: APNs, FCM, SMS, and email specifics
Each channel is a different integration with its own limits, and pretending they are interchangeable is a common mistake. APNs uses an HTTP/2 connection with token-based JWT auth (ES256, refreshed at least hourly) and a 4KB payload cap; it returns per-request status and reports dead tokens you must stop using. FCM targets a token, a topic, or a device group, supports high and normal priority, collapsible messages, and a TTL, and covers Android and web. SMS goes through an aggregator like Twilio, is metered per segment (160 GSM-7 characters, fewer for Unicode), and has real cost and carrier throughput limits, so batching and number pooling matter. Email through SES or SendGrid is cheap but needs SPF/DKIM/DMARC alignment, bounce and complaint handling, and reputation management or you land in spam. Because payload limits are tight on push, the pattern is to send a small notification with an id and let the app fetch heavy content on tap. Workers keep persistent connections and respect each provider's rate limits, and token invalidation from any provider feeds back to flip device_token.is_valid.
User preferences, quiet hours, and aggregation
Delivery is not just technical routing; it is a product decision about whether to send at all. Before any send, the worker checks the user's preferences: has this category been opted out, is this channel muted, is it inside the user's quiet hours in their local timezone, has this user already hit a daily frequency cap. LinkedIn's Air Traffic Controller is the reference design here: it is explicitly not the thing that sends the email or push, it is the decision layer that decides which channel, whether to send, and when, using per-member state co-located by partitioning all of a member's signals to the same host. Aggregation matters too: three people liking a post should become one "3 people liked your post" notification, not three buzzes, which requires buffering and a short delay window keyed by user. This is a direct tension with latency, so aggregation applies to social and promotional traffic, never to transactional. Doing preferences on the hot path (worker time) rather than at enqueue time means a user who mutes a channel stops receiving on it immediately, even for already-queued messages.
Delivery tracking and status reconciliation
Proving a notification was delivered is harder than sending it, because delivery is asynchronous and the truth arrives later through separate channels. The synchronous provider response only tells you the message was accepted for delivery, not that it reached the device. Real delivery and engagement come back as webhooks: APNs and FCM report unregistered tokens, email providers post bounces, complaints, and open events, and SMS aggregators send delivery receipts. A webhook ingestion service consumes these and updates a write-optimized delivery-status store keyed by message id, moving a message from sent to delivered, failed, or opened. This store is high write volume (at least two writes per message plus receipts), so it favors a wide-column database like Cassandra over a relational one, and status is often kept for a bounded retention window. On top of it you build a status API for support ("did the user get their receipt?"), per-channel dashboards, and alerting on delivery-rate drops that often signal a provider incident or a bad template before users complain.
Trade-offs to discuss
Every senior interviewer expects you to surface at least 3 of these. Pick the decisions, state the alternatives, and justify your choice.
At-least-once with idempotent consumers versus exactly-once delivery
True exactly-once across a network and third-party providers is effectively impossible, since you cannot atomically send a push and record that you sent it. The practical choice is at-least-once delivery plus an idempotency key and a dedup store, which turns unavoidable duplicates into no-ops. You accept some storage cost for dedup keys in exchange for correctness that actually holds under failure.
Separate priority queues versus a single shared queue
A shared queue is simpler and cheaper to operate, but it lets bulk promotional traffic delay time-critical transactional messages during a spike. Separate topics and consumer pools per priority class cost more moving parts but guarantee an OTP is never stuck behind a newsletter. For anything with security or OTP traffic, the isolation is worth it.
Fanout on write versus fanout on read for broadcasts
Fanout on write (materialize a job per recipient up front) gives fast, trackable per-user delivery but amplifies storage and queue load massively for huge audiences. Fanout on read (compute recipients lazily) saves storage but complicates status tracking and rate limiting. Most notification systems fan out on write for targeted sends and stream-shard very large broadcasts to bound the burst.
Checking preferences at enqueue time versus at send time
Evaluating preferences when the job is created is cheaper because you do it once, but a user who mutes a channel afterward still gets already-queued messages. Checking at send time in the worker costs an extra lookup per message but honors preference changes immediately, which is the behavior users expect and regulators sometimes require.
Managed provider (SNS, OneSignal) versus building the pipeline in-house
A managed service gets you multi-channel delivery quickly and offloads provider integrations, but you cede control over priority lanes, dedup semantics, delivery-data ownership, and per-message cost at very high volume. Companies like Uber and LinkedIn build in-house precisely because at their scale they need the routing intelligence and the cost control that a generic service will not give.
Storing delivery status in a wide-column store versus a relational database
Status is write-heavy, append-mostly, and queried by message or user id, which fits Cassandra-style stores that scale writes horizontally. A relational database gives richer queries and transactions but becomes a write bottleneck at tens of thousands of status updates per second. The usual answer is a wide-column store for status with a bounded retention window.
How Notification System actually does it
Uber's real-time push platform, RAMEN, is a strong public reference: it started on Server-Sent Events and later moved to gRPC bidirectional streams, uses a Fireball service to decide when to push and an API gateway to decide what to push, shards connections with Apache Helix and ZooKeeper via Streamgate workers, and has reported peaks above 250,000 messages per second with more than 1.5M concurrent connections and priority queues for time-critical traffic. LinkedIn's Air Traffic Controller is the reference for the decision layer: built on Apache Samza and Kafka, it partitions all of a member's signals to one host so channel selection, aggregation, delivery-time optimization, and relevance filtering run against co-located state, processing over a billion notification decisions a day. On the delivery edge, Apple's APNs is an HTTP/2 provider API with token-based ES256 JWT auth and a 4KB payload limit, and Google's FCM handles Android and web with token, topic, and device-group targeting plus message priority, TTL, and collapsible messages. Amazon SNS is the common managed building block for pub/sub fanout and mobile push when teams do not build their own.
Lessons to study before this interview
If any of these topics are fuzzy, the interviewer will catch it. Each lesson is 15 to 60 minutes with diagrams, code, and a quiz.
Related system design interview questions
Practice these next. They lean on the same core building blocks as Notification System.