System design interview guide
Design ChatGPT: LLM Serving System Design Interview
ChatGPT has to answer a question in the second or two before a user feels the pause, and it has to do that for a userbase OpenAI put at about 900 million weekly actives in February 2026, sending on the order of 2.5 billion messages a day. That averages to nearly 30,000 messages a second before you account for peaks. The hard part is not the web tier, it is that every one of those messages occupies a GPU for several seconds while the model emits tokens one at a time, and the memory that holds each conversation's attention state grows with every token generated. A large model does not even fit on one card: a 70B-class model's weights alone are about 140 GB, so it is served across a node of several GPUs, and after the weights are loaded the memory that is left is what holds the per-conversation KV cache. A single 8K-token conversation can hold about 2.5 GB of that cache, so the node runs out of memory long before it runs out of compute. Capacity, latency, and cost in this system are all decided by how well you manage GPU memory.
Designing ChatGPT looks like designing a chat app until you notice the model is stateless and the GPU is the entire problem. Every turn resends the whole conversation, so the model reprocesses the full history to produce the next reply, and the cost of a conversation grows as it gets longer. Inference itself splits into two phases with opposite bottlenecks: prefill reads the prompt in parallel and saturates compute, while decode emits one token at a time and is limited by memory bandwidth, because each token has to re-read the model weights. That asymmetry drives nearly every decision. The KV cache that makes decode affordable is also what fills the GPU, so PagedAttention borrows virtual-memory paging to stop fragmentation from wasting most of the card, and continuous batching swaps finished sequences out at each iteration instead of waiting for the slowest one. Prefix caching turns a shared system prompt and a repeated conversation history into a cache hit rather than recomputation. Speculative decoding exploits the fact that verifying several tokens costs about the same as generating one. Then the product layer adds its own constraints: streaming so the answer starts before it is finished, moderation on both the input and the streaming output, token-based quotas rather than request-based ones, and routing that treats a GPU holding long conversations as full even when its request count looks low. The chat CRUD is a weekend project. The interview is about GPU memory, batching, and the latency budget.
Where it shows up
Asked at OpenAI, Anthropic, Google, Meta, Microsoft, NVIDIA, Databricks, Perplexity, and the large set of companies now serving their own LLM products or internal assistants. It appears as a standalone AI infrastructure question, as an inference-serving question for platform and performance roles, and as the generation half of broader designs such as an AI agent platform or an enterprise copilot.
Why this question is asked
It is the cleanest way to find out whether a candidate understands that serving a large language model breaks the assumptions behind ordinary web architecture. The instincts that work everywhere else mislead here. Requests are not uniform, they occupy a GPU for seconds and hold memory that grows the whole time. Load balancing on request count or CPU is wrong, because the real measure of how full a server is is KV cache occupancy. Autoscaling in seconds is impossible, because loading a hundred gigabytes of weights takes minutes. Caching a response by request hash almost never hits, but caching the prefix of a prompt hits constantly. An interviewer can watch a candidate discover, or fail to discover, that the binding constraint is memory rather than compute, and that latency has two separate numbers, time to first token and time per output token, which are governed by different phases and pull against each other. It also rewards honest cost reasoning, because at these volumes the difference between a well-batched cluster and a poorly batched one is a very large amount of money, and it forces the candidate to talk about safety and abuse in a system whose output is generated rather than retrieved.
Requirements
Always clarify these in the first 5 minutes of the interview. Do not start drawing boxes until both lists are agreed.
Functional requirements
- Hold a multi-turn conversation, using the full history so follow-up questions resolve correctly
- Stream the response token by token so the user sees an answer forming rather than waiting for a complete reply
- Support a reasoning mode where the model can think for many tokens before answering, without that hidden work stalling other users
- Let the model call tools mid-generation (web search, code execution, retrieval) and resume with the results
- Accept image and voice input, not only text, routing each through the right encoder before generation
- Let a user stop a generation mid-stream, regenerate a response, and edit an earlier message to branch the conversation
- Persist conversations so a user can leave and return to them across devices
- Offer a choice of models with different capability, latency, and cost profiles
- Moderate both the incoming prompt and the outgoing response, including while it is still streaming
- Enforce per-user quotas and tiering so free, subscriber, and API traffic get different service
Non-functional requirements
- Per-class latency targets, not one number: interactive chat wants time to first token under about 1 second and time per output token around 30ms, while a reasoning or batch request trades latency for throughput and is scheduled differently
- Steady time per output token rather than jittery, since a stalled token stream is more noticeable to a reader than a slightly slower but even one
- Handle tens of thousands of messages per second with hundreds of thousands of concurrent generations at peak
- Maximize goodput, the tokens per GPU-second that actually meet their latency target, rather than raw throughput that may be violating SLOs
- Degrade gracefully under overload by queueing and shedding to lower tiers rather than failing everyone
- Survive a GPU failing mid-generation without losing the user's answer, and keep conversation data durable and isolated per user with deletion that actually removes it
Back-of-envelope scale estimates
Show your math. Pulling numbers from thin air signals you have not thought about the load.
User and message volume
~900M weekly actives, ~2.5B messages/day
OpenAI put weekly actives at about 900 million in February 2026, and has separately stated roughly 2.5 billion messages a day. That averages to roughly 29,000 messages per second, and real traffic peaks well above its average, so the cluster must be sized for a multiple of that.
Concurrent generations
~450K concurrent streams at average load
At the interactive latency target of about 30ms per output token, a typical 500-token reply occupies a GPU slot for roughly 15 seconds. 29,000 new messages per second times a 15 second hold is on the order of 450,000 sequences generating at any instant, and several times that at peak. This is Little's Law: concurrency is arrival rate times how long each request stays in the system.
KV cache per conversation
~320 KB per token, ~2.5 GB at 8K context
For a 70B-class model with 80 layers using grouped-query attention with 8 KV heads and a head dimension of 128 at FP16: 2 (K and V) x 80 x 8 x 128 x 2 bytes is 320 KB per token. An 8,000 token conversation therefore holds about 2.5 GB, which is why GPU memory, not FLOPs, sets the concurrency limit.
GPU fleet floor
~2,000 8-GPU nodes, on the order of ten thousand GPUs, for decode alone
An 8-GPU node serving a quantized 70B has roughly a couple of hundred concurrent sequences worth of KV cache after the weights are loaded. 450,000 concurrent streams divided by ~200 per node is on the order of 2,000 nodes, so roughly ten thousand GPUs, just to hold the KV cache, before peak headroom, prefill capacity, and redundancy. Halve the batching efficiency and you double that number, which is why batching is worth so much money.
Cost per million output tokens
order of $0.50 at cost, versus dollars at list
Ten thousand H100s at a rough $2 per GPU-hour is about $20,000 an hour. If that fleet sustains on the order of 10 million output tokens a second across all its batches, an hour produces tens of billions of tokens, which lands the raw serving cost in the region of a dollar or less per million output tokens against a list price measured in dollars. The exact figure depends entirely on batching efficiency, which is the same point the fleet estimate makes: the gross margin on inference is almost entirely a function of how full the batches run.
A reasoning request holds a slot far longer
minutes, not the ~15 seconds of a chat reply
A reasoning request can emit tens of thousands of hidden thinking tokens before it produces a visible answer. At the same ~30ms per token that is minutes of GPU occupancy for a single request, tens of times a normal chat turn, so a small fraction of reasoning traffic can consume a large fraction of the fleet and has to be scheduled as its own class.
High-level architecture
The product tier looks conventional and is deliberately kept that way. A client opens a streaming connection, an API gateway authenticates the user, checks their tier and token quota, and applies input moderation. A conversation service loads the message history from a database, assembles the prompt, and hands the request to the inference layer, then persists the completed response and updates usage accounting. Everything interesting is behind that handoff. An inference router picks a serving replica, and its scheduling input is not request count or CPU but KV cache occupancy, because a replica holding a few long conversations is genuinely fuller than one holding many short ones. The router also prefers a replica that already holds the prefix of this conversation in its cache, so session affinity turns the repeated history into a cache hit instead of recomputation. Inside a replica the model is sharded across GPUs with tensor parallelism inside a node, where NVLink makes the per-layer all-reduce cheap, and the serving engine runs continuous batching: at every decode iteration it admits newly arrived requests and evicts finished ones, rather than locking a batch until its slowest member completes. KV cache memory is managed by PagedAttention in fixed-size blocks so a long conversation does not need one contiguous reservation, and blocks holding an identical shared prefix are referenced by several sequences at once. Prefill and decode either run on separate GPU pools, so a long prompt cannot stall other users' token streams, or are interleaved with chunked prefill on the same pool. Generated tokens stream back through the gateway over server-sent events, passing an output moderation classifier as they go, and a cancelled stream immediately frees its GPU slot and KV blocks. Around all of this sits the ordinary but necessary machinery: conversation storage, usage metering for billing and quotas, feedback capture for evaluation, and capacity management that has to be predictive because a new replica takes minutes to load its weights.
In a real interview, sketch this on the whiteboard before diving into any single box.
Core components
Walk through each service. The interviewer wants to hear what each one owns, not just the names.
API gateway and streaming layer
Terminates the client connection, authenticates, enforces tier and token quota, and streams tokens back over server-sent events. It matters more than it looks: it must propagate cancellation promptly, because an abandoned stream that keeps generating is pure wasted GPU, and it must hold a very large number of mostly-idle long-lived connections cheaply.
Conversation service
Owns history and prompt assembly. Because the model is stateless, this service resends the conversation on every turn and decides what to do when the history outgrows the context window: truncate the oldest turns, summarize them, or retrieve only the relevant ones. It is also where branching from an edited message is handled, which makes a conversation a tree rather than a list.
Inference router
Chooses a replica using KV cache occupancy and prefix locality rather than round-robin, since the ordinary load signals are actively misleading here. Routing a follow-up turn to the replica that already holds the conversation's prefix converts an expensive prefill into a cache hit, so affinity is a throughput feature and not just a nicety.
Serving engine (continuous batching + PagedAttention)
The core. It runs iteration-level scheduling so requests join and leave the batch every decode step, and manages KV cache as fixed-size pages so memory is not fragmented by variable-length conversations. These two techniques together are the difference between a GPU fleet that is mostly busy and one that is mostly idle.
Model execution layer
Shards the model with tensor parallelism inside a node and pipeline parallelism only when a model cannot fit in one node, since pipeline stages introduce bubbles, the idle gaps while one stage waits for the one before it. Quantization to FP8 or INT8 shrinks the weights and frees memory for more KV cache. It speeds up decode too, though the gain is modest once the batch is large, because at that point each sequence's own KV read, not the shared weight read, is the bigger part of the memory traffic.
Prefix cache
Keeps the KV state of shared prompt prefixes so the system prompt and the conversation history are not recomputed on every turn. This is the cache that actually hits in an LLM product, unlike a response cache keyed on the whole request, and it is why providers can offer a discount on cached input tokens.
Moderation and safety
Classifies the input before it reaches a GPU, which is also a cheap way to reject abuse before paying for it, and screens the output while it streams, which is harder because tokens have already been sent. In practice you buffer a small window before emitting so a violation can be caught before the user sees it, trading a little latency for the ability to stop mid-answer.
Metering, quotas, and evaluation
Counts tokens rather than requests, because a request can be a hundred tokens or a hundred thousand and only tokens map to cost. It also captures feedback and samples conversations for offline evaluation, so a change to the model, the prompt, or the truncation policy can be measured instead of guessed at.
Tool orchestrator
Runs the loop when the model decides to call a tool. It pauses generation, executes the web search, retrieval, or sandboxed code, and feeds the result back so generation resumes. The design question it forces is what happens to the KV cache during a multi-second tool call: hold it and the memory sits idle destroying occupancy, or evict it and pay a re-prefill, which the prefix cache makes survivable.
Multimodal encoders
A vision encoder turns an image into tokens that enter the prompt, and voice mode adds a low-latency speech path with its own duplex and barge-in requirements. Each is a separate pool with its own scaling, and image tokens count as prefill cost, so a picture is not free context, it is a chunk of the compute budget before a single output token is produced.
Data model
Pick the right store per table. Justify each choice with the access pattern, not by reflex.
conversationsconversation_iduser_idtitlemodelcreated_atupdated_atOne row per thread, indexed by user and recency for the sidebar. Deletion has to cascade for real, since users expect a deleted conversation to be gone rather than hidden.
messagesmessage_idconversation_idparent_message_idrolecontenttoken_countcreated_atThe parent pointer is what makes editing and regenerating work: a conversation is a tree, and the visible thread is one path through it. Storing the token count per message lets the prompt assembler fit the context window without re-tokenizing the history each turn.
usage_eventsuser_idrequest_idmodelprompt_tokenscached_tokenscompletion_tokenstsAppend-only and the basis for both billing and quota. Separating cached from uncached prompt tokens matters because they do not cost the same, and it is the metric that shows whether prefix caching is actually working.
feedbackmessage_iduser_idratingreasontsThumbs up and down joined back to the exact model version and prompt that produced the message, which is the only way a quality regression can be attributed to a specific change.
Deep dives
These are the conversations the interviewer is steering you toward. Practice each one until you can talk through it without notes.
Prefill and decode have opposite bottlenecks
Inference runs in two phases that behave nothing alike, and one mechanical fact explains the difference. A forward pass has to read every weight in the model once. In prefill, that single read of the weights is used to process every token of the prompt in parallel, so it is a big matrix multiply doing hundreds of arithmetic operations per byte fetched, and it saturates the GPU's compute units. In decode, that same full read of the weights produces just one token for one sequence, a matrix times a single vector, a handful of operations per byte, so the GPU sits mostly idle waiting on memory. Decode is therefore limited by memory bandwidth, not arithmetic. This single fact explains the shape of the whole system. It is why the two user-facing latency numbers are different: time to first token is dominated by queueing and prefill, while time per output token is dominated by decode. It is why batching helps so much, since 200 sequences decoding together share one read of the weights, turning that matrix-vector into a matrix-matrix multiply that does far more useful work per byte fetched. Batching does not make decode purely compute-bound, though: each sequence still has to read its own KV cache every step, so past a modest batch the bottleneck moves from reading the weights to reading the caches, and stays in memory. And it is why quantization speeds up decode, because smaller weights mean fewer bytes to read per step.
The KV cache is the real capacity limit
Attention needs the keys and values of every previous token, so the engine caches them rather than recomputing the whole history for each new token, turning generation from quadratic into linear work. The cost is memory that grows with every token and lives for the whole generation. For a 70B-class model with grouped-query attention that is about 320 KB per token, so an 8K conversation holds roughly 2.5 GB and a 128K one would hold about 40 GB for a single conversation. To put 40 GB in perspective, the model's own weights are already about 140 GB, so it is served across a node of several GPUs, and one maxed-out conversation can eat a large slice of the node's remaining memory on its own. Grouped-query attention exists largely to fight this: with 64 query heads sharing 8 KV heads it cuts the cache by exactly 8x versus full multi-head attention, which would put the same 8K conversation at 20 GB. Since a node's memory is spent first on weights and only what remains holds KV cache, the arithmetic of weights plus cache, not FLOPs, sets how many users it can serve at once.
PagedAttention: virtual memory for the KV cache
The obvious way to allocate KV cache is a contiguous buffer sized for the longest possible output, but you never know in advance how long a reply will be, so that reservation is mostly empty and the memory is both internally wasted and externally fragmented. The vLLM paper measured that naive schemes waste the majority of the cache. PagedAttention applies the operating system's answer: split the cache into fixed-size blocks, let a sequence's blocks be scattered anywhere in memory, and keep a per-sequence block table mapping logical positions to physical blocks. Waste drops to at most one partly-filled block per sequence, which means far more sequences fit at once, which raises the batch size, which is precisely what makes decode efficient. It also enables sharing: several sequences with an identical prefix can point at the same physical blocks and copy on write only when they diverge, which is what makes a shared system prompt nearly free.
Continuous batching beats static batching
A static batch is assembled, run to completion, and returned, so every sequence in it is held hostage by the longest one. When replies vary from twenty tokens to two thousand, most of the batch's slots sit finished but occupied, and the GPU does padding work instead of useful work. Continuous batching schedules at the granularity of a single decode iteration: after every token, finished sequences leave and waiting ones join. Utilization rises sharply and, just as importantly, a newly arrived request no longer waits for a whole batch to drain, which improves time to first token as well as throughput. This is the default in modern engines and is the reason a naive request-per-GPU design is off by an order of magnitude on cost.
Prefix caching is the cache that actually hits
Caching a whole response keyed on the request almost never hits, because prompts are rarely identical. But the beginning of the prompt is highly repetitive: the system prompt is the same for everyone, and turn five of a conversation starts with the exact tokens of turns one through four, which were already processed a moment ago. Caching the KV state of those prefixes means each new turn only prefills the newly added tokens. The gain compounds over a long conversation, where the history dwarfs the new question, and it is why routing a follow-up back to the replica that holds the prefix is worth real engineering. Systems such as SGLang manage this with a radix tree over cached prefixes so the longest match is found automatically, and providers expose the benefit directly as cheaper cached input tokens.
Speculative decoding buys latency from spare compute
Because decode is memory-bound, a forward pass that checks several candidate tokens costs almost the same as one that produces a single token; the weights are read once either way. Speculative decoding exploits that. A small, cheap draft model proposes the next few tokens, and the large model verifies all of them in one pass. Each drafted token is accepted with a probability set by how the large model's own likelihood for it compares to the draft's, so a token the large model would never have chosen can still be rejected, and at the first rejection the large model resamples that position from a corrected distribution. That acceptance rule is precisely what makes the output distribution provably identical to running the large model alone, so this is a pure speed optimization rather than a quality trade, and every pass yields at least one token. When the draft is good, several tokens land per verification and the stream can run two to three times faster. The catch is that the win comes from spending spare compute, so it is largest on lightly-batched, latency-sensitive traffic; at the very large batches a busy cluster runs, the spare compute is already gone and the benefit shrinks.
Prefill and decode interfere, so separate or interleave them
Running both phases on the same GPU creates a scheduling conflict with a visible symptom. A long prefill is a compute-heavy job that occupies the GPU for a while, and every user currently streaming tokens on that card stalls until it finishes, so their output stutters. There are two accepted answers. Disaggregation, as in DistServe and Splitwise, puts prefill and decode on separate GPU pools sized independently and ships the KV cache between them, which protects steady token streams and lets each pool be tuned for its own bottleneck, at the cost of a cache transfer across the interconnect. Chunked prefill, as in Sarathi-Serve, instead splits a long prompt into pieces and interleaves them with decode steps, so no single prefill can monopolize the GPU. Both are choosing to protect time per output token, and knowing why that choice exists is what separates a serious answer.
Everything is a throughput versus latency dial
Larger batches amortize the weight read across more sequences and raise tokens per GPU-second, which lowers cost, but each individual token now waits behind more work, so per-user latency rises. This is the central tension of the system and it does not have a single right setting: it has a setting per tier. Interactive chat needs a latency ceiling and accepts lower utilization; asynchronous or batch API traffic can be packed aggressively because nobody is watching it arrive. That is a real architectural reason to run separate pools with different batching policies rather than one cluster with an averaged compromise, and it is also why the honest answer to how many users a GPU serves is always a question about the latency target.
Capacity cannot scale in seconds
Normal services autoscale in response to load because a new instance is ready in seconds. An inference replica has to acquire a scarce GPU, pull a hundred gigabytes or more of weights, and load them into memory, which takes minutes. That makes reactive autoscaling useless as a defence against a spike. The practical design is predictive: provision to forecast demand, keep warm pools, and treat overload as a queueing and prioritization problem instead of a scaling one. When the cluster saturates you protect the paying tiers, shed or queue the free tier with an honest wait, and where possible fall back to a smaller model, because a fast answer from a lesser model usually beats an error.
The model is stateless, so context management is a product decision
The model remembers nothing between calls. Every turn resends the entire conversation, which means a long chat gets steadily more expensive and eventually exceeds the context window, and something has to give. Truncating the oldest turns is simple and silently drops facts the user thinks the assistant knows. Summarizing older turns preserves the gist and loses detail, and the summary itself costs a model call. Retrieving only the relevant earlier turns is the most scalable and turns the conversation into a small retrieval problem. Persistent memory across conversations is a further step with its own privacy and correctness burden. There is no neutral choice here, since each one changes what the assistant appears to know, which is why it belongs in the design discussion rather than being waved away.
A frontier model is a mixture of experts, which changes the math
The dense model used above to make the arithmetic clean is not what a current frontier system serves. Production models are mixtures of experts: each layer has many expert sub-networks, and a router sends each token to only a few of them, so a model with a trillion total parameters may activate only tens of billions per token. That decouples capacity from per-token compute, but it moves the serving problem rather than removing it. The experts still all have to live in GPU memory, so the weights now span many GPUs and add a third kind of parallelism, expert parallelism, on top of tensor and pipeline. Routing tokens to their chosen experts turns the tidy per-layer all-reduce into an all-to-all shuffle across the fleet, which is bandwidth-hungry and sensitive to imbalance when a popular expert becomes a hotspot. And the batching economics invert: because decode must read every activated expert but each expert serves only the tokens routed to it, you need a large batch just to keep each expert busy, which pushes MoE serving toward bigger batches than a dense model would want. Attention variants like multi-head latent attention and sliding-window attention push the KV cache down further than grouped-query attention does. The dense numbers are the right way to learn the shape; the MoE reality is what an interviewer at a frontier lab will ask you to redo.
Reasoning traffic breaks the single latency model
A reasoning request thinks before it answers, emitting a long internal chain of tokens the user never sees, sometimes tens of thousands of them, before the visible reply begins. That one feature breaks several assumptions at once. The concurrency estimate built on a ten to fifteen second hold is wrong for these requests, which hold a slot for minutes, so a small share of reasoning traffic can eat a large share of the fleet. Output length is now unpredictable and unbounded, which defeats any scheduler that wanted to run short jobs first, and it creates head-of-line blocking where one giant generation sits in a batch slot while shorter requests wait. The clean answer is to treat reasoning as its own traffic class with its own SLO and its own pool, to make reasoning effort a dial the product can turn down under load, and to accept that time to first token for a reasoning request is really time to first visible token, which can be many seconds of legitimate thinking. A system designed only for short chat replies will fall over the first time reasoning traffic spikes.
The unit economics are the whole argument
Everything above claims that GPU cost dominates, so it is worth actually closing the loop with numbers. Put ten thousand H100s at roughly two dollars per GPU-hour and the fleet costs about twenty thousand dollars an hour to run. What it produces is tokens, and the rate depends entirely on how full the batches are: a well-batched decode fleet can sustain millions of output tokens a second, which over an hour is tens of billions of tokens, landing the raw cost somewhere around or below a dollar per million output tokens against a list price measured in dollars. The margin, in other words, is almost entirely batching efficiency. This is why the right serving metric is goodput, the tokens per GPU-second that actually met their latency target, not raw throughput that may be quietly violating SLOs to look good. It is also why the interactive tier and the batch API tier are priced and scheduled differently: the batch tier can pack aggressively because nobody is watching each token arrive, so it runs at higher utilization and lower cost, while the interactive tier pays for latency headroom. When someone asks how many users a GPU serves, the honest answer routes straight back through this arithmetic.
Continuous batching needs a scheduler, and hardware fails mid-answer
Iteration-level batching is often described as a free win, but admitting and evicting requests every token is a scheduling policy, and the policy is where the hard problems live. Admission control decides when the batch is too full to take another request without blowing the latency target. Priority and fairness decide whose request runs when the paid tier and the free tier both want the same GPU, and without a starvation guard a stream of short high-priority requests can leave a long generation waiting forever. Preemption of a long generation means choosing between swapping its KV cache out to CPU memory and recomputing it later, a real tradeoff between memory and compute. And the observability signal that catches a prefill stalling everyone is not average latency, which hides it, but the ninety-ninth percentile of inter-token latency. Then there is failure. A generation runs on a tensor-parallel group of GPUs in lockstep, so one card dying takes the whole replica and every stream on it with it. You cannot bit-exactly replay a half-streamed answer, because batch composition changes the reduction order, so the recovery is to re-prefill the tokens already sent to the user on another replica and continue from there. None of this appears in a diagram, and all of it appears in a senior interview.
ChatGPT is not a chat completion: tools, agents, and other inputs
The product is a loop with a model in it, not a single call, and that changes the infrastructure. When the model calls a tool, generation pauses for a web fetch or a sandboxed code run that can take seconds, and the design question is what happens to that conversation's KV cache in the gap: holding it wastes the scarcest resource on an idle request, while evicting and re-prefilling on return is only affordable because the prefix cache makes the replay cheap. Multimodal input adds encoder pools and makes an image a real prefill cost rather than free context. Voice mode is a different latency problem again, closer to real-time duplex with barge-in than to request-response. Safety shifts too: the dangerous input is no longer only the user's prompt but the untrusted text a tool returns, which is the vector for prompt injection in an agentic system, and a prefix cache shared across tenants becomes a timing side channel that can leak whether someone else has asked a given question, which is why the cache is partitioned per tenant despite the efficiency cost. The chat box is the visible end of a pipeline that has grown well past text in, text out.
Trade-offs to discuss
Every senior interviewer expects you to surface at least 3 of these. Pick the decisions, state the alternatives, and justify your choice.
Route on KV cache occupancy rather than request count
Requests are wildly unequal in the memory and time they consume, so request-based load balancing sends work to a replica that looks idle and is actually nearly full. Occupancy is the signal that reflects the binding constraint, at the cost of the router needing live memory telemetry from every replica.
Session affinity for prefix reuse, accepting weaker load spreading
Pinning a conversation to the replica holding its prefix turns each follow-up prefill into a cache hit, which is a large saving on long chats. The price is a hotspot risk and a cold prefill when the replica dies or the cache evicts, so affinity has to be a preference rather than a hard binding.
Disaggregate prefill and decode, or use chunked prefill
Separate pools give the cleanest isolation and independent tuning but add a KV cache transfer and more operational surface. Chunked prefill needs no new topology and protects token streams almost as well, which usually makes it the right first move, with disaggregation reserved for scale where the interference is worth the complexity.
Quantize the model, accepting a small quality cost
FP8 or INT8 weights halve the weight read and free memory for more KV cache. At serving batch sizes the freed memory is the larger of the two wins, because per-sequence KV reads dominate the traffic there and shrinking the weights speeds each token only modestly. Modern quantization loses little quality on most tasks, but it does lose some, so this is a measured decision per model and per tier rather than a free win. Quantizing the KV cache itself is the further lever once weights are already small.
Tensor parallelism inside a node, pipeline parallelism only across nodes
Tensor parallelism all-reduces on every layer, which is only cheap over NVLink, so it belongs inside a node where it cuts latency. Pipeline parallelism tolerates slower links but introduces bubbles and raises latency, so it is what you reach for when a model genuinely will not fit in one node, not as a first choice.
Buffer a small window before emitting moderated output
Screening the output while streaming means the decision arrives after some tokens are already generated. Holding a short window lets a violation be caught before the user sees it, at the cost of slightly worse time to first token, which is usually the right trade for a consumer product and the wrong one for a latency-critical API.
Add expert parallelism for a mixture-of-experts model
An MoE model's experts do not fit on one node, so they are spread across the fleet and routing becomes an all-to-all shuffle rather than a per-layer all-reduce. That buys far more capacity per unit of per-token compute, at the price of interconnect pressure and sensitivity to expert load imbalance, and it pushes the ideal batch size up because each expert needs enough routed tokens to stay busy.
Preempt a long generation by recompute or by swap
When a higher-priority request needs the slot, you can drop a running generation's KV cache and recompute it later, spending compute to reclaim memory, or swap it out to CPU memory and copy it back, spending bandwidth to preserve it. Recompute wins when the prefix is cheap to rebuild, swap wins when it is not, and a scheduler that never preempts lets long generations starve everyone else.
How ChatGPT actually does it
OpenAI does not publish ChatGPT's serving architecture, so treat the specific topology above as a reasoned design rather than a description of their internals. What is public and well documented is the inference science that any such system must obey, and it is unusually open. PagedAttention and continuous batching come from the vLLM paper by Kwon et al. at SOSP 2023, which measured that naive KV cache allocation wastes most of the memory and showed the throughput gain from fixing it; iteration-level scheduling was introduced earlier by Orca at OSDI 2022. Speculative decoding with its rejection sampling proof of distributional equivalence is from Leviathan et al. in 2023. The prefill and decode interference problem and the disaggregation answer are documented in DistServe (OSDI 2024) and Splitwise, and the chunked prefill alternative in Sarathi-Serve. Prefix caching with a radix tree is from SGLang. Grouped-query attention, the reason the KV cache numbers above are merely large instead of impossible, is from Ainslie et al. in 2023 and is used by Llama and most current open models. The mixture-of-experts discussion generalizes the sparse-routing idea from Switch Transformers (Fedus et al., 2021); the specific expert counts and routing behavior of any given frontier model are not public, so treat that section as the shape of the problem rather than a description of one vendor's model. The cost arithmetic uses round public numbers for GPU rental and known token economics to show the method, not to price any specific provider. The user-facing figures cited here come from different public snapshots and should be read with their dates: OpenAI stated about 900 million weekly actives in February 2026, and separately about 2.5 billion messages a day in mid 2025. The prompt caching discount on cached input tokens is documented in OpenAI's own API docs, which is a useful confirmation that prefix caching is running in production at that scale.
Sources
- Kwon et al., Efficient Memory Management for LLM Serving with PagedAttention (vLLM, SOSP 2023)
- Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI 2022)
- Leviathan et al., Fast Inference from Transformers via Speculative Decoding (2023)
- Zhong et al., DistServe: Disaggregating Prefill and Decoding (OSDI 2024)
- Agrawal et al., Sarathi-Serve: Chunked Prefill (2024)
- Ainslie et al., GQA: Grouped-Query Attention (2023)
- Fedus et al., Switch Transformers: scaling mixture-of-experts (2021)
- Zheng et al., SGLang and RadixAttention prefix caching (2023)
- OpenAI prompt caching documentation
Lessons to study before this interview
If any of these topics are fuzzy, the interviewer will catch it. Each lesson is 15 to 60 minutes with diagrams, code, and a quiz.
Related system design interview questions
Practice these next. They lean on the same core building blocks as ChatGPT.