System design interview guide
Code Deployment System Design Interview: Shipping a Commit to 100,000 Servers Safely
A large fleet can run well over 100,000 hosts across several regions, and a single service artifact can be hundreds of megabytes. Twitter reported that switching to a BitTorrent based distribution model turned a 40 minute deploy into a 12 second one. Facebook engineers described pushing a build of roughly 1.5 GB to their web tier, with peer to peer propagation reaching all servers in about 20 minutes instead of hours. The hard part is not building the code, it is moving it everywhere and turning it on without taking the site down.
A code deployment system takes a commit and carries it all the way to production: it builds the code, runs tests, produces an immutable artifact, stores that artifact, distributes it to every target host, and then flips traffic over in a controlled way. The design splits cleanly into a control plane that decides what should run where, and a data plane that actually moves bytes and swaps processes on each machine. The interesting engineering lives in two places. First, distribution: pushing a large artifact to tens of thousands of hosts from a central store saturates the origin, so real systems use peer to peer swarms or a tree of regional caches so that hosts pull from each other. Second, safety: you never flip all hosts at once, you use rolling, blue-green, or canary strategies with automated health checks and metric comparison so a bad build is caught on a small blast radius and rolled back fast. Everything hinges on immutable, versioned artifacts and a deployment state machine that can always answer what version is on which host and can drive it back to a known good version.
Asked at: Asked at Netflix, Meta, Google, Amazon, Uber, and most infrastructure or platform focused interviews. It also shows up for SRE and release engineering roles as design your own CD pipeline or deploy to a million machines.
Why this question is asked
It is a rare interview problem where the write path is the whole story, and where safety matters more than raw throughput. Interviewers like it because it forces you to separate a control plane from a data plane, to reason about distributing a large binary without melting your origin, and to design for failure as the normal case rather than the exception. It touches consensus and leader election for the orchestrator, peer to peer or CDN style fan out for artifacts, health checking and observability for rollout decisions, and the classic rollback question. A strong candidate talks about blast radius, immutability, and idempotency without being prompted. A weak candidate treats deploy as copy a file and restart, and never mentions what happens when the file is bad.
Requirements
Always clarify these in the first 5 minutes of the interview. Do not start drawing boxes until both lists are agreed.
Functional requirements
- Trigger a pipeline on a commit or tag: build the code, run tests, and produce a versioned artifact
- Store artifacts immutably with content addressing and full version history
- Distribute an artifact to thousands of target hosts across multiple regions
- Support rolling, blue-green, and canary deployment strategies with configurable batch size and pace
- Run health checks on each host and each batch before advancing the rollout
- Automatically halt and roll back to the last known good version when health checks or canary metrics fail
- Manage per environment and per service configuration separately from the artifact
- Expose live rollout status: which version is on which host, progress, and per batch health
- Provide audit history of who deployed what, when, and the outcome
Non-functional requirements
- Distribution to the full fleet should complete in minutes, not hours, even for large artifacts
- A bad deployment must be detectable and reversible within seconds to a few minutes
- The control plane must be highly available and survive the loss of a region
- Deployments must be idempotent and safe to retry after a partial failure
- Blast radius must be bounded: a failure should never take down more than a small, configurable slice at once
- Artifacts and the pipeline must be tamper resistant, with signing and provenance
- The system must degrade gracefully: the fleet keeps serving traffic even if the deploy control plane is down
Back-of-envelope scale estimates
Show your math. Pulling numbers from thin air signals you have not thought about the load.
Target fleet size
100,000+ hosts
A large internet company runs six figures of hosts across regions. Twitter and Facebook both described fleets and deploy tooling operating at the tens of thousands to hundreds of thousands scale. Treat the exact number as a company specific estimate.
Artifact size
200 MB to 1.5 GB per service
Facebook engineers cited a web tier build near 1.5 GB including the server and compiled app. A typical microservice container image is a few hundred MB. We use 500 MB as a working midpoint estimate for arithmetic.
Central push bandwidth without peering
~50,000 GB of egress for one push
Derived: 500 MB times 100,000 hosts equals 50,000,000 MB, about 50 PB moved if every host pulls from the origin. This is why a central store cannot fan out directly and why peer to peer or tiered caching is mandatory.
Peer to peer distribution time
single digit minutes to ~20 minutes
Twitter reported 12 second deploys for their payload with Murder, and Facebook reported roughly 20 minutes for a ~1.5 GB artifact to reach the whole fleet with BitTorrent. Time scales with log of the fleet size in a healthy swarm rather than linearly.
Deploys per day
1,000 to 10,000+ across all services
Estimate. With hundreds to thousands of services each deploying several times a day in a continuous delivery shop, total daily deployments land in the thousands. Netflix has publicly described millions of deployments over the life of Spinnaker.
Canary evaluation window
10 to 60 minutes per canary stage
Estimate based on published canary practice. You need enough traffic through the canary instances to gather statistically meaningful metrics before the judge compares canary against baseline, so a stage typically waits tens of minutes.
High-level architecture
A commit lands and fires a trigger into the build service, which checks out the code, compiles it, runs the test suites, and produces a single immutable artifact tagged with a content hash and a semantic version. That artifact is written once to an artifact store, which is the source of truth for every version that has ever existed. A user or an automated pipeline then creates a deployment: it names the target service, the artifact version, the environment, and a strategy such as rolling or canary. The control plane, an orchestrator backed by a replicated, consensus based datastore, records the desired state and breaks the rollout into batches. For each batch it instructs the data plane to fetch the artifact. Here the distribution layer takes over: rather than every host pulling from the origin store, hosts join a peer to peer swarm or pull from a tree of regional and rack local caches, so the bytes propagate host to host and the origin serves each chunk only a few times. A lightweight agent on every host receives the artifact, verifies its hash and signature, unpacks it next to the running version, applies the environment config, and starts the new process. The agent then reports health back up. The orchestrator watches health checks and, for canary strategies, feeds live metrics from the new instances into an automated judge that compares them against a baseline. If the batch is healthy the orchestrator drains connections from the old version, shifts traffic, and advances to the next batch. If health checks fail or the canary judge flags a regression, the orchestrator stops advancing and drives every touched host back to the last known good version. Throughout, the control plane and the data plane are separate: the fleet keeps serving user traffic even if the orchestrator is unavailable, because the running processes do not depend on it.
In a real interview, sketch this on the whiteboard before diving into any single box.
Core components
Walk through each service. The interviewer wants to hear what each one owns, not just the names.
Build and pipeline service
Watches source control, runs the build and test stages in isolated workers, and emits a versioned artifact only when every gate passes. It is the entry point of the pipeline and enforces that nothing untested becomes deployable. Each pipeline run is reproducible and recorded for provenance.
Artifact store
An immutable, content addressed repository for build outputs, container images, and their metadata. Artifacts are written once and never mutated, so a version always means exactly the same bytes. It holds full history so any prior version is instantly available for rollback.
Control plane orchestrator
The brain that turns a desired deployment into an ordered set of batch operations and tracks the actual state of every host. It is backed by a consensus replicated datastore and uses leader election so exactly one orchestrator drives a given rollout. It decides when to advance, pause, or roll back based on health signals.
Distribution layer
Moves artifact bytes from the store to every target host without saturating the origin. It uses peer to peer swarming, like Twitter's Murder or Facebook's internal BitTorrent, or a tiered set of regional and rack local caches so hosts largely pull from each other. Chunks are content addressed so partial and resumed transfers are safe.
Host agent
A daemon on every target host that fetches the artifact, verifies its hash and signature, stages the new version alongside the current one, applies config, and manages process start, drain, and stop. It reports health and installed version upward. Because it runs locally, deployments proceed even under intermittent control plane connectivity.
Health and canary evaluation
Combines per host health checks, service level probes, and an automated canary judge that statistically compares metrics from new instances against a baseline. It gates the rollout: a batch only advances when it is provably healthy. This is what converts deployment from a blind push into a controlled experiment.
Config and secrets management
Stores environment specific configuration and secrets separately from the immutable artifact, so the same build promotes cleanly from staging to production by changing only config. It versions config changes and treats a config push as its own deployable event with its own rollback path.
Data model
Pick the right store per table. Justify each choice with the access pattern, not by reflex.
artifactsartifact_idcontent_hashservice_nameversionsize_bytescreated_atOne row per immutable build output. content_hash is the content address and doubles as an integrity check. Never updated after insert; rollback just points at an older row.
deploymentsdeployment_idservice_nameartifact_idenvironmentstrategystatusinitiated_byThe intent record for a rollout. status moves through a state machine (pending, in_progress, paused, succeeded, rolled_back, failed). strategy is rolling, blue_green, or canary.
deployment_batchesbatch_iddeployment_idbatch_indexhost_countstatushealth_resultA rollout is split into ordered batches to bound blast radius. The orchestrator advances batch_index only after health_result on the prior batch passes.
host_statehost_idregionservice_namecurrent_versiondesired_versionagent_statuslast_heartbeatActual versus desired version per host, the core of reconciliation. Answers what is running where and lets the orchestrator detect drift and stragglers.
canary_metricsdeployment_idmetric_namebaseline_valuecanary_valueclassificationjudged_atPer metric comparison between baseline and canary clusters. classification is pass, fail, or nodata; the aggregate pass ratio produces the canary score that gates promotion.
config_versionsconfig_idservice_nameenvironmentversionchecksumapplied_atVersioned environment config kept separate from artifacts. A config change is deployable and rollbackable on its own, independent of the code artifact.
Deep dives
These are the conversations the interviewer is steering you toward. Practice each one until you can talk through it without notes.
Distributing a large artifact to the whole fleet with peer to peer swarming
The naive design has every host pull the artifact from a central store, which does not survive contact with a real fleet. At 500 MB per host and 100,000 hosts you would try to move roughly 50 PB out of the origin for a single push, and the store's network becomes the bottleneck while most hosts sit idle waiting. The fix is to make every host a peer. Twitter's Murder and Facebook's internal system both used BitTorrent: the artifact is split into chunks, a seeder announces it to a tracker, and hosts download chunks from each other rather than all from the origin. Because the datacenter has high bandwidth, low latency, and no NAT or ISP shaping, the swarm saturates internal links efficiently and propagation time grows roughly with the logarithm of fleet size instead of linearly. Twitter reported this cut a 40 minute deploy to 12 seconds for their payload, and Facebook reported about 20 minutes for a 1.5 GB artifact to reach the fleet. A production tuning detail is topology awareness: you bias the swarm toward rack local and cluster local peers first so that most traffic stays on cheap intra rack links and you avoid flooding cross datacenter backbones. An alternative to a pure swarm is a tree of regional and rack local caches, which is simpler to operate and integrates naturally with a container registry, at the cost of provisioning and warming those caches.
Control plane versus data plane separation
The single most important structural decision is that the thing which decides deployments and the thing which serves user traffic must not share a fate. The control plane, meaning the orchestrator, its datastore, and the pipeline, is where you accept some unavailability. The data plane, meaning the running application processes and their host agents, must keep serving traffic even if the entire control plane is down. That is why the host agent stages and runs the process locally and does not phone home on every request. If the orchestrator dies mid rollout, hosts that already switched keep running the new version and hosts that did not keep running the old one, and nothing user facing breaks. When the orchestrator recovers, it reconciles: it reads actual host_state, compares against desired state, and resumes. This separation also shapes availability targets. The control plane needs a consensus replicated store and leader election so exactly one orchestrator owns a rollout, but it can tolerate seconds of failover because it is not on the request path. Candidates who blur these two planes end up designing a system where a control plane outage becomes a site outage, which is exactly the failure you are supposed to prevent.
Rolling, blue-green, and canary strategies and when each fits
Rolling deployment updates the fleet in batches, taking a slice out of rotation, upgrading it, health checking it, and returning it before moving to the next slice. It uses no extra capacity but the fleet runs mixed versions during the rollout, so your code and data formats must be backward and forward compatible. Blue-green stands up a full second environment on the new version, verifies it, then flips traffic all at once, which gives an instant and clean rollback by flipping back, at the cost of running double capacity during the switch. Canary is the most careful: you send a small fraction of real traffic to new instances, watch them closely against a baseline, and only widen if they behave. In practice large systems compose these, for example a canary stage followed by a rolling deploy region by region. The choice is driven by blast radius tolerance, capacity budget, and how reversible the change is. Stateful or schema changing deploys lean toward blue-green or heavily gated canaries; stateless services deploy fine with rolling. The orchestrator encodes the chosen strategy as the batch plan and the gating rules between batches.
Automated canary analysis and the judgment problem
Canary only helps if a machine, not a tired human, decides whether the canary is healthy. Netflix built Kayenta for exactly this. The idea is to run two comparable clusters, a baseline on the current version and a canary on the new one, both taking production traffic, and to compare their metrics rather than comparing the canary against absolute thresholds. Comparing against a baseline controls for time of day and traffic mix. Kayenta pulls key metrics such as error rate, latency, CPU, and custom business metrics from a time series store, then for each metric runs a statistical test, using a Mann-Whitney U test to decide whether the canary differs significantly from the baseline. Each metric is classified pass or fail, and the canary score is the ratio of passing metrics, for example nine of ten passing gives ninety percent. If the score clears a configured bar the orchestrator promotes, if it falls below it fails and rolls back, and a middle band can route to human approval. The hard parts are picking metrics that actually predict user pain, giving the canary enough traffic and time to be statistically meaningful, and handling nodata cleanly so a missing metric is not silently treated as a pass.
Fast, safe rollback
Rollback is a first class operation, not an afterthought, and immutable versioned artifacts are what make it cheap. Because every prior version still exists byte for byte in the artifact store and the host agent can keep the previous version staged on disk, rolling back is usually just point desired_version at the last known good and let the agent restart against the already present bits, which can complete in seconds. Blue-green makes this even faster because the old environment is still running, so rollback is a traffic flip. The subtlety is that not everything rolls back by swapping a binary. Database schema migrations, message format changes, and stateful side effects can be one way doors, so you enforce discipline like expand and contract migrations, where you add the new column or field, deploy code that tolerates both shapes, and only remove the old shape in a later deploy once nothing references it. Config changes need their own rollback path because a bad config push can break a service without any code change. The orchestrator ties this together by always tracking the last known good version per service so a rollback target is unambiguous, and by making the rollback itself a normal, health checked, batched deployment rather than a panicked manual scramble.
Health checks, draining, and orchestrating across regions
Every advance decision rests on health signals, and there are layers. A liveness check tells you the process is up, a readiness check tells you it can serve, and deeper service level probes and real metrics tell you it is actually correct. The orchestrator should never advance a batch on liveness alone, because a process can be up and still returning errors. Before taking an old instance out of rotation you drain it: stop sending new requests, let in flight requests finish within a timeout, then stop the process, which avoids dropping user requests during the swap. Across regions you deploy sequentially, not simultaneously, so a region acts as a natural blast radius boundary. A common pattern is deploy to one small region, bake and watch, then fan out to larger regions, with the option to halt the whole train if any region regresses. This region by region train, seen in Spinnaker style pipelines at Netflix, means a bad build is caught while it is still confined to a fraction of users. The orchestrator holds the global view of which regions are done, in progress, or pending, and enforces the ordering and the bake times between them.
Idempotency, reconciliation, and handling partial failures
At fleet scale, partial failure is the normal state: some hosts are unreachable, some agents crash mid update, some transfers stall. The system has to be designed so that retrying is always safe. That means operations are idempotent, keyed by deployment and version, so telling a host to install version X twice results in version X installed once, not a corrupted half state. Rather than a fire and forget push, the orchestrator runs a reconciliation loop: it continuously compares desired_version against current_version for every host and issues corrective actions until they match, which naturally handles stragglers, flapping hosts, and hosts that were offline during the original push and rejoin later. Content addressed chunks let a host resume an interrupted transfer instead of restarting it. The orchestrator tracks per host progress so it knows the difference between a host that failed to update and a host that is simply slow, and it applies a threshold: if more than a small configured fraction of a batch fails to reach healthy, it stops advancing and raises the alarm rather than pressing on. This reconciliation model, desired state plus a control loop, is what lets a deployment survive the messy reality of a hundred thousand imperfect machines.
Trade-offs to discuss
Every senior interviewer expects you to surface at least 3 of these. Pick the decisions, state the alternatives, and justify your choice.
Peer to peer swarm versus tiered CDN style caches for distribution
A swarm like Murder or Facebook's BitTorrent minimizes origin load and scales sublinearly with fleet size, but it is complex to tune and reason about. A tree of regional and rack local caches is easier to operate and pairs naturally with a container registry, but you must provision and warm the caches and it can strain cross region links. Very large fleets tend to favor swarming; smaller ones do fine with tiered caches.
Blue-green versus rolling deployment
Blue-green gives instant rollback and no mixed version window, but doubles capacity during the switch, which is expensive at fleet scale. Rolling uses no extra capacity and is the default for stateless services, but forces backward and forward compatibility because versions coexist. Choose by capacity budget and how compatible successive versions are.
Automated canary judgment versus manual approval gates
Automated analysis like Kayenta scales to thousands of deploys a day and removes human latency and bias, but it can be fooled by low traffic canaries or badly chosen metrics. Manual gates catch things metrics miss but do not scale and slow everything down. Most mature systems automate the common case and reserve human approval for a scored middle band or high risk services.
Immutable artifacts versus in place patching
Immutable, content addressed artifacts make every version reproducible and make rollback a pointer change, at the cost of storing full builds and moving more bytes. In place patching moves less data but destroys the guarantee that a version means one exact set of bytes, which makes rollback and debugging unreliable. Immutability is almost always worth the storage.
Strong consistency in the control plane versus eventual consistency
The orchestrator needs a consistent, consensus backed record of desired state and a single leader per rollout to avoid two orchestrators fighting over the same fleet. The data plane, meaning what each host reports, is fine being eventually consistent and reconciled over time. Putting strong consistency on the control path and eventual consistency on the reporting path gives correctness where it matters without paying for it everywhere.
Coupling config to the artifact versus keeping it separate
Baking config into the artifact makes a deploy fully self contained and reproducible, but forces a full rebuild and redistribute for a one line config change. Keeping config separate and versioned lets the same build promote across environments and lets config roll back on its own, at the cost of an extra moving part to track. Separating them is standard, with the discipline that config changes are themselves deployable events.
How Code Deployment System actually does it
The peer to peer distribution idea is not hypothetical. Twitter open sourced Murder, a set of Python and Ruby scripts built on the BitTornado library, and reported it cut a datacenter deploy from about 40 minutes to 12 seconds by turning every server into a peer. Facebook engineers described using BitTorrent internally to push web tier builds around 1.5 GB, configured with cluster and rack affinity to keep traffic local, reaching the full fleet in roughly 20 minutes. On the orchestration and safety side, Netflix built and open sourced Spinnaker, a multi cloud continuous delivery platform that models deployments as pipelines of stages and supports blue-green, rolling, and canary strategies with region by region rollouts. Netflix also built Kayenta for automated canary analysis, which compares canary metrics against a baseline using statistical tests such as the Mann-Whitney U test and produces a pass ratio score that gates promotion. These systems together illustrate the split this design is built around: peer to peer or tiered distribution for the data plane, and a pipeline driven orchestrator with automated canary judgment for the control plane.
Sources
- Twitter Engineering: Murder, fast datacenter code deploys using BitTorrent
- Murder source (BitTorrent based large scale deploy) on GitHub
- Netflix TechBlog: Global Continuous Delivery with Spinnaker
- Netflix TechBlog: Automated Canary Analysis at Netflix with Kayenta
- Development and Deployment at Facebook (paper)
Lessons to study before this interview
If any of these topics are fuzzy, the interviewer will catch it. Each lesson is 15 to 60 minutes with diagrams, code, and a quiz.
Related system design interview questions
Practice these next. They lean on the same core building blocks as Code Deployment System.