System design interview guide
API Gateway System Design Interview: Building the Single Front Door for Hundreds of Microservices
When you split a monolith into two hundred services, every client suddenly needs to know where each service lives, how to authenticate to it, and how to retry it. An API gateway collapses that into one address. Netflix's Zuul gateway fleet handles more than one million requests per second across roughly 80 clusters, fronting about 100 backend service clusters (Netflix engineering, estimates vary by year). The gateway is the busiest single tier in the whole architecture, so its design decisions ripple through everything behind it.
An API gateway is the single entry point that sits between external clients and a fleet of internal microservices. It terminates TLS, authenticates and authorizes each request, applies rate limits and quotas, routes to the correct upstream based on path or host or headers, and often transforms or aggregates responses before returning them. The hard part is that it is on the critical path for every request, so it must be fast, non-blocking, and horizontally scalable without becoming a single point of failure. Good designs treat the gateway as a thin, stateless, policy-enforcing proxy built around a chain of composable filters or plugins, with heavy work (business logic, data ownership) pushed back into the services. The interview is really about where responsibility belongs: what the gateway should own versus what it must never own. Candidates who put too much logic in the gateway rebuild a distributed monolith with a new name. The reference implementations to reason from are Netflix Zuul 2, Kong, Envoy, and AWS API Gateway, each of which models the request lifecycle as an ordered set of pluggable stages.
Asked at: Asked at Amazon, Netflix, Uber, Stripe, Atlassian, and most companies running a microservices or platform team. It shows up both as a standalone gateway design and as a component inside larger designs like 'design Uber' or 'design a payments platform'.
Why this question is asked
Interviewers like this problem because it forces a candidate to separate cross-cutting concerns from business logic, which is the central skill in microservices design. It tests whether you understand the request lifecycle in depth: TLS termination, authentication, authorization, rate limiting, routing, transformation, and observability, and in what order they must run. It also probes reliability thinking, because the gateway is a shared dependency for every team, so a candidate has to reason about how to keep one bad plugin or one slow upstream from taking down the whole front door. Finally it reveals judgment about scope. A weak answer piles orchestration, retries with business meaning, and data joins into the gateway. A strong answer keeps it thin and pushes ownership to the services.
Requirements
Always clarify these in the first 5 minutes of the interview. Do not start drawing boxes until both lists are agreed.
Functional requirements
- Route incoming requests to the correct backend service based on path, host, HTTP method, and headers
- Terminate TLS and optionally re-encrypt to upstreams, presenting a single certificate and domain to clients
- Authenticate requests (API keys, JWT, OAuth2 tokens, mTLS) and pass a verified identity to upstreams
- Authorize requests against scopes or roles before they ever reach a service
- Enforce rate limits, quotas, and throttling per client, per API key, and per route
- Transform requests and responses: rewrite paths, inject or strip headers, translate protocols (REST to gRPC, HTTP/1.1 to HTTP/2)
- Aggregate or compose multiple upstream calls into one client response (backend-for-frontend pattern)
- Integrate with service discovery so routes resolve to healthy instances without hardcoded IPs
- Emit structured logs, metrics, and distributed traces for every request that passes through
Non-functional requirements
- Low added latency: the gateway should add single-digit milliseconds in the common path, not tens of milliseconds
- High availability with no single point of failure: run many stateless instances behind a load balancer across zones
- Horizontal scalability: capacity grows by adding instances, not by making one instance bigger
- Fault isolation: a slow or failing upstream must not exhaust gateway threads and block unrelated traffic
- Dynamic reconfiguration: routes, limits, and plugins update without restarting or dropping connections
- Security by default: reject malformed or oversized requests early, hide internal topology from clients
- Observability: per-route latency, error rate, and saturation metrics that make the gateway debuggable under load
Back-of-envelope scale estimates
Show your math. Pulling numbers from thin air signals you have not thought about the load.
Peak request rate (large gateway fleet)
~1,000,000+ requests/sec
Netflix has publicly described its Zuul fleet handling over 1M req/sec across roughly 80 clusters. This is a real published figure for a very large deployment, not typical. Most gateways serve far less.
Per-instance throughput
~5,000 to 20,000 req/sec
Derived estimate. A non-blocking Netty or Nginx-based gateway on a modern 8-core instance commonly sustains low tens of thousands of simple proxy requests per second. To serve 1M req/sec at ~10k per instance you need ~100 instances, matching the order of magnitude of large fleets.
Added latency budget (p50)
~1 to 5 ms
Estimate based on a thin proxy path. TLS is already terminated per connection, so per-request overhead is header parsing, an auth token check (often a cached lookup), a rate-limit counter read, and route selection. Each is sub-millisecond when backed by in-memory or Redis lookups.
AWS API Gateway account throttle (default)
10,000 req/sec, 5,000 burst
Published AWS default account-level throttle per Region, using a token bucket. Some newer Regions default to 2,500 req/sec with 1,250 burst. These are managed-service ceilings you would design around, not build.
Rate-limit counter operations
~1 to 3 Redis ops per request
Derived. A token-bucket or sliding-window limiter typically does one atomic increment plus an occasional refill read. At 1M req/sec that is roughly 1 to 3M Redis ops/sec, which forces sharding the counter store by key rather than a single node.
Route table size
hundreds to low thousands of routes
Estimate. One route per public API surface across ~100 to a few hundred services, with a handful of paths each. This fits comfortably in memory as a compiled trie or radix tree, so route matching is not the bottleneck.
High-level architecture
A client resolves one DNS name and connects to a layer 4 or layer 7 load balancer, which spreads connections across a pool of stateless gateway instances running in multiple availability zones. The gateway instance terminates TLS, then runs the request through an ordered chain of filters. Inbound filters parse and validate the request, authenticate the caller by verifying a JWT signature or looking up an API key, check authorization scopes, and apply rate limiting by reading and incrementing a counter in a shared store like Redis. If any of these reject the request, the gateway returns an error immediately and never touches a backend. If they pass, a routing filter matches the request against a compiled route table (path, host, method, headers) to pick a target service, then consults service discovery to get a healthy instance and applies a load-balancing policy. An endpoint filter proxies the request upstream using a non-blocking HTTP client, optionally rewriting the path, injecting the verified identity as a trusted header, and translating protocols. When the response returns, outbound filters run in reverse: they can strip internal headers, compress or reshape the body, add caching headers, and record metrics and traces. For aggregation or backend-for-frontend routes, a composition filter fans out several upstream calls concurrently and stitches the results into one response. Every stage emits logs, metrics, and a trace span so the whole path is observable. Configuration (routes, limits, plugin settings) is pushed dynamically from a control plane so instances reconfigure without restarts.
In a real interview, sketch this on the whiteboard before diving into any single box.
Core components
Walk through each service. The interviewer wants to hear what each one owns, not just the names.
Load balancer (in front of the gateway)
A layer 4 or layer 7 balancer distributes client connections across gateway instances and removes unhealthy ones from rotation. It is what makes the gateway itself horizontally scalable and free of a single point of failure. The gateway is deliberately stateless so any instance can serve any request.
TLS termination and connection handling
The gateway terminates client TLS once per connection and reuses that connection for many requests via keep-alive. It presents a single certificate and hostname to the outside world while the internal topology stays hidden. Non-blocking I/O (Netty in Zuul 2, Nginx event loop in Kong) lets one instance hold many thousands of concurrent connections cheaply.
Filter or plugin chain
The heart of the gateway is an ordered pipeline of small, composable filters that each own one concern: auth, rate limiting, routing, transformation, logging. Zuul calls them inbound, endpoint, and outbound filters; Kong calls them plugins that run in access, rewrite, header_filter, body_filter, and log phases; Envoy calls them HTTP filters in a decode and encode chain. New behavior is added by writing a filter, not by editing a monolith.
Authentication and authorization module
Verifies the caller's identity from an API key, JWT, OAuth2 token, or client certificate, then checks that the identity holds the scopes required for the route. It offloads this cross-cutting concern from every service so backends can trust a signed internal header. Token verification is usually a fast local signature check or a cached lookup rather than a database call on every request.
Rate limiter and quota enforcer
Enforces per-client, per-key, and per-route limits using algorithms like token bucket or sliding window, backed by a shared counter store so limits hold across all gateway instances. It protects upstreams from overload and abusive callers. Because counters are hot and shared, they are sharded by key and often use atomic operations in Redis.
Router and service discovery integration
Matches each request to a route using a compiled trie or radix tree over path, host, method, and headers, then resolves the target to a live instance via service discovery (Consul, Eureka, Kubernetes endpoints) rather than a hardcoded address. It applies a load-balancing policy and health awareness so traffic avoids dead instances. Route and endpoint data refresh dynamically as services scale up and down.
Control plane and observability pipeline
A control plane distributes route tables, rate-limit rules, and plugin configuration to every instance without restarts, so operators change behavior at runtime. Every request produces structured logs, latency and error metrics tagged by route, and a distributed trace span. This is what lets a platform team debug which upstream or which plugin is slow when the front door misbehaves.
Data model
Pick the right store per table. Justify each choice with the access pattern, not by reflex.
routesroute_idmatch_hostmatch_path_prefixmatch_methodsupstream_servicestrip_prefixDefines how an incoming request maps to a backend. Loaded into memory as a compiled trie for fast matching. In managed gateways this is config, not a hot database read.
api_consumersconsumer_idnameapi_key_hashauth_typetierstatusRepresents a client or partner allowed to call the gateway. api_key_hash is stored hashed, never in plaintext. Tier drives which rate-limit and quota rules apply.
rate_limit_policiespolicy_idscopelimit_per_windowwindow_secondsburstalgorithmDeclares the allowed rate for a scope such as per-consumer, per-route, or global. The gateway reads these at config load time; the live counters live in a separate fast store.
rate_limit_counterscounter_keycurrent_tokenswindow_startlast_refill_tsHot, ephemeral state in Redis or a similar store, keyed by consumer plus route. Updated atomically on every request. Sharded by key so no single node is the bottleneck.
plugin_configsplugin_idplugin_typescope_refconfig_jsonenabledpriorityBinds a plugin (auth, cors, transform, logging) to a route or consumer with its settings and execution priority. Priority determines position in the filter chain.
upstream_targetsservice_nameinstance_addrweighthealth_statuslast_check_tsDiscovered set of live instances per service, usually populated from service discovery rather than hand-edited. Weight and health drive load-balancing and failover decisions.
Deep dives
These are the conversations the interviewer is steering you toward. Practice each one until you can talk through it without notes.
The request lifecycle and why filter order matters
A request flows through the gateway as an ordered pipeline, and the order is not arbitrary. Cheap rejections must come first so the gateway spends as little work as possible on requests it will deny. A typical order is: parse and validate the request, authenticate, authorize, rate limit, route, then proxy, then run response filters in reverse. Authentication comes before rate limiting because per-consumer limits need a known identity, but coarse global or per-IP limits can run even earlier to shed obvious floods. Envoy makes this explicit: decoder (request) filters run A, B, C and encoder (response) filters run in reverse C, B, A, so a filter that added a header on the way in can remove it on the way out. Kong exposes named phases (rewrite, access, header_filter, body_filter, log) and each plugin declares which phase and priority it runs at. Getting the order wrong causes real bugs, for example rate limiting before auth lets an unauthenticated attacker consume a legitimate user's quota, or transforming a body before validating it wastes CPU on garbage.
Plugin and filter chain architecture (Zuul, Kong, Envoy)
The reason these gateways scale as products is that behavior is composed from small, independent filters rather than baked into one program. Netflix Zuul 2 models this as inbound filters, an endpoint filter that proxies upstream, and outbound filters, all running on a non-blocking Netty event loop; filters can be added or reloaded dynamically as Groovy or Java classes. Kong is built on OpenResty, which is Nginx plus LuaJIT, and each plugin is essentially a handler.lua with functions for the phases it cares about plus a schema.lua that validates its config. Envoy compiles filters as native or WASM extensions attached to an HTTP connection manager. The shared idea is a clean contract: a filter receives the request or response context, can read and mutate it, and can short-circuit the chain by returning a response early. This is what lets a platform team ship a new auth scheme or logging format to the whole fleet by deploying one filter, without asking every service team to change code. The tradeoff is that a badly written filter, for example one that blocks on I/O in a non-blocking event loop, can degrade every request, so filter isolation and review matter.
Rate limiting across a distributed gateway fleet
A single gateway instance can rate limit with an in-memory counter, but a fleet of a hundred instances cannot, because a client's requests land on different instances and no single instance sees the full rate. The counters must live in a shared, fast store, usually Redis, and be updated atomically. Token bucket and sliding-window log or sliding-window counter are the common algorithms; token bucket is popular because it allows short bursts while capping the sustained rate, which is exactly what AWS API Gateway does with its account-level 10,000 req/sec limit and 5,000 burst. The hard problems are hot keys and latency. A very active consumer creates a hot counter that can bottleneck one Redis shard, so you shard by key and sometimes use local approximate counters that sync periodically to trade a little accuracy for a lot of throughput. You also have to decide the failure mode: if Redis is unreachable, do you fail open (allow traffic, risking overload) or fail closed (reject, risking an outage from the limiter itself)? Most production gateways fail open for availability and treat the limiter as best-effort protection rather than a hard security boundary.
Keeping the gateway from becoming a bottleneck or single point of failure
The gateway is on the path of every request, so its failure is everyone's failure. The defenses are structural. First, keep it stateless: no request should depend on session state pinned to a specific instance, so any instance can serve any request and you can add or kill instances freely behind a load balancer. Second, run many instances across multiple availability zones so losing a zone loses capacity, not the service. Third, use non-blocking I/O so one slow upstream does not tie up a thread and starve unrelated routes; this is exactly why Netflix rewrote Zuul from a blocking, thread-per-request model to the async Netty-based Zuul 2, reporting roughly a 25% throughput gain and matching CPU reduction. Fourth, isolate failure with per-upstream circuit breakers, bulkheads, timeouts, and concurrency limits so a failing service is quickly cut off with fast errors instead of dragging the gateway down with piled-up waiting requests. Fifth, shed load deliberately: when saturated, reject or queue the least important traffic rather than degrading everything. The gateway should also protect itself with tight request timeouts and maximum body sizes.
Response caching at the gateway
The cheapest request is the one that never reaches a backend. For read-heavy, cacheable endpoints, the gateway can cache upstream responses keyed by method, path, query, and relevant headers, honoring Cache-Control and TTL. This cuts latency and shields services from repeated identical reads, which is valuable during traffic spikes. The subtlety is correctness. You must not cache authenticated or personalized responses across users, so the cache key has to include or the policy has to exclude anything varying by identity. You need a clear invalidation story, whether TTL-based expiry, explicit purge on write, or cache tags, because a stale gateway cache is a bug that every client sees at once. Caching also interacts with rate limiting and metrics: a cache hit should still be counted and still respect quotas if that is the intent. In practice teams cache aggressively for truly public content (product catalogs, config, static reference data) and avoid caching anything user-specific at the gateway, pushing that to per-service or per-user caches instead.
Aggregation, protocol translation, and the backend-for-frontend pattern
A mobile screen might need data from five services. Making the client fan out five round trips over a slow mobile network is wasteful, so the gateway (or a backend-for-frontend built on top of it) can accept one client call, fan out the five upstream calls concurrently, and stitch the results into one response tailored to that client. This is where a gateway earns real latency wins, because the fan-out happens inside the data center on fast links. Protocol translation lives here too: the gateway can expose REST or GraphQL to the outside while talking gRPC or HTTP/2 to services, or terminate HTTP/3 from clients and use HTTP/1.1 internally. The danger is scope creep. Once the gateway starts composing responses it is tempting to put business rules, joins, and orchestration in it, and then the gateway becomes a shared monolith that every team is blocked on. The disciplined pattern is a separate BFF per client type (one for web, one for mobile) owned by the client team, doing presentation-shaped aggregation only, while the core gateway stays a thin policy-and-routing layer. Keep durable business logic and data ownership in the services.
How a gateway differs from a plain load balancer
Interviewers often probe this because the two overlap. A load balancer operates mostly at layer 4 or basic layer 7 and answers one question: which healthy instance of this pool should get this connection? It spreads traffic and does health checks, and that is largely it. An API gateway is application-aware and identity-aware. It understands routes and APIs, authenticates and authorizes callers, enforces per-consumer rate limits and quotas, transforms and aggregates messages, translates protocols, and applies per-route policy through plugins. Put differently, a load balancer distributes load, while a gateway enforces the contract between clients and services. In a real deployment they stack: a load balancer sits in front of the gateway fleet to distribute connections across gateway instances, and the gateway then does the application-level work and forwards to services (which may themselves sit behind internal load balancers or a service mesh). A gateway is not a replacement for a mesh either. The gateway handles north-south traffic (clients entering the system) while a service mesh handles east-west traffic (service-to-service inside the system), and many architectures run both.
Trade-offs to discuss
Every senior interviewer expects you to surface at least 3 of these. Pick the decisions, state the alternatives, and justify your choice.
Thin routing gateway versus fat orchestration gateway
A thin gateway (routing, auth, limits only) keeps teams decoupled and the front door fast and safe, at the cost of pushing aggregation into separate BFF layers. A fat gateway that orchestrates and joins data is convenient at first but becomes a shared monolith and a deployment bottleneck. Prefer thin, and add BFFs owned by client teams for composition.
Non-blocking async (Zuul 2, Envoy, Nginx) versus blocking thread-per-request
Async event-loop I/O handles far more concurrent connections and isolates slow upstreams from unrelated traffic, which is why Netflix moved Zuul to Netty and reported around 25% more throughput. The cost is a harder programming model where any blocking call in a filter poisons the event loop. Blocking is simpler to write but scales worse and couples request fates together.
Fail open versus fail closed when the rate-limit store is down
Failing open keeps traffic flowing if Redis is unreachable but risks overloading upstreams during the outage. Failing closed protects upstreams but turns a limiter outage into a full front-door outage. Most teams fail open and treat rate limiting as best-effort protection, reserving fail-closed for hard security or billing quotas.
Managed gateway (AWS API Gateway) versus self-hosted (Kong, Envoy, Zuul)
Managed gateways remove operational burden and scale automatically but impose ceilings (for example AWS's default 10,000 req/sec per Region), pricing per request, and less control over the filter chain. Self-hosted gives full control, custom plugins, and no per-request vendor cost, but you own the uptime, scaling, and upgrades of a tier that everything depends on.
Caching responses at the gateway versus only in services
Gateway caching gives the biggest latency and load win for public, cacheable reads because it stops requests at the edge, but it is dangerous for personalized data and needs careful cache keys and invalidation. Per-service caching is safer and closer to the data owner but does not shield the network hop. Cache public content at the gateway, keep user-specific caching in services.
Gateway per client type versus one gateway for all clients
Separate gateways or BFFs for web, mobile, and partners let each evolve independently and shape responses for their needs, at the cost of more infrastructure to run. One shared gateway is simpler operationally but forces one team's changes to consider every client. Large orgs usually split by client to keep teams autonomous.
How API Gateway actually does it
Netflix Zuul is the most documented large-scale gateway. Zuul 1 was a blocking, thread-per-request system; Netflix rewrote it as Zuul 2 on Netty with inbound, endpoint, and outbound filters running asynchronously, and reported roughly a 25% throughput increase with a matching drop in CPU. Their gateway fleet has been described as handling over a million requests per second across dozens of clusters fronting around a hundred backend clusters. Kong is built on OpenResty (Nginx plus LuaJIT) and models everything as plugins that run in named phases (rewrite, access, header_filter, body_filter, log), with each plugin a handler.lua plus a schema.lua. Envoy, which powers Istio and many service meshes, structures processing as listeners, network filters, and an HTTP connection manager that holds an ordered chain of HTTP filters, invoking decoder filters in order and encoder filters in reverse. AWS API Gateway is the managed reference point, throttling with a token bucket at a default account level of 10,000 requests per second with a 5,000 burst per Region (lower in some newer Regions). Across all four, the common thread is the same: a thin, non-blocking, horizontally scaled proxy whose behavior is composed from an ordered chain of pluggable stages.
Lessons to study before this interview
If any of these topics are fuzzy, the interviewer will catch it. Each lesson is 15 to 60 minutes with diagrams, code, and a quiz.
Related system design interview questions
Practice these next. They lean on the same core building blocks as API Gateway.