System design interview guide
Online Judge System Design Interview: Running Untrusted Code Safely at Contest Scale
A large online judge fields tens of millions of submissions a year, and almost none of that load is uniform. When a rated contest opens, thousands of people hit Submit in the same few minutes, so a queue that idles at a handful of jobs per second can spike past a thousand. Every one of those jobs is a stranger's program that you have to compile and run on your own hardware without letting it read the hidden tests, phone home, or take down the box. Figures here are illustrative estimates unless a source is cited.
An online judge accepts source code, runs it against a set of hidden test cases, and returns a verdict such as Accepted, Wrong Answer, Time Limit Exceeded, or Memory Limit Exceeded. The hard part is not the web app. It is executing untrusted code safely and cheaply at high concurrency. A submission is accepted quickly, persisted, and dropped onto a queue, then a pool of judge workers pulls jobs and runs each one inside a locked-down sandbox with hard limits on CPU time, wall time, memory, process count, and output size. The judge compiles the code, runs it test by test against stored inputs, compares output to the expected answer with a checker, and emits the first failing verdict. Results flow back through a results channel that the client polls or receives over a websocket. The interesting design pressure comes from contests, where a synchronized burst of submissions creates a thundering herd on the queue and the worker pool, and from security, where a single sandbox escape leaks every hidden test on the machine. Good answers treat isolation, back-pressure, and fair scheduling as first-class, not as an afterthought.
Asked at: Asked at Google, Meta, Amazon, and many competitive-programming and ed-tech companies such as LeetCode, HackerRank, Codeforces, and Codechef. It is a favorite for backend and infrastructure roles because it forces a candidate to reason about running untrusted workloads.
Why this question is asked
Interviewers like this problem because it looks like a simple CRUD app and turns out to be an operating-systems and distributed-systems question in disguise. It tests whether a candidate understands that you cannot just call docker run on a stranger's binary and hope for the best. A strong answer has to cover async job processing with a queue and worker pool, real resource isolation using kernel primitives, a threat model for untrusted code, fair scheduling so one abusive user or one contest does not starve everyone else, and the bursty load pattern of contests. Weak answers stop at the request-response API. Strong answers get into seccomp, cgroups, fork bombs, TLE detection, checker programs, and how the leaderboard stays consistent while thousands of verdicts land per second.
Requirements
Always clarify these in the first 5 minutes of the interview. Do not start drawing boxes until both lists are agreed.
Functional requirements
- Accept a code submission with a chosen language, problem id, and optional contest id, and return a submission token immediately.
- Run the submission against a set of hidden test cases and produce a verdict: Accepted, Wrong Answer, Time Limit Exceeded, Memory Limit Exceeded, Runtime Error, or Compilation Error.
- Enforce per-problem time and memory limits, with the language runtime taken into account (an interpreted language gets a multiplier).
- Support a run-with-custom-input mode so users can test code against their own input without submitting.
- Store problems, their test cases, expected outputs, checker or comparator, and per-language limits.
- Maintain per-user submission history and, for contests, a live leaderboard ranked by score and penalty time.
- Support a custom checker or special judge for problems where more than one output is correct.
- Detect likely plagiarism across submissions to the same problem and flag suspicious pairs for review.
- Deliver the verdict back to the client through polling and, where possible, a push channel.
Non-functional requirements
- Isolation: untrusted code must not read hidden tests, other users' data, or the host filesystem, and must not open network connections.
- Availability: submission intake stays up even when the judge fleet is saturated, so nothing is silently dropped.
- Elastic throughput: absorb a contest-start burst that is one to two orders of magnitude above baseline without unbounded queue latency.
- Fairness: no single user, problem, or contest can starve the judge pool; work is scheduled so everyone makes progress.
- Determinism: the same submission against the same tests yields the same verdict, so time limits must be measured in a stable way.
- Bounded verdict latency: a submission gets a verdict within seconds off-peak and within a small number of minutes at contest peak.
- Durability: submissions, verdicts, and contest standings survive worker crashes and are exactly-once with respect to scoring.
- Security defense in depth: a single sandbox escape should not compromise the host or leak the whole test corpus.
Back-of-envelope scale estimates
Show your math. Pulling numbers from thin air signals you have not thought about the load.
Baseline submissions per second
~10 to 30 submissions/sec
Estimate. Roughly 1 to 2 million submissions/day spread over active hours. 1.5M / 86400 is about 17/sec average, higher in the evening. This is the easy case.
Contest peak submissions per second
~1,000 to 1,500 submissions/sec
Derived estimate. If 100,000 contestants each submit a few times in the final rush, a 400,000-submission surge compressed into the last five minutes is 400000 / 300 which is about 1,330/sec, roughly 50x to 100x baseline.
Concurrent sandboxes at peak
~3,000 to 4,000 active sandboxes
Derived estimate. At 1,330 jobs/sec with an average judged runtime near 2 seconds per submission, Little's law gives 1330 x 2 which is about 2,660 in flight, rounded up for compile time and headroom to 3,000 to 4,000.
Judge worker machines needed at peak
~250 to 500 machines
Derived estimate. If a machine safely runs 8 to 16 concurrent sandboxes without CPU contention skewing time limits, 3,500 sandboxes / 10 per box is about 350 machines, which you pre-warm before a contest and scale down after.
Test case and problem storage
~a few TB
Estimate. Tens of thousands of problems, each with test suites from a few KB to hundreds of MB for heavy inputs. 50,000 problems x an average of 20 MB is about 1 TB, kept in object storage and cached on the judges.
Leaderboard read traffic during a contest
~100,000 to 200,000 reads/sec
Estimate. Contestants refresh standings constantly. 100,000 viewers polling every 1 to 2 seconds is 50,000 to 100,000 reads/sec, more with auto-refresh, which is why the board is served from cache, not the primary database.
High-level architecture
A submission enters through an API gateway that authenticates the user, applies per-user rate limits, and hands off to a submission service. That service does the cheap work synchronously: validate the language and payload size, write a submission row in the primary database with status Queued, upload the source to object storage, and return a submission token. It then publishes a submission event to a durable queue, partitioned so that all of a single user's submissions stay ordered. A fleet of judge workers pulls from the queue. Each worker fetches the source and the problem's test bundle, warming a local cache so popular problems are not re-downloaded, then spins up a sandbox. Inside the sandbox the worker compiles the code, and if compilation succeeds it runs the binary once per test case under strict CPU, wall-clock, memory, process, and output limits. After each run it compares the program's stdout against the expected output using either an exact comparator or a custom checker, and it stops at the first failing test to save compute. The worker writes the final verdict and resource usage back to the database, updates the contest scoring stream, and publishes a result event. The client learns the outcome by polling the submission token or by receiving a websocket push. A separate leaderboard service consumes the scoring stream and keeps a cached ranking that thousands of viewers read without touching the primary store. A background plagiarism pipeline runs after the fact, comparing accepted submissions per problem.
In a real interview, sketch this on the whiteboard before diving into any single box.
Core components
Walk through each service. The interviewer wants to hear what each one owns, not just the names.
Submission service and API gateway
The front door. It authenticates, rate limits per user and per IP, validates the language and code size, persists the submission, uploads source to object storage, and enqueues a job. It returns fast and never blocks on judging, so intake stays available even when the judge fleet is backed up.
Submission queue
A durable message queue such as Kafka or a Redis-backed job queue sits between intake and judging. It absorbs contest bursts, decouples producers from consumers, and provides at-least-once delivery. Partitioning by user id keeps a single user's submissions in order and spreads load across many partitions.
Judge worker pool
Stateless workers that pull jobs, run them in a sandbox, and report verdicts. Workers are grouped or pre-warmed by language so a contest that is mostly C++ does not wait on cold Java toolchains. The pool auto-scales ahead of scheduled contests and scales down afterward.
Sandbox and execution engine
The security core. Each run happens in an isolated environment built from Linux namespaces, cgroups, and seccomp filters, or a microVM for stronger isolation. It enforces hard limits on time, memory, processes, and output, blocks network access, and mounts the filesystem read-only so nothing leaks or persists between runs.
Test case and problem store
Object storage holds immutable, versioned problem bundles: inputs, expected outputs, checker binaries, and per-language limits. Bundles are content-addressed by hash so the judge can cache them locally and trust the cache. Large heavy-input problems are streamed rather than loaded whole.
Verdict and scoring pipeline
Consumes result events, writes final verdicts, and for contests updates each user's score and penalty time. It must be idempotent so a retried job does not double-count. It publishes to the leaderboard stream and to any push channel that notifies the waiting client.
Leaderboard and plagiarism services
The leaderboard service maintains a cached, sorted ranking updated from the scoring stream and served to a flood of readers. The plagiarism service runs asynchronously, fingerprinting accepted solutions per problem and flagging similar pairs for human review.
Data model
Pick the right store per table. Justify each choice with the access pattern, not by reflex.
submissionssubmission_iduser_idproblem_idcontest_idlanguagestatussource_urlcreated_atOne row per submission. status moves Queued to Running to a terminal verdict. source lives in object storage, not the row. Partition or shard by user_id; index by (contest_id, user_id) for standings.
verdictssubmission_idresultfailed_test_indextime_msmemory_kbscorejudged_atThe judged outcome. result is one of AC, WA, TLE, MLE, RE, CE. Stores the first failing test and measured resource usage. Written once and treated as immutable so scoring can be recomputed if needed.
problemsproblem_idslugtime_limit_msmemory_limit_kbchecker_typetest_bundle_hashProblem metadata and limits. checker_type distinguishes exact match from a custom special judge. test_bundle_hash points at an immutable, versioned bundle in object storage so changing tests never rewrites old rows.
test_casesproblem_idtest_indexinput_refexpected_refis_samplegroup_idOrdering and grouping of tests. Inputs and outputs are references into object storage, not inline blobs. is_sample marks the few public examples; group_id supports subtask scoring where a group must fully pass.
contest_scorescontest_iduser_idproblem_idbest_scorepenalty_secondssolved_atattemptsDenormalized per-user, per-problem standing driven by the scoring stream. The leaderboard aggregates from here. Updated idempotently keyed on submission_id so a redelivered result cannot inflate a score.
plagiarism_flagsproblem_idsubmission_asubmission_bsimilaritydetectorreviewedOutput of the offline similarity pipeline. Stores a similarity score and which detector produced it. reviewed tracks moderator action. Kept separate from verdicts so accusations never affect automated scoring.
Deep dives
These are the conversations the interviewer is steering you toward. Practice each one until you can talk through it without notes.
Sandboxing untrusted code
This is the heart of the system. You are running code written by anyone, including people actively trying to break out. The standard approach layers Linux kernel primitives. Namespaces give the process its own view of the process tree, mounts, and network, so it cannot see or signal other jobs and has no route to the internet. Cgroups cap memory, CPU, and the number of processes, which is how you both enforce the memory limit and stop a fork bomb from spawning until the machine dies. Seccomp-BPF applies a syscall whitelist so the program can read, write, and exit but cannot, for example, call ptrace or open a socket. The isolate sandbox, originally built for the International Olympiad in Informatics and used by Judge0, packages exactly this: namespaces plus cgroups plus a read-only filesystem plus resource limits. For a stronger boundary, many teams now run each job in a microVM such as Firecracker, which gives hardware-level KVM isolation with a boot time around 150 milliseconds, or in gVisor, which intercepts syscalls in userspace. The rule is defense in depth: assume any one layer can fail and make sure a breach of it still does not hand over the host or the hidden tests.
The queue and judge worker pool
Judging is slow and bursty, so it must be asynchronous. Intake writes the submission and drops an event on a durable queue, then a pool of stateless workers pulls jobs. Decoupling this way means intake stays fast and available even when every worker is busy, and it lets you scale the two sides independently. Partition the queue by user id so a single user's submissions are evaluated in order while different users spread across partitions. Workers should be grouped or pre-warmed by language because compiler and runtime startup dominates for short programs, and a contest is often 60 percent one language. At-least-once delivery means a job can be redelivered after a worker crash, so the verdict write and the score update both have to be idempotent, keyed on submission id. The pool auto-scales on queue lag: when the backlog clearing time crosses a threshold you add workers, and you pre-provision capacity minutes before a scheduled contest rather than reacting after the herd arrives.
The verdict pipeline: compile, run, compare
Once a worker has a job it runs a fixed pipeline. First compile inside the sandbox; a failure here is Compilation Error and the job ends. Then run the compiled program once per test case, feeding the test input on stdin and capturing stdout under the time and memory limits. If a run exceeds the CPU or wall clock limit it is Time Limit Exceeded, if it trips the memory ceiling it is Memory Limit Exceeded, and if it crashes or returns a nonzero exit it is Runtime Error. If it finishes cleanly, compare its output to the expected answer. For most problems that is a normalized exact match ignoring trailing whitespace, but many problems allow multiple valid answers, so they ship a custom checker or special judge that reads the input, the contestant output, and the reference output and decides correctness. The pipeline is fail-fast: it stops at the first test that is not Accepted and reports that verdict, which cuts wasted compute dramatically since most wrong submissions fail early. The final verdict plus measured time and memory is written back and streamed to scoring.
Measuring time limits fairly
A time limit is only meaningful if it is measured consistently, and that is harder than it looks. If two heavy sandboxes share a core, each one runs slower through no fault of the code, so wall-clock time alone will produce flaky TLE verdicts. The usual fix is to measure CPU time consumed by the process, not just elapsed time, and to also cap wall time as a backstop against a program that sleeps or blocks. You pin or limit the number of concurrent sandboxes per physical core so measurements stay stable, which is a real cost driver: you deliberately underpack machines to keep judging deterministic. Interpreted languages get a limit multiplier, since the same algorithm in Python is legitimately slower than in C++. Some judges also re-run a borderline submission or use a reference machine to calibrate, because a verdict that flips between runs is worse than one that is slightly strict. The design tension is throughput versus fairness: the more you isolate for accurate timing, the fewer jobs per machine you can run.
Surviving the contest thundering herd
Baseline traffic is calm; contests are not. When a round opens, and again in the final minutes, thousands of people submit within the same short window, and everyone refreshes the leaderboard between submissions. Two herds hit at once: writes to the judge queue and reads on standings. For the write side, the queue is the shock absorber. It accepts everything, and the worker pool drains it, so latency degrades gracefully instead of intake failing. You pre-scale the pool before the scheduled start rather than waiting for autoscaling to notice. One well-known optimization, used by Codeforces, is to judge live submissions only against a small pretest set during the contest, then run the full exhaustive system tests after it ends, which slashes peak compute. For the read side, the leaderboard is served entirely from a cache updated by the scoring stream, never by querying the primary database per refresh, and clients get a sensible poll interval or a push feed so a million refreshes do not become a million database queries.
Test case storage and distribution
Test data is large, immutable, and read constantly by the judges, which makes object storage the right home for it. Each problem's tests, expected outputs, and checker are packaged into a versioned bundle addressed by a content hash. Because the hash changes whenever the tests change, a judge that has the bundle cached on local SSD can trust it without revalidation, and old submissions keep pointing at the exact bundle they were judged against. Before a contest you can pre-seed every judge with the relevant bundles so the first submissions do not stampede object storage. Heavy problems whose inputs run to hundreds of megabytes are streamed test by test rather than loaded whole into memory. The security angle matters here too: the hidden tests are valuable, so the sandbox that runs the code must never be able to read the bundle path, which is why the test files are mounted read-only outside the contestant's reach and the process has no network to exfiltrate them.
Plagiarism detection
Cheating is common, so judges run similarity detection, but it belongs off the hot path. After submissions land, a batch pipeline groups accepted solutions per problem and compares them pairwise. The classic technique, from Stanford's MOSS, is to tokenize the source so variable renaming and reformatting do not hide a copy, hash overlapping windows of tokens, and select a stable subset of fingerprints using winnowing so two files that share long token sequences share fingerprints. Pairs above a similarity threshold are flagged for human review rather than auto-penalized, because false positives are easy when everyone implements the same standard algorithm. Contest settings tighten this: submissions within one contest are compared against each other and sometimes against public editorials and past submissions. Keeping this asynchronous means it never slows judging, and keeping flags separate from verdicts means an accusation never silently changes someone's automated score.
Trade-offs to discuss
Every senior interviewer expects you to surface at least 3 of these. Pick the decisions, state the alternatives, and justify your choice.
Container sandbox such as isolate versus microVM such as Firecracker
Namespaces plus cgroups plus seccomp are lightweight and start in milliseconds, which is cheap per job but shares the host kernel, so a kernel exploit is a full escape. A microVM gives hardware-backed isolation and a much smaller trust surface at the cost of higher memory per job and a heavier boot. Many judges use the container path for throughput and reserve microVMs when the threat model demands a hard boundary.
Polling versus websocket push for verdicts
Polling on the submission token is dead simple, works through any proxy, and scales with a cache, but it wastes requests and adds latency. A websocket push delivers the verdict the instant it is ready and feels better during a contest, but it needs sticky connection handling and a fallback. A common answer is push when a connection is live, poll as the fallback.
Fail-fast on first failing test versus running the full suite
Stopping at the first non-Accepted test can cut judging compute by up to about 90 percent because most wrong submissions fail early, and it is what contestants expect. Running the whole suite gives richer feedback and a stable per-test breakdown, which practice platforms may want. Contests almost always fail-fast; learning modes sometimes run everything.
Pretests during a contest versus full system tests live
Judging against a small pretest set during the contest and running exhaustive tests afterward, the Codeforces model, dramatically lowers peak compute and keeps verdict latency low under the herd. The cost is that a solution can pass live and fail final system tests, which changes standings after the fact and adds the complexity of a second judging pass. Judging the full suite live avoids surprises but needs far more capacity at peak.
Dedicated pre-warmed judge fleet versus pure on-demand autoscaling
Pre-provisioning machines before a scheduled contest guarantees capacity the moment the herd arrives, but you pay for idle hardware around the event. Pure on-demand scaling is cheaper at rest but reacts only after the backlog grows, and bare-metal or microVM hosts can take minutes to come online. Because contests are scheduled, pre-warming to a forecast and autoscaling on top is the pragmatic mix.
Exact output comparison versus a custom checker per problem
Exact match after whitespace normalization is trivial and fast and covers most problems. But many problems accept multiple correct answers, such as any valid shortest path, so they need a special judge that validates the answer against the input. Checkers add per-problem code to maintain and another untrusted-ish binary to run, but without them a whole class of problems is unjudgeable.
How Online Judge actually does it
The building blocks here are real and open. Judge0 is a widely used open-source execution engine that runs submissions inside the isolate sandbox, supports more than 90 languages, and exposes an async submit-then-poll API with webhooks. Isolate itself was written for the International Olympiad in Informatics and relies on Linux namespaces, cgroups, and a read-only chroot to bound untrusted programs, which is the same recipe most judges use. Codeforces publicly documents its two-phase model, pretests during the round and full system tests after, precisely to survive contest-scale bursts. For stronger isolation, teams increasingly reach for Firecracker microVMs or gVisor instead of plain containers. On the anti-cheating side, Stanford's MOSS pioneered the tokenize-fingerprint-winnow approach that most code-similarity detectors still follow. None of these components is exotic; the engineering is in wiring them together so the queue absorbs the herd, the sandbox never leaks, and the leaderboard stays fast while thousands of verdicts land.
Lessons to study before this interview
If any of these topics are fuzzy, the interviewer will catch it. Each lesson is 15 to 60 minutes with diagrams, code, and a quiz.
Related system design interview questions
Practice these next. They lean on the same core building blocks as Online Judge.