How grading works

Code submissions compile and run remotely in an isolated grading environment. System Design uses a separate data-only path: the server validates a bounded topology and evaluates deterministic graph rules without executing learner code. Both return the evidence that applies to the challenge. Your browser and network latency are not included in code timings.

Use Run while iterating on visible samples and custom input. Use Submit when you want the full grade, hidden tests, solved status, best-time update, learning-path progress, and signed-in rewards. Only Submit uses the Free plan's daily submission allowance.

The overall result passes only when every required dimension passes. Correct output does not override a timeout, allocation gate, query-plan rule, design rule, security check, or test-writing target.

Every result opens with a verdict line on both the Output and Tests tabs. It says whether the result passed and names the gate that decided it: a hidden test, the performance budget, a query-plan rule, the mutation score, and so on. A failure adds what to inspect next. When hidden tests failed, the Output tab offers a button that jumps to them on Tests.

On a failure, the Tests tab puts the evidence first: the failing test rows, plan rules, or code rules, then what passed, then a compact recap titled Graded result (Run result after Run). On a passing Submit the recap leads, with one card per dimension. The recap shows only the dimensions that matter for the selected puzzle. It can include:

• Correctness or behavior preserved;
• Runtime budget and allocation budget;
• Adversarial checks for secure-coding work;
• Query-plan rules for database work;
• Code-quality, architecture, or design rules;
• Mutants caught for test-writing work;
• Best-time standing after an eligible graded result.

A green recap means all graded checks passed. An amber or red dimension points to the kind of change to make next. The detailed test rows contain the individual evidence.

Each test has its own wall-clock budget, measured on the server, immediately around your code. The results panel shows a bar of how much budget you used. Passing close to the limit is a hint even when the test is technically green. Hidden performance tests use inputs large enough that complexity decides the outcome: an efficient solution passes comfortably, and a naive one cannot.

The slowest value in the graded result is not a sum of every test. Use the detailed test rows to find which input consumed the largest share of its own budget.

Tests also report bytes allocated on the managed heap during your run, and some puzzles set an allocation budget. Watch this number even where it isn't enforced. Allocation pressure is the silent performance tax in .NET.

When no allocation limit applies, memory can appear as Measured rather than Passed or Failed. That is still useful feedback, but it does not affect the verdict.

Each track adds its own gate on top of the behavioral tests:

• Database puzzles can grade the execution plan the engine chose: a full-table scan where an index should be used fails, even when the rows come back right. The Database track guide lists every plan rule and how to read the outline.
• Refactoring puzzles enforce structural metrics measured from your source: method length, cyclomatic complexity, nesting depth, and duplicate blocks, each against an explicit limit.
• Architecture katas verify design rules: dependency direction, layer isolation, required abstractions, sealed types.
• Secure-coding puzzles run a separate adversarial suite of real attack payloads alongside the functional tests, both must pass.
• Design Principles puzzles check focused constraints such as immutability, sealing, role placement, and dependency direction.
• Test Writing puzzles first require your tests to pass against the correct implementation, then measure how many planted-bug versions your tests reject. See the Test Writing track guide.
• System Design challenges verify semantic graph properties: required paths, forbidden bypasses, redundancy, fan-out, boundary containment, and typed request, messaging, storage, secrets, telemetry, and replication connections. The pass verdict is structural, not a load test. When the structure holds, a deterministic capacity simulation adds p99 latency, throughput, error rate, and modeled cost under the challenge's traffic scenarios; the System Design Studio guide explains how to read it.

Metric gates report the measured value next to the limit, so a failure tells you exactly how far off you are, and a pass tells you how close you came.

Hidden tests never reveal their input, expected output, or your actual output. They show the test name, status, timing, and applicable budget or rule evidence. This keeps the suite meaningful: you cannot hardcode your way past it. The test names are written as useful hints, for example a large input that requires linear complexity.

Visible tests show their input, expected answer, actual answer, and character-level difference. Custom Run cases show input, return value, and console output but have no expected answer.

The Output and Tests tabs surface different details by track:

• Database work can list generated queries and display the graded query-plan outline.
• Secure-coding work separates functional cases from exploit cases.
• Refactoring and design work lists each structural rule and its measured detail.
• Test-writing work lists planted bugs as caught or survived without exposing their implementation.
• System Design work lists each structural rule, the property it expected, and whether the current topology satisfied it.

These are the public learning signals you need to act on a grade. Katabench does not expose its private test inputs or the internal implementation of the grading service.

A puzzle becomes solved after a clean graded Submit, not after Run and not after revealing the reference solution. A successful solve can also:

• update your saved best time and puzzle standing;
• complete a step in one or more learning paths;
• extend signed-in activity, XP, streak, quest, and achievement progress;
• unlock the post-solve Why this works explanation;
• offer the next puzzle in the active path or catalog.

Timings come from the grading servers, not your browser or network, so results are consistent and comparable. Everyone on the leaderboard is measured by the same clock.

Timing always has some normal variation. Compare algorithms by clear differences and budget headroom, not by treating a tiny change in one run as a benchmark conclusion. Your persisted best eligible time is what drives puzzle speed standing.