How grading works

What Submit does

Submit compiles and runs your code remotely in an isolated grading environment, then returns the evidence that applies to that puzzle. Your browser and network latency are not included in the code timings.

Use Run while iterating on visible samples and custom input. Use Submit when you want the full grade, hidden tests, solved status, best-time update, learning-path progress, and signed-in rewards. Only Submit uses the Free plan's daily submission allowance.

Result statuses

Status	Meaning
Passed	Correct output, within the time budget.
Failed	Wrong output, visible tests show a character-level diff of expected vs. actual.
Timeout	Correct or not, it exceeded the per-test time budget. Usually means your complexity is wrong, not your logic.
Error	The code threw an exception, you'll see the type and message.
Compile error	The project did not compile. Read the diagnostics in the result and inline editor markers.
Rejected	The submission uses an operation that the code-safety rules do not allow. Read each reason and remove the blocked operation.
Platform error	The grade could not complete for a reason outside your solution. Retry once, then contact support if it repeats.

The overall result passes only when every required dimension passes. Correct output does not override a timeout, allocation gate, query-plan rule, design rule, security check, or test-writing target.

The quality scorecard

The top of the Tests tab summarizes only the dimensions that matter for the selected puzzle. It can include:

• Correctness or behavior preserved;
• Runtime budget and allocation budget;
• Adversarial checks for secure-coding work;
• Query-plan rules for database work;
• Code-quality, architecture, or design rules;
• Mutants caught for test-writing work;
• Best-time standing after an eligible graded result.

A green scorecard means all graded checks passed. An amber or red dimension points to the kind of change to make next. The detailed rows below it contain the individual evidence.

Time budgets

Each test has its own wall-clock budget, measured on the server, immediately around your code. The results panel shows a bar of how much budget you used. Passing close to the limit is a hint even when the test is technically green. Hidden performance tests use inputs large enough that complexity decides the outcome: an efficient solution passes comfortably, and a naive one cannot.

The slowest value in the scorecard is not a sum of every test. Use the detailed test rows to find which input consumed the largest share of its own budget.

Memory

Tests also report bytes allocated on the managed heap during your run, and some puzzles set an allocation budget. Watch this number even where it isn't enforced. Allocation pressure is the silent performance tax in .NET.

When no allocation limit applies, memory can appear as Measured rather than Passed or Failed. That is still useful feedback, but it does not affect the verdict.

Beyond time and memory

Each track adds its own gate on top of the behavioral tests:

• Database puzzles can grade the execution plan the engine chose: a full-table scan where an index should be used fails, even when the rows come back right.
• Refactoring puzzles enforce structural metrics measured from your source: method length, cyclomatic complexity, nesting depth, and duplicate blocks, each against an explicit limit.
• Architecture katas verify design rules: dependency direction, layer isolation, required abstractions, sealed types.
• Secure-coding puzzles run a separate adversarial suite of real attack payloads alongside the functional tests, both must pass.
• Design Principles puzzles check focused constraints such as immutability, sealing, role placement, and dependency direction.
• Test Writing puzzles first require your tests to pass against the correct implementation, then measure how many planted-bug versions your tests reject.

Metric gates report the measured value next to the limit, so a failure tells you exactly how far off you are, and a pass tells you how close you came.

Hidden tests

Hidden tests never reveal their input, expected output, or your actual output. They show the test name, status, timing, and applicable budget or rule evidence. This keeps the suite meaningful: you cannot hardcode your way past it. The test names are written as useful hints, for example a large input that requires linear complexity.

Visible tests show their input, expected answer, actual answer, and character-level difference. Custom Run cases show input, return value, and console output but have no expected answer.

Track-specific evidence

The Output and Tests tabs surface different details by track:

• Database work can list generated queries and display the graded query-plan outline.
• Secure-coding work separates functional cases from exploit cases.
• Refactoring and design work lists each structural rule and its measured detail.
• Test-writing work lists planted bugs as caught or survived without exposing their implementation.

These are the public learning signals you need to act on a grade. Katabench does not expose its private test inputs or the internal implementation of the grading service.

A successful solve

A puzzle becomes solved after a clean graded Submit, not after Run and not after revealing the reference solution. A successful solve can also:

• update your saved best time and puzzle standing;
• complete a step in one or more learning paths;
• extend signed-in activity, XP, streak, quest, and achievement progress;
• unlock the post-solve Why this works explanation;
• offer the next puzzle in the active path or catalog.

Fairness

Timings come from the grading servers, not your browser or network, so results are consistent and comparable. Everyone on the leaderboard is measured by the same clock.

Timing always has some normal variation. Compare algorithms by clear differences and budget headroom, not by treating a tiny change in one run as a benchmark conclusion. Your persisted best eligible time is what drives puzzle speed standing.