The README covers what the tool does and how to run it. This page is the part that needs an argument rather than an instruction: the pipeline, the exact strength of the legality guarantee, and the measurements behind the kill-rate table.
Three steps. Two of them are deterministic; only the middle one is an agent.
statement (+ a reference solution — optional)
│
├─ 1. VALIDATOR a testlib validator from the constraints — ranges AND prose (distinct,
│ tree, |S|=N, primality…), rewritten until it accepts the statement's samples
├─ 2. CONSTRUCT an agent reasons from the STATEMENT in a shell workspace and builds a
│ discriminating set — edge cases, per-failure-mode inputs, and max-N stress
└─ 3. RENDER re-check every test through the validator (+ reference, if present), then write
tests/ (+ .ans when answered) + a Polygon package + a per-test REPORT.md
Steps 1 and 3 live in scripts/finalize.py and are shared by both entry points, so a set built by
your coding agent and a set built by the CLI pass through the same gates and come out in the same
shape. Step 2 is the only place a model decides anything about the tests.
That middle step is deliberately an agent, not a generator. The construction doctrine — what to build
and why — is SKILL.md, and it's the same document whether your own coding agent follows it or the
CLI hands it to mini-SWE-agent. The agent works in a
scratch workspace with the compiled validator and reference, writes generators, probes edge cases,
and checks its own candidates as it goes.
The model is the knob that matters. The scaffold is thin and the doctrine is fixed, so what you change to get better tests is the construction model. DeepSeek-V4-Pro scores a few points above DeepSeek-V4-Flash at a few times the cost. In a contest, where one missed wrong solution can flip a verdict, that's usually the knob to turn first.
There is no wrong-solution / kill-signal machinery: the agent does not draft its own wrong solutions to mine disagreements. That channel was measured in the companion study and adds nothing here while biasing the set toward a sparse set of targets.
Step 1 reads the constraints out of the statement — not only the ranges in the Constraints block but
the ones stated in prose (distinctness, a < b, permutation, tree / connected / simple graph, sum
bounds, character set, field relations like |S| = N, "no contradiction" premises) — and writes a real
testlib validator. It then repairs that validator until it accepts every sample input printed in the
statement. The samples are parsed out automatically; you place nothing, and it costs nothing, because
the statement already contains them.
If you happen to have a real test suite — richer than the 2–3 samples — put it in official_in/ and
those inputs are used instead. That's an override for people who have one, not a requirement.
Now the precise version of the guarantee. The validator is written by a model, from the statement. A test passing it proves the test is consistent with that validator — not that it is legal against the problem. Concretely:
| failure | caught? |
|---|---|
validator too strict (e.g. a missing readSpace/readEoln that rejects legal input) |
yes — the sample check catches it, and the repair loop fixes it |
| validator too loose (a constraint buried in prose, never exercised) | no — nothing here detects it |
| the statement ships no samples at all | the run says so: REPORT.md warns ⚠ UNVERIFIED |
So the honest framing is: a mechanical, repeatable legality gate you can inspect, not a proof.
validator.cpp ships with the package precisely so you can read it. For a high-stakes problem, that
read is cheap and worth it.
The same asymmetry applies to expected outputs: your reference is the sole oracle, and nothing cross-checks it against a brute force or a second solution. A wrong reference produces a wrong answer set that every gate here will wave through.
Kill-rate = the fraction of a real pool of wrong solutions that a test set catches, over 105–106 AtCoder problems (105 for the Flash row measured in this repo; ≈106 for the paper's arms).
| method | mean kill-rate | cost / problem |
|---|---|---|
| official test suite | 0.933 | — |
| Claude Code (Claude Opus 4.8) | 0.930 | ≈$1.6 |
| Codex (GPT-5.4) | 0.929 | ≈$1.4 |
| DeepSeek-V4-Pro + mini-SWE | 0.918 | ≈$0.04 (est.) |
| DeepSeek-V4-Flash + mini-SWE | 0.895 | ≈$0.02 |
| random generation | 0.823 | ≈$0 |
The pool is real rejected submissions to these problems. The wrong solutions are there because the official judge rejected them, which makes the official test suite the ceiling of this measurement — not a method we beat.
It still doesn't score 1.000, and the reason is the scoring rule: only deterministic kills count. A kill is credited when a test provokes a wrong answer or a runtime error. TLE kills earn no credit, and neither do environment-dependent failures (e.g. a stack-limit RE that depends on the machine), so that a kill always reflects a real logic bug rather than a flaky or platform artifact.
Measured over the pool: ≈91% of the rows official "misses" are ones it originally rejected on time, not on a wrong answer. That cuts both ways — every method is credited for the logic bugs it provokes and none for timing — and it's why the agent rows land where they do.
- The Flash + mini-SWE row is measured in this repo (105 problems,
runs_flash_mini/); the rows above it come from the companion paper's panel. The two pools differ by about a point on the methods common to both — official scores 0.946 on the local pool vs 0.933 on the panel — so the Flash row is not strictly on the same pool as the rows above it. Read its gap to them as approximate. - Point estimates within ≈1.5 points are not a claim of statistical equivalence. This page reports the headline; the companion paper carries the rigorous version — full per-problem data, the scaffold-vs-model and doctrine ablations, false-kill / legality audits, and significance tests.
- Cost figures are estimates from measured token and credit use, not a billed per-problem invoice.
No — which is why it doesn't by default. Both arms below kept the reference as the output oracle and differ only in whether the construction model saw its source. Same pipeline, same problems, paired:
| construction model | construction reads it | hidden from construction | Δ | 95% CI |
|---|---|---|---|---|
| Pro (31 problems) | 0.727 | 0.725 | −0.002 | [−0.027, +0.023] |
| Flash (10 problems) | 0.560 | 0.601 | +0.041 | [−0.007, +0.099] |
Neither Δ is distinguishable from zero at this sample size. That is not a proof of equivalence — the Pro interval still admits a ≈2.7-point drop, and the Flash arm is only 10 problems — but it is enough that there's no reason to pay for reading it: construction is ≈30% cheaper without the reference in context. Hence statement-first by default.
A reference-free run (no reference at all, so no .ans) builds its inputs the same way the hidden arm
does, so we expect the same input quality there. Note that this last step is an extrapolation from the
table, not a separate measurement of it.
Paired same-pool driver kill-rate, excl-TLE, on a subset of problems with its own baseline — a different measurement from the 105–106-problem table above. The columns here (0.727 / 0.560) are not comparable to the 0.918 / 0.895 there.
scripts/build_testset.py the CLI — validator + agentic construction + render, end to end
scripts/finalize.py the deterministic tools (render · optional validator writer); the CLI
imports the same code, so both paths produce identical artifacts
SKILL.md the construction doctrine — the agent-facing document
vendor/testlib.h testlib, vendored unmodified
examples/demo-output/ a real build's output, pruned for the repo: the validator, the per-test
REPORT.md, and a sample-tests/ excerpt standing in for the full tests/
examples/polygon_push.py reference only — the Polygon API call sequence + HMAC signing
Pushing the package to Polygon is out of scope; examples/polygon_push.py exists to show the call
sequence, not as a supported path.