agentic-testgen

Generate the test data for a competitive-programming problem you already have. Point it at the statement (and your reference solution, if you have one) and you get back a testlib validator, a set of tests built to catch wrong solutions, and a Polygon-ready package.

On a 105-problem AtCoder benchmark its tests caught 0.90–0.92 of a real pool of wrong submissions — against 0.93 for the problems' own official test data — at $0.02–0.04 per problem. Fuller numbers in How good is it.

Any LLM will hand you a bag of test data. The difference here: every test is checked by a real testlib validator before it ships, and the tests are reasoned about — an agent works out from the statement where a wrong solution would break, then builds inputs to hit exactly that — instead of being sampled at random.

It generates the test data. It does not invent problems, and it stops at the package — pushing to Polygon is your step.

Research context

This repository is the companion skill and CLI for the paper Coding Agents as Test-Suite Auditors: Finding What Official Suites Miss While Approaching What They Catch. The paper's reproduction materials — ledgers, kill matrices, validators, judging harness, and analysis code — live in the separate xieTwim/test-suite-auditors research-artifact repository. Use this repository when you want to generate tests for a problem; use the artifact repository when you want to inspect or reproduce the paper's evidence.

Install

git clone https://github.com/xieTwim/agentic-testgen && cd agentic-testgen

Two things, both probably already on your machine:

• g++ — to build the validator, your reference solution, and the agent's generators. Any recent one; a distro package on Linux, MinGW-w64 or WSL on Windows. testlib is bundled.
• python3 — the tool itself is written in Python (the driver and the legality checks). Standard library only, so nothing to pip install if you're going the coding-agent route.

Everything is compiled as C++17 by default. If your reference uses C++20 features (ranges, <bit>, std::span, <=>) it won't build — set CXXSTD=c++20 and it will.

_{Pin a compiler with GXX=/path/to/g++. macOS only: /usr/bin/g++ is Apple clang and has no <bits/stdc++.h>, so most reference solutions won't compile — brew install gcc and it's picked up automatically.}

Set up your problem

Put your files in a directory. Names don't matter — it works out which is which:

my-problem/
  <anything>.md      the problem statement — .md, .txt or .pdf — required
  <anything>.cpp     a correct reference solution — optional

Nothing else to prepare. The sample inputs printed in your statement are pulled out automatically and used to check the validator. With a reference you get expected outputs too; without one you get the inputs alone, and you bring your own oracle (your solution, a brute force, a checker) — or just use them as stress inputs.

PDF statements work — contest problems usually come as PDFs. They're converted with pdftotext (apt install poppler-utils / brew install poppler), which is reliable for the prose and the constraints but can mangle a two-column sample table. If the run reports that it found no samples, either save the statement as text or drop a couple of legal inputs into official_in/ by hand.

_{Only ambiguity needs your help: several .cpp files and none obviously the reference → pass --ref path/to/it (same for --statement).}

Run it

Two ways in, depending on what you already have. Both build the same thing.

If you use a coding agent (Claude Code, Cursor, Codex, …)

Cheapest path — no extra install, no extra API key. Your agent is the construction agent; SKILL.md tells it how. For Claude Code:

ln -sfn "$PWD" ~/.claude/skills/agentic-testgen    # then invoke the `agentic-testgen` skill

Other agents have their own skill / rules mechanism — register SKILL.md the same way, or just paste it in. Then point it at your problem dir and say what you have:

"build a test set for this problem" · "…I have no reference, inputs only"

It writes the validator, builds the tests, and runs the deterministic packager itself to finish.

If you'd rather run a CLI

Drives a cheap model itself. Needs an API key and one pip install:

pip install mini-swe-agent
export DEEPSEEK_API_KEY=…          # or any OpenAI-compatible provider:
                                   #   OPENAI_API_BASE=https://your-endpoint/v1  OPENAI_API_KEY=…

python3 scripts/build_testset.py my-problem/ --out out/

The knob that matters is the model: --model deepseek-v4-pro costs a few times more than the default deepseek-v4-flash and catches a few more wrong solutions. Otherwise you'll mostly want --budget N (how many tests to keep, default 50) and --out DIR. --help has the rest.

If mini isn't on your PATH after installing, point MINI_BIN at it.

How your reference is used. By default it answers the tests — it produces every .ans and gates every input — but the agent doesn't read your code; it reasons from the statement, which measures no worse and costs less. Two flags move that independently:

--read-ref        also let the agent read your reference's source
--no-ref          ignore the reference entirely: inputs only, no .ans

What you get

tests/NN.in                 the test inputs
tests/NN.ans                expected outputs — when you supplied a reference
validator.cpp               the testlib input validator
ref.cpp                     a copy of your reference — when you supplied one
polygon-manifest.json       validator + tests + main solution + suggested checker, for Polygon
REPORT.md                   one row per test: its size, and what wrong behaviour it targets

⚠ Output goes into your problem dir by default and overwrites those names. Use --out DIR for a clean run. Stale tests/NN.in above the new count aren't deleted.

Supported: ordinary batch problems — one input file, one exact expected output, non-interactive. Not interactors, custom checkers, subtasks, or multiple-valid-output problems.

How good is it

Kill-rate = the share of a real pool of wrong submissions that a test set catches. Over 105 AtCoder problems, TLE and environment-dependent failures excluded, so a kill means a real logic bug:

test data	kill-rate	cost / problem
the problems' official test suite	0.933	—
Claude Code (Claude Opus 4.8)	0.930	≈$1.6
Codex (GPT-5.4)	0.929	≈$1.4
agentic-testgen · DeepSeek-V4-Pro	0.918	≈$0.04
agentic-testgen · DeepSeek-V4-Flash	0.895	≈$0.02
random generation	0.823	≈$0

The wrong solutions are in the pool because the official judge rejected them, so official test data is the ceiling here — not something we beat. What the table says is that a cheap model gets within a point or two of it, and comfortably past random.

→ docs/DETAILS.md for how this is measured, why official isn't 1.000, the per-problem data, and the ablations.

Before you ship it

Roughly in the order these should change what you do.

⚠️ The model can be wrong — don't ship a high-stakes problem unreviewed. Two mistakes propagate quietly into data that looks fine: the validator is written by a model, so if it's too loose, illegal tests ship as legal and nothing here catches that (too strict is caught — see DETAILS); and your reference is the only oracle, so a wrong reference means a wrong answer set. Stronger models make fewer mistakes, but not zero. Treat every build as a candidate: read validator.cpp against your statement, and run your intended solution plus a few known-wrong ones against tests/ before publishing. Minutes of work on a build that cost cents.

⚠️ It optimises for catching solutions, not for your intent — so prune it. The agent will happily max out every size parameter and pile on adversarial cases. You often want less: to let an intended-but-slower solution pass, or to keep the problem about the algorithm rather than the constant factor. That's what REPORT.md is for — one row per test with its measured size and what it targets. Read it, delete the tests/NN.in you don't want (drop the matching row from polygon-manifest.json), ship the rest.

⚠️ Your problem leaves your machine. The statement goes to whichever model endpoint is configured — your coding agent's provider, or OPENAI_API_BASE. --read-ref sends your reference too. For an unreleased problem, check the provider's retention and training terms, or point at a local endpoint.

⚠️ It runs model-authored shell commands on your machine. Construction is an agent writing and running code. This repo is not a sandbox and adds no denylist. Via a coding agent you get that agent's permission prompts; the CLI runs unattended, so that's the one to put in a container or VM if you don't fully trust the model.

⚠️ A run can finish with zero tests. If everything the agent built gets rejected by the validator or crashes the reference, you get an empty package. Check the reported test count and the dropped counts in REPORT.md rather than assuming it worked.

Citation

If you use this skill, please cite the accompanying paper. CITATION.cff provides the same preferred citation to GitHub's Cite this repository menu.

@misc{xie2026codingagents,
  title         = {Coding Agents as Test-Suite Auditors:
                   Finding What Official Suites Miss While Approaching What They Catch},
  author        = {Xie, Shuyang and Xie, Shuxiao and Zhu, Feng and Ji, Yanli and Zuo, Wangmeng},
  year          = {2026},
  eprint        = {2608.01715},
  archivePrefix = {arXiv},
  primaryClass  = {cs.SE},
  url           = {https://arxiv.org/abs/2608.01715}
}

License and third-party notices

vendor/testlib.h is testlib by Mike Mirzayanov, vendored unmodified under its upstream MIT license; see THIRD_PARTY_NOTICES.md. Construction is driven by mini-SWE-agent (Kilian Lieret & Carlos E. Jimenez; MIT), installed separately. This project's own code is MIT — see LICENSE.