benchmark

Benchmark: does a security skill actually reduce vulnerabilities?

This is the honest test that shaped paranoid. It answers one question with a reproducible harness instead of a claim — and the answer is why paranoid is a doing-tool (/hack-me) rather than a "write secure code" prompt.

The question

For the same security-sensitive tasks, does an agent with the skill loaded produce fewer exploitable solutions than the same agent without it, holding functional correctness equal?

Two numbers per condition:

• exploit rate among correct solutions — of the code that actually works, how much is still exploitable. Lower is better.
• correctness — the skill mustn't win on security by breaking functionality.

Method

• Tasks (tasks/): each has a neutral spec (no security hints), a functional check, and an exploit check for a specific class (IDOR, missing authz, SQLi, mass assignment, path traversal, SSRF, XSS, command injection, open redirect, JWT auth verification, leaked secrets, CSRF, template/format-string injection, XXE, unrestricted upload, permissive CORS, weak password storage, unverified webhooks, ReDoS, insecure deserialization, SSRF via DNS-rebinding, route-wiring IDOR, session cookie flags).
• Conditions: identical base model; baseline = no skill, paranoid = skill in context. The generated solutions live in solutions/<condition>/.
• Score: harness/run.py runs the functional check, then the exploit check, for each solution and reports per-condition rates.

Blinding (and a correction to the earlier numbers)

The task ids name their own vulnerability — sqli_login, xxe_item_parse, insecure_deserialization. Telling a model to write sqli_login.py is a security hint, arguably the strongest one available, because it names the exact weakness to defend against. The earlier three-class numbers were generated that way, and they were written into .../paranoid/benchmark/solutions/<condition>/, so the control arm could also infer it was the control arm of a security benchmark. Both leaks push the baseline toward looking secure — that is, toward the null result this page already reports. That is the worst direction for a bias to run, so the 22-class run was done twice:

• unblinded — specs as-is, vulnerability-named files, written inside the repo
• blinded — the same specs presented as task_01.py ... task_22.py, spec titles stripped, generated outside the repo, with no mention of a benchmark, a baseline, or security anywhere in the instructions

Both landed on +0pp. Quote the blinded row.

Residual limitation, stated rather than hidden: 22 security-shaped specs presented together still hint at the theme. The blinded baseline volunteered that "several specs were written in a way that invites an unsafe reading" — it inferred the subject from content alone. Removing that entirely would need one task per isolated context. So the baseline here is an upper bound on how secure an unprompted model looks, and the true delta could only be smaller, not larger.

Result

Matched-model runs, skill vs no-skill:

Model	Task set	exploit rate (baseline)	exploit rate (paranoid)	delta
Opus	all 22 classes, blinded (2026-09-21)	0%	0%	+0pp
Opus	all 22 classes, unblinded (2026-09-21)	0%	0%	+0pp
Fable 5.1	3 isolated fns, leading specs	0%	0%	+0pp
Opus	3 isolated fns, leading specs	0%	0%	+0pp
Opus	3 isolated fns, neutral/tempting specs	0%	0%	+0pp

A capable model already writes the secure version of an isolated function unprompted. The skill has no headroom to add value at this granularity — now measured across all 22 classes, not three, and with the vulnerability names hidden from both arms (see Blinding below).

It does cost correctness. In both 22-class runs the skill condition scored 21/22 functional against the baseline's 22/22, failing the same task each time: leaked_secrets_client_config, where it over-filtered until the legitimate public settings were stripped along with the secrets. A safety win that breaks the feature is not a win, and it is the trade-off BaxBench reports too.

Specifically, it dropped STRIPE_PUBLISHABLE_KEY — a credential Stripe designs to ship to the browser — because the name looks like a secret. The skill has since been given the missing counterweight: which credentials are meant to ship and the rule that you decide by what a credential is, not by whether its name contains KEY. Whether that actually closes the gap is unmeasured — it needs a re-run, and the 21/22 above stands until someone produces a new number (#17).

Raw solutions for every condition are committed under solutions/ so anyone can re-score them.

This is not a broken harness. Against deliberately-insecure vs secure reference solutions (solutions/selftest_insecure, solutions/selftest_secure) it reports 100% and 0% exploit rates respectively — it detects the difference when there is one. This self-test runs in CI on every push, so a regression that makes the harness miss a known bug fails the build.

Task classes

Twenty-four classes have a neutral spec + functional + exploit check today: idor_invoices, idor_session_only, missing_auth_admin, sqli_login, mass_assignment_update, path_traversal_note, ssrf_url_preview, xss_comment_render, command_injection_ping, open_redirect_login, jwt_verify_identity, leaked_secrets_client_config, csrf_state_change, template_injection_notice, xxe_item_parse, unrestricted_file_upload, permissive_cors_origin, weak_password_storage, redos_username_validate, webhook_event_apply, insecure_deserialization, ssrf_dns_rebinding, session_cookie_flags, idor_route_wiring. All twenty-four are covered by the insecure/secure self-test above (100% / 0%). As of 2026-09-21 twenty-two of them have also been scored against a model in both conditions — see the blinded run in Result; session_cookie_flags and idor_route_wiring are harness-verified and awaiting a model-condition run (honesty rule: no model number until the harness produces one).

idor_route_wiring is the first task whose bug isn't visible in any single function — it only exists in how two routes are wired, one checking ownership and the other not. That is the regime the +0pp result above does not cover, so it is the one most worth scoring next.

Machine-readable output: add --json to any run.py invocation to get a single JSON summary (per-condition rates + delta) instead of the human tables — handy for CI and for diffing conditions.

What it means

Isolated functions are the wrong unit. The vulnerabilities that ship in real apps live in cross-cutting wiring — an auth check present on one route and missing on the next, a request body spread into an update, a search that builds SQL — context a model can't hold perfectly across a whole codebase. That is what /hack-me targets by attacking the running app, and where it demonstrably finds real bugs (see ../examples/ledgerlite).

This matches the literature

The null result isn't an artifact of a tiny harness — it lines up with recent work:

• Prompt Structure Redistributes, Not Reduces — security-aware prompting changes which weakness categories appear but does not reliably lower the overall vulnerability rate.
• BaxBench — generic security reminders help reasoning models modestly and barely move instruction-following ones; naming the exact vulnerability helps but costs functional correctness.

The consistent takeaway: guidance nudges, it doesn't fix. That's why paranoid's value is in doing (/hack-me), not advising.

Reproduce

cd benchmark

# sanity-check the harness itself:
python3 harness/run.py --only mass_assignment_update,idor_session_only,path_traversal_note \
    solutions/selftest_insecure solutions/selftest_secure     # expect 100% vs 0%

# score a pair of generated conditions:
python3 harness/run.py solutions/opus_baseline solutions/opus_paranoid
python3 harness/run.py --only mass_assignment_update,idor_session_only,path_traversal_note \
    solutions/hard_opus_baseline solutions/hard_opus_paranoid

To add your own condition, drop one <task_id>.py per task into solutions/<name>/ (each defining the function named in that task's spec.md) and pass that dir to run.py. Generate them with any model, with or without the skill in context.

Honesty rules for this benchmark

• No number in a README until the harness produced it here. (The +0pp above is measured, not assumed.)
• Report the losses too — this whole page is a negative result, kept because it's true and it's why the project pivoted.
• Report the model and date; a number without them is meaningless as models move.
• A fuller run belongs on realistic, multi-file apps (BaxBench-style), not just isolated functions — that's the regime where a security tool has room to help.