bench: multi-prior hallucination scorecard (issue #4) - #10
Merged
Conversation
2akouwu
approved these changes
Sep 4, 2026
2akouwu
left a comment
2akouwu
left a comment
Owner
There was a problem hiding this comment.
Reviewed and verified: four blind priors with an independent re-check of every VERIFIED verdict, synthetic PE/ELF fixtures pinning the guard, CI green on all nine jobs. Merging. I'll follow up by wiring the scorecard into the same record format as prologue_prior.py (--json / --markdown / --fail-on-false-verified, per-platform corpora) so it runs in CI next to the other gates — no action needed from you. Thanks for taking #4.
2akouwu
added a commit
to IMGillusion/reverify
that referenced
this pull request
Sep 4, 2026
…outputs, temp cleanup, TESTED tier and weight Native re-execution of candidate C is a different risk class from Unicorn emulation: off by default, INCONCLUSIVE with a hint unless the environment opts in. MinGW/MSVC append .exe; work happens in a TemporaryDirectory; stdin is closed; the claim records native_execution and a weight basis; the ledger files it as TESTED. CHANGELOG entries for 2akouwu#10 and 2akouwu#11. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V7iJFjziYzeBUwjqB5HkKw
2akouwu
added a commit
that referenced
this pull request
Sep 4, 2026
…corded I/O, opt-in native execution (#11, issue #2) * feat: ExeBench re-executability adapter + exebench claim kind (issue #2) * feat: exebench adapter — compile candidate C, re-run against I/O pairs * test: ExeBench adapter — real-compiler + gated paths * exebench: opt-in native execution (REVERIFY_ALLOW_NATIVE_EXEC), .exe outputs, temp cleanup, TESTED tier and weight Native re-execution of candidate C is a different risk class from Unicorn emulation: off by default, INCONCLUSIVE with a hint unless the environment opts in. MinGW/MSVC append .exe; work happens in a TemporaryDirectory; stdin is closed; the claim records native_execution and a weight basis; the ledger files it as TESTED. CHANGELOG entries for #10 and #11. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V7iJFjziYzeBUwjqB5HkKw --------- Co-authored-by: 2akouwu <14812897+2akouwu@users.noreply.github.com>
2akouwu
added a commit
that referenced
this pull request
Sep 4, 2026
… CLI, OpenCode Version bump and CHANGELOG. The headline is the universal rollover: state lives in files, the conversation is a cache, and the session is replaced rather than summarized. Also in this release: exebench claim kind (#11), multi-prior hallucination scorecard (#10), and the rollover controller. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V7iJFjziYzeBUwjqB5HkKw
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #5 (arm64 arch routing). Extends the single-prior
prologue_prior.pybenchmark into a scorecard over four model priors reflecting real LLM hallucination patterns, each applied blind and measured through the verifier with a per-probe false-VERIFIED guard.benchmarks/hallucination_probes.py: probes for the textbook frame-pointer prologue, a misidentified MD5 initial constant (A0 = 0x67452301), the deprecatedgets()import, and the ELF.rodatasection name applied to PE. x86-only probes are gated by arch so they run cleanly on ARM corpora too.reverify/tests/test_probes.py: synthetic PE32+/ELF fixtures pin the guard logic — a VERIFIED verdict whose evidence contradicts the claim must be flagged as a false accept (this is where a first draft of the prologue guard was inverted).BENCHMARK.md: documents the scorecard and the Linux ELF field run (8 real arm64 ELFs, import_gets refuted 8/8, 0 false VERIFIED).Field run:
global false VERIFIED (must be 0): 0. Full suite 200 passed; the one environmental failure (objdump oracle hardcodes an x86 target) is the same class #5 fixes.