benchmarks: add elf_shoff prior probe (guessed ELF64 section-table offset) + fix --probe filter - #13
Merged
Conversation
Issue 2akouwu#4 asks for more prior probes that reflect real hallucination patterns; the struct-layout one was missing. probe_elf_shoff asserts e_shoff == 0x1000 (the memorized 'typical small ELF' section-table offset, u64 at file offset 0x28 of the ELF64 header) for ELF64 binaries only (EI_CLASS == 2), with the usual raw-bytes guard. Also fixes the --probe CLI filter in the same file: the value loop started at the flag itself, so `only` was always None and the flag silently ran every probe.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #4 — the multi-prior scorecard was missing the guessed struct field offsets
pattern named in the issue.
What
probe_elf_shoff(new): the model memorizes a textbook ELF64 layout in which a"typical small ELF" keeps its section header table at 0x1000, and asserts
e_shoff == 0x1000(the u64 at file offset 0x28 of the ELF64 header) for anyELF64 binary. The prior is blind (never reads the section table) and the verdict
goes through the verifier with the usual false-VERIFIED guard that re-reads the raw
bytes. ELF64 only (
EI_CLASS == 2): the ELF32 header layse_shoffoutdifferently, so the 64-bit memorized value is not a sound prior there.
tests/test_probes.py:TestElfShoffProbe— refuted when the slot is not 0x1000,verified (not a false accept) when it is, inapplicable for ELF32 / PE, plus the
guard-inversion case (a buggy VERIFIED whose raw bytes contradict the claim is
flagged).
BENCHMARK.md: probe table row + one aarch64 field run (33 ELFs under/usr/bin + /usr/lib on a Jetson): prior refuted on all 33, 0 false VERIFIED.
Across the ~1,800 ELF64 binaries there, observed
e_shoffranges from 0x2a8 to0xbba8d38 and none equals 0x1000.
Bonus fix in the same file
The
--probeCLI filter never worked: the value-collection loop started at the--probetoken itself (which starts with--, so the loop body never ran),onlywas always
None, and the flag silently ran every probe. One-line fix (i = argv.index("--probe") + 1). Verified:--probe elf_shoffnow runs that probe only.Local verification