Skip to content

benchmarks: add elf_shoff prior probe (guessed ELF64 section-table offset) + fix --probe filter - #13

Merged
2akouwu merged 1 commit into
2akouwu:mainfrom
IMGillusion:issue4-elf-shoff-probe
Sep 5, 2026
Merged

benchmarks: add elf_shoff prior probe (guessed ELF64 section-table offset) + fix --probe filter#13
2akouwu merged 1 commit into
2akouwu:mainfrom
IMGillusion:issue4-elf-shoff-probe

Conversation

@IMGillusion

Copy link
Copy Markdown
Contributor

Part of #4 — the multi-prior scorecard was missing the guessed struct field offsets
pattern named in the issue.

What

  • probe_elf_shoff (new): the model memorizes a textbook ELF64 layout in which a
    "typical small ELF" keeps its section header table at 0x1000, and asserts
    e_shoff == 0x1000 (the u64 at file offset 0x28 of the ELF64 header) for any
    ELF64 binary. The prior is blind (never reads the section table) and the verdict
    goes through the verifier with the usual false-VERIFIED guard that re-reads the raw
    bytes. ELF64 only (EI_CLASS == 2): the ELF32 header lays e_shoff out
    differently, so the 64-bit memorized value is not a sound prior there.
  • tests/test_probes.py: TestElfShoffProbe — refuted when the slot is not 0x1000,
    verified (not a false accept) when it is, inapplicable for ELF32 / PE, plus the
    guard-inversion case (a buggy VERIFIED whose raw bytes contradict the claim is
    flagged).
  • BENCHMARK.md: probe table row + one aarch64 field run (33 ELFs under
    /usr/bin + /usr/lib on a Jetson): prior refuted on all 33, 0 false VERIFIED.
    Across the ~1,800 ELF64 binaries there, observed e_shoff ranges from 0x2a8 to
    0xbba8d38 and none equals 0x1000.

Bonus fix in the same file

The --probe CLI filter never worked: the value-collection loop started at the
--probe token itself (which starts with --, so the loop body never ran), only
was always None, and the flag silently ran every probe. One-line fix (i = argv.index("--probe") + 1). Verified: --probe elf_shoff now runs that probe only.

Local verification

$ python -m unittest reverify/tests/test_probes.py -v
Ran 16 tests ... OK

$ python benchmarks/hallucination_probes.py /usr/bin /usr/lib --per-dir 20 --probe elf_shoff
elf_shoff       33/33 = 100%   0
global false VERIFIED (must be 0): 0

Issue 2akouwu#4 asks for more prior probes that reflect real hallucination
patterns; the struct-layout one was missing. probe_elf_shoff asserts
e_shoff == 0x1000 (the memorized 'typical small ELF' section-table
offset, u64 at file offset 0x28 of the ELF64 header) for ELF64
binaries only (EI_CLASS == 2), with the usual raw-bytes guard.

Also fixes the --probe CLI filter in the same file: the value loop
started at the flag itself, so `only` was always None and the flag
silently ran every probe.
@2akouwu
2akouwu merged commit d9b92d3 into 2akouwu:main Sep 5, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants