Skip to content

docs(model-notes): ohalloran-demonstrator was a manifest defect, not two model failures - #37

Merged
usjoh merged 2 commits into
mainfrom
docs/model-notes-ohalloran-false-failure
Aug 16, 2026
Merged

docs(model-notes): ohalloran-demonstrator was a manifest defect, not two model failures#37
usjoh merged 2 commits into
mainfrom
docs/model-notes-ohalloran-false-failure

Conversation

@usjoh

@usjoh usjoh commented Aug 16, 2026

Copy link
Copy Markdown
Owner

claude-opus-4.8 and claude-sonnet-5 each read 0% first-try on site-build, and that is the whole of their site-build record. Neither model failed.

What the run actually was

ohalloran-demonstrator-20260816T022403Z-p82264 (meridian ms-20260815-1624), four lanes:

c-drawn         claude-opus-4.8   FAIL x2   (91.7k tok)
d-conversation  claude-sonnet-5   FAIL x2   (115.0k tok)
a-reveal        gpt-5.6-sol       PASS
b-archive       gpt-5.6-sol       PASS

Two opencode lanes fail, two codex lanes pass — a shape that reads instantly as "the cheap lane can't cope." It is nothing of the kind.

Both opencode workers exited cleanly (worker_returncode=0). The checks failed only on missing_expect_files. The manifest declared each deliverable at an absolute path inside the calling Meridian session's scratchpad, which the opencode Seatbelt wrapper denies — it confines writes to the task dir plus the per-run scratch. The models did the work and physically could not deliver it. The codex lanes passed because they were never behind that profile.

Filed as dated bullets under both model headings, so the warning reaches whoever routes site-build work next, plus a process-lessons section recording the rule this confirms for the third time in this file: read worker_returncode before calling anything a model failure.

It also corrects a claim this file made yesterday

The opus-5 entry from #35 said models --attributable "excludes them and shows 100%". It does show 100% — but not because it identified the OpenRouter outage.

db_attempt_rows never SELECTs the failure_class column. On the DB read path — the default — every row reaches the filter unclassified, model_log_row_failure_class returns None, and model_log_row_is_model_attributable rejects it. Measured here:

total attempts : 371
non-PASS       : 137
--attributable : "dropped 137 row(s)"      ← every failure, not the unattributable ones
result         : all 33 model rows read 100% first-try

The column is populated correctly in both runs.jsonl and ringer.db — only the reader drops it, so no db rebuild fixes it.

The note now says not to cite an --attributable rate as evidence until that's fixed, including the 2026-07-28 figures elsewhere in this file, which predate the DB read path and should be re-derived rather than trusted.

That reader bug is not fixed here — this PR is notes only.

Verification

  • Every claim checked against runs.jsonl, ringer.db, and the recorded run state, not against the prose
  • Both new bullets confirmed surfacing on the site-build scoreboard rows (models --task-type site-build)
  • Full suite: 352 pass

🤖 Generated with Claude Code

…two model failures

claude-opus-4.8 and claude-sonnet-5 each read 0% first-try on site-build, and
that is the whole of their site-build record. Neither failed.

Run ohalloran-demonstrator-20260816T022403Z-p82264 (meridian ms-20260815-1624):
both opencode workers exited CLEANLY (worker_returncode=0) and their checks
failed only on missing_expect_files. The manifest declared each deliverable at
an absolute path inside the calling Meridian session's scratchpad, which the
opencode Seatbelt wrapper denies — it confines writes to the task dir plus the
per-run scratch. The models did the work and physically could not deliver it.
The two codex lanes in the same run passed because they were never behind that
profile, which is what makes the run read as "the cheap lane cannot cope".

Files dated bullets under both model headings so the warning reaches whoever
routes site-build work, plus a process-lessons section recording the diagnostic
rule this confirms for the third time in this file: read worker_returncode
before calling anything a model failure.

Also corrects a claim this file made yesterday. The opus-5 entry said
`models --attributable` "excludes [the OpenRouter outage rows] and shows 100%".
It does show 100% — but not because it identified the outage. db_attempt_rows
never SELECTs the failure_class column, so on the DB read path every row
reaches the filter unclassified and the flag drops EVERY failure: 137 of 371
attempts, exactly the non-PASS count, after which all 33 model rows read 100%
first-try. The column is populated correctly in both runs.jsonl and ringer.db;
only the reader drops it, so no rebuild fixes it. The note now says not to cite
an --attributable rate as evidence until that is fixed, including the
2026-07-28 figures elsewhere in this file.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This branch was written while db_attempt_rows still dropped the failure_class
column, so it told the reader not to cite an --attributable rate at all. PR #38
fixes the reader: the flag now drops 46 rows (2 engine-error + 44 unknown)
instead of all 137 failures, matching the JSONL path.

Rewrites the warning as history rather than standing advice, and narrows the
"do not cite" to what it should always have been: any attributable figure read
BEFORE that fix, including the 2026-07-28 ones elsewhere in this file.

The opus-5 entry gets the same treatment. Its 100% over one completed task is
now genuine — those two tasks are classed `unknown` and really are excluded —
where before it was an artifact of every failure being dropped.

Also records the lesson that outlives the bug: a filter that reports how much
it removed can still be removing the wrong things, so check the retained
distribution. A uniform 100% across every model is not a result, it is a
symptom.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@usjoh
usjoh merged commit ef1c729 into main Aug 16, 2026
2 of 3 checks passed
@usjoh
usjoh deleted the docs/model-notes-ohalloran-false-failure branch August 16, 2026 13:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant