docs(model-notes): ohalloran-demonstrator was a manifest defect, not two model failures - #37
Merged
Merged
Conversation
…two model failures claude-opus-4.8 and claude-sonnet-5 each read 0% first-try on site-build, and that is the whole of their site-build record. Neither failed. Run ohalloran-demonstrator-20260816T022403Z-p82264 (meridian ms-20260815-1624): both opencode workers exited CLEANLY (worker_returncode=0) and their checks failed only on missing_expect_files. The manifest declared each deliverable at an absolute path inside the calling Meridian session's scratchpad, which the opencode Seatbelt wrapper denies — it confines writes to the task dir plus the per-run scratch. The models did the work and physically could not deliver it. The two codex lanes in the same run passed because they were never behind that profile, which is what makes the run read as "the cheap lane cannot cope". Files dated bullets under both model headings so the warning reaches whoever routes site-build work, plus a process-lessons section recording the diagnostic rule this confirms for the third time in this file: read worker_returncode before calling anything a model failure. Also corrects a claim this file made yesterday. The opus-5 entry said `models --attributable` "excludes [the OpenRouter outage rows] and shows 100%". It does show 100% — but not because it identified the outage. db_attempt_rows never SELECTs the failure_class column, so on the DB read path every row reaches the filter unclassified and the flag drops EVERY failure: 137 of 371 attempts, exactly the non-PASS count, after which all 33 model rows read 100% first-try. The column is populated correctly in both runs.jsonl and ringer.db; only the reader drops it, so no rebuild fixes it. The note now says not to cite an --attributable rate as evidence until that is fixed, including the 2026-07-28 figures elsewhere in this file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This branch was written while db_attempt_rows still dropped the failure_class column, so it told the reader not to cite an --attributable rate at all. PR #38 fixes the reader: the flag now drops 46 rows (2 engine-error + 44 unknown) instead of all 137 failures, matching the JSONL path. Rewrites the warning as history rather than standing advice, and narrows the "do not cite" to what it should always have been: any attributable figure read BEFORE that fix, including the 2026-07-28 ones elsewhere in this file. The opus-5 entry gets the same treatment. Its 100% over one completed task is now genuine — those two tasks are classed `unknown` and really are excluded — where before it was an artifact of every failure being dropped. Also records the lesson that outlives the bug: a filter that reports how much it removed can still be removing the wrong things, so check the retained distribution. A uniform 100% across every model is not a result, it is a symptom. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
claude-opus-4.8andclaude-sonnet-5each read 0% first-try on site-build, and that is the whole of their site-build record. Neither model failed.What the run actually was
ohalloran-demonstrator-20260816T022403Z-p82264(meridianms-20260815-1624), four lanes:Two opencode lanes fail, two codex lanes pass — a shape that reads instantly as "the cheap lane can't cope." It is nothing of the kind.
Both opencode workers exited cleanly (
worker_returncode=0). The checks failed only onmissing_expect_files. The manifest declared each deliverable at an absolute path inside the calling Meridian session's scratchpad, which the opencode Seatbelt wrapper denies — it confines writes to the task dir plus the per-run scratch. The models did the work and physically could not deliver it. The codex lanes passed because they were never behind that profile.Filed as dated bullets under both model headings, so the warning reaches whoever routes site-build work next, plus a process-lessons section recording the rule this confirms for the third time in this file: read
worker_returncodebefore calling anything a model failure.It also corrects a claim this file made yesterday
The opus-5 entry from #35 said
models --attributable"excludes them and shows 100%". It does show 100% — but not because it identified the OpenRouter outage.db_attempt_rowsneverSELECTs thefailure_classcolumn. On the DB read path — the default — every row reaches the filter unclassified,model_log_row_failure_classreturnsNone, andmodel_log_row_is_model_attributablerejects it. Measured here:The column is populated correctly in both
runs.jsonlandringer.db— only the reader drops it, so nodb rebuildfixes it.The note now says not to cite an
--attributablerate as evidence until that's fixed, including the 2026-07-28 figures elsewhere in this file, which predate the DB read path and should be re-derived rather than trusted.That reader bug is not fixed here — this PR is notes only.
Verification
runs.jsonl,ringer.db, and the recorded run state, not against the prosemodels --task-type site-build)🤖 Generated with Claude Code