Skip to content

Fix model invocation attribution - #106

Draft
ppkulk-oss wants to merge 2 commits into
NateBJones-Projects:mainfrom
ppkulk-oss:agent/model-invocation-attribution
Draft

Fix model invocation attribution#106
ppkulk-oss wants to merge 2 commits into
NateBJones-Projects:mainfrom
ppkulk-oss:agent/model-invocation-attribution

Conversation

@ppkulk-oss

@ppkulk-oss ppkulk-oss commented Aug 19, 2026

Copy link
Copy Markdown

What changed

  • Separate invocation evidence from performance evidence in the model scoreboard.
  • Preserve requested and harness-reported model identities, with alias-aware matching.
  • Register Fugu Ultra, current Grok models, and Claude Opus 5.
  • Recognize Grok's selectable grok-4.6 model and reported grok-4.6-build usage alias.
  • Attribute Claude from the primary assistant event's message.model, not the first modelUsage key (which can be an auxiliary Haiku call).
  • Capture effective reasoning effort across Codex, Grok, OpenCode, and Claude invocation styles.
  • Keep mismatched or unattributed rows unranked instead of silently crediting them.

Why

Grok runs without an explicit model were being stored with a blank identity and displayed as unattributed legacy rows. Fugu Ultra and Claude Opus 5 had no registry entries. Claude JSON can list an auxiliary Haiku model before the requested Opus model. Together, those gaps made the proven/probation table ambiguous because routing evidence and task-performance evidence were presented as one concept.

Impact

Future runs show whether an identity was harness-reported and matched, only configured/pinned, mismatched, legacy, or unattributed. Performance remains a separate proven/probation/unranked signal. Existing rows without defensible identity evidence are not rewritten.

The installed Claude lane pins claude-opus-5, explicit effort max, one attempt at the manifest level for the acceptance canary, and a four-turn CLI ceiling. It parses stream JSON assistant events, so auxiliary usage cannot be credited as the primary model.

Validation

  • python3 -m unittest discover -s tests -p 'test_*.py' — 259 tests passed.
  • Hardened Claude integration suite — 13/13 passed, including the fenced access-token-only OAuth-status check; no inference occurred in that suite.
  • Live bounded Claude canary — PASS on attempt 1 in 2 turns.
    • requested model: claude-opus-5
    • reported assistant model: claude-opus-5
    • Ringer invocation: reported match · max
    • performance: probation (one task; not yet enough evidence for proven)
  • Offline before/after Grok fixture:
    • before: unattributed / unranked
    • after: Grok 4.6 · high / reported match / proven
  • Fugu launcher offline preflight passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant