fix(schema): do not invent max attempts - #258
Conversation
Leave max_attempts unset when a source does not report an attempt limit, while preserving explicitly observed values. Assisted-by: OpenAI Codex
|
CI was never run here either — the workflow was queued as This closes #236. The change is right: a record carrying Flagging the lane rather than the content: this edits |
What / source
Changes the optional generation_args.max_attempts schema default from 1 to null and regenerates eval_types.py. Sources that do not report an attempt limit no longer materialize a claim that one attempt was allowed. Explicitly observed Inspect values remain unchanged.
Fixes #236.
Review lane
Design agreed in: #236, including collaborator confirmation that this is worth doing.
Checklist
The adapter report, registry, and source-row checklist items are not applicable because this changes schema default serialization and drops no source records.
Decisions & coverage
Chose / instead of: keep the current version because the field remains optional and accepts the same integer values; only its non-required default annotation changes
Confidence: high
General: no
Chose / instead of: regenerate eval_types.py from the schema and run post_codegen, rather than hand-editing generated behavior
Confidence: high
General: yes
Chose / instead of: preserve source-provided Inspect max_attempts values while omitting unobserved values
Confidence: high
General: yes
Coverage: 2 lm-eval source tasks to 2 valid records, 0 dropped; 1 Inspect source run to 1 valid record, 0 dropped. The lm-eval outputs contain no max_attempts key; the Inspect fixture that explicitly reports 1 still emits 1.
Commands run:
AI assistance
OpenAI Codex assisted with implementation, tests, and validation. I reviewed the diff and the generated output.