Build AI check versions exclude max_retries and request_timeout_seconds, so evaluations that can differ share one version identity

[Sagar-024]

Current behavior

_check_version (src/hflow/build_ai_vlm_checks.py:652-685) derives the version identity stamped into every Build AI check run. The OpenAI-compatible branch includes endpoint, model, response_format, temperature, and max_tokens. It excludes max_retries — validated at :164-167, defaulted at :143, and live at :715, where it is passed into the OpenAI client and decides whether transient errors become retried requests or terminal failures. The hosted branch includes only the execution label and endpoint; it excludes request_timeout_seconds — validated at :190-195 and live at :626 as the HTTP timeout.

Controlled result

Two executions that differ only in max_retries, and two that differ only in request_timeout_seconds, run through the real _check_version:

=== OpenAI path: same version identity, different completeness ===
max_retries=0 -> build-ai-single-frame-v1-eabed1f2e257cc07
max_retries=5 -> build-ai-single-frame-v1-eabed1f2e257cc07
same version: True

=== hosted path: same version identity, different completeness ===
request_timeout_seconds=1.0  -> build-ai-single-frame-v1-ac6fa96b9b859bec
request_timeout_seconds=60.0 -> build-ai-single-frame-v1-ac6fa96b9b859bec
same version: True

Why this is observable semantics, not reliability noise

The distinction that decides this issue: sampling noise changes the value of an answer; retries and timeouts change which items produce answers at all.

Temperature, which is already in the contract, adds sampling noise to a response that exists either way. max_retries and request_timeout_seconds change the failure mode and the completeness of a corpus: with max_retries=0, one transient 500 turns into a failed check run with no result; with max_retries=5, the same request becomes a parsed prediction. A short timeout drops slow-but-valid responses that a long timeout includes. Two corpora evaluated under these different settings have systematically different result sets — different error rates, different episode coverage — and a buyer comparing them under one version identity is told they are comparable when they are not.

That is the rule stated on #365: if the effective measurement path changes observable semantics, version and cache/provenance identity must change with it. The contract's own inclusion of temperature and max_tokens already establishes that behavioral knobs are version-worthy; these two knobs meet the same standard.

What to build

Open call, not assumed:

1. Add the fields to the contract: max_retries in the OpenAI-compatible branch, request_timeout_seconds in the hosted branch. The precedent is internal (temperature, max_tokens), so this is a consistency fix rather than a design change.
2. Record the behavior as evidence instead: leave the contract alone and stamp attempts/timeouts as measurements on every check run, so the difference is visible in the record without moving identity.

The first matches the contract's own logic; the second matches a "retries are infrastructure" reading. What cannot stand is the current middle: knobs that change outcomes, invisible to both the identity and the record.

Definition of done

1. Every execution knob that changes which items produce results is either in the version contract or recorded per run as a measurement.
2. Two configurations differing only in the newly covered fields produce different version identities, pinned by a test through the real _check_version.
3. Configurations that are identical in every covered field keep their existing version, so unchanged methodology does not silently invalidate.
4. The hosted and OpenAI branches treat the rule symmetrically.
5. The demo above lands as a test.

Non-goals

• Changing the max_retries or request_timeout_seconds defaults
• Changing how temperature participates in the contract
• Restructuring step_version_from_contract or the fingerprint machinery

Validation

uv run ruff check --fix
uv run ruff format
uv run ty check
uv run pytest -q tests/test_build_ai_vlm_checks.py
uv run pytest -q

[Sagar-024]

Taking this if accepted. Same trust standard as #376/#389/#395, applied to the version identity of the eval family: a check version that can cover different result sets is not an identity. The demo above lands as a test.

[kstonekuan]

Accepted, labelled bug / advanced. Both collisions reproduce here, and I added a control so the machinery is not in question:

=== OpenAI path: differing only in max_retries ===
  max_retries=0   -> build-ai-single-frame-v1-338ae9c5b40ab295
  max_retries=5   -> build-ai-single-frame-v1-338ae9c5b40ab295      same version

=== hosted path: differing only in request_timeout_seconds ===
  request_timeout_seconds=1.0   -> build-ai-single-frame-v1-6d7879ea74099688
  request_timeout_seconds=60.0  -> build-ai-single-frame-v1-6d7879ea74099688   same version

=== control: a knob already in the contract ===
  max_tokens=32   -> build-ai-single-frame-v1-338ae9c5b40ab295
  max_tokens=64   -> build-ai-single-frame-v1-ef35fc167126dd4d      different

Direction: take option 1, add both fields to the contract. Your framing settles it, and it is the sharpest sentence anyone has written on this thread: sampling noise changes the value of an answer, retries and timeouts change which items produce answers at all. A knob that decides whether an episode yields a prediction or a failed check run has a stronger claim to version identity than temperature, which is already in the contract and only perturbs a result that exists either way.

max_retries in the OpenAI branch, request_timeout_seconds in the hosted branch, and DoD item 4 is the one to hold yourself to: the two branches must treat the rule the same way, because the asymmetry is how this got missed.

Option 2 is worth doing and is not an alternative. Stamping attempts and timeouts as per-run measurements is independently useful (a corpus where a third of runs consumed four retries is telling you something about the endpoint, not the data), but it does not fix the identity problem. Two configurations would still be indistinguishable at the version level while producing different result sets. If you want it, file it separately; it is a genuine enhancement rather than a fix.

One cost your DoD does not name. The version is stamped into every check run and participates in cache and dedupe identity, so adding contract fields re-mints every existing Build AI check version. Nothing is wrong with that, it is the correct consequence, but it means existing results become non-matching on the next run rather than being silently reused. DoD item 3 is the guard that makes it survivable, so keep it strict: a configuration identical in every covered field must keep its current version. That is a test worth writing before the change, so you can see it go red and then green.

Third instance of the version-identity family today, after #390's episode/v1 stamps and #403's success label, which is starting to look like a pattern worth a shared note somewhere rather than three separate discoveries.

Worth checking while you are in _check_version: the OpenAI branch's comment says the contract shape was kept deliberately so migrating to the explicit execution value would not invalidate identical results. Make sure whatever you add does not read as though that constraint still applies to the new fields, since it does not.

[Sagar-024]

Demo run against hflow at 8044159 (Python 3.12, WSL2 Ubuntu). The demo script is 60 lines against the public _check_version; happy to attach it as a gist or land it with the fix.