Commit 5cfa8af
authored
fix(ci): accept tagged no-traffic Cloud Run candidate with omitted percent (#10267)
## Summary
Candidate-mode `verify_backend_release_vector.py` rejected a valid
no-traffic
Cloud Run candidate because it read an omitted/`null` traffic `percent`
(a
tagged, zero-allocation candidate) as positive pre-promotion traffic.
This
blocked the automatic dev deploy at the "Accept no-traffic Cloud Run
candidate"
gate for all four backend services. One-line-classification fix plus the
regression test that reproduces the exact failed-development shape.
Failure-Class: FC-nonrecoverable-promotion
## Failure mechanism
Cloud Run serializes a tagged, no-allocation candidate revision with
`percent: null` in `status.traffic` — the revision exists and is
`Ready`, is
reachable via its tag for acceptance probing, but carries 0% of
default-domain
traffic until promotion (the prior serving revision keeps 100%). This is
the
intended pre-promotion state the candidate gate is supposed to *accept*.
The verifier's candidate check was:
```python
if not require_serving_traffic and any(_mapping(entry).get('percent') != 0 for entry in expected_traffic):
errors.append('... expected revision carries traffic before promotion')
```
For an omitted/null percent, `_mapping(entry).get('percent')` is `None`,
and
`None != 0` is `True`, so a Ready zero-allocation candidate was falsely
flagged
as "carries traffic before promotion". Sibling scripts already handle
this
correctly (`cloud_run_traffic_snapshot.py` requires `isinstance(percent,
int)`;
`deploy_status_report.py`/`repair_cloud_run_traffic.py` use `int(... or
0)`),
so the verifier was the lone outlier — the fix is localized to it.
Exact failing run (development, SHA `2904ea7`):
https://github.com/BasedHardware/omi/actions/runs/29891331283 — each of
the four
services printed `percent: null` for the candidate entry and `percent:
100` for
the prior serving revision, then failed with "expected revision carries
traffic
before promotion".
## Root cause confirmed
Cloud Run allocation semantics, distinguished before patching:
- Explicit numeric `percent` → that exact allocation.
- Omitted/`null` `percent` on a `revisionName`-pinned target → 0%
(tagged-only,
no default-domain traffic). This is the candidate shape.
- Omitted `percent` on a `latestRevision: true` target → receives the
remainder
(may be > 0). These never reach this check: `expected_traffic` is
filtered by
`revisionName == expected_revision`, and `latestRevision` targets carry
no
literal `revisionName`, so they cannot match the candidate name.
So treating omitted percent as 0% **only for revision-pinned candidate
entries**
is semantically correct — not a blanket `int(... or 0)` coercion.
## Change
`backend/scripts/verify_backend_release_vector.py`
- New `_effective_candidate_traffic_percent(percent)`: `None` → `0.0`; a
real
non-negative number → its allocation; bool/string/negative → `None`
(ambiguous, must not be trusted as zero).
- Candidate-mode loop now: reject only a positive allocation ("carries
traffic
before promotion"); reject ambiguous shapes with a distinct "candidate
traffic
allocation is ambiguous" error; accept `None`/`0`.
`backend/tests/unit/test_verify_backend_release_vector.py`
-
`test_candidate_evaluation_accepts_a_tagged_candidate_with_omitted_percent`:
RED→GREEN regression reproducing run 29891331283 (prior 100%, candidate
`percent: None`, Ready=True) → candidate passes; serving mode still
rejects it
until promoted to 100%.
-
`test_candidate_evaluation_rejects_positive_or_ambiguous_candidate_traffic`:
positive (5, 100) → "carries traffic before promotion"; ambiguous
(`"0"`,
`True`, `-1`) → "ambiguous". Proves the safety guard stays
mutation-sensitive.
## TDD evidence
- RED (before fix): the omitted-percent acceptance test failed with the
exact
production error; the ambiguous cases mis-reported as "carries traffic".
- GREEN (after fix): `tests/unit/test_verify_backend_release_vector.py`
→ 48
passed; production boundary test → 3 passed;
`tests/unit/test_workflow_contracts.py` → 21 passed.
- Re-ran the verifier against a sanitized fixture equivalent to the
failed dev
shape → candidate PASS; flipped to serving mode → still fails until
100%.
## Safety invariants (unchanged)
- Serving/post-promotion check is untouched: it still requires the
expected
revision's traffic entry at exactly `percent == 100` and rejects
everything
else (`None`, mixed, ambiguous). The fix touches only the `not
require_serving_traffic` branch.
- Candidate readiness, latest-created-revision, immutable image,
timeout, and
`OMI_ENV_STAGE` checks are unchanged.
- No new dependency, no workflow redesign, no refactor.
## Rollback impact
Reverting restores the false-positive candidate rejection (auto-dev
stays
blocked on the same gate) but does not weaken any promotion/serving gate
— a
revert cannot let traffic move or weaken the final 100% vector check.
## Why production behavior is not weakened
Production serving verification always runs in serving mode
(`require_serving_traffic=True`), whose check is byte-for-byte
unchanged.
Candidate mode is used only for the pre-promotion acceptance gate, which
this
fix makes correctly recognize the intended no-traffic candidate. The
`FC-nonrecoverable-promotion` contract — preserve pre-promotion traffic
state
until the serving vector is proven by accepting an immutable no-traffic
candidate — is what this fix restores.
<!-- This is an auto-generated description by cubic. -->
<a
href="https://cubic.dev/pr/BasedHardware/omi/pull/10267?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->2 files changed
Lines changed: 83 additions & 2 deletions
File tree
- backend
- scripts
- tests/unit
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
271 | 271 | | |
272 | 272 | | |
273 | 273 | | |
274 | | - | |
275 | | - | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
276 | 281 | | |
277 | 282 | | |
278 | 283 | | |
| |||
432 | 437 | | |
433 | 438 | | |
434 | 439 | | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
| 454 | + | |
| 455 | + | |
| 456 | + | |
435 | 457 | | |
436 | 458 | | |
437 | 459 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
678 | 678 | | |
679 | 679 | | |
680 | 680 | | |
| 681 | + | |
| 682 | + | |
| 683 | + | |
| 684 | + | |
| 685 | + | |
| 686 | + | |
| 687 | + | |
| 688 | + | |
| 689 | + | |
| 690 | + | |
| 691 | + | |
| 692 | + | |
| 693 | + | |
| 694 | + | |
| 695 | + | |
| 696 | + | |
| 697 | + | |
| 698 | + | |
| 699 | + | |
| 700 | + | |
| 701 | + | |
| 702 | + | |
| 703 | + | |
| 704 | + | |
| 705 | + | |
| 706 | + | |
| 707 | + | |
| 708 | + | |
| 709 | + | |
| 710 | + | |
| 711 | + | |
| 712 | + | |
| 713 | + | |
| 714 | + | |
| 715 | + | |
| 716 | + | |
| 717 | + | |
| 718 | + | |
| 719 | + | |
| 720 | + | |
| 721 | + | |
| 722 | + | |
| 723 | + | |
| 724 | + | |
| 725 | + | |
| 726 | + | |
| 727 | + | |
| 728 | + | |
| 729 | + | |
| 730 | + | |
| 731 | + | |
| 732 | + | |
| 733 | + | |
| 734 | + | |
| 735 | + | |
| 736 | + | |
| 737 | + | |
| 738 | + | |
| 739 | + | |
681 | 740 | | |
682 | 741 | | |
683 | 742 | | |
| |||
0 commit comments