Skip to content

Staged validation topologies can never ship: git-pusher counts only VALIDATION_RESULT #1018

Description

@EivMeyer

Recording this so it is not rediscovered the hard way. Not a bug on main
today
— nothing on main publishes STAGE_<n>_VALIDATION_RESULT. It becomes
one the moment a topology that validates in stages is used, which is what
eivind/topology-generator produces.

What happens

git-pusher is what commits, pushes and opens the PR in ship mode. It fires
only once every validator has approved, established by counting
VALIDATION_RESULT messages since the last IMPLEMENTATION_READY:

if (latestByValidator.size < validators.length) return false;

A staged topology publishes STAGE_<n>_VALIDATION_RESULT for every stage
except the last. Those validators never count, the check can never be
satisfied, and git-pusher stays idle forever.

Observed on a real two-stage cluster (violet-aether-70), both approving:

STAGE_1_VALIDATION_RESULT  <- toolchain-gate           ← not counted
VALIDATION_RESULT          <- cancel-behaviour-audit   ← counted

latestByValidator.size = 1  <  validators.length = 2   →  no ship

The change was written, every validator approved it, and it was abandoned
uncommitted in the worktree while the run reported retries exhausted. It
repeated across three clusters and roughly $15 before the cause was found.
There is no error anywhere — the failure is silent.

Where the fix belongs

Not in git-pusher. I tried that in #1017 and closed it: tolerating extra
topics there means matching validation topics loosely, and this repo has
QUICK_VALIDATION_RESULT and HEAVY_VALIDATION_RESULT as tiers of the same
check — a suffix match lets a cheap pre-check satisfy the ship gate, which is
strictly worse than not shipping. It also costs the indexed topic query.

The generator is what broke the existing contract, by hijacking the validation
topic to chain stages. It can satisfy that contract instead: have every stage's
validators publish VALIDATION_RESULT and carry the stage in the payload,
with stage N+1 triggering on that field rather than on a renamed topic.
Staging still works, git-pusher needs no change, and the gate stays exact.

If someone does patch git-pusher anyway

Match exactly — ^(?:STAGE_[0-9]+_)?VALIDATION_RESULT$ — never by suffix. And
note SHARED_TRIGGER_SCRIPT is a template literal, so \d is consumed before
the script compiles and silently becomes d; use [0-9].

The test from #1017 (tests/git-pusher-staged-validators.test.js) covers all of
this and is worth keeping whichever layer the fix lands in: it compiles the
trigger through vm the way the runtime does, and fails against the current
implementation.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions