Recording this so it is not rediscovered the hard way. Not a bug on main
today — nothing on main publishes STAGE_<n>_VALIDATION_RESULT. It becomes
one the moment a topology that validates in stages is used, which is what
eivind/topology-generator produces.
What happens
git-pusher is what commits, pushes and opens the PR in ship mode. It fires
only once every validator has approved, established by counting
VALIDATION_RESULT messages since the last IMPLEMENTATION_READY:
if (latestByValidator.size < validators.length) return false;
A staged topology publishes STAGE_<n>_VALIDATION_RESULT for every stage
except the last. Those validators never count, the check can never be
satisfied, and git-pusher stays idle forever.
Observed on a real two-stage cluster (violet-aether-70), both approving:
STAGE_1_VALIDATION_RESULT <- toolchain-gate ← not counted
VALIDATION_RESULT <- cancel-behaviour-audit ← counted
latestByValidator.size = 1 < validators.length = 2 → no ship
The change was written, every validator approved it, and it was abandoned
uncommitted in the worktree while the run reported retries exhausted. It
repeated across three clusters and roughly $15 before the cause was found.
There is no error anywhere — the failure is silent.
Where the fix belongs
Not in git-pusher. I tried that in #1017 and closed it: tolerating extra
topics there means matching validation topics loosely, and this repo has
QUICK_VALIDATION_RESULT and HEAVY_VALIDATION_RESULT as tiers of the same
check — a suffix match lets a cheap pre-check satisfy the ship gate, which is
strictly worse than not shipping. It also costs the indexed topic query.
The generator is what broke the existing contract, by hijacking the validation
topic to chain stages. It can satisfy that contract instead: have every stage's
validators publish VALIDATION_RESULT and carry the stage in the payload,
with stage N+1 triggering on that field rather than on a renamed topic.
Staging still works, git-pusher needs no change, and the gate stays exact.
If someone does patch git-pusher anyway
Match exactly — ^(?:STAGE_[0-9]+_)?VALIDATION_RESULT$ — never by suffix. And
note SHARED_TRIGGER_SCRIPT is a template literal, so \d is consumed before
the script compiles and silently becomes d; use [0-9].
The test from #1017 (tests/git-pusher-staged-validators.test.js) covers all of
this and is worth keeping whichever layer the fix lands in: it compiles the
trigger through vm the way the runtime does, and fails against the current
implementation.
Recording this so it is not rediscovered the hard way. Not a bug on
maintoday — nothing on main publishes
STAGE_<n>_VALIDATION_RESULT. It becomesone the moment a topology that validates in stages is used, which is what
eivind/topology-generatorproduces.What happens
git-pusheris what commits, pushes and opens the PR in ship mode. It firesonly once every validator has approved, established by counting
VALIDATION_RESULTmessages since the lastIMPLEMENTATION_READY:A staged topology publishes
STAGE_<n>_VALIDATION_RESULTfor every stageexcept the last. Those validators never count, the check can never be
satisfied, and
git-pusherstaysidleforever.Observed on a real two-stage cluster (
violet-aether-70), both approving:The change was written, every validator approved it, and it was abandoned
uncommitted in the worktree while the run reported retries exhausted. It
repeated across three clusters and roughly $15 before the cause was found.
There is no error anywhere — the failure is silent.
Where the fix belongs
Not in
git-pusher. I tried that in #1017 and closed it: tolerating extratopics there means matching validation topics loosely, and this repo has
QUICK_VALIDATION_RESULTandHEAVY_VALIDATION_RESULTas tiers of the samecheck — a suffix match lets a cheap pre-check satisfy the ship gate, which is
strictly worse than not shipping. It also costs the indexed
topicquery.The generator is what broke the existing contract, by hijacking the validation
topic to chain stages. It can satisfy that contract instead: have every stage's
validators publish
VALIDATION_RESULTand carry the stage in the payload,with stage N+1 triggering on that field rather than on a renamed topic.
Staging still works,
git-pusherneeds no change, and the gate stays exact.If someone does patch git-pusher anyway
Match exactly —
^(?:STAGE_[0-9]+_)?VALIDATION_RESULT$— never by suffix. Andnote
SHARED_TRIGGER_SCRIPTis a template literal, so\dis consumed beforethe script compiles and silently becomes
d; use[0-9].The test from #1017 (
tests/git-pusher-staged-validators.test.js) covers all ofthis and is worth keeping whichever layer the fix lands in: it compiles the
trigger through
vmthe way the runtime does, and fails against the currentimplementation.