ci(hub): deploy the Agent Hub Worker before publishing to it - #2991
Conversation
Publishing hub components has been broken since July and the error blamed the
wrong thing. Every run died on:
invalid_manifest: language "go" is not supported. Use one of: cpp, python
but that list is rendered from VALID_LANGUAGES, which has included `go` and
`typescript` since #2530 — the commit that added terminal-hub and agent-ui as
R2 packages in the first place. The message quoted code that no longer exists,
because the deployed Worker was months behind the manifests it was validating.
There was no way for it not to drift. agent_hub_worker_ci.yml only type-checks
and runs vitest; the sole deploy path is a human running `wrangler deploy`. So
the validator silently ages until a release trips over it, which is what
happened three times before someone disabled the workflow.
release_components.yml now deploys the Worker before it uploads anything, and
both publish jobs gate on that. Deploying here rather than on push-to-main is
deliberate: the job sits in the agent-publish environment, so a production
Worker deploy still needs a reviewer and lands in the same approval as the
upload it has to match.
Dry runs skip the deploy and still run every build and validation step — the
publish jobs admit a *skipped* dependency but never a failed one.
Needs CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID on the agent-publish
environment; the job fails with both names and where to get them rather than
publishing to a stale Worker. A post-deploy health check on hub.amd-gaia.ai
asserts what is actually live, since the whole failure mode was that nobody
ever checked.
|
Verdict: Request changes — the idea is right and two of the fixes are one-liners. Releases were failing because the live Hub service was older than the manifests it was being asked to validate, and nothing in CI ever redeployed it. Deploying it as an approval-gated step immediately before the upload is the right shape, and the reasoning is written down well. Three things to settle before merge:
Smaller but worth folding in: the deploy runs a type-check yet skips the service's own test suite, which already exists and already runs on PRs. Real-world evidenceN/A — CI workflow + README only; no runnable GAIA surface, so the verdict rests on static review. One caveat the author should know: the new job is skipped on a dry run, so the dispatch dry run can't rehearse it — the first time this code ever executes will be a live release against production. That's the one place I'd want a rehearsal path before merging (see the technical notes for a cheap one). 🔍 Technical details🟡 Important1.
2. The verification step can't detect the failure it's written to detect ( The comment says it proves "the deployed Worker is the one we just built, not a cached edge version," but 3. Deploy gates on The step's own rationale — a dispatch can run from any ref, and a broken Worker breaks publishing for every agent — argues harder for the vitest suite than for the type-check. 4. New hard dependency on secrets that don't exist yet
🟢 Minor5. The health check targets a different host than the publish (
6. The new job can never be rehearsed (
7. Drifts from the sibling Worker workflow (
8. 9. The guard checks one of the two secrets it names ( Strengths
|
…ehearsal
Five fixes from review, one of which was a genuine hole.
`always()` on the publish jobs survives a cancelled run, so hitting Cancel
mid-release could still POST to /publish — into an immutable path. Now
`!cancelled()`, which keeps the skipped-dependency handling while restoring
Cancel as a stop.
The post-deploy check could not detect the failure it was written for. /health
returned a fixed `{"status":"ok"}`, which the months-stale Worker would have
answered exactly the same way. Deploys now stamp the commit into WORKER_BUILD,
/health reports it, and the workflow asserts the live build equals the one it
just pushed. It also curls the same base URL the publish steps resolve rather
than the custom domain, because CI publishes through workers.dev — the WAF on
hub.amd-gaia.ai blocks large multipart uploads — so that is the origin whose
freshness actually matters.
The deploy job could never be rehearsed: it is skipped on a dry run, so its
first execution would have been a live release. Split out a worker-check job
that runs on every dispatch, unapproved, and does typecheck + the vitest suite
+ `deploy:dry-run`. That also answers the review's point that gating on `tsc`
while skipping the suite guarding the /publish contract was backwards; the
deploy job now inherits a checked bundle instead of re-deriving a weaker one.
Smaller: validate both Cloudflare secrets rather than one and let wrangler fail
obscurely on the other; setup-node v7 with npm caching to match the sibling
workflow; and correct agent_hub_worker_ci.yml's header, which still claimed
deploys were manual.
|
All five code points fixed in
🔍 Technical details1 — cancel safety. You're right that this was a regression I introduced: before the PR neither publish job had an 2 — the check now checks something. Deploys run 5 — right origin. Now resolves 3 + 6 — one job solves both. New Verified locally: typecheck clean on both tsconfigs, 204 tests pass, 4 — still needs you. Worth flagging something I found while testing this, since it affects merge order: terminal-hub has never actually reached the hub. |
Why this matters
Publishing hub components has been broken since July, and the error blamed the wrong thing. Every run died on
invalid_manifest: language "go" is not supported. Use one of: cpp, python— but that list is rendered fromVALID_LANGUAGES, which has includedgoandtypescriptsince #2530, the commit that added terminal-hub and agent-ui as R2 packages in the first place. The message quoted code that no longer exists, because the deployed Worker was months behind the manifests it was validating. Nothing could have caught it:agent_hub_worker_ci.ymlonly type-checks and tests, so the only deploy path is a human runningwrangler deploy, and the validator ages silently until a release trips over it. That happened three times before someone disabled the workflow.The publish now deploys the Worker before uploading to it, so the validator always matches the manifests it's judging.
Test plan
CLOUDFLARE_API_TOKENandCLOUDFLARE_ACCOUNT_IDto the agent-publish environment (see the table added toworkers/agent-hub/README.md)dry_run=true—deploy-workeris skipped, both publish jobs still run every build and validation step (a skipped dependency must not skip them)dry_run=false— approve at the reviewer gate, confirmdeploy-workerruns first and the two publish jobs then succeed where they previously failed oninvalid_manifest🔍 Design notes
Why deploy on publish rather than on push-to-main. Push-to-main would deploy production with no human in the loop. This job sits in the
agent-publishenvironment, so it inherits the existing reviewer gate and lands in the same approval as the upload it has to match. It also deploys exactly the ref being published, which is the property that actually matters.Dry-run safety. GitHub skips a job whose dependency was skipped, so gating the publishes on
deploy-workerwould have silently killed dry runs. Both publish jobs usealways() && version == success && (deploy-worker == success || skipped)— a skipped deploy is admitted, a failed one is not.Post-deploy health check.
hub.amd-gaia.ai/health, retried five times. The entire failure mode was that nothing ever asserted what was live, so the job now proves it before anything immutable is written.Type-check before deploy. A dispatch can run from any ref, and shipping a Worker that fails
tscwould break publishing for every agent, not just this release.