Skip to content

ci(hub): deploy the Agent Hub Worker before publishing to it - #2991

Merged
kovtcharov-amd merged 2 commits into
mainfrom
ci/deploy-agent-hub-worker-on-publish
Aug 18, 2026
Merged

ci(hub): deploy the Agent Hub Worker before publishing to it#2991
kovtcharov-amd merged 2 commits into
mainfrom
ci/deploy-agent-hub-worker-on-publish

Conversation

@kovtcharov-amd

Copy link
Copy Markdown
Collaborator

Why this matters

Publishing hub components has been broken since July, and the error blamed the wrong thing. Every run died on invalid_manifest: language "go" is not supported. Use one of: cpp, python — but that list is rendered from VALID_LANGUAGES, which has included go and typescript since #2530, the commit that added terminal-hub and agent-ui as R2 packages in the first place. The message quoted code that no longer exists, because the deployed Worker was months behind the manifests it was validating. Nothing could have caught it: agent_hub_worker_ci.yml only type-checks and tests, so the only deploy path is a human running wrangler deploy, and the validator ages silently until a release trips over it. That happened three times before someone disabled the workflow.

The publish now deploys the Worker before uploading to it, so the validator always matches the manifests it's judging.

Test plan

  • Add CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID to the agent-publish environment (see the table added to workers/agent-hub/README.md)
  • Dispatch with dry_run=truedeploy-worker is skipped, both publish jobs still run every build and validation step (a skipped dependency must not skip them)
  • Dispatch with dry_run=false — approve at the reviewer gate, confirm deploy-worker runs first and the two publish jobs then succeed where they previously failed on invalid_manifest
  • Remove one secret and dispatch — the job fails naming both secrets and where to get them, rather than publishing to a stale Worker
🔍 Design notes

Why deploy on publish rather than on push-to-main. Push-to-main would deploy production with no human in the loop. This job sits in the agent-publish environment, so it inherits the existing reviewer gate and lands in the same approval as the upload it has to match. It also deploys exactly the ref being published, which is the property that actually matters.

Dry-run safety. GitHub skips a job whose dependency was skipped, so gating the publishes on deploy-worker would have silently killed dry runs. Both publish jobs use always() && version == success && (deploy-worker == success || skipped) — a skipped deploy is admitted, a failed one is not.

Post-deploy health check. hub.amd-gaia.ai/health, retried five times. The entire failure mode was that nothing ever asserted what was live, so the job now proves it before anything immutable is written.

Type-check before deploy. A dispatch can run from any ref, and shipping a Worker that fails tsc would break publishing for every agent, not just this release.

Publishing hub components has been broken since July and the error blamed the
wrong thing. Every run died on:

    invalid_manifest: language "go" is not supported. Use one of: cpp, python

but that list is rendered from VALID_LANGUAGES, which has included `go` and
`typescript` since #2530 — the commit that added terminal-hub and agent-ui as
R2 packages in the first place. The message quoted code that no longer exists,
because the deployed Worker was months behind the manifests it was validating.

There was no way for it not to drift. agent_hub_worker_ci.yml only type-checks
and runs vitest; the sole deploy path is a human running `wrangler deploy`. So
the validator silently ages until a release trips over it, which is what
happened three times before someone disabled the workflow.

release_components.yml now deploys the Worker before it uploads anything, and
both publish jobs gate on that. Deploying here rather than on push-to-main is
deliberate: the job sits in the agent-publish environment, so a production
Worker deploy still needs a reviewer and lands in the same approval as the
upload it has to match.

Dry runs skip the deploy and still run every build and validation step — the
publish jobs admit a *skipped* dependency but never a failed one.

Needs CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID on the agent-publish
environment; the job fails with both names and where to get them rather than
publishing to a stale Worker. A post-deploy health check on hub.amd-gaia.ai
asserts what is actually live, since the whole failure mode was that nobody
ever checked.
@github-actions github-actions Bot added the devops DevOps/infrastructure changes label Aug 18, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Verdict: Request changes — the idea is right and two of the fixes are one-liners.

Releases were failing because the live Hub service was older than the manifests it was being asked to validate, and nothing in CI ever redeployed it. Deploying it as an approval-gated step immediately before the upload is the right shape, and the reasoning is written down well. Three things to settle before merge:

  1. Cancelling a release no longer stops it. The two publish jobs now carry a condition that GitHub deliberately treats as "run even if the run was cancelled." Hitting Cancel mid-release can still upload — and Hub storage paths are permanent, so that upload can't be taken back. One-word fix.
  2. The new "did the deploy land?" check can't tell a new service from an old one. The probe it calls answers with the same fixed "ok" whether or not the deploy took effect, so the stale-service failure this PR exists to catch would still sail through it. Either make the probe report which build is live and assert on that, or drop the claim from the comment.
  3. Two new Cloudflare credentials have to exist on the release environment before the next tag. If they aren't there, every component publish now stops dead at a job that didn't exist yesterday. Worth confirming they're provisioned at merge time rather than discovering it during a release.

Smaller but worth folding in: the deploy runs a type-check yet skips the service's own test suite, which already exists and already runs on PRs.

Real-world evidence

N/A — CI workflow + README only; no runnable GAIA surface, so the verdict rests on static review. One caveat the author should know: the new job is skipped on a dry run, so the dispatch dry run can't rehearse it — the first time this code ever executes will be a live release against production. That's the one place I'd want a rehearsal path before merging (see the technical notes for a cheap one).

🔍 Technical details

🟡 Important

1. always() publishes through a cancelled run (release_components.yml:192-195 and :305-308)

always() is true even when the run is cancelled — GitHub's own docs recommend !cancelled() instead for exactly this reason. Before this PR neither publish job had an if:, so cancelling the run stopped them; now a cancel can still let terminal-hub / agent-ui start and POST to /publish. Given R2 paths are immutable, that's the one failure mode worth being conservative about. Same edit in both places:

    if: |
      !cancelled()
      && needs.version.result == 'success'
      && (needs.deploy-worker.result == 'success' || needs.deploy-worker.result == 'skipped')

2. The verification step can't detect the failure it's written to detect (release_components.yml:169-184)

The comment says it proves "the deployed Worker is the one we just built, not a cached edge version," but /health is json({ status: "ok" }) with no version, commit, or build field (workers/agent-hub/src/index.ts:41-43). The stale Worker that produced language "go" is not supported would have answered this probe 200 the whole time. To actually assert it, stamp the build and echo it — npx wrangler deploy --var WORKER_BUILD:"${GITHUB_SHA}", add WORKER_BUILD to Env in types.ts, return it from /health, then grep the response for the SHA. If that's more than you want here, at least soften the comment so the next reader doesn't trust a check that only proves something is listening.

3. Deploy gates on tsc but not on the Worker's test suite (release_components.yml:151-153)

The step's own rationale — a dispatch can run from any ref, and a broken Worker breaks publishing for every agent — argues harder for the vitest suite than for the type-check. agent_hub_worker_ci.yml:49-53 runs both, and npm run typecheck covers tsconfig.test.json too, which npx tsc --noEmit misses:

      - name: Type-check
        working-directory: workers/agent-hub
        run: npm run typecheck

      - name: Unit tests (vitest)
        working-directory: workers/agent-hub
        run: npm test

4. New hard dependency on secrets that don't exist yet

CLOUDFLARE_API_TOKEN / CLOUDFLARE_ACCOUNT_ID appear nowhere else in the repo, so they're new to the agent-publish environment. Because terminal-hub and agent-ui now require deploy-worker to be success or skipped, a missing secret converts "publish works" into "publish is blocked" on the next v* tag. The failure is loud and documented, which is right — this is a merge-ordering note, not a code change: confirm both secrets (and that the token covers R2 for the bucket binding) are on the environment before this lands.

🟢 Minor

5. The health check targets a different host than the publish (:176 vs :286)

wrangler.toml:20-24 says CI publishes through workers.dev because the WAF on the proxied custom domain blocks large multipart uploads — so GAIA_HUB_PUBLISH_URL likely isn't hub.amd-gaia.ai. Curling the same resolved base URL the publish steps use would verify the endpoint you're about to upload to, and would stop an unrelated custom-domain hiccup from failing an otherwise fine release.

6. The new job can never be rehearsed (:134)

if: needs.version.outputs.dry_run == 'false' means a dry-run dispatch skips it entirely, so its first execution is a production deploy. package.json already has deploy:dry-run (wrangler deploy --dry-run --outdir dist) — running that branch on a dry run would at least prove the bundle builds and the config resolves.

7. Drifts from the sibling Worker workflow (:140)

actions/setup-node@v6 here vs @v7 in agent_hub_worker_ci.yml:40, and no cache: npm / cache-dependency-path despite package-lock.json being present.

8. agent_hub_worker_ci.yml:8 is now stale — "Deploys stay manual; this only tests" stopped being true with this PR. Per CLAUDE.md's rule about updating every doc that describes a behavior, that header should move with it.

9. The guard checks one of the two secrets it names (:162) — the error message tells the maintainer to add CLOUDFLARE_ACCOUNT_ID too, but only CLOUDFLARE_API_TOKEN is validated, so a half-configured environment fails later inside wrangler with a less useful message.

Strengths

  • The block comments explain the incident, not the mechanism — the language "go" is not supported quote makes the whole job self-justifying to the next maintainer, and the note on why this deploys here rather than on push-to-main pre-empts the obvious review question.
  • Error messages follow the Fail-Loudly contract properly: what failed, what to do about it, and where to read more.
  • workers/agent-hub/README.md is updated in the same change with a provisioning table for the new secrets, instead of leaving the secret requirement discoverable only by reading YAML.
  • The success' || 'skipped' handling on the dependency, and the comment explaining why skipped has to be admitted, is the correct and commonly-missed detail.

…ehearsal

Five fixes from review, one of which was a genuine hole.

`always()` on the publish jobs survives a cancelled run, so hitting Cancel
mid-release could still POST to /publish — into an immutable path. Now
`!cancelled()`, which keeps the skipped-dependency handling while restoring
Cancel as a stop.

The post-deploy check could not detect the failure it was written for. /health
returned a fixed `{"status":"ok"}`, which the months-stale Worker would have
answered exactly the same way. Deploys now stamp the commit into WORKER_BUILD,
/health reports it, and the workflow asserts the live build equals the one it
just pushed. It also curls the same base URL the publish steps resolve rather
than the custom domain, because CI publishes through workers.dev — the WAF on
hub.amd-gaia.ai blocks large multipart uploads — so that is the origin whose
freshness actually matters.

The deploy job could never be rehearsed: it is skipped on a dry run, so its
first execution would have been a live release. Split out a worker-check job
that runs on every dispatch, unapproved, and does typecheck + the vitest suite
+ `deploy:dry-run`. That also answers the review's point that gating on `tsc`
while skipping the suite guarding the /publish contract was backwards; the
deploy job now inherits a checked bundle instead of re-deriving a weaker one.

Smaller: validate both Cloudflare secrets rather than one and let wrangler fail
obscurely on the other; setup-node v7 with npm caching to match the sibling
workflow; and correct agent_hub_worker_ci.yml's header, which still claimed
deploys were manual.
@kovtcharov-amd

Copy link
Copy Markdown
Collaborator Author

All five code points fixed in 39cb4f48. Two were sharper than I'd credited — the cancel hole is a real one, and the verification step genuinely could not see the failure it was written for.

# Finding Resolution
1 always() publishes through a cancelled run Fixed — !cancelled()
2 Health check can't detect a stale Worker Fixed — build stamp, asserted
3 Gates on tsc, skips the suite Fixed — npm run typecheck + npm test
4 Secrets don't exist yet Merge-ordering; both now validated up front
5 Checks a different host than the publish Fixed — same resolved base URL
6 Job can never be rehearsed Fixed — new worker-check job
7 Drifts from the sibling workflow Fixed — setup-node@v7 + npm cache
8 agent_hub_worker_ci.yml header stale Fixed
9 Validates one of two secrets Fixed
🔍 Technical details

1 — cancel safety. You're right that this was a regression I introduced: before the PR neither publish job had an if:, so Cancel stopped them. !cancelled() keeps the skipped-dependency handling and restores Cancel as a stop, which matters most here precisely because the paths are immutable.

2 — the check now checks something. Deploys run wrangler deploy --var WORKER_BUILD:"${GITHUB_SHA}"; WORKER_BUILD is on Env; /health returns {status, build}; the step polls until build == GITHUB_SHA and fails loudly otherwise. Two tests pin it — one that the stamp is echoed, one that it degrades to "unknown" rather than vanishing when unset, so a hand-run deploy is visibly distinguishable from a CI one.

5 — right origin. Now resolves GAIA_HUB_PUBLISH_URL || GAIA_HUB_BASE_URL || hub.amd-gaia.ai, the same expression the publish steps use. Good catch: verifying the custom domain while uploading through workers.dev would have proved the wrong thing, and could have failed a fine release on an unrelated WAF hiccup.

3 + 6 — one job solves both. New worker-check: no environment, so it runs unapproved on every dispatch including dry runs, and does typecheck + vitest + deploy:dry-run. The deploy job depends on it, so it inherits a bundle that has already been built and tested rather than re-deriving a weaker check. Both publish jobs also require worker-check == success, otherwise a failed check would leave deploy-worker skipped and the publishes would happily proceed.

Verified locally: typecheck clean on both tsconfigs, 204 tests pass, deploy:dry-run bundles and resolves the R2 binding.

4 — still needs you. CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID must be on agent-publish before the next v* tag, and the token needs R2 coverage for the bucket binding. Both are validated up front now, so a half-configured environment fails with both names rather than dying inside wrangler.

Worth flagging something I found while testing this, since it affects merge order: terminal-hub has never actually reached the hub. hub.amd-gaia.ai/index.json lists only email, and agents/terminal-hub/0.23.0/* is 404 — consistent with this workflow never having completed. The flagship agent's release verifies its tui lane is fetchable at the public origin, so gaia cannot publish until terminal-hub does. That makes this PR the head of the chain rather than a cleanup.

@kovtcharov-amd
kovtcharov-amd added this pull request to the merge queue Aug 18, 2026
Merged via the queue into main with commit 9c89cb4 Aug 18, 2026
26 of 28 checks passed
@kovtcharov-amd
kovtcharov-amd deleted the ci/deploy-agent-hub-worker-on-publish branch August 18, 2026 16:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

devops DevOps/infrastructure changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant