feat(hub): publish oversized artifacts direct to R2, bypassing the Worker body cap - #3025
Conversation
…at all
The Agent UI installers have never been publishable. Every one is 106-135 MiB
and a Worker request body is capped by the Cloudflare plan — 100 MB on Free and
Pro — so they 413 at the edge before the Worker runs. MAX_ARTIFACT_BYTES was
never the constraint and raising it changes nothing; the HTML error body is
Cloudflare's, not ours. terminal-hub and email only work because they are small.
POST /publish now accepts an artifact either inline, as today, or by reference:
CI PUTs the bytes straight into R2 over the S3 API, which has no such cap, then
sends `artifact_ref_{filename,sha256,size,content_type}` instead of the file.
The Worker not seeing the bytes must not become "the publisher said so". Before
recording anything it heads the object and checks the size and the SHA-256
against what R2 itself stored at PUT time. R2 keeps a whole-object SHA-256 only
for single-part uploads, so an object without one is REFUSED rather than
accepted on trust — CI has to upload single-part with x-amz-checksum-sha256.
Immutability needed rethinking rather than reusing. Inline, the object's
presence is the record, so heading it is the right check. By reference the
object always exists by the time we are called, so that check would 409 against
the caller's own upload; the record is the agent manifest, which only lists
artifacts this endpoint accepted.
10 tests cover the new path, including the two that matter: a hash that does not
match the stored bytes, and an object R2 could not checksum. Both were confirmed
to fail when the verifier is mutated to trust the caller's claim.
… the Worker The publisher now routes any artifact at or above 90 MiB into R2 over the S3 API and publishes it by reference, rather than streaming it through a Worker request that Cloudflare rejects at 100 MB. The threshold sits below the real cap on purpose: an artifact growing into the limit should change lanes before it starts 413ing mid-release, not after. put_object, never upload_file. R2 records a whole-object SHA-256 only for single-part uploads, and the Worker refuses to publish an object it cannot verify — upload_file switches to multipart above its own threshold and drops the checksum, which would turn a green release into an unverifiable one. Shared by every hub publisher, so email and terminal-hub inherit it; both stay on the inline path today because they are small, and a test pins that so this does not quietly change how the working agents publish. Needs R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY and CLOUDFLARE_ACCOUNT_ID. These are R2 S3 credentials, NOT the CLOUDFLARE_API_TOKEN that deploys the Worker. Missing any of them fails loudly naming all three rather than falling back to the Worker path, which cannot work for these sizes.
github-actions
Bot
added
devops
Verdict: Request changesThis makes the Agent UI installers publishable at all — they're 106–135 MiB and Cloudflare rejects them at the edge before the Worker runs — by uploading them straight to R2 and having the Worker verify the stored object instead of the bytes on the wire. The verification design is solid: the hub checks size and hash against what R2 itself recorded, and refuses anything it can't verify rather than trusting the publisher. One thing needs fixing before merge:
Worth addressing in the same pass: the size ceiling that guards the normal publish path isn't applied to the new one, so an artifact of any size can be recorded. The README section added here still describes that ceiling as if it covered everything. Real-world evidenceThe evidence bundle exercised both lanes against a live Worker (
Marked not exercised: the boto3 🔍 Technical details🔴 CriticalThe R2 PUT precedes the immutability check, so a re-publish overwrites published bytes and reports success (
The inline lane is immune (the Worker rejects before Fix: skip the upload when the filename is already published, which also restores the 409 mismatch check's meaning: (A HEAD against the public download route is the smallest change; reading the catalog manifest works too. Either way the existing 409 branch then compares stored vs. local bytes and fails loudly on divergence.) 🟡 ImportantBy-reference publishes bypass
🟢 MinorNo preflight for the three new R2 secrets ( The sibling publisher didn't get the change (
Strengths
|
…gaps The critical one was mine and it failed silently, which is the worst shape. The publisher PUT to R2 before asking the Worker to record the artifact, but the by-reference immutability guard keys on the agent manifest and so fires only after that PUT. Re-publishing different bytes for a filename already published therefore replaced the stored object, left the catalog holding the OLD hash and size, and then the 409 handler re-downloaded the bytes it had just written — agreed with itself, printed "already published with identical bytes", and exited 0. Install-time verification would have been broken for every user with nothing failing anywhere. The publisher now checks before uploading, which also restores the meaning of that 409. The by-reference lane also skipped MAX_ARTIFACT_BYTES, so the ceiling the README calls the artifact size cap did not cover the one lane that exists for the largest artifacts. Enforced there too; the 250 MiB default clears the 135 MiB installers. agent-ui checked its credentials after waiting up to an hour for release assets and pulling ~400 MB. Moved ahead of the wait and extended to the R2 pair, so a missing secret fails in seconds. The flagship publisher is a separate script without this lane, and its frozen sidecar is the other artifact plausibly heading past 100 MB. Rather than duplicate the implementation it now refuses loudly at the same threshold and names what to port, instead of letting a release discover an HTML 413.
|
All four fixed in You're right about the ordering. Uploading before the Worker's manifest-keyed immutability check meant a re-publish with different bytes replaced the stored object, left the catalog holding the old hash and size, and then the 409 handler re-downloaded the bytes it had just written — agreed with itself, printed "already published with identical bytes", and exited 0. Install-time verification broken for every user, with nothing failing anywhere. Took your fix: HEAD the download route first, skip the upload if it's there, which also restores the meaning of that 409.
🔍 DetailsTwo tests for the 🔴, and I verified they fail against the old ordering: one that an already-published object is never overwritten, one that a first publish still uploads (so the guard doesn't break the normal path). Preflight — rather than just extending the step in place, I moved it ahead of "Wait for the installer assets". It was sitting after a 60-minute wait and a ~400 MB download, so a missing secret cost the whole job; now it fails in seconds. It covers the R2 pair as well as Sibling publisher — porting would have meant a second copy of the upload logic in a 347-line near-duplicate. Instead it now refuses at the same threshold with a message naming exactly what to port ( Ceiling — 215 Worker tests and 9 Python tests pass; typecheck and lint clean. One thing this surfaced that isn't in the diff. The secrets are now on |
|
Verdict: Approve This cleanly solves the real shipping blocker (Agent UI installers at 106-135 MiB exceed Cloudflare's 100 MB Free/Pro request-body cap) without relaxing integrity: the Worker refuses any by-reference artifact whose R2-recorded SHA-256 or size doesn't match the claim, and objects uploaded without a checksum are rejected rather than trusted. The pre-flight HEAD check before the R2 PUT, the manifest-keyed immutability for by-reference vs. bucket-keyed for inline, and the early credential gate in CI are all sound. Test coverage on both the Python and TypeScript sides is thorough. Two 🟢 nits before you ship:
🔍 Technical detailsNit 1 — bucket = os.environ.get("R2_BUCKET", "gaia-hub")The Nit 2 — No correctness, security, or architecture issues found. |
|
Verdict: Approve with suggestions The core design is sound: single-part PUT to get an R2-recorded checksum, base64 encoding (not hex) for the One finding worth addressing before the next release cycle: 🟡 The 409 handler downloads the full artifact to verify its sha256, with a 120-second timeout. A re-run of a release job takes the by-reference path for installers ≥ 90 MiB: HEAD returns 200 (already in R2), the R2 upload is skipped, the POST to the Worker still fires, the Worker sees the filename in the manifest and returns 409, and Two small nits: 🟢 🟢 🔍 Technical details🟡 409-handler timeout on re-runs of large artifacts
The path that reaches this code on a re-run:
Simplest fix: when elif resp.status_code == 409:
if by_reference:
print(f"[publish] OK 409 — {filename} already published (by-reference, idempotent no-op).", flush=True)
else:
remote_sha = _download_sha256(base_url, agent_id, version, filename)
if remote_sha != local_sha:
raise SystemExit(...)
print(f"[publish] OK 409 — already published with identical bytes (idempotent no-op).", flush=True)Or stream-hash the response to stay under the timeout without buffering 135 MiB. 🟢 R2_BUCKET default — 🟢 Unwrapped boto3 exceptions — |
1 participant
Why this matters
The Agent UI installers have never been publishable to the hub, and the error blamed the wrong thing. Each one is 106–135 MiB, and Cloudflare caps a Worker request body at 100 MB on Free/Pro, so they are rejected with a
413by the edge before the Worker executes.MAX_ARTIFACT_BYTES(250 MiB) was never consulted — the HTML error body is Cloudflare's, not ours — so raising it would have changed nothing. email and terminal-hub only work because they are small (43.5 MiB and under).Artifacts at or above 90 MiB now go straight into the same R2 bucket over the S3 API, which has no such cap, and are published by reference: the POST carries the artifact's coordinates instead of its bytes.
Test plan
R2_ACCESS_KEY_ID/R2_SECRET_ACCESS_KEYrepo secrets (Cloudflare → R2 → Manage API Tokens → Object Read & Write ongaia-hub)cd workers/agent-hub && npm test— 214 pass, 10 of them newpytest hub/agents/email/python/tests/test_publish_to_r2_by_reference.py— 7 passrelease_components.ymlwithdry_run=falseand confirm agent-ui publishes where it previously 413'd🔍 How the integrity guarantee survives
The Worker no longer sees these bytes, so "the publisher said so" would be an easy accidental outcome. It isn't: before recording anything, the Worker heads the object and checks its size and SHA-256 against what R2 itself stored at PUT time.
R2 keeps a whole-object SHA-256 only for single-part uploads. An object without one is refused (
artifact_unverifiable) rather than trusted — so the uploader usesput_objectwithChecksumSHA256, neverupload_file, which would switch to multipart and silently drop the checksum. A test pins that call shape, because the failure mode is a green upload followed by a rejected publish, mid-release.Immutability needed rethinking rather than reusing. Inline, the object's presence in R2 is the record, so heading it is the right check. By reference the object always exists by the time the Worker is called, so that same check would 409 against the caller's own upload — the record is the agent manifest, which only lists artifacts this endpoint accepted.
Both new-path tests were verified non-vacuous by mutating the verifier to trust the caller's claim; the hash-mismatch and missing-checksum cases fail as they should.
New secrets are R2 S3 credentials, distinct from the
CLOUDFLARE_API_TOKENthat deploys the Worker. Missing any of the three fails loudly naming all of them — never a fallback to the Worker path, which cannot work at these sizes.Stacked on #2991 (merged) and independent of #3018.