fix(hub): stream large whole-package zip to R2 instead of buffering (502 OOM)

[github-actions]

The email-agent v0.2.1 release died at the "Assemble + publish the whole-package zip" step with 502 Bad Gateway from Cloudflare, which skipped the npm publish and website redeploy that run after it — leaving the release half-done (per-platform binaries on the Hub, no npm package, no catalog update). The Agent Hub worker buffered the entire upload in memory (request.formData() → arrayBuffer()), so the 177 MB whole-package zip (added in #1843) needed ~350 MB working set, far past Cloudflare's hard 128 MB per-isolate limit — the isolate was OOM-killed before the worker's own size guard could even run. After this change the large zip is streamed straight to R2 with no buffering on either side, so it stays well under the memory ceiling and the release completes end-to-end.

Closes #1848

The existing multipart path for the ~40 MB per-platform binaries (which stay under the limit and carry README/CHANGELOG) is left byte-identical; only a new application/octet-stream branch is added. R2 verifies the client-supplied SHA-256 as it streams, so integrity is preserved and a mismatch fails loudly (422, nothing committed) — no silent fallback.

Test plan

• cd workers/agent-hub && npm run typecheck && npm test — 74 pass (10 new streaming-path tests incl. SHA mismatch → 422, up-front Content-Length 413, auth/immutability/traversal still enforced)
• python -m pytest hub/agents/python/email/tests/test_publish_to_r2.py -q — 4 pass (zip→octet-stream headers, binary→multipart, chunked SHA, fail-loud integrity)
• Maintainer prod gate: wrangler dev locally, stream a ~180 MB file via the new octet-stream path → expect 201, object stored, no OOM
• Maintainer prod gate: re-run the email-agent release workflow (CI holds the publish token) and confirm the zip step, npm publish, and website redeploy all complete

⚠️ Needs manual validation — the automated checks here confirm no regression in the worker (vitest) or client (pytest) logic, but can't exercise the real Cloudflare isolate-memory behavior or a live R2 streaming PUT. A maintainer should run the two prod-gate steps above before merging.

🔍 Technical details

Root cause. workers/agent-hub/src/publish.ts did request.formData() then new Uint8Array(await artifactFile.arrayBuffer()) — buffering the whole body. The 250 MiB MAX_ARTIFACT_BYTES guard runs after buffering, so it never fired; the OOM happened during buffering. The 502 (not 413) confirms the body cleared Cloudflare's edge and the isolate died mid-request.

Worker (src/publish.ts). handlePublish now dispatches on Content-Type:

• multipart/form-data → handleMultipartPublish (the original logic, unchanged — server-computed SHA-256, README/CHANGELOG parts).
• application/octet-stream → new handleStreamingPublish: metadata from x-gaia-manifest-b64 / x-gaia-artifact-filename / x-gaia-artifact-sha256 / optional x-gaia-package-files-b64 headers; size limit enforced up front from Content-Length (over → 413); body stored via env.BUCKET.put(key, request.body, { sha256 }) so R2 verifies the client digest as it streams (mismatch caught → 422 integrity_mismatch, nothing stored). Auth, manifest scope, filename validation, and per-filename immutability all still run (immutability head() checked before consuming the body so an idempotent re-run 409s without uploading).
• The per-version record writes + catalog rebuild + response are factored into a shared finalizeAndRespond both paths call, so the catalog contract is identical regardless of encoding.

Client (packaging/publish_to_r2.py). publish_one branches on .zip: per-platform binaries keep using _post_multipart; the whole-package zip uses _post_streaming, which passes the open file handle as data= (requests streams it and sets Content-Length) with the x-gaia-* metadata headers. _sha256_file now hashes in 1 MiB chunks, so the client never fully buffers the file either. The 201/409 integrity verification is unchanged and shared by both paths.

Docs. Worker README.md gains an "Upload encodings" section documenting both paths and the x-gaia-* header contract; the packaging README.md line for publish_to_r2.py is corrected (server-side checksum now describes only the multipart path).

Tests. test/publish.test.ts adds a streaming suite via a new streamingPublishRequest helper (Web-globals only — crypto.subtle/btoa — to satisfy the worker tsconfig); fake-r2.ts now consumes a ReadableStream put value and already honored options.sha256 for the mismatch test. test_publish_to_r2.py follows the existing importlib-by-path pattern used by test_gen_binaries_lock.py.

[itomek]

@claude it looks like PR 1849 already took care of this, maybe we close this #1850 PR?

[github-actions]

You're right — #1849 and this PR are duplicates. Both fix #1848 the same way (route the whole-package zip to a raw application/octet-stream body streamed straight to BUCKET.put(..., { sha256 }) so the isolate never buffers it, multipart path untouched), and both are still open/unmerged.

Worth noting before you close: neither has actually landed the fix yet, so make sure the one you keep gets merged. #1849 is the slightly fuller version — it carries a couple more streaming tests and a "follow-ups" note (chunked idempotency re-check, presigned direct-to-R2 escape hatch). If you go with #1849, closing this autofix PR is the clean move.

I can't close PRs myself — a maintainer will need to do that.

🔍 Technical details

Both diffs touch the same surface:

File	#1850	#1849
workers/agent-hub/src/publish.ts	+266/-20	+269/-20
workers/agent-hub/test/publish.test.ts	+182 (10 streaming tests)	+270 (12 streaming tests)
workers/agent-hub/test/fake-r2.ts	+61	+61
hub/agents/python/email/packaging/publish_to_r2.py	+124/-29	+140/-14
client tests	hub/agents/python/email/tests/test_publish_to_r2.py	tests/unit/test_publish_to_r2_stream.py

Minor divergence worth a glance when you pick: this PR returns 422 integrity_mismatch on a SHA mismatch; #1849 returns 400 integrity_check_failed. Same fail-loud behavior, different status code — whichever lands sets the contract.

[itomek]

Closing based on comments above.