feat(email-release): publish frozen binaries to R2 via rclone (#1688)

## Why this matters
Before: the email release workflow targeted the agent-hub Worker `POST
/publish` API, and `@amd-gaia/agent-email` shipped with placeholder
hashes — not actually installable. After: one `agent-pkg-email-v*` tag
freezes all four platforms, uploads them by `rclone` to a **dedicated**
`gaia-hub` R2 bucket (served at its own subdomain `hub.amd-gaia.ai`),
regenerates `binaries.lock.json` with **real** SHA-256s, fetch-verifies
every platform via the real `fetch` CLI, then `npm publish`es via OIDC
trusted publishing. This is the last code piece before email v0.1.0 can
ship for real (#1685).

Distribution model: a plain public GET serves each binary at
`https://hub.amd-gaia.ai/agents/python/email/<version>/...`; the SHA-256
in `binaries.lock.json` is the integrity gate the `fetch` CLI enforces
(no server-side checksum). Atomic + idempotent: rclone copy is
content-addressed by the pre-computed hash, the post-upload fetch-verify
fails the release on any mismatch, and `npm publish` is skipped if the
version already exists.

The hub gets its own bucket + subdomain rather than riding on the
videos' `amd-gaia`/`assets.amd-gaia.ai` store — so its API token is
scoped to `gaia-hub` only and can never touch the docs assets.

`publish_to_r2.py` (the Worker `POST /publish` path) is **kept as a
testing helper** — its summary JSON is a valid `--meta` for the lock
generator.

> Maintainer prerequisites (one-time, before tagging): create the
`gaia-hub` R2 bucket and connect the `hub.amd-gaia.ai` custom domain to
it; create an Object-Read/Write token scoped to `gaia-hub` and set repo
secrets `R2_ACCESS_KEY_ID` / `R2_SECRET_ACCESS_KEY` / `R2_ACCOUNT_ID`;
optional var `HUB_BASE_URL` (defaults to `https://hub.amd-gaia.ai`);
register the npm trusted publisher for `@amd-gaia/agent-email` against
`release_agent_email.yml`. First release ships **unsigned** binaries
(#732 / #733 / #1650).

## Test plan
- [x] `gen_binaries_lock.py` functional round-trip: verbatim
`--base-url`, meta-driven win32 entry, placeholders preserved for absent
platforms, bad/non-http URL fails loudly
- [x] `python -m py_compile` + `black` + `isort` clean on changed Python
- [x] `bash -n upload_to_r2.sh` + `release_agent_email.yml` parses as
valid YAML
- [ ] Dry-run a real `agent-pkg-email-v0.1.0` once the `gaia-hub` bucket
+ repo secrets are set (CI freezes 4 platforms → R2 → lock →
fetch-verify → npm)

---------

Co-authored-by: Ovtcharov <kovtchar@amd.com>
Co-authored-by: Kalin Ovtcharov <kalin@extropolis.ai>

Author: 3 people

.github/workflows/release_agent_email.yml

@@ -5,31 +5,42 @@
 # (milestone #49, issue #1648). One namespaced tag does the WHOLE release:
 #
 #   1. build   — freeze the email REST sidecar into a native one-file binary on
-#                each of the 4 target platforms (no cross-compile), compute its
-#                SHA-256 + size, upload the binary + a meta sidecar.
-#   2. publish — POST /publish every binary to the Agent Hub R2 Worker (Bearer
-#                token from a repo secret), regenerate binaries.lock.json with
-#                the REAL hashes, then `npm publish` via npm trusted publishing
-#                (OIDC, provenance — NO npm token).
+#                each of the 4 target platforms (no cross-compile), smoke-test it,
+#                compute SHA-256 + size, upload the binary + a meta sidecar.
+#   2. publish — upload every binary to the GAIA hub R2 bucket (gaia-hub,
+#                served at hub.amd-gaia.ai) via rclone,
+#                regenerate binaries.lock.json with the REAL hashes, verify every
+#                published object is fetchable + hash-matches via the real fetch
+#                CLI, then `npm publish` via npm trusted publishing (OIDC,
+#                provenance — NO npm token).
 #
-# Atomic + idempotent: the Worker computes SHA-256 server-side and rejects any
-# attempt to overwrite a published filename, so a re-run of an already-published
-# version is a no-op (publish_to_r2.py treats 409-with-identical-bytes as
-# success and fails loudly only on a genuine hash divergence). The npm publish
-# is skipped if that exact version already exists on the registry.
+# Distribution model: the gaia-hub R2 bucket is a public static object store.
+# Binaries are served by a plain public
+# GET at hub.amd-gaia.ai/...; the SHA-256 in binaries.lock.json is the
+# integrity gate the fetch CLI enforces on download (no server-side checksum).
+#
+# Atomic + idempotent: rclone copy is overwrite-by-key (re-uploading identical
+# bytes is a no-op); the post-upload fetch-verify — not rclone's own size/ETag
+# skip check — is the real integrity gate and fails the release if any object's
+# bytes don't match the lock; npm publish is skipped if that exact version
+# already exists on the registry.
 #
 # Tag namespace: `agent-pkg-email-*` (NOT `v*`). This is the established Agent
 # Hub package-share namespace (see build_agent_package.yml) and deliberately
 # does NOT fire publish.yml (v* only) or the paused publish_agents.yml (no tag
 # trigger). Example: `git tag agent-pkg-email-v0.1.0 && git push origin --tags`.
 #
 # Maintainer setup (one-time):
-#   * Repo secret  AGENT_HUB_PUBLISH_TOKEN  — Bearer token for POST /publish.
-#   * Repo variable AGENT_HUB_BASE_URL      — Worker origin, e.g.
-#       https://gaia-agent-hub.<acct>.workers.dev  (no trailing slash, no /publish).
-#   * npm trusted publisher for @amd-gaia/agent-email registered against THIS
-#     workflow file name: release_agent_email.yml (repo amd/gaia). The OIDC
-#     subject is tied to the exact filename — renaming this file breaks publish.
+#   Secrets:
+#     R2_ACCESS_KEY_ID       — R2 S3 API token (Object Read & Write on gaia-hub).
+#     R2_SECRET_ACCESS_KEY   — its secret.
+#     R2_ACCOUNT_ID          — Cloudflare account id (for the R2 S3 endpoint).
+#   Variables:
+#     HUB_BASE_URL           — public origin, default https://hub.amd-gaia.ai
+#                              (set only to override).
+#   npm trusted publisher for @amd-gaia/agent-email registered against THIS
+#   workflow file name: release_agent_email.yml (repo amd/gaia). The OIDC subject
+#   is tied to the exact filename — renaming this file breaks publish.
 
 name: Release Agent (email)
 
@@ -55,6 +66,9 @@ env:
   PKG_DIR: hub/agents/npm/agent-email
   MANIFEST: hub/agents/python/email/gaia-agent.yaml
   FREEZE_DIST: hub/agents/python/email/packaging/dist
+  # R2 bucket + key prefix (layout mirrors the GitHub hub/agents/<lang>/<id> tree).
+  R2_BUCKET: gaia-hub
+  HUB_PREFIX: agents/email
 
 jobs:
   # ── Stage 1: native freeze on every platform ───────────────────────
@@ -144,7 +158,7 @@ jobs:
             staging/${{ matrix.platform }}.meta.json
           if-no-files-found: error
 
-  # ── Stage 2: publish to R2 + npm (single atomic step) ──────────────
+  # ── Stage 2: upload to R2 + publish to npm (single atomic step) ─────
   publish:
     name: Publish to R2 + npm
     runs-on: ubuntu-latest
@@ -169,40 +183,50 @@ jobs:
       - name: Resolve + validate release version
         id: ver
         shell: bash
+        # User-controlled inputs go through env (never inlined into the script)
+        # so a value can't inject shell — GitHub Actions script-injection
+        # hardening. The tag path uses GITHUB_REF_NAME (also env), and the
+        # version is still triple-checked against package.json + manifest below.
         env:
           EVENT_NAME: ${{ github.event_name }}
-          # Routed through env (never inlined into the script) so an input value
-          # cannot inject shell — GitHub Actions script-injection hardening.
           INPUT_VERSION: ${{ github.event.inputs.version }}
+          HUB_BASE_URL: ${{ vars.HUB_BASE_URL }}
         run: |
           set -euo pipefail
           if [ "$EVENT_NAME" = "workflow_dispatch" ]; then
             VERSION="$INPUT_VERSION"
           else
-            # Strip the agent-pkg-email- prefix and an optional leading 'v'.
             TAG="${GITHUB_REF_NAME}"
             VERSION="${TAG#agent-pkg-email-}"
             VERSION="${VERSION#v}"
           fi
           PKG_VER="$(node -p "require('./${PKG_DIR}/package.json').version")"
-          MAN_VER="$(python -c "import yaml,sys; print(yaml.safe_load(open('${MANIFEST}'))['version'])")"
+          MAN_VER="$(python -c "import yaml; print(yaml.safe_load(open('${MANIFEST}'))['version'])")"
           echo "release version: ${VERSION} (package.json=${PKG_VER}, manifest=${MAN_VER})"
           if [ "$VERSION" != "$PKG_VER" ] || [ "$VERSION" != "$MAN_VER" ]; then
             echo "::error::version mismatch — tag/input=${VERSION}, package.json=${PKG_VER}, gaia-agent.yaml=${MAN_VER}. Align all three before tagging."
             exit 1
           fi
           echo "version=${VERSION}" >> "$GITHUB_OUTPUT"
+          echo "base_url=${HUB_BASE_URL:-https://hub.amd-gaia.ai}/${HUB_PREFIX}/${VERSION}" >> "$GITHUB_OUTPUT"
 
-      - name: Assert release config present
+      - name: Assert R2 credentials present
         shell: bash
+        # Secrets go through env (never inlined into the script text) — same as
+        # the rclone step — so a value with shell metacharacters can't break
+        # parsing or defeat log masking.
+        env:
+          R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
+          R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
+          R2_ACCOUNT_ID: ${{ secrets.R2_ACCOUNT_ID }}
         run: |
           set -euo pipefail
-          if [ -z "${{ vars.AGENT_HUB_BASE_URL }}" ]; then
-            echo "::error::repo variable AGENT_HUB_BASE_URL is not set (Worker origin, e.g. https://hub.example)."
-            exit 1
-          fi
-          if [ -z "${{ secrets.AGENT_HUB_PUBLISH_TOKEN }}" ]; then
-            echo "::error::repo secret AGENT_HUB_PUBLISH_TOKEN is not set (Bearer publish token)."
+          missing=""
+          [ -z "${R2_ACCESS_KEY_ID:-}" ] && missing="$missing R2_ACCESS_KEY_ID"
+          [ -z "${R2_SECRET_ACCESS_KEY:-}" ] && missing="$missing R2_SECRET_ACCESS_KEY"
+          [ -z "${R2_ACCOUNT_ID:-}" ] && missing="$missing R2_ACCOUNT_ID"
+          if [ -n "$missing" ]; then
+            echo "::error::missing repo secret(s):$missing — needed to upload to the gaia-hub R2 bucket."
             exit 1
           fi
 
@@ -212,56 +236,55 @@ jobs:
           pattern: email-agent-*
           path: artifacts
 
-      - name: Install publish deps
-        run: python -m pip install --upgrade requests pyyaml
+      - name: Install deps (PyYAML for the lock generator) + rclone
+        shell: bash
+        run: |
+          set -euo pipefail
+          python -m pip install --upgrade pyyaml
+          curl -fsSL https://rclone.org/install.sh | sudo bash
+          rclone version | head -1
 
-      - name: Collect binaries
-        id: collect
+      - name: Collect binaries + metas
         shell: bash
         run: |
           set -euo pipefail
           mkdir -p bins
-          # Each artifact dir holds the binary + a <platform>.meta.json.
           find artifacts -type f ! -name '*.meta.json' -exec cp {} bins/ \;
           find artifacts -name '*.meta.json' -exec cp {} bins/ \;
+          cp "${MANIFEST}" bins/gaia-agent.yaml
           echo "=== collected ==="
           ls -la bins/
-          # Build the repeatable --artifact arg list (path=platform from meta).
-          ARGS=""
-          for meta in bins/*.meta.json; do
-            plat="$(python -c "import json,sys; print(json.load(open(sys.argv[1]))[0]['platform'])" "$meta")"
-            fn="$(python -c "import json,sys; print(json.load(open(sys.argv[1]))[0]['filename'])" "$meta")"
-            ARGS="$ARGS --artifact bins/${fn}=${plat}"
-          done
-          echo "artifact_args=$ARGS" >> "$GITHUB_OUTPUT"
 
-      - name: Publish binaries to R2
-        env:
-          AGENT_HUB_PUBLISH_TOKEN: ${{ secrets.AGENT_HUB_PUBLISH_TOKEN }}
-          AGENT_HUB_BASE_URL: ${{ vars.AGENT_HUB_BASE_URL }}
+      - name: Upload binaries to R2 (rclone)
         shell: bash
+        env:
+          # rclone reads the remote 'R2' entirely from env — nothing on disk.
+          RCLONE_CONFIG_R2_TYPE: s3
+          RCLONE_CONFIG_R2_PROVIDER: Cloudflare
+          RCLONE_CONFIG_R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
+          RCLONE_CONFIG_R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
+          RCLONE_CONFIG_R2_ENDPOINT: https://${{ secrets.R2_ACCOUNT_ID }}.r2.cloudflarestorage.com
         run: |
           set -euo pipefail
-          # artifact_args is generated by the previous step from the build job's
-          # own meta files (platform keys + filenames) — intentionally word-split.
-          python hub/agents/python/email/packaging/publish_to_r2.py \
-            --base-url "$AGENT_HUB_BASE_URL" \
-            --manifest "${MANIFEST}" \
-            ${{ steps.collect.outputs.artifact_args }} \
-            --summary-out published.json
+          DEST="R2:${R2_BUCKET}/${HUB_PREFIX}/${{ steps.ver.outputs.version }}"
+          echo "uploading binaries + manifest -> ${DEST}/"
+          # Upload only the binaries + manifest (not the *.meta.json sidecars).
+          for f in bins/email-agent-* bins/gaia-agent.yaml; do
+            [ -f "$f" ] || continue
+            rclone copyto "$f" "${DEST}/$(basename "$f")" --s3-no-check-bucket
+          done
+          echo "=== R2 listing ==="
+          rclone lsl "${DEST}/" --s3-no-check-bucket
 
       - name: Regenerate binaries.lock.json with real hashes
         shell: bash
-        env:
-          AGENT_HUB_BASE_URL: ${{ vars.AGENT_HUB_BASE_URL }}
-          REL_VERSION: ${{ steps.ver.outputs.version }}
         run: |
           set -euo pipefail
           python hub/agents/python/email/packaging/gen_binaries_lock.py \
-            --base-url "$AGENT_HUB_BASE_URL" \
-            --version "$REL_VERSION" \
+            --base-url "${{ steps.ver.outputs.base_url }}" \
+            --version "${{ steps.ver.outputs.version }}" \
             --lock "${PKG_DIR}/binaries.lock.json" \
-            $(for m in bins/*.meta.json; do echo --summary "$m"; done)
+            $(for m in bins/*.meta.json; do echo --meta "$m"; done)
           echo "=== regenerated lock ==="
           cat "${PKG_DIR}/binaries.lock.json"
 
@@ -271,17 +294,39 @@ jobs:
           npm ci
           npm run build
 
+      - name: Verify every published object via the real fetch CLI
+        working-directory: ${{ env.PKG_DIR }}
+        shell: bash
+        run: |
+          set -euo pipefail
+          # Atomicity gate: the public object's bytes must hash to the lock value
+          # for EVERY platform before we publish the package that points at them.
+          # Bounded retry: a just-uploaded object can briefly 404 / serve a stale
+          # negative cache at the Cloudflare edge (hub.amd-gaia.ai) before it
+          # propagates — that edge is separate from R2's read-after-write store.
+          # Explicit, caller-requested retry that still fails loudly once exhausted.
+          for plat in win32-x64 darwin-arm64 darwin-x64 linux-x64; do
+            echo "--- verifying $plat ---"
+            attempt=1; max=5
+            until node dist/cli.js fetch --out "/tmp/verify-$plat" --platform "$plat"; do
+              if [ "$attempt" -ge "$max" ]; then
+                echo "::error::fetch-verify failed for $plat after ${max} attempts — the published object never became fetchable/valid at the public origin."
+                exit 1
+              fi
+              echo "attempt ${attempt}/${max} failed; object may not have propagated to the edge yet — retrying in 10s."
+              attempt=$((attempt + 1)); sleep 10
+            done
+          done
+
       - name: Upgrade npm (trusted publishing support)
         run: npm install -g npm@latest
 
       - name: Publish to npm (OIDC trusted publishing, provenance)
         working-directory: ${{ env.PKG_DIR }}
         shell: bash
-        env:
-          REL_VERSION: ${{ steps.ver.outputs.version }}
         run: |
           set -euo pipefail
-          VERSION="$REL_VERSION"
+          VERSION="${{ steps.ver.outputs.version }}"
           if npm view "@amd-gaia/agent-email@${VERSION}" version >/dev/null 2>&1; then
             echo "@amd-gaia/agent-email@${VERSION} already on npm — skipping (idempotent)."
             exit 0

docs/plans/package-publishing.mdx

@@ -29,7 +29,7 @@ We adopt **option (b)** from the design discussion:
 > desktop installers). The paused `publish_agents.yml` (PyPI wheels, #1179) is
 > **left untouched** — the PyPI channel stays paused; the new publisher does
 > **npm + binary** first. Agent UI is **promoted to a registry package** so it
-> releases on its own cadence. R2 backend is **rclone → `assets.amd-gaia.ai`**.
+> releases on its own cadence. R2 backend is **rclone → `hub.amd-gaia.ai`**.
 
 ## Core principle: version-as-signal + idempotent publish
 
@@ -185,10 +185,10 @@ interact with:
 
 ## Open items to resolve during implementation
 
-- **R2 backend** — confirm rclone → `assets.amd-gaia.ai` (bucket `amd-gaia`) as
+- **R2 backend** — confirm rclone → `hub.amd-gaia.ai` (bucket `gaia-hub`) as
   the binary origin; the `binary` channel's upload + the lock `baseUrl` depend on
   it. (Repo secrets `R2_ACCESS_KEY_ID` / `R2_SECRET_ACCESS_KEY` /
-  `R2_ACCOUNT_ID`; var `ASSETS_BASE_URL`.)
+  `R2_ACCOUNT_ID`; var `HUB_BASE_URL`.)
 - **Approval gate** — `publish.yml` uses a manual `publish` environment gate. Do
   package releases need the same gate, or is the namespaced tag + idempotency
   enough? (Recommendation: gate the `npm`/`pypi` publish step only, not the

hub/agents/npm/agent-email/binaries.lock.json

@@ -1,8 +1,7 @@
 {
   "schemaVersion": "1.0",
   "agentVersion": "0.1.0",
-  "_comment": "Per-platform frozen email-agent binary manifest. baseUrl + sha256 values are PLACEHOLDERS pending real R2 publishing (issue #1648). fetch is intentionally blocked while sha256 is a placeholder so a bad binary can never be trusted. Override baseUrl at fetch time and replace these hashes once artifacts are published. NO binaries are committed to the repo or the npm tarball — this manifest + the fetch CLI pull them at build time.",
-  "baseUrl": "https://PENDING-R2-BUCKET.example/email-agent/0.1.0",
+  "baseUrl": "https://hub.amd-gaia.ai/agents/email/0.1.0",
   "binaries": {
     "win32-x64": {
       "filename": "email-agent-win32-x64.exe",

hub/agents/python/email/packaging/HUB-UPLOAD.md

@@ -0,0 +1,96 @@
+# Manual R2 upload — email agent binaries
+
+How to publish frozen email-agent binaries to the GAIA hub bucket **by hand**
+(the "I run rclone myself" path). This produces the exact same objects and lock
+as the CI release (`.github/workflows/release_agent_email.yml`) — same bucket,
+same `agents/email` prefix, same `hub.amd-gaia.ai` origin — so a hand-upload
+and a CI release are interchangeable.
+
+> **One-time rclone setup** (remote name, R2 credentials, endpoint) follows the
+> same steps as [`scripts/video-demo/R2-SETUP.md`](../../../../../scripts/video-demo/R2-SETUP.md),
+> but targets the dedicated `gaia-hub` bucket (custom domain `hub.amd-gaia.ai`)
+> rather than `amd-gaia`/`assets.amd-gaia.ai`. Configure a remote named `gaia`
+> against `gaia-hub`. This page assumes that remote exists.
+
+## Layout
+
+Binaries are served by a plain public GET; the SHA-256 in `binaries.lock.json`
+is the integrity gate the npm `fetch` CLI enforces (no server-side checksum).
+
+```
+R2:  gaia-hub/agents/email/<version>/email-agent-<platform>[.exe]
+URL: https://hub.amd-gaia.ai/agents/email/<version>/email-agent-<platform>[.exe]
+```
+
+Platforms: `win32-x64` (`.exe`), `darwin-arm64`, `darwin-x64`, `linux-x64`.
+
+## One command
+
+Stage the per-platform binaries into a folder named `email-agent-<platform>[.exe]`,
+then:
+
+```bash
+# from the repo root, with rclone remote 'gaia' configured
+hub/agents/python/email/packaging/upload_to_r2.sh 0.1.0 ./staging
+```
+
+The script:
+
+1. asserts the version matches `package.json` **and** `gaia-agent.yaml` (fails loudly otherwise),
+2. hashes each binary it is about to upload (so the lock can never drift from the bytes),
+3. `rclone copyto`s each binary + `gaia-agent.yaml` to `gaia:gaia-hub/agents/email/<version>/`,
+4. regenerates `hub/agents/npm/agent-email/binaries.lock.json` with the real hashes.
+
+Absent platforms keep their existing lock entry, so a **Windows-only** hand-upload
+won't wipe the mac/linux entries — upload the rest later and rerun.
+
+Env overrides: `R2_REMOTE` (default `gaia`), `R2_BUCKET` (default `gaia-hub`),
+`HUB_BASE_URL` (default `https://hub.amd-gaia.ai`).
+
+## Where the binaries come from
+
+PyInstaller does **not** cross-compile, so each platform must be frozen natively
+(`freeze.py` — see [`README.md`](README.md)). You can only build `win32-x64` on a
+Windows box. For an all-platform release without four machines, let CI build them
+and **download its build artifacts**, then rclone them up yourself:
+
+```bash
+# download the 4 platform artifacts from a release_agent_email.yml run into ./staging,
+# unzip so files are named email-agent-<platform>[.exe], then:
+hub/agents/python/email/packaging/upload_to_r2.sh 0.1.0 ./staging
+```
+
+This is the clean split: **CI builds, you publish.**
+
+## Verify, then publish npm
+
+```bash
+cd hub/agents/npm/agent-email
+npm ci && npm run build
+node dist/cli.js fetch --out ./verify --platform win32-x64   # --out is required; repeat per platform — downloads + checks SHA-256
+npm publish --access public                                  # or let CI do it via OIDC trusted publishing
+```
+
+If `fetch` errors with a hash mismatch, the uploaded bytes don't match the lock —
+re-run the upload; do **not** hand-edit the lock.
+
+## Manual upload without the script
+
+Equivalent raw commands (the script just wraps these + the hashing + lock regen):
+
+```bash
+VER=0.1.0
+DEST="gaia:gaia-hub/agents/email/$VER"
+# --exclude '*.json' keeps the *.meta.json sidecars out of the public dir.
+rclone copy ./staging/ "$DEST/" --s3-no-check-bucket --progress --exclude '*.json'
+# Upload the manifest too, so the hand path matches CI byte-for-byte.
+rclone copyto hub/agents/python/email/gaia-agent.yaml "$DEST/gaia-agent.yaml" \
+  --s3-no-check-bucket
+python hub/agents/python/email/packaging/gen_binaries_lock.py \
+  --base-url "https://hub.amd-gaia.ai/agents/email/$VER" \
+  --version "$VER" --lock hub/agents/npm/agent-email/binaries.lock.json \
+  --meta staging/<platform>.meta.json   # one --meta per platform
+```
+
+`--s3-no-check-bucket` is required for Object-Read/Write tokens (no bucket-create
+permission) — see [`R2-SETUP.md`](../../../../../scripts/video-demo/R2-SETUP.md).