Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 26 additions & 5 deletions .github/workflows/harness-canary.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,11 +13,17 @@
# issue for that leg. Coverage table, secrets and the workspace flag:
# docs/harness-canary.md.
#
# Drift pairing: the drift scout (gatewaystack-connect, apps/tenant-gateway/
# src/drift) dispatches `harness-release` with client_payload
# {harness, drift_issue, version, drift_kind, close_on_pass} when an upstream
# release signal moves; the reporter then comments the verdict on that drift
# issue and, for release-type drift, closes it on a pass.
# Drift sweep: the drift scout (gatewaystack-connect, apps/tenant-gateway/
# src/drift) files one `drift` issue per upstream release with the marker
# `<!-- drift-scout:key=release:<entryId>@<version> -->`. No dispatch token:
# after each leg reports, scripts/canary-report.mjs lists the open `drift`
# issues and resolves the ones for this leg's registry entries at the
# version the run just tested (pass -> comment + close; fail -> one comment
# per failure). The scheduled run therefore answers a release issue within
# ~6 h. SDK legs pass the upstream versions they installed as --tested
# pairs, since their own --version is the ACP package's. Dispatch pairing
# (repository_dispatch harness-release with client_payload.drift_issue) is
# still honoured for runs triggered by hand.
#
# Live legs (one governed `echo <marker>`, then the audit-row assert):
# claude-code claude-acp -p, BYO subscription OAuth (CLAUDE_CODE_OAUTH_TOKEN)
Expand Down Expand Up @@ -340,6 +346,19 @@ jobs:
esac
echo "resolved: $v"
echo "version=$v" >> "$GITHUB_OUTPUT"
# Upstream versions this leg actually installed, as drift-registry
# entryId=version pairs, so the reporter's drift sweep can match
# the scout's release issues (its own version above is the ACP
# package's). Entry ids and leg mapping: registry.ts canaryLeg.
pyver() { pip show "$1" 2>/dev/null | sed -n 's/^Version: //p' || true; }
u=""
case "$LEG" in
sdk-langchain) u="langchain-core=$(pyver langchain-core),langgraph=$(pyver langgraph)" ;;
sdk-crewai) u="crewai=$(pyver crewai)" ;;
sdk-governance-anthropic) u="anthropic-sdk-typescript=$(npm ls @anthropic-ai/sdk --depth=1 2>/dev/null | grep -o '@anthropic-ai/sdk@[0-9.]*' | head -1 | sed 's/.*@//' || true)" ;;
esac
echo "upstream: ${u:-(none mapped)}"
echo "upstream=$u" >> "$GITHUB_OUTPUT"
echo "CANARY_SINCE=$(date -u +%Y-%m-%dT%H:%M:%S.000Z)" >> "$GITHUB_ENV"

- name: One governed tool-call check
Expand Down Expand Up @@ -372,6 +391,7 @@ jobs:
CANARY_ISSUES_TOKEN: ${{ secrets.CANARY_ISSUES_TOKEN }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
PKG_VERSION: ${{ steps.pkg.outputs.version }}
UPSTREAM_VERSIONS: ${{ steps.pkg.outputs.upstream }}
DRIFT_ISSUE: ${{ github.event.client_payload.drift_issue || '' }}
DRIFT_VERSION: ${{ github.event.client_payload.version || '' }}
DRIFT_KIND: ${{ github.event.client_payload.drift_kind || '' }}
Expand All @@ -394,5 +414,6 @@ jobs:
done
node scripts/canary-report.mjs --harness "$LEG" --status "$status" --version "$PKG_VERSION" \
--run-url "$RUN_URL" --failed-step "${failed:-none}" ${log:+--log "$log"} \
${UPSTREAM_VERSIONS:+--tested "$UPSTREAM_VERSIONS"} \
${DRIFT_ISSUE:+--drift-issue "$DRIFT_ISSUE"} ${DRIFT_VERSION:+--drift-version "$DRIFT_VERSION"} \
${DRIFT_KIND:+--drift-kind "$DRIFT_KIND"} ${DRIFT_CLOSE:+--drift-close "$DRIFT_CLOSE"}
75 changes: 53 additions & 22 deletions docs/harness-canary.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,30 +114,61 @@ Triggers: `schedule: 13 */6 * * *`, `workflow_dispatch` (input `harness`:
`harness-release` (optional `client_payload.harness`, same values).
Concurrency is per leg; legs do not cancel each other.

### Drift pairing
### Drift sweep (how a drift issue gets resolved)

The drift scout (gatewaystack-connect, `apps/tenant-gateway/src/drift/`,
`docs/drift-scout.md` there) sends the `harness-release` dispatch when an
upstream release signal (npm latest, GitHub release/tag, PyPI) of a harness
or SDK moves, with `client_payload`
`{harness, drift_issue, version, drift_kind: "release"|"docs", close_on_pass}`.
The workflow passes those through to the reporter as `--drift-issue`,
`--drift-version`, `--drift-kind`, `--drift-close`, and the reporter comments
the verdict on that drift issue in `davidcrowe/gatewaystack-connect`:

- pass: `Canary <leg> on <version>: ✅ passed: safe to close` and, when
closing is on, closes the drift issue (`state_reason: completed`);
- fail: `Canary <leg> on <version>: ❌ failed at <step>: see <canary issue>`
(the run URL when no canary issue exists).

Closing on pass defaults ON for `drift_kind: release` (the only kind the scout
dispatches) and OFF for `drift_kind: docs`; `close_on_pass: false` in the
payload (or `--drift-close off`) turns it off either way. The drift comment is
best effort: a failure there is logged and never fails the reporter, so the
canary issue stays the primary record. `CANARY_ISSUES_TOKEN` already covers
it (same repo, Issues: read and write). The scout's own dispatch credential
(`CANARY_DISPATCH_TOKEN`, Actions: read and write on this repo) is documented
in gatewaystack-connect `docs/drift-scout.md`.
`docs/drift-scout.md` there) files **one issue per upstream release** in
`davidcrowe/gatewaystack-connect` when a release signal (npm latest, GitHub
release/tag, PyPI) of a harness or SDK moves: label `drift`, body marker
`<!-- drift-scout:key=release:<entryId>@<version> -->`. It does not trigger
the canary. **The scheduled canary resolves matching drift issues within
~6 h; no extra token.** Each leg installs `@latest`, so by the next scheduled
run it is testing exactly the version the scout saw.

After every leg reports, `scripts/canary-report.mjs` lists the open `drift`
issues, keeps the ones whose release marker names a registry entry mapped to
this leg (`LEG_ENTRIES` in the script mirrors `canaryLeg` in the scout's
`registry.ts`: `claude-code`, `codex`, `opencode`, `qwen-code`, `pi`,
`prime-agent`, `grok`, `dsh`, `hermes`, `openclaw`, `muse` map to the leg of
the same name; `crewai` to `sdk-crewai`; `langchain-core` and `langgraph` to
`sdk-langchain`; `anthropic-sdk-typescript` to `sdk-governance-anthropic`)
and whose normalised version (first dotted number, the scout's
`normalizeVersion` rule, so `v1.1.0`, `rust-v1.1.0`, `1.1.0 (Claude Code)`
all read `1.1.0`) equals the version this run tested, and posts the verdict:

- pass: `Canary <leg> on <version>: ✅ passed — closing.` and closes the
issue (`state_reason: completed`);
- fail: `Canary <leg> on <version>: ❌ failed at <step>: <canary issue or
run link>`, once per failure: when the reporter's last verdict comment on
that issue already says failed for the same version, a later red run on
the same version does not repeat it.

Docs-type drift issues (markers that are not `release:`) are left alone: a
green canary says nothing about a changed docs page. For harness legs the
tested version is the harness's own `--version` output. For SDK legs the
`--version` is the ACP package's, so the `install-package` step also records
the upstream versions it installed (`langchain-core`, `langgraph`, `crewai`,
`@anthropic-ai/sdk`) and passes them as `--tested entryId=version,...`; an
SDK entry with no recorded version is skipped. The sweep is best effort: a
failure there is logged and never fails the reporter, so the canary issue
stays the primary record. `CANARY_ISSUES_TOKEN` covers it (same repo,
Issues: read and write).

Dispatch pairing is kept for hand-triggered runs: a `repository_dispatch`
`harness-release` with `client_payload`
`{harness, drift_issue, version, drift_kind: "release"|"docs", close_on_pass}`
reaches the reporter as `--drift-issue`, `--drift-version`, `--drift-kind`,
`--drift-close`, and the verdict is posted on that issue (closed on pass for
`release` drift unless `close_on_pass: false`); the sweep skips that issue
so it is not commented twice.

Preview the sweep locally with a fixture (`{drift: [...], canary: [...],
comments: [...]}` is what dry-run `GET`s return):

```
node scripts/canary-report.mjs --harness qwen-code --status success --version "0.9.0 (Qwen Code)" \
--run-url https://example.invalid/run --dry-run --fixture fixture.json
```

## Secrets (repository secrets on agentic-control-plane/acp-install)

Expand Down
Loading
Loading