feat(qualification): black-box end-to-end qualification suite

.agents/skills/run-guardian-qualification/SKILL.md

@@ -0,0 +1,218 @@
+---
+name: run-guardian-qualification
+description: Set up and run the end-to-end qualification suite in this repository: the deterministic profile against a provisioned stack, and the live profile driving real Miden transactions from a CI-held treasury. Use when Codex needs to run or debug `qualification/stack/run.sh`, add or change a scenario, drive one SDK's live scenarios against testnet, reproduce a qualification failure locally, or interpret a run result's outcome classes and qualification claim.
+---
+
+# Run Guardian Qualification
+
+Two profiles. **Deterministic** provisions a stack and drives fixture flows with
+no chain; it is fast enough to gate merges. **Live** drives real transactions on
+a public Miden network and spends real funds.
+
+Read [`docs/QUALIFICATION.md`](../../../docs/QUALIFICATION.md) for what the suite
+covers and what it deliberately does not. This skill is how to run it.
+
+## Pick the smallest thing that answers the question
+
+| Question | Command |
+|---|---|
+| Does the manifest hold together? | `./target/debug/qualification-driver validate` |
+| Does driver logic still work? | `cargo test -p guardian-qualification-driver` |
+| Do the shell libs still work? | `qualification/stack/tests/harness-test.sh` |
+| Does a scenario body work? | one scenario, one SDK, against a hand-started server |
+| Does the assembled system work? | `qualification/stack/run.sh` with Docker |
+
+Do not reach for the full stack to debug a scenario body. It rebuilds an image.
+
+## Build the binary first, always
+
+```bash
+cargo build -p guardian-qualification-driver
+```
+
+Every entry point shells out to `target/debug/qualification-driver`. Going
+through `cargo run` instead costs about 3.5s of build-graph checking per call
+and can stall on a rebuild in the middle of a chain operation.
+
+## Fast path: one scenario, no Docker
+
+The quickest loop for scenario work. Start a server with `cargo` directly, then
+drive one scenario.
+
+Deterministic scenarios need the **fixture** acknowledgement key: the committed
+fixture account binds one specific guardian identity, and a server booted with a
+generated key rejects it with `403 ... not an authorized signer`. Point
+`GUARDIAN_ACK_FALCON_SECRET_PATH` at a file containing `guardian_secret_key`
+from `crates/server/src/testing/fixtures/keys.json`, or accept that only the
+live and error-envelope scenarios will pass.
+
+```bash
+# Rust, one live scenario
+export QUAL_TREASURY_KEY="$QUAL_TREASURY_KEY_testnet"   # from qualification/.treasury-secrets.env
+./target/debug/qualification-driver run \
+  --profile live --network testnet --sdk rust \
+  --scenario live-execute-2of3-falcon \
+  --run-id local-$(date +%s) \
+  --http-endpoint http://127.0.0.1:3000 \
+  --grpc-endpoint http://127.0.0.1:50051 \
+  --out /tmp/qual-results/result.json
+```
+
+```bash
+# TypeScript, same scenario
+cd packages/miden-multisig-client
+QUAL_PROFILE=live QUAL_NETWORK=testnet \
+QUAL_SCENARIOS=live-execute-2of3-falcon \
+QUAL_HTTP_ENDPOINT=http://127.0.0.1:3000 \
+QUAL_GRPC_ENDPOINT=http://127.0.0.1:50051 \
+QUAL_OUT_DIR=/tmp/qual-results QUAL_RUN_ID=local-$(date +%s) \
+  npx vitest run --config vitest.qualification.config.ts
+```
+
+The two drivers reach GUARDIAN over **different transports**: Rust speaks gRPC,
+TypeScript speaks HTTP. Passing one the other's endpoint fails in ways that look
+like an outage.
+
+## Full stack
+
+```bash
+qualification/stack/run.sh --profile deterministic
+qualification/stack/run.sh --profile live --network testnet
+```
+
+`--keep-stack` leaves the containers up so you can read logs; without it the
+stack is torn down even on failure, and diagnostics land in
+`<out>/diagnostics/`.
+
+**Any change under `crates/` rebuilds the server image.** The build context
+copies the workspace, so a one-line driver edit costs a full release build
+before the stack starts. This is the main reason to iterate against a
+hand-started server and use the stack only to confirm.
+
+**Never edit `run.sh` while a run is in flight.** Bash reads scripts
+incrementally, so an edit mid-run produces a syntax error at a stale byte offset
+in a file that passes `bash -n`. Wait for the run, then edit.
+
+The stack starts a **second GUARDIAN** (`server-migration-target`) with its own
+acknowledgement identity and its own database, because migrating an account to
+the GUARDIAN it already uses is not a state change and the kernel rejects it. It
+exports `QUAL_GUARDIAN_MIGRATION_GRPC` and `QUAL_GUARDIAN_MIGRATION_HTTP`; the
+migration scenario reports environment-blocked when they are absent.
+
+## Treasury
+
+Live runs need `QUAL_TREASURY_KEY`. Locally it lives in
+`qualification/.treasury-secrets.env` (gitignored, mode 0600) with one key per
+network.
+
+```bash
+set -a && . qualification/.treasury-secrets.env && set +a
+export QUAL_TREASURY_KEY="$QUAL_TREASURY_KEY_testnet"
+./target/debug/qualification-driver treasury-status --network testnet
+```
+
+`treasury-address` prints the address to fund from the faucet;
+`treasury-bootstrap` consumes the faucet notes into the vault. The treasury is a
+**public** single-signature wallet on purpose: a private one keeps its state only
+in the local store, which a fresh CI runner does not have, so the next
+transaction would build on a stale nonce.
+
+Do not arm the GitHub secret until the `qualification-{devnet,testnet}`
+environments have their deployment branches locked to the default branch.
+
+## Changing scenarios
+
+Scenario sources are data, not code: `qualification/manifest/scenarios.toml` and
+`matrix.toml`. After editing either:
+
+```bash
+./target/debug/qualification-driver export-manifest --out qualification/manifest/manifest.json
+./target/debug/qualification-driver validate
+```
+
+A committed `manifest.json` that does not match the sources fails a test. The
+validator also refuses a scenario whose step budget exceeds a network's
+historical window while required there, so exclude it for that network rather than
+shrinking a budget the flow cannot meet.
+
+A new action needs **six edits across five files**, and missing any one fails
+closed on a required scenario rather than silently skipping: the `Action`
+variant and its `From<String>` arm in `manifest/mod.rs`, the Rust dispatch in
+`scenario/mod.rs`, the Rust body in `scenario/{live,account,identity,error_envelope}.rs`,
+the TypeScript dispatch in `runner.ts`, and the TypeScript body in
+`actions/{live,account,identity,errorEnvelope,operator}.ts`. The body goes in
+the file for the action's family, not always `live`.
+
+A scenario that only composes existing actions needs no driver code at all. The
+full procedure, including the matrix and the negative control, is in
+`docs/QUALIFICATION.md`, "Adding a scenario".
+
+## Reading a result
+
+`conclusion` is the gate. `qualification_claim` is the coverage statement, and
+the two are independent: a run can conclude successfully and still claim only
+`partial`, because a required scenario that skipped or was environment-blocked
+forfeits the claim without failing the run.
+
+| Outcome | Meaning |
+|---|---|
+| `passed` | the flow completed and was verified |
+| `failed` + `product` | the system under test is wrong |
+| `failed` + `setup` | the harness or environment is wrong |
+| `skipped` | a known capability gap, named in the reason |
+| `environment_blocked` | the network or a dependency stopped it |
+
+Exit codes: 0 concluded, 1 product failure, 2 setup failure, 3 environment
+blocked and nothing else ran. **A zero exit is not full coverage**, so read the
+claim.
+
+The TypeScript leg writes `<run-id>-typescript.json` carrying only `run_id` and
+`scenario_results`; the Rust merger folds it into the run and restates the claim
+over both SDKs. Merge with `qualification-driver report --results <dir>`.
+
+## Negative controls
+
+A suite that has never failed is not known to detect anything. Before trusting a
+green run after changing assertion logic, break the thing deliberately and
+confirm a **named** scenario fails. The procedure and the two recorded controls
+are in [`qualification/README.md`](../../../qualification/README.md).
+
+Do not ship a fault-injection switch to make this easier. A switch that disables
+an assertion is a switch that can be left on.
+
+## Gotchas that cost real time
+
+- **TypeScript needs an HTTP/2 shim** to reach Miden at all from Node, and
+  without a reachable remote prover the SDK silently falls back to in-WASM
+  proving at roughly twenty-five times the CPU. `tests/qualification/h2Fetch.ts`
+  is the workaround.
+- **The qualification tests typecheck under their own config.** `tsconfig.json`
+  covers only `src/`. Run `npm run typecheck:tests` in
+  `packages/miden-multisig-client`, or a missing import ships.
+- **A stalled image build is usually the network, not the Dockerfile.** A cargo
+  fetch through a VM NAT can stall with zero bytes transferred. Re-run it before
+  changing anything; `CARGO_HTTP_MULTIPLEXING=false` in the build environment
+  trades roughly a third more fetch time for one connection per download, and is
+  worth reaching for only if the stall reproduces.
+- **GUARDIAN holds one acknowledgement identity per signature scheme.** Asking
+  for the pubkey without a scheme returns the default, and an ECDSA account built
+  from it is refused at registration with a message about authorization rather
+  than the mismatch.
+- **A proposal leaving the pending set is not completion.** A discarded delta is
+  removed the same way. Assert through `verify_state_commitment` /
+  `verifyStateCommitment` plus canonical delta history, which is what the drivers
+  now do.
+
+## Before calling a change done
+
+```bash
+cargo test -p guardian-qualification-driver
+cargo clippy -p guardian-qualification-driver --all-targets
+cargo fmt --all -- --check
+qualification/stack/tests/harness-test.sh
+cd packages/miden-multisig-client && npm run typecheck:tests && npx vitest run
+```
+
+Live scenario changes are not verified by any of the above. Run the affected
+scenario on testnet for **both** SDKs, since the two drivers diverge in ways the
+type system does not catch.

.agents/skills/run-guardian-qualification/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Run Guardian Qualification"
+  short_description: "Set up and run the end-to-end qualification suite"
+  default_prompt: "Use $run-guardian-qualification to set up and run the Guardian qualification suite, or to debug a scenario or a run result."

.github/workflows/ci.yml

@@ -334,11 +334,15 @@ jobs:
           EOF
 
       # Workspaces link the package sources, but the multisig SDK imports the
-      # guardian-client's generated dist output.
+      # generated dist output of both. The operator client is a test-only
+      # dependency: the qualification suite's dashboard scenarios drive the
+      # operator surface through it.
       - name: Build workspace dependencies
         if: matrix.package == 'miden-multisig-client'
         working-directory: packages
-        run: npm run build -w @openzeppelin/guardian-client
+        run: |
+          npm run build -w @openzeppelin/guardian-client
+          npm run build -w @openzeppelin/guardian-operator-client
 
       - name: Build
         working-directory: packages
@@ -555,3 +559,73 @@ jobs:
 
       - name: Check OpenAPI spec is up to date
         run: cargo run --features evm --bin gen-openapi -- --check docs
+
+  qualification-harness:
+    name: Qualification Harness
+    runs-on: ubuntu-latest
+    if: github.repository_owner == 'OpenZeppelin'
+    steps:
+      - name: Harden the runner (Audit all outbound calls)
+        uses: step-security/harden-runner@95d9a5deda9de15063e7595e9719c11c38c90ae2 # v2.13.2
+        with:
+          egress-policy: audit
+
+      - name: Checkout code
+        uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0
+        with:
+          persist-credentials: false
+
+      - name: Install Rust
+        uses: dtolnay/rust-toolchain@351f82a4dc29e4159746a068ed925da17341219f # 1.89.0
+        with:
+          toolchain: 1.98.1
+
+      - name: Install system dependencies
+        run: sudo apt-get update && sudo apt-get install -y protobuf-compiler libpq-dev
+
+      - name: Cache cargo (rust-cache)
+        uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
+        with:
+          prefix-key: v1-rust
+          cache-on-failure: true
+
+      # The scenario manifest is the contract both drivers read. A run that
+      # discovers it is malformed has already built an image and started a
+      # database to find out.
+      - name: Validate the scenario manifest
+        run: cargo run --quiet -p guardian-qualification-driver -- validate
+
+      # These need the Docker CLI for the Compose cases but never a daemon.
+      - name: Test the stack shell libraries
+        run: ./qualification/stack/tests/harness-test.sh
+
+      # `tsconfig.json` covers only `src/`, so the qualification driver
+      # typechecks under its own config and nothing else was running it. A
+      # missing import in a scenario therefore reached a run rather than a
+      # build, which has already happened once.
+      - name: Install Node
+        uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
+        with:
+          node-version: "24"
+          cache: npm
+          cache-dependency-path: packages/package-lock.json
+
+      - name: Install workspace dependencies
+        working-directory: packages
+        run: npm ci
+
+      # Every `dist/` the typecheck resolves through, and all three are
+      # gitignored. The two GUARDIAN clients because the driver imports the
+      # operator client at module scope, and the multisig client itself because
+      # `tests/browser/harness.ts` imports the package's own build output: a
+      # test tree that typechecks against `dist/` needs `dist/` to exist.
+      - name: Build the workspace packages
+        working-directory: packages
+        run: |
+          npm run build -w @openzeppelin/guardian-client
+          npm run build -w @openzeppelin/guardian-operator-client
+          npm run build -w @openzeppelin/miden-multisig-client
+
+      - name: Typecheck the qualification driver
+        working-directory: packages/miden-multisig-client
+        run: npm run typecheck:tests