Add reusable E2E AI failure-triage adjudication workflow

.github/workflows/ci.yml

@@ -0,0 +1,66 @@
+---
+# CI for the toolkit's own scripts.
+#
+# The policy engine in scripts/triage-policy.js decides whether an E2E failure
+# blocks a merge across every repo that calls the reusable workflow. Shipping it
+# without its tests running would mean a regression in "when is a red allowed to
+# turn green" reaches consumers silently — which is the one failure mode the
+# whole design is built to prevent.
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+permissions:
+  contents: read
+
+jobs:
+  test:
+    name: Script tests
+    runs-on: ubuntu-24.04
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+
+      - uses: actions/setup-node@2028fbc5c25fe9cf00d9f06a71cc4710d4507903 # v6.0.0
+        with:
+          node-version: '22'
+
+      - name: ci/test
+        # Files are listed explicitly rather than passing the directory: node's
+        # directory runner also executes non-test sources, which turns a plain
+        # `require` into a spurious failure.
+        run: |
+          node --test \
+            scripts/triage-policy.test.js \
+            scripts/triage-apply.test.js \
+            scripts/triage-override.test.js \
+            scripts/triage-blame.test.js
+
+  actionlint:
+    name: Workflow lint
+    runs-on: ubuntu-24.04
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+
+      - name: ci/actionlint
+        run: |
+          set -euo pipefail
+          # Pinned by commit, not by branch. `.../main/scripts/...` re-fetches
+          # whatever that branch holds at the moment CI runs, so this step used to
+          # pipe a mutable remote script straight into bash on a runner holding
+          # the workflow token. The commit ref is immutable, and the version the
+          # script then downloads is fixed rather than "latest".
+          INSTALLER_REF=3795ba2f6cb243eeca54c9d22e5c531cb9dcfb4a
+          ACTIONLINT_VERSION=1.7.12
+          curl -fsSL -o download-actionlint.bash \
+            "https://raw.githubusercontent.com/rhysd/actionlint/${INSTALLER_REF}/scripts/download-actionlint.bash"
+          bash download-actionlint.bash "${ACTIONLINT_VERSION}"
+          # shellcheck is not installed on the runner image by default; the
+          # workflow-level checks are what matter here.
+          ./actionlint -shellcheck= .github/workflows/*.yml

.github/workflows/e2e-ai-triage-override.yml

@@ -0,0 +1,122 @@
+---
+# Human override of an automated triage verdict.
+#
+# Corrections are the only ground truth this system gets — everything else is the
+# triage grading its own homework. A maintainer saying "that was actually a real
+# bug" is the one input that can prove a waiver wrong, and it is exactly what the
+# false-green metric counts.
+#
+# The correction is recorded in the TSIO ledger *and* the checks are brought into
+# line with what the human said. If the ledger write fails the checks are still
+# updated — the maintainer's intent must be honoured — but the reply says so,
+# because an unrecorded correction is a data point permanently lost.
+name: E2E AI Triage Override (Reusable)
+
+on:
+  workflow_call:
+    inputs:
+      target_repo:
+        description: "Full repo name the PR and statuses belong to"
+        required: true
+        type: string
+      pr_number:
+        required: true
+        type: string
+      comment_body:
+        description: "Raw comment body, parsed for /e2e-triage-override <verdict> <reason>"
+        required: true
+        type: string
+      comment_id:
+        description: "Comment to react to, so the author sees it was picked up"
+        required: false
+        type: string
+        default: ""
+      sender:
+        description: "Who issued the override — recorded as corrected_by"
+        required: true
+        type: string
+      tsio_url:
+        required: false
+        type: string
+        default: "https://test-io.test.mattermost.com"
+    secrets:
+      GH_TOKEN:
+        description: "Token for statuses, labels, and comments on target_repo"
+        required: true
+      TSIO_API_KEY:
+        description: "Optional. Without it the ledger write uses a minted OIDC token."
+        required: false
+
+permissions:
+  contents: read
+  # The ledger write authenticates with a minted OIDC token by default.
+  id-token: write
+
+jobs:
+  override:
+    runs-on: ubuntu-24.04
+    steps:
+      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+
+      - uses: actions/setup-node@2028fbc5c25fe9cf00d9f06a71cc4710d4507903 # v6.0.0
+        with:
+          node-version: '22'
+
+      # Verify the sender here rather than trusting the caller to have done it.
+      #
+      # `sender` is just a string input: this workflow is reusable, so whether it
+      # was gated upstream is a property of each caller, not of this file. A
+      # caller that forgets — or one that passes github.actor from a trigger that
+      # any user can fire — would let an outside contributor overturn a red E2E
+      # verdict and apply E2E/AI-Waived to their own PR. The permission is
+      # therefore checked against the repository the override targets, before
+      # anything is written.
+      - name: ci/verify-sender-can-write
+        env:
+          GH_TOKEN: ${{ secrets.GH_TOKEN }}
+          TARGET_REPO: ${{ inputs.target_repo }}
+          SENDER: ${{ inputs.sender }}
+        run: |
+          set -euo pipefail
+          if [ -z "${SENDER}" ]; then
+            echo "::error::sender is empty — refusing to apply an unattributed override"
+            exit 1
+          fi
+          PERM=$(gh api "repos/${TARGET_REPO}/collaborators/${SENDER}/permission" \
+            --jq '.permission' 2>/dev/null || echo "none")
+          echo "sender ${SENDER} has '${PERM}' on ${TARGET_REPO}"
+          case "${PERM}" in
+            admin|maintain|write) ;;
+            *)
+              echo "::error::${SENDER} needs write access to ${TARGET_REPO} to override a triage verdict (has '${PERM}')"
+              exit 1
+              ;;
+          esac
+
+      - name: ci/apply-override
+        env:
+          GH_TOKEN: ${{ secrets.GH_TOKEN }}
+          TSIO_API_KEY: ${{ secrets.TSIO_API_KEY }}
+          # Passed through the environment, never interpolated into the script
+          # arguments: a comment body is attacker-controlled text and must not be
+          # able to reach a shell as anything but data.
+          COMMENT_BODY: ${{ inputs.comment_body }}
+          # Same reasoning for the rest: sender is a GitHub login and the others
+          # are caller-supplied, so none of them are pasted into the script text.
+          TARGET_REPO: ${{ inputs.target_repo }}
+          PR_NUMBER: ${{ inputs.pr_number }}
+          SENDER: ${{ inputs.sender }}
+          COMMENT_ID: ${{ inputs.comment_id }}
+          TSIO_URL: ${{ inputs.tsio_url }}
+          RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
+        run: |
+          set -euo pipefail
+          node scripts/triage-override.js \
+            --repo="${TARGET_REPO}" \
+            --pr="${PR_NUMBER}" \
+            --actor="${SENDER}" \
+            --comment-id="${COMMENT_ID}" \
+            --tsio-url="${TSIO_URL}" \
+            --run-url="${RUN_URL}"

.github/workflows/e2e-ai-triage.md

@@ -0,0 +1,199 @@
+# E2E AI Triage (reusable)
+
+Adjudicates E2E failures a caller repo's deterministic rules could not decide,
+then posts `e2e-test/ai-triage`.
+
+## Division of labour
+
+The caller owns everything device- and repo-specific; this workflow owns
+everything repo-agnostic.
+
+| Stage | Where | What |
+|---|---|---|
+| collect, cluster, rule-classify, enrich with history | **caller repo** | needs its spec layout, its artifact names, and its failure-signature catalogue |
+| adjudicate the residue, apply policy, post status/label/comment/ledger | **here** | operates purely on the normalized `evidence.json` contract |
+
+The contract between them is one file. Any framework that can produce it can use
+this workflow.
+
+## Design rules
+
+These are what make an automated green trustworthy. Change them deliberately.
+
+**Fail closed.** No evidence bundle, unparseable model output, unknown verdict,
+confidence under the bar, API error, job timeout — all resolve red. There is no
+path where "we don't know" produces green.
+
+**Asymmetric bars.** A verdict that would waive a failure needs 0.85 confidence;
+one that keeps it red needs 0.7. The errors are not symmetric: a false red costs
+a rerun, a false green ships a bug.
+
+**Two citations minimum.** A verdict citing fewer than two independent evidence
+items is downgraded to `INCONCLUSIVE` before policy ever sees it. A single
+citation is an assertion, not corroboration.
+
+**The model never decides its own authority.** It emits a verdict; the
+deterministic, unit-tested policy engine in `scripts/triage-policy.js` decides
+what that means for the merge button. The model never calls the status API.
+
+**One unwaived cluster keeps the run red.** A run is green only when *every*
+cluster is waived. Greening because the majority was flaky is exactly the failure
+mode that would make the system untrustworthy.
+
+**Baseline branches never auto-waive.** On `MAIN` and `RELEASE` runs, a flake
+verdict is recorded but stays red. Baseline health has to reflect reality — it is
+also the comparison every PR's verdict is drawn from.
+
+**AI waivers are labelled separately.** `E2E/AI-Waived`, never the human
+`E2E/Override`. Conflating them makes the false-green metric uncomputable.
+
+## Modes
+
+| Mode | Behaviour | Use when |
+|---|---|---|
+| `shadow` | posts its own status and comment; never waives | always, first. Measure accuracy before granting authority. |
+| `assist` | additionally applies `E2E/AI-Waived`, which the caller's status reporter honours | once `false_greens` has been 0 over a real sample |
+| `gate` | reserved for making `e2e-test/ai-triage` the required check | only after sustained assist-mode metrics |
+
+Promotion is a repo-variable change (`E2E_AI_TRIAGE_MODE`), not a code change, so
+rolling back is instant.
+
+## Usage
+
+```yaml
+adjudicate:
+  uses: mattermost/mattermost-test-automation-toolkit/.github/workflows/e2e-ai-triage.yml@main
+  permissions:
+    contents: read
+    actions: read
+    statuses: write
+    pull-requests: write
+    issues: write
+    id-token: write
+  with:
+    target_repo: ${{ github.repository }}
+    commit_sha: ${{ inputs.commit_sha }}
+    pr_number: ${{ inputs.pr_number }}
+    run_type: PR
+    evidence_artifact: e2e-triage-evidence-${{ github.run_id }}
+    evidence_run_id: ${{ github.run_id }}
+    mode: ${{ vars.E2E_AI_TRIAGE_MODE || 'shadow' }}
+    diff_overlaps_failure: ${{ needs.plan.outputs.diff_overlaps == 'true' }}
+  secrets:
+    GH_TOKEN: ${{ secrets.GH_TOKEN }}
+    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+    # Optional — without it the ledger write uses a minted OIDC token.
+    TSIO_API_KEY: ${{ secrets.TSIO_API_KEY }}
+    WEBHOOK_URL: ${{ secrets.WEBHOOK_URL }}
+```
+
+`permissions` must be granted at every level down from the root workflow — a
+reusable workflow cannot escalate past its caller, and a missing scope makes the
+nested step no-op silently rather than fail.
+
+## `evidence.json` contract
+
+```jsonc
+{
+  "tier": 1,                       // 0-4 volume tier; 4 = the run itself is broken
+  "tier_reason": "...",
+  "summary": {"totalTests": 600, "passed": 590, "failed": 10, "shards": [...]},
+  "suite_verdict": null,           // set when a suite-shape rule already decided the run
+  "needs_ai": true,
+  "clusters": [{
+    "signature_hash": "a1b2c3d4e5f6",
+    "signature_label": "...",
+    "member_count": 7,
+    "spans_shards": true,
+    "spans_platforms": false,
+    "shards": ["1"], "platforms": ["ios"], "specs": ["..."],
+    "matched_signatures": [{"id": "device.adb-offline", "weight": 0.9, "verdict": "FLAKY_INFRA"}],
+    "rule_verdict": null,          // non-null means the rules decided; the model is skipped
+    "confidence": 0.25,
+    "needs_ai": true,
+    "representative": {"error_message": "...", "device_log_excerpt": "...", "screenshot": "..."},
+    "member_test_ids": ["MM-T4783_1"],
+    "history": [...],              // per-test TSIO history + amnesty
+    "all_failing_on_baseline": false,
+    "any_failing_elsewhere": false,
+    "amnesty_exhausted": false
+  }]
+}
+```
+
+Clusters with `needs_ai: false` and a `rule_verdict` are already decided and are
+never sent to the model. A `suite_verdict` replaces per-cluster adjudication
+entirely — when every shard died, the individual assertion messages are symptoms,
+not causes.
+
+## Verdicts
+
+| Verdict | Waivable | Meaning |
+|---|---|---|
+| `PR_REGRESSION` | no | the change under test broke it |
+| `MAIN_REGRESSION` | yes\* | already failing on the baseline branch |
+| `FLAKY_TEST` | yes | test-side non-determinism |
+| `FLAKY_INFRA` | yes | runner, emulator, or simulator |
+| `FLAKY_SERVER` | yes | test server or its provisioning |
+| `BUILD_OR_ENV_ERROR` | no | bundler/dependency/signing — looks like infra, is a code problem |
+| `TEST_DEBT` | no | the test is wrong and the app is right |
+| `INCONCLUSIVE` | no | evidence bar not met |
+
+\* only when `diff_overlaps_failure` is false. If the PR touches the same area,
+attribution is ambiguous and ambiguity is red.
+
+## Human override
+
+`/e2e-triage-override <verdict> <reason>` on the PR, from an OWNER, MEMBER, or
+COLLABORATOR. Handled by `e2e-ai-triage-override.yml`.
+
+The verdict is case- and dash-insensitive (`flaky-infra` and `FLAKY_INFRA` both
+work). A reason is mandatory — the correction's value is as a labelled example,
+and a bare verdict records that triage was wrong while discarding the only part
+that says how.
+
+Correcting to a waivable verdict greens the check and applies the waiver label;
+correcting to anything else reds it and **withdraws** the label. The withdrawal
+matters: the label is sticky across pushes and the status reporter honours it
+unconditionally, so leaving it applied would keep greening later commits.
+
+The correction is written to the ledger first, because that is the part that
+outlives the PR. If the ledger write fails, the checks are still updated — the
+maintainer's intent is honoured — but the reply says so explicitly, since an
+unrecorded correction is a data point permanently lost.
+
+## Main-regression blame
+
+When triage concludes `MAIN_REGRESSION` the PR is innocent, but someone's change
+did break the baseline. TSIO already knows the last commit where the test passed
+and the first where it failed, so the suspect range is whatever landed between —
+no bisect, no builds.
+
+- **One commit in the range** → that is attribution, and the author is named in
+  the PR comment and the channel notification.
+- **Two to eight** → candidates are listed, nobody is singled out.
+- **More than eight** → not attributed at all.
+
+Naming the wrong author is worse than naming nobody: it burns the one thing the
+callout needs, which is people trusting it enough to look. Merge commits are
+excluded, and only `MAIN_REGRESSION` clusters are blamed — attributing a flake to
+a commit is a false accusation.
+
+## Metrics
+
+Every verdict is recorded in the TSIO ledger. `GET /api/v1/triage/accuracy`
+returns `false_greens` — waived verdicts a human later reclassified as a real
+bug. **That number decides whether this system is allowed to gate anything.** It
+must be zero.
+
+Human corrections come from `/e2e-triage-override <verdict> <reason>` on the PR
+and are the only ground truth available; recurring ones should become signature
+entries in the caller's catalogue, which shrinks the model's share of the work
+over time.
+
+## Testing
+
+```bash
+node --test scripts/triage-policy.test.js scripts/triage-apply.test.js \
+  scripts/triage-override.test.js scripts/triage-blame.test.js
+```