Skip to content

Commit db76ce2

Browse files
sarthakaggarwal97sarthakaggarwal97
authored
Add ci_fix: AI-assisted CI test-failure fixes via @valkeyrie-bot (#33)
A maintainer comments `@valkeyrie-bot fix <ci-run-url>` on a backport PR. The bot downloads the failing run's logs, diagnoses the failure, applies a fix, builds and runs the failing test to confirm it passes, has a second agent review the diff, then pushes to the PR branch and comments with the evidence. Every step fails closed and posts a comment explaining why. Pushes are gated: commenter must be in valkey-io/contributors, the run must be a completed failure whose SHA still matches the PR head, and the target must be an agent/backport/ branch on the PR's own head repo. The push runs from a fresh clone at the gated SHA so untrusted test code never touches credentials. The verification command runs with no GitHub token or AWS credentials in its environment. Shared helpers (run_git, filter_env, extract_json_object) are consolidated in scripts/common and reused by the backport and fuzzer paths. Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>
1 parent 20b6276 commit db76ce2

44 files changed

Lines changed: 5533 additions & 113 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/ci-fix-verify-macos.yml‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
name: CI Fix Verify macOS
2+
3+
# Verifies a ci_fix candidate patch on a real macOS runner. The main agent
4+
# dispatches this with the target repo, the PR head SHA, the approved patch
5+
# (base64), and the targeted verification command. The job checks out the
6+
# target repo at the head SHA, applies the patch, and runs the command. Its
7+
# conclusion (success/failure) is the verdict; the agent reads it and only then
8+
# decides whether to push. The correlation token is placed in the run name so
9+
# the agent can match this exact run.
10+
#
11+
# No secrets are used and permissions are minimal: this job only reads a public
12+
# repo and runs a build. It never pushes.
13+
14+
run-name: "verify-macos [token:${{ inputs.correlation }}]"
15+
16+
on:
17+
workflow_dispatch:
18+
inputs:
19+
target_repo:
20+
description: "Target repo (owner/name) to verify"
21+
required: true
22+
type: string
23+
head_sha:
24+
description: "Commit SHA to check out and apply the patch onto"
25+
required: true
26+
type: string
27+
patch_b64:
28+
description: "base64-encoded patch to apply"
29+
required: true
30+
type: string
31+
verify_command:
32+
description: "Targeted verification command to run"
33+
required: true
34+
type: string
35+
workdir:
36+
description: "Relative working directory for the command"
37+
required: false
38+
type: string
39+
default: ""
40+
correlation:
41+
description: "Correlation token (echoed in the run name)"
42+
required: true
43+
type: string
44+
45+
permissions: {}
46+
47+
jobs:
48+
verify-macos:
49+
runs-on: macos-latest
50+
timeout-minutes: 90
51+
steps:
52+
- name: Validate inputs
53+
env:
54+
TARGET_REPO: ${{ inputs.target_repo }}
55+
HEAD_SHA: ${{ inputs.head_sha }}
56+
PATCH_B64: ${{ inputs.patch_b64 }}
57+
VERIFY_WORKDIR: ${{ inputs.workdir }}
58+
run: |
59+
set -euo pipefail
60+
if [[ ! "${TARGET_REPO}" =~ ^[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+$ ]]; then
61+
echo "Invalid target_repo" >&2; exit 1
62+
fi
63+
if [[ ! "${HEAD_SHA}" =~ ^[0-9a-fA-F]{7,40}$ ]]; then
64+
echo "Invalid head_sha" >&2; exit 1
65+
fi
66+
if [[ -z "${PATCH_B64}" || ! "${PATCH_B64}" =~ ^[A-Za-z0-9+/]+={0,2}$ ]]; then
67+
echo "Invalid patch_b64" >&2; exit 1
68+
fi
69+
if [[ "${VERIFY_WORKDIR}" == *..* || "${VERIFY_WORKDIR}" == /* || "${VERIFY_WORKDIR}" == *$'\n'* ]]; then
70+
echo "Invalid workdir" >&2; exit 1
71+
fi
72+
73+
- name: Check out target repo at head SHA
74+
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.0
75+
with:
76+
repository: ${{ inputs.target_repo }}
77+
ref: ${{ inputs.head_sha }}
78+
persist-credentials: false
79+
path: target
80+
81+
- name: Apply candidate patch
82+
working-directory: target
83+
env:
84+
PATCH_B64: ${{ inputs.patch_b64 }}
85+
run: |
86+
set -euo pipefail
87+
printf '%s' "${PATCH_B64}" | base64 --decode > "${RUNNER_TEMP}/ci-fix.patch"
88+
git -c core.hooksPath=/dev/null apply --index --whitespace=nowarn "${RUNNER_TEMP}/ci-fix.patch"
89+
90+
- name: Run targeted verification
91+
working-directory: target
92+
env:
93+
VERIFY_COMMAND: ${{ inputs.verify_command }}
94+
VERIFY_WORKDIR: ${{ inputs.workdir }}
95+
run: |
96+
set -euo pipefail
97+
cd "${VERIFY_WORKDIR:-.}"
98+
/bin/sh -c "${VERIFY_COMMAND}"

‎.github/workflows/ci-fix.yml‎

Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
name: CI Fix Bot
2+
3+
# A maintainer triggers a fix manually via workflow_dispatch (the Actions UI or
4+
# `gh workflow run`), supplying the PR and the failing run URL. A comment-based
5+
# trigger (`@valkeyrie-bot fix ...` on a valkey-io/valkey PR) needs a thin
6+
# wrapper in the target repo to forward the event here, since issue_comment
7+
# events only fire in the repo where the comment is made; that wrapper is a
8+
# follow-up. See README for the dispatch invocation.
9+
on:
10+
workflow_dispatch:
11+
inputs:
12+
repo:
13+
description: "PR repository, e.g. valkey-io/valkey"
14+
required: true
15+
type: string
16+
pr:
17+
description: "Backport PR number"
18+
required: true
19+
type: number
20+
run_url:
21+
description: "Failed GitHub Actions run URL"
22+
required: true
23+
type: string
24+
hint:
25+
description: "Optional diagnosis hint"
26+
required: false
27+
type: string
28+
29+
permissions: {}
30+
31+
env:
32+
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true
33+
34+
jobs:
35+
ci-fix:
36+
runs-on: ubuntu-latest
37+
timeout-minutes: 90
38+
permissions:
39+
contents: read
40+
id-token: write
41+
concurrency:
42+
# Serialize ci-fix runs for the same PR so two dispatches don't push at
43+
# once. This does not coordinate with other workflows that write the same
44+
# branch; the fast-forward-only push (see push.py) is what keeps a
45+
# concurrent branch update from being clobbered.
46+
group: ci-fix-${{ inputs.pr }}
47+
cancel-in-progress: false
48+
env:
49+
AWS_REGION: ${{ vars.AWS_REGION || 'us-east-1' }}
50+
CLAUDE_CODE_USE_BEDROCK: "1"
51+
CI_AGENT_EVIDENCE_DIR: agent-evidence
52+
steps:
53+
- name: Check out agent repository
54+
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.0
55+
with:
56+
persist-credentials: false
57+
fetch-depth: 1
58+
59+
- name: Set up Python 3.11
60+
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.0.0
61+
with:
62+
python-version: "3.11"
63+
cache: "pip"
64+
65+
- name: Install dependencies
66+
run: |
67+
pip install -r requirements.txt
68+
npm install -g @anthropic-ai/claude-code@2.1.153
69+
70+
- name: Generate GitHub App token
71+
id: generate-token
72+
uses: actions/create-github-app-token@d72941d797fd3113feb6b93fd0dec494b13a2547 # v1.12.0
73+
with:
74+
app-id: ${{ secrets.VALKEYRIE_BOT_APP_ID }}
75+
private-key: ${{ secrets.VALKEYRIE_BOT_PRIVATE_KEY }}
76+
owner: valkey-io
77+
repositories: valkey
78+
# members:read for the contributors-team auth check; contents:write
79+
# to push the fix; pull-requests:write + issues:write to comment (the
80+
# PR comment uses the Issues comments API); actions:read for logs.
81+
permission-members: read
82+
permission-actions: read
83+
permission-contents: write
84+
permission-pull-requests: write
85+
permission-issues: write
86+
permission-metadata: read
87+
88+
- name: Generate agent-repo token (macOS verify dispatch)
89+
id: agent-token
90+
uses: actions/create-github-app-token@d72941d797fd3113feb6b93fd0dec494b13a2547 # v1.12.0
91+
with:
92+
app-id: ${{ secrets.VALKEYRIE_BOT_APP_ID }}
93+
private-key: ${{ secrets.VALKEYRIE_BOT_PRIVATE_KEY }}
94+
owner: valkey-io
95+
repositories: valkey-ci-agent
96+
# actions:write to dispatch the verify-macos workflow and read its
97+
# run status; nothing else on the agent repo.
98+
permission-actions: write
99+
permission-metadata: read
100+
101+
- name: Configure AWS credentials
102+
uses: aws-actions/configure-aws-credentials@61815dcd50bd041e203e49132bacad1fd04d2708 # v5.1.1
103+
with:
104+
role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
105+
role-session-name: valkey-ci-fix-${{ github.run_id }}
106+
aws-region: ${{ env.AWS_REGION }}
107+
108+
- name: Run the CI fix
109+
shell: bash
110+
env:
111+
TARGET_TOKEN: ${{ steps.generate-token.outputs.token }}
112+
CI_FIX_MACOS_AGENT_REPO: valkey-io/valkey-ci-agent
113+
CI_FIX_MACOS_AGENT_REF: main
114+
CI_FIX_MACOS_TOKEN: ${{ steps.agent-token.outputs.token }}
115+
CI_FIX_REPO: ${{ inputs.repo }}
116+
CI_FIX_PR: ${{ inputs.pr }}
117+
CI_FIX_RUN_URL: ${{ inputs.run_url }}
118+
CI_FIX_COMMENTER: ${{ github.actor }}
119+
CI_FIX_HINT: ${{ inputs.hint || '' }}
120+
run: |
121+
set -euo pipefail
122+
if [[ ! "${CI_FIX_REPO}" =~ ^[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+$ ]]; then
123+
echo "Invalid repo: ${CI_FIX_REPO}" >&2
124+
exit 1
125+
fi
126+
if [[ "${CI_FIX_REPO}" != "valkey-io/valkey" ]]; then
127+
echo "ci-fix.yml is currently scoped to valkey-io/valkey (got: ${CI_FIX_REPO})" >&2
128+
exit 1
129+
fi
130+
if [[ ! "${CI_FIX_PR}" =~ ^[0-9]+$ ]]; then
131+
echo "Invalid PR number: ${CI_FIX_PR}" >&2
132+
exit 1
133+
fi
134+
args=(
135+
-m scripts.ci_fix.main
136+
--repo "${CI_FIX_REPO}"
137+
--pr "${CI_FIX_PR}"
138+
--run-url "${CI_FIX_RUN_URL}"
139+
--commenter "${CI_FIX_COMMENTER}"
140+
)
141+
if [[ -n "${CI_FIX_HINT}" ]]; then
142+
args+=(--hint "${CI_FIX_HINT}")
143+
fi
144+
python "${args[@]}"
145+
146+
- name: Upload agent evidence
147+
if: always()
148+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
149+
with:
150+
name: ci-fix-agent-evidence-${{ github.run_id }}
151+
path: agent-evidence
152+
if-no-files-found: ignore
153+
retention-days: 30

0 commit comments

Comments
 (0)