-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Onboard the Build Failure Analysis agentic workflow #55539
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
2a58a8c
d5e3d2f
830e840
8ba5507
b7123f8
a374ba6
ccfc1d7
f730952
2932c65
bc719fc
a86c702
358e607
f36b646
57c8e21
14fce28
c267d2a
2503d5a
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,157 @@ | ||
| --- | ||
| name: "Build Failure Analysis (command)" | ||
| description: >- | ||
| Rerun the build-failure analysis on a pull request when a maintainer comments | ||
| `/analyze-build-failure`. Same body as `build-failure-analysis.md` — it does | ||
| NOT rebuild: it inspects the PR's **latest** Azure Pipelines `dotnet-sdk-public-ci` | ||
| build and, **only when that latest build has failed** (it stops if the | ||
| newest build is still running or has succeeded), downloads the binary logs | ||
| that build already produced (all build legs) and delegates to the | ||
| `build-failure-analyst` agent (which queries the binlogs live via the | ||
| containerized `binlog-mcp` MCP server). Useful when a previous run was | ||
| cancelled, the analysis comment was dismissed, or the agent needs another | ||
| pass. Like the auto workflow it performs **no build**; the generated jobs do | ||
| check out the repository (and, for the slash-command event, the PR branch) | ||
| for agent tooling only — the PR's code is never built or executed. | ||
|
|
||
| on: | ||
| slash_command: | ||
| name: analyze-build-failure | ||
| events: [pull_request_comment] | ||
| roles: [admin, maintainer, write] | ||
|
Copilot marked this conversation as resolved.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. RE your question about
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks — and the exposure is narrower than It fires on The on-demand path ( |
||
| reaction: "eyes" | ||
| # Gate the AI pipeline on the fetch job so the agent only runs when a binlog | ||
| # was actually retrieved from a failed Azure DevOps build. | ||
| needs: [fetch-binlog] | ||
|
|
||
| # Skip activation (and the agent) unless a binlog was retrieved — e.g. if the | ||
| # PR's latest Azure DevOps build did not fail, or the PR is out of scope. | ||
| if: needs.fetch-binlog.outputs.binlog-found == 'true' | ||
|
|
||
| # Least-privilege for the workflow/agent jobs. The agent runs read-only; it | ||
| # does NOT post directly. All PR writes it produces (summary comment + inline | ||
| # review suggestions) go through gh-aw **safe-outputs**, which the compiler | ||
| # emits as a separate `safe_outputs` job granted `pull-requests: write` + | ||
| # `issues: write` in the generated lock. (The slash-command trigger also adds | ||
| # an acknowledgement reaction to the command comment; gh-aw emits that in its | ||
| # own generated job with the scope it needs — it is not driven by this agent | ||
| # job.) Keep `pull-requests: read` here so the AI agent job stays | ||
| # least-privilege — do NOT raise it to `write`, that would hand PR-write scope | ||
| # to the agent job unnecessarily. | ||
| permissions: | ||
| contents: read | ||
| pull-requests: read | ||
| copilot-requests: write | ||
|
|
||
| concurrency: | ||
| # Distinct from the automatic workflow's group (`build-failure-analysis-<pr>`). | ||
| # Concurrency groups are repository-global, so sharing the name made the two | ||
| # workflows cancel each other for the same PR: a newly failing build would | ||
| # kill an on-demand analysis a maintainer had just asked for. Each still | ||
| # collapses its own repeat invocations for a PR. | ||
| group: build-failure-analysis-cmd-${{ github.event.issue.number || github.event.pull_request.number || fromJSON(github.event.inputs.aw_context || github.event.client_payload.aw_context || '{}').item_number || github.run_id }} | ||
| cancel-in-progress: true | ||
|
|
||
| timeout-minutes: 30 | ||
|
|
||
|
|
||
| # Narrow the safe-output target for the on-demand path. | ||
| # | ||
| # The rest of `safe-outputs` comes from shared/build-failure-analysis-shared.md; | ||
| # gh-aw lets the main workflow override an individual safe-output type, and only | ||
| # `target` is intended to differ here. Unlike the automatic (check_run/dispatch) | ||
| # workflow, this one HAS a triggering item — the PR the command was typed on — | ||
| # and `fetch-binlog` resolves that very same PR from `github.event.issue.number`, | ||
| # so `triggering` is equivalent by construction while removing the agent's | ||
| # ability to name a different issue/PR. `"*"` stays unavoidable on the automatic | ||
| # path, which has no triggering item; see the note in the shared file. | ||
| # | ||
| # KEEP IN SYNC with `safe-outputs` in shared/build-failure-analysis-shared.md: | ||
| # overriding a type replaces it wholesale, so `max` and `hide-older-comments` | ||
| # are restated verbatim and must not be allowed to drift. | ||
| safe-outputs: | ||
| add-comment: | ||
| max: 5 | ||
| target: "triggering" | ||
| hide-older-comments: | ||
| enabled: true | ||
| match: | ||
| - build-failure-analysis | ||
| - build-failure-analysis-command | ||
| create-pull-request-review-comment: | ||
| max: 25 | ||
| target: "triggering" | ||
|
|
||
| imports: | ||
| - uses: shared/pat_pool.md | ||
| with: | ||
| environment: copilot-pat-pool | ||
| - shared/build-failure-analysis-fetch.md | ||
| - shared/build-failure-analysis-shared.md | ||
|
|
||
| environment: copilot-pat-pool | ||
|
|
||
| engine: | ||
| id: copilot | ||
| env: | ||
| COPILOT_GITHUB_TOKEN: ${{ case(needs.pat_pool.outputs.pat_number == '0', secrets.COPILOT_PAT_0, needs.pat_pool.outputs.pat_number == '1', secrets.COPILOT_PAT_1, needs.pat_pool.outputs.pat_number == '2', secrets.COPILOT_PAT_2, needs.pat_pool.outputs.pat_number == '3', secrets.COPILOT_PAT_3, needs.pat_pool.outputs.pat_number == '4', secrets.COPILOT_PAT_4, needs.pat_pool.outputs.pat_number == '5', secrets.COPILOT_PAT_5, needs.pat_pool.outputs.pat_number == '6', secrets.COPILOT_PAT_6, needs.pat_pool.outputs.pat_number == '7', secrets.COPILOT_PAT_7, needs.pat_pool.outputs.pat_number == '8', secrets.COPILOT_PAT_8, needs.pat_pool.outputs.pat_number == '9', secrets.COPILOT_PAT_9, 'NO COPILOT PAT AVAILABLE') }} | ||
|
|
||
|
|
||
| # Custom job that reuses the binlogs from the PR's most recent failed Azure | ||
| # DevOps `dotnet-sdk-public-ci` build instead of rebuilding. Mirrors the fetch-binlog job | ||
| # in build-failure-analysis.md; it locates the build by the PR's merge branch | ||
| # (no `check_run` payload is available on a slash command). | ||
| # Steps that run in the agent job. The top-level `if:` gates these on binlogs | ||
| # having been retrieved, so the agent never runs without something to analyse. | ||
| steps: | ||
| - name: Download analysis artifact | ||
| uses: actions/download-artifact@v8.0.1 | ||
| with: | ||
| name: build-failure-analysis-data | ||
| path: /tmp/binlogs | ||
|
|
||
| - name: Export agent context | ||
| shell: bash | ||
| env: | ||
| GH_AW_BINLOG_FOUND_VALUE: ${{ needs.fetch-binlog.outputs.binlog-found }} | ||
| GH_AW_PR_NUMBER_VALUE: ${{ needs.fetch-binlog.outputs.pr-number }} | ||
| GH_AW_PR_HEAD_SHA_VALUE: ${{ needs.fetch-binlog.outputs.pr-head-sha }} | ||
| GH_AW_PR_MERGE_SHA_VALUE: ${{ needs.fetch-binlog.outputs.pr-merge-sha }} | ||
| GH_AW_ADO_BUILD_URL_VALUE: ${{ needs.fetch-binlog.outputs.ado-build-url }} | ||
| GH_AW_MISSING_LEGS_VALUE: ${{ needs.fetch-binlog.outputs.missing-legs }} | ||
| GH_AW_GITHUB_WORKSPACE: ${{ github.workspace }} | ||
| run: | | ||
| # See build-failure-analysis.md for the binlog path conventions. The | ||
| # per-leg binlogs are read through the binlog-mcp MCP server (mounted at | ||
| # `/data/binlogs`); GH_AW_BINLOG_HOST_PATH points at the Azure DevOps | ||
| # build for human-facing references. | ||
| BINLOG_DIR="/data/binlogs" | ||
| LIST="" | ||
| if [ "${GH_AW_BINLOG_FOUND_VALUE:-false}" = "true" ] && [ -d /tmp/binlogs ]; then | ||
| for f in /tmp/binlogs/*.binlog; do | ||
| [ -f "$f" ] || continue | ||
| LIST="${LIST}${BINLOG_DIR}/$(basename "$f")"$'\n' | ||
| done | ||
| fi | ||
| FIRST=$(printf '%s' "$LIST" | head -1) | ||
| { | ||
| echo "GH_AW_BUILD_OUTCOME=failure" | ||
| echo "GH_AW_BINLOG_DIR=${BINLOG_DIR}" | ||
| echo "GH_AW_BINLOG_PATH=${FIRST}" | ||
| echo "GH_AW_BINLOG_HOST_PATH=${GH_AW_ADO_BUILD_URL_VALUE}" | ||
| echo "GH_AW_PR_NUMBER=${GH_AW_PR_NUMBER_VALUE}" | ||
| echo "GH_AW_PR_HEAD_SHA=${GH_AW_PR_HEAD_SHA_VALUE}" | ||
| echo "GH_AW_PR_MERGE_SHA=${GH_AW_PR_MERGE_SHA_VALUE}" | ||
| echo "GH_AW_MISSING_LEGS=${GH_AW_MISSING_LEGS_VALUE}" | ||
| echo "GH_AW_WORKSPACE=${GH_AW_GITHUB_WORKSPACE}" | ||
| echo "GH_AW_BINLOG_LIST<<GH_AW_EOF" | ||
| printf '%s' "$LIST" | ||
| echo "GH_AW_EOF" | ||
| } >> "$GITHUB_ENV" | ||
|
|
||
|
|
||
| --- | ||
|
|
||
| <!-- | ||
| Body provided by shared/build-failure-analysis-shared.md. | ||
| --> | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why was the command and analysis file put into separate workflows? A workflow can have multiple triggers. If this was for different security contexts I think that's more interesting but I'd try to consolidate if at all possible.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You guessed right — it is the different security contexts, and it turns out gh-aw makes consolidation impossible rather than just awkward.
roles:is workflow-scoped, not per-trigger. In the schema it is a sibling of the trigger keys inside the singleon:object (pkg/parser/schemas/main_workflow_schema.json), and the compiler reads it exactly once:pkg/workflow/role_checks.gopullsfrontmatter["on"]["roles"]into a single[]string, emits it as oneGH_AW_REQUIRED_ROLESenv var, and generates onepre_activationjob for the entire workflow (compiler_pre_activation_job.go). There is no syntax for attaching a different role set to a different trigger.Combining the triggers would be fine —
slash_commandcoexists happily withcheck_run/workflow_dispatch, nothing in the validator rejects it. It is only the security context that cannot be shared:build-failure-analysischeck_run(automatic)allbuild-failure-analysis-command/build-failure-analysiscomment[admin, maintainer, write]Merging them forces one of two bad outcomes: either external contributors lose automatic analysis, or any commenter can trigger a 600 MB download. So they stay split.
The duplication that split caused was a real problem though, and that part is now fixed — see my reply on the review body.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a great improvement. I do wonder if workflow dispatch could create the security scope changes for us and then dispatch the same build failure .md out, though I'm unsure whether this would work.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The mechanism works. The docs are explicit:
So a command workflow really could authorize, then
gh workflow run build-failure-analysis.lock.yml -f pr-number=Nwith the plainGITHUB_TOKEN. The automatic workflow already exposes exactly those inputs (pr-number,ado-build-id) — that's how I've been fork-testing it.Why I don't think it's the right trade here. The command file has to keep existing regardless, because it's what hosts the
slash_commandtrigger androles:. If its only job becomes "authorize and dispatch", it can't be a gh-aw workflow anymore — gh-aw always compiles an agent job — so it becomes plain YAML, and I'd have to hand-write both the role check and the command-position parsing thatpre_activationgenerates for me today. That's the exact hand-written-authorization surface that just got picked apart in review on the arcade copy of this (dotnet/arcade#17228, where arole_namecheck I'd written turned out to widen the gate). Trading compiler-generated authorization for more hand-written authorization seems like the wrong direction. It also costs a second run per invocation, and the analysis run is no longer linked to the comment that asked for it.But the duplication you're pointing at was real, and I've now removed most of it (
14fce28).I diffed the two frontmatters key by key. Four blocks were byte-identical and in gh-aw's importable field set:
network,mcp-servers,tools,safe-outputs. Those now live inshared/build-failure-analysis-shared.md, which both workflows already imported. Net -60 lines.What genuinely cannot be shared, and why:
on:/roles:/concurrency:permissions:engine:/environment:/timeout-minutes:/steps:Verified this is a pure refactor: both regenerated
.lock.ymlfiles are byte-identical to the previous ones apart from thefrontmatter_hashin the metadata comment (normalized that field and diffed — 0 differing lines in each).gh aw compile --strictv0.82.9, 0 errors / 0 warnings.So the two files are now down to their genuinely per-trigger parts. Happy to revisit the dispatch shape if gh-aw ever grows per-trigger
roles:— that's the thing that would actually collapse them into one file.