Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
2a58a8c
Onboard the Build Failure Analysis agentic workflow
YuliiaKovalova Jul 30, 2026
d5e3d2f
Make the slash-command workflow self-contained
YuliiaKovalova Jul 30, 2026
830e840
Fix stale artifact-name reference in fetch-binlog comment
YuliiaKovalova Jul 30, 2026
8ba5507
Require a trusted author before fetching binlogs on the slash command
YuliiaKovalova Jul 30, 2026
b7123f8
Detect failed legs that published no logs, and bound the download
YuliiaKovalova Jul 31, 2026
a374ba6
Support the release/* artifact layout so servicing PRs are analyzed
YuliiaKovalova Jul 31, 2026
ccfc1d7
Resolve the commenter's real repository permission before downloading…
YuliiaKovalova Jul 31, 2026
f730952
Accept maintain/custom roles in the permission pre-gate
YuliiaKovalova Jul 31, 2026
2932c65
Drop a stale duplicated artifact-naming comment
YuliiaKovalova Jul 31, 2026
bc719fc
Report an unreadable build timeline as unknown, not as complete
YuliiaKovalova Jul 31, 2026
a86c702
Ask the timeline whether each leg published logs, and fix the downloa…
YuliiaKovalova Jul 31, 2026
358e607
Share one fetch-binlog job between the two analysis workflows
Aug 3, 2026
f36b646
Re-read the PR head before the pre-download staleness check
YuliiaKovalova Aug 3, 2026
57c8e21
Restrict the binlog pre-gate to `.permission` and separate the concur…
YuliiaKovalova Aug 3, 2026
14fce28
Move the shared agent configuration into the shared import
YuliiaKovalova Aug 4, 2026
c267d2a
Address the second Evangelink review round on the build-failure workf…
YuliiaKovalova Aug 5, 2026
2503d5a
Merge branch 'main' into onboard-build-failure-analysis
YuliiaKovalova Aug 5, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
214 changes: 214 additions & 0 deletions .github/agents/build-failure-analyst.agent.md

Large diffs are not rendered by default.

2,530 changes: 2,530 additions & 0 deletions .github/workflows/build-failure-analysis-command.lock.yml

Large diffs are not rendered by default.

157 changes: 157 additions & 0 deletions .github/workflows/build-failure-analysis-command.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,157 @@
---
name: "Build Failure Analysis (command)"
description: >-
Rerun the build-failure analysis on a pull request when a maintainer comments

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why was the command and analysis file put into separate workflows? A workflow can have multiple triggers. If this was for different security contexts I think that's more interesting but I'd try to consolidate if at all possible.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You guessed right — it is the different security contexts, and it turns out gh-aw makes consolidation impossible rather than just awkward.

roles: is workflow-scoped, not per-trigger. In the schema it is a sibling of the trigger keys inside the single on: object (pkg/parser/schemas/main_workflow_schema.json), and the compiler reads it exactly once: pkg/workflow/role_checks.go pulls frontmatter["on"]["roles"] into a single []string, emits it as one GH_AW_REQUIRED_ROLES env var, and generates one pre_activation job for the entire workflow (compiler_pre_activation_job.go). There is no syntax for attaching a different role set to a different trigger.

Combining the triggers would be fine — slash_command coexists happily with check_run/workflow_dispatch, nothing in the validator rejects it. It is only the security context that cannot be shared:

workflow trigger roles why
build-failure-analysis check_run (automatic) all must fire on every failing PR, including external contributors'
build-failure-analysis-command /build-failure-analysis comment [admin, maintainer, write] downloads ~600 MB of binlogs on demand

Merging them forces one of two bad outcomes: either external contributors lose automatic analysis, or any commenter can trigger a 600 MB download. So they stay split.

The duplication that split caused was a real problem though, and that part is now fixed — see my reply on the review body.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a great improvement. I do wonder if workflow dispatch could create the security scope changes for us and then dispatch the same build failure .md out, though I'm unsure whether this would work.

@YuliiaKovalova YuliiaKovalova Aug 4, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The mechanism works. The docs are explicit:

When you use the repository's GITHUB_TOKEN to perform tasks, events triggered by the GITHUB_TOKEN will not create a new workflow run, with the following exceptions:

  • workflow_dispatch and repository_dispatch events always create workflow runs.

So a command workflow really could authorize, then gh workflow run build-failure-analysis.lock.yml -f pr-number=N with the plain GITHUB_TOKEN. The automatic workflow already exposes exactly those inputs (pr-number, ado-build-id) — that's how I've been fork-testing it.

Why I don't think it's the right trade here. The command file has to keep existing regardless, because it's what hosts the slash_command trigger and roles:. If its only job becomes "authorize and dispatch", it can't be a gh-aw workflow anymore — gh-aw always compiles an agent job — so it becomes plain YAML, and I'd have to hand-write both the role check and the command-position parsing that pre_activation generates for me today. That's the exact hand-written-authorization surface that just got picked apart in review on the arcade copy of this (dotnet/arcade#17228, where a role_name check I'd written turned out to widen the gate). Trading compiler-generated authorization for more hand-written authorization seems like the wrong direction. It also costs a second run per invocation, and the analysis run is no longer linked to the comment that asked for it.

But the duplication you're pointing at was real, and I've now removed most of it (14fce28).

I diffed the two frontmatters key by key. Four blocks were byte-identical and in gh-aw's importable field set: network, mcp-servers, tools, safe-outputs. Those now live in shared/build-failure-analysis-shared.md, which both workflows already imported. Net -60 lines.

What genuinely cannot be shared, and why:

Field Why it stays per-workflow
on: / roles: / concurrency: the reason the two files exist at all
permissions: imports are validated against the caller, not merged into it — each main workflow must re-declare its own
engine: / environment: / timeout-minutes: / steps: not in gh-aw's importable set; the engine identifier is "always inherited from the importing workflow"

Verified this is a pure refactor: both regenerated .lock.yml files are byte-identical to the previous ones apart from the frontmatter_hash in the metadata comment (normalized that field and diffed — 0 differing lines in each). gh aw compile --strict v0.82.9, 0 errors / 0 warnings.

So the two files are now down to their genuinely per-trigger parts. Happy to revisit the dispatch shape if gh-aw ever grows per-trigger roles: — that's the thing that would actually collapse them into one file.

`/analyze-build-failure`. Same body as `build-failure-analysis.md` — it does
NOT rebuild: it inspects the PR's **latest** Azure Pipelines `dotnet-sdk-public-ci`
build and, **only when that latest build has failed** (it stops if the
newest build is still running or has succeeded), downloads the binary logs
that build already produced (all build legs) and delegates to the
`build-failure-analyst` agent (which queries the binlogs live via the
containerized `binlog-mcp` MCP server). Useful when a previous run was
cancelled, the analysis comment was dismissed, or the agent needs another
pass. Like the auto workflow it performs **no build**; the generated jobs do
check out the repository (and, for the slash-command event, the PR branch)
for agent tooling only — the PR's code is never built or executed.

on:
slash_command:
name: analyze-build-failure
events: [pull_request_comment]
roles: [admin, maintainer, write]
Comment thread
Copilot marked this conversation as resolved.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RE your question about roles: all - as long as we're not allowing folks to trigger the workflow themselves and spam AI token cost on us, I don't see any issue with this. Triage workflows already do this across the org.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — and the exposure is narrower than roles: all makes it look, because this workflow is not comment-triggered at all.

It fires on check_run, and the agent job is gated on needs.fetch-binlog.outputs.binlog-found == 'true'. That output only becomes true when there is a genuinely failed dotnet-sdk-public-ci build (definition 101) for an in-scope PR (base main or release/*) that actually published binlog artifacts, and whose head still matches the PR. Nobody can self-trigger it to burn tokens — you would have to push a commit that really breaks the SDK build.

The on-demand path (/build-failure-analysis), which is comment-triggered and does the expensive fetch, is the one restricted to [admin, maintainer, write] — enforced inside fetch-binlog itself, before the download, since gh-aw's own pre_activation role check runs after it.

reaction: "eyes"
# Gate the AI pipeline on the fetch job so the agent only runs when a binlog
# was actually retrieved from a failed Azure DevOps build.
needs: [fetch-binlog]

# Skip activation (and the agent) unless a binlog was retrieved — e.g. if the
# PR's latest Azure DevOps build did not fail, or the PR is out of scope.
if: needs.fetch-binlog.outputs.binlog-found == 'true'

# Least-privilege for the workflow/agent jobs. The agent runs read-only; it
# does NOT post directly. All PR writes it produces (summary comment + inline
# review suggestions) go through gh-aw **safe-outputs**, which the compiler
# emits as a separate `safe_outputs` job granted `pull-requests: write` +
# `issues: write` in the generated lock. (The slash-command trigger also adds
# an acknowledgement reaction to the command comment; gh-aw emits that in its
# own generated job with the scope it needs — it is not driven by this agent
# job.) Keep `pull-requests: read` here so the AI agent job stays
# least-privilege — do NOT raise it to `write`, that would hand PR-write scope
# to the agent job unnecessarily.
permissions:
contents: read
pull-requests: read
copilot-requests: write

concurrency:
# Distinct from the automatic workflow's group (`build-failure-analysis-<pr>`).
# Concurrency groups are repository-global, so sharing the name made the two
# workflows cancel each other for the same PR: a newly failing build would
# kill an on-demand analysis a maintainer had just asked for. Each still
# collapses its own repeat invocations for a PR.
group: build-failure-analysis-cmd-${{ github.event.issue.number || github.event.pull_request.number || fromJSON(github.event.inputs.aw_context || github.event.client_payload.aw_context || '{}').item_number || github.run_id }}
cancel-in-progress: true

timeout-minutes: 30


# Narrow the safe-output target for the on-demand path.
#
# The rest of `safe-outputs` comes from shared/build-failure-analysis-shared.md;
# gh-aw lets the main workflow override an individual safe-output type, and only
# `target` is intended to differ here. Unlike the automatic (check_run/dispatch)
# workflow, this one HAS a triggering item — the PR the command was typed on —
# and `fetch-binlog` resolves that very same PR from `github.event.issue.number`,
# so `triggering` is equivalent by construction while removing the agent's
# ability to name a different issue/PR. `"*"` stays unavoidable on the automatic
# path, which has no triggering item; see the note in the shared file.
#
# KEEP IN SYNC with `safe-outputs` in shared/build-failure-analysis-shared.md:
# overriding a type replaces it wholesale, so `max` and `hide-older-comments`
# are restated verbatim and must not be allowed to drift.
safe-outputs:
add-comment:
max: 5
target: "triggering"
hide-older-comments:
enabled: true
match:
- build-failure-analysis
- build-failure-analysis-command
create-pull-request-review-comment:
max: 25
target: "triggering"

imports:
- uses: shared/pat_pool.md
with:
environment: copilot-pat-pool
- shared/build-failure-analysis-fetch.md
- shared/build-failure-analysis-shared.md

environment: copilot-pat-pool

engine:
id: copilot
env:
COPILOT_GITHUB_TOKEN: ${{ case(needs.pat_pool.outputs.pat_number == '0', secrets.COPILOT_PAT_0, needs.pat_pool.outputs.pat_number == '1', secrets.COPILOT_PAT_1, needs.pat_pool.outputs.pat_number == '2', secrets.COPILOT_PAT_2, needs.pat_pool.outputs.pat_number == '3', secrets.COPILOT_PAT_3, needs.pat_pool.outputs.pat_number == '4', secrets.COPILOT_PAT_4, needs.pat_pool.outputs.pat_number == '5', secrets.COPILOT_PAT_5, needs.pat_pool.outputs.pat_number == '6', secrets.COPILOT_PAT_6, needs.pat_pool.outputs.pat_number == '7', secrets.COPILOT_PAT_7, needs.pat_pool.outputs.pat_number == '8', secrets.COPILOT_PAT_8, needs.pat_pool.outputs.pat_number == '9', secrets.COPILOT_PAT_9, 'NO COPILOT PAT AVAILABLE') }}


# Custom job that reuses the binlogs from the PR's most recent failed Azure
# DevOps `dotnet-sdk-public-ci` build instead of rebuilding. Mirrors the fetch-binlog job
# in build-failure-analysis.md; it locates the build by the PR's merge branch
# (no `check_run` payload is available on a slash command).
# Steps that run in the agent job. The top-level `if:` gates these on binlogs
# having been retrieved, so the agent never runs without something to analyse.
steps:
- name: Download analysis artifact
uses: actions/download-artifact@v8.0.1
with:
name: build-failure-analysis-data
path: /tmp/binlogs

- name: Export agent context
shell: bash
env:
GH_AW_BINLOG_FOUND_VALUE: ${{ needs.fetch-binlog.outputs.binlog-found }}
GH_AW_PR_NUMBER_VALUE: ${{ needs.fetch-binlog.outputs.pr-number }}
GH_AW_PR_HEAD_SHA_VALUE: ${{ needs.fetch-binlog.outputs.pr-head-sha }}
GH_AW_PR_MERGE_SHA_VALUE: ${{ needs.fetch-binlog.outputs.pr-merge-sha }}
GH_AW_ADO_BUILD_URL_VALUE: ${{ needs.fetch-binlog.outputs.ado-build-url }}
GH_AW_MISSING_LEGS_VALUE: ${{ needs.fetch-binlog.outputs.missing-legs }}
GH_AW_GITHUB_WORKSPACE: ${{ github.workspace }}
run: |
# See build-failure-analysis.md for the binlog path conventions. The
# per-leg binlogs are read through the binlog-mcp MCP server (mounted at
# `/data/binlogs`); GH_AW_BINLOG_HOST_PATH points at the Azure DevOps
# build for human-facing references.
BINLOG_DIR="/data/binlogs"
LIST=""
if [ "${GH_AW_BINLOG_FOUND_VALUE:-false}" = "true" ] && [ -d /tmp/binlogs ]; then
for f in /tmp/binlogs/*.binlog; do
[ -f "$f" ] || continue
LIST="${LIST}${BINLOG_DIR}/$(basename "$f")"$'\n'
done
fi
FIRST=$(printf '%s' "$LIST" | head -1)
{
echo "GH_AW_BUILD_OUTCOME=failure"
echo "GH_AW_BINLOG_DIR=${BINLOG_DIR}"
echo "GH_AW_BINLOG_PATH=${FIRST}"
echo "GH_AW_BINLOG_HOST_PATH=${GH_AW_ADO_BUILD_URL_VALUE}"
echo "GH_AW_PR_NUMBER=${GH_AW_PR_NUMBER_VALUE}"
echo "GH_AW_PR_HEAD_SHA=${GH_AW_PR_HEAD_SHA_VALUE}"
echo "GH_AW_PR_MERGE_SHA=${GH_AW_PR_MERGE_SHA_VALUE}"
echo "GH_AW_MISSING_LEGS=${GH_AW_MISSING_LEGS_VALUE}"
echo "GH_AW_WORKSPACE=${GH_AW_GITHUB_WORKSPACE}"
echo "GH_AW_BINLOG_LIST<<GH_AW_EOF"
printf '%s' "$LIST"
echo "GH_AW_EOF"
} >> "$GITHUB_ENV"


---

<!--
Body provided by shared/build-failure-analysis-shared.md.
-->
Loading
Loading