Weekly Eval #7
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Copyright(C) 2025-2026 Advanced Micro Devices, Inc. All rights reserved. | |
| # SPDX-License-Identifier: MIT | |
| # | |
| # Weekly eval sweep (#1964). Runs the email eval suite (triage + drafting + | |
| # action-item) once a week on the self-hosted Lemonade runner and OPENS A GITHUB | |
| # ISSUE if the eval does not pass, so a regression that the report-mode run only | |
| # logs is escalated into the tracker once a week instead of scrolling past. | |
| # | |
| # This calls the SAME reusable workflow the nightly and the release gates use | |
| # (test_email_agent_eval.yml), so there is one eval definition, not a fork. That | |
| # callee owns the `lemonade-eval` concurrency group, which is what keeps every | |
| # eval serial on the single Lemonade slot — this caller MUST NOT also claim it | |
| # (see the concurrency block below). | |
| # | |
| # A non-pass here is always a real signal, never a silent skip: the action-item | |
| # eval REQUIRES the Claude judge (no fuzzy fallback), so a missing/broken judge, a | |
| # Lemonade/infra error, or an enforced (enforce:true) gate breach all fail the job. | |
| # A CANCELLED callee counts too — see the `if:` on the notify job. | |
| name: Weekly Eval | |
| on: | |
| # Sundays 08:17 UTC — off the nightly's 07:00 slot; the callee's concurrency | |
| # group serializes them if they ever overlap. | |
| schedule: | |
| - cron: '17 8 * * 0' | |
| workflow_dispatch: | |
| concurrency: | |
| # MUST NOT be `lemonade-eval`. A called reusable workflow's concurrency is | |
| # evaluated in the caller's context, so if this caller held the same group the | |
| # callee could never acquire it — it would die before a job record was ever | |
| # created, and `needs.email-eval.result` would report 'failure' for an eval | |
| # that never ran (the phantom-job bug: every Weekly Eval run 2026-07-09..19 | |
| # ended in <60s with only the notify job, filing issue #2026 on a no-op). | |
| # Serialization on the Lemonade slot belongs to the callee, which owns it. | |
| group: weekly-eval | |
| cancel-in-progress: false | |
| permissions: | |
| contents: read | |
| jobs: | |
| # Reuse the one eval definition. secrets: inherit passes ANTHROPIC_API_KEY | |
| # (REQUIRED by the action-item judge — the suite fails loudly without it). | |
| email-eval: | |
| name: Weekly email eval suite | |
| uses: ./.github/workflows/test_email_agent_eval.yml | |
| secrets: inherit | |
| open-issue-on-failure: | |
| name: Open an issue if the weekly eval did not pass | |
| needs: email-eval | |
| # always() so this runs even though the needed job failed; then gate on the | |
| # result being anything other than a clean pass. | |
| # | |
| # `!= 'success'` rather than `== 'failure'`: the callee's job now shares the | |
| # `lemonade-eval` group with PR-triggered runs (test_email_agent_eval.yml | |
| # gained a `pull_request` trigger), and that group holds one running plus one | |
| # pending run — a newly-queued run cancels the pending one. An evicted weekly | |
| # eval reports 'cancelled', not 'failure', so the old condition filed nothing | |
| # and the week's eval vanished with zero signal. A week with no eval is a week | |
| # with no evidence; say so out loud. | |
| if: ${{ always() && needs.email-eval.result != 'success' }} | |
| runs-on: ubuntu-latest | |
| permissions: | |
| contents: read | |
| issues: write | |
| steps: | |
| - name: Checkout | |
| uses: actions/checkout@v7 | |
| - name: File (or update) the rolling eval-failure issue | |
| env: | |
| GH_TOKEN: ${{ github.token }} | |
| RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} | |
| EVENT_NAME: ${{ github.event_name }} | |
| EVAL_RESULT: ${{ needs.email-eval.result }} | |
| run: | | |
| set -euo pipefail | |
| # STABLE DEDUP KEY — do not reword. The rolling-issue lookup below | |
| # searches on this exact title; changing it orphans the open issue and | |
| # files a duplicate. The body, not the title, says what actually happened. | |
| TITLE="Weekly eval failure: email eval suite" | |
| # Idempotent label so --label never errors (create-or-update). | |
| gh label create eval-failure --color B60205 \ | |
| --description "Automated weekly eval failure" --force | |
| BODY="$(cat <<EOF | |
| The weekly email eval suite (\`test_email_agent_eval.yml\`, called by \`weekly_eval.yml\`) did not pass. | |
| - Run: ${RUN_URL} | |
| - Trigger: ${EVENT_NAME} | |
| - Result: \`${EVAL_RESULT}\` | |
| If the result is \`failure\`: the email eval REQUIRES the Claude equivalence judge (there is no fuzzy fallback), | |
| so it is one of a Lemonade / infra error, the Claude judge being unavailable or broken, or an enforced | |
| (\`enforce: true\`) gate breach. Open the run log above to see which stage failed (triage / drafting / | |
| action-item) and why. | |
| If the result is \`cancelled\`: the run was most likely evicted from the shared \`lemonade-eval\` concurrency | |
| slot — it holds one running plus one pending run, and a newly-queued run cancels the pending one (PR-triggered | |
| email evals now queue there too). No eval ran this week; re-dispatch \`weekly_eval.yml\` when the slot is free. | |
| EOF | |
| )" | |
| # One rolling issue: comment on the open one if it exists, else open it. | |
| EXISTING="$(gh issue list --state open --label eval-failure \ | |
| --search "${TITLE} in:title" --json number --jq '.[0].number // empty')" | |
| if [ -n "${EXISTING}" ]; then | |
| gh issue comment "${EXISTING}" --body "${BODY}" | |
| echo "commented on existing issue #${EXISTING}" | |
| else | |
| gh issue create --title "${TITLE}" --body "${BODY}" --label eval-failure | |
| fi |