fix(ci): post PR reviews from the workflow instead of hoping the model does (#2719)

The Claude reviewer has posted **nothing since 2026-07-27 08:34 UTC**
while every `pr-review` job reported success — 84 PRs opened in that
window, **50 already merged unreviewed**, each having burned ~\$1.80 of
Opus writing a review no one ever saw. The cause is that posting was
never part of the harness: the prompt asked the model to finish by
running `gh pr comment` itself, and when that last call stops happening
the job still exits clean, because the only health check asks "did
Claude error?" and never "did a comment appear?". After this change
Claude writes the review to a file and the workflow posts it, and a
review that fails to materialise turns the job red instead of green.

The trigger was #2506 bumping `claude-code-action` v1.0.178 → v1.0.183
(the last review landed 4m25s before it merged; nothing else touched
review CI in that window). That bump carries no comment-posting changes
of its own — only a bundled runtime bump — which is exactly why a
harness that depends on the model remembering a shell command should not
be restored as-is.

Scoped deliberately: posting is gated on the new `comment_file` input,
so `issue-handler`, `pr-comment` and `auto-fix` keep their existing
in-prompt posting and cannot double-post. `pr-rereview` opts out of
`require_comment` because `REVIEW.md` makes silence correct there.

> [!IMPORTANT]
> `pull_request_target` reads workflows from the **base** branch, so
this PR is still reviewed by the old, broken `main` copy — its own
silence proves nothing. The fix only takes effect once merged.

## Test plan

- [x] `pytest tests/unit/test_claude_review_posting.py` — 6 passed.
Locks the invariant: neither review prompt may hand the model a `gh …
comment … --body-file` line, and both jobs must set `comment_file`.
- [x] Posting step exercised locally across all six branches: review
written → posts; missing file + `require_comment` → **exit 1**; empty
file + `require_comment` → **exit 1**; missing file without it → clean
no-op; issue event → `gh issue comment`.
- [x] Both workflows parse (`yaml.safe_load`).
- [ ] After merge: close/reopen one PR and confirm a review comment
appears, and that the run is red if it doesn't.
- [ ] After merge: backfill reviews on the ~32 still-open unreviewed
PRs.

---------

Co-authored-by: Ovtcharov <kovtchar@amd.com>

Author: kovtcharov

.github/workflows/claude-run.yml

@@ -12,7 +12,11 @@
 #      errored) writes execution_file and is NOT retried. The retried case is thus a
 #      hard crash with no output, overwhelmingly the pre-Claude install ENOENT (no
 #      comment posted). See the attempt-2 step for the one vanishingly-rare residual.
-#   3. artifact-based verification: the action only sets execution_file once Claude
+#   3. posting: when a caller passes `comment_file`, Claude WRITES its comment to that
+#      path and a workflow step POSTS it. Callers must not ask the model to run
+#      `gh pr comment` itself — a skipped final tool call is invisible (green job, no
+#      comment). With `require_comment: true` an absent file FAILS the job instead.
+#   4. artifact-based verification: the action only sets execution_file once Claude
 #      ran to completion, and is_error flags a mid-flight error. Either gap emits a
 #      loud ::warning:: (the step outcome is unreliable — continue-on-error masks it
 #      and an install crash exits before outcome is set).
@@ -58,6 +62,16 @@ on:
         required: false
         type: boolean
         default: false
+      comment_file:
+        description: "Path Claude writes its comment to (e.g. /tmp/review.md). When set, a workflow step posts it — the model must NOT post it itself. Empty keeps the legacy in-prompt `gh` posting."
+        required: false
+        type: string
+        default: ""
+      require_comment:
+        description: "With comment_file set: true fails the job when Claude produced no comment (pr-review — a review is mandatory); false treats an absent file as legitimate silence (pr-rereview)."
+        required: false
+        type: boolean
+        default: false
 
 jobs:
   run:
@@ -345,6 +359,35 @@ jobs:
           prompt: ${{ inputs.prompt }}
           claude_args: ${{ inputs.claude_args }}
 
+      # --- Post the comment Claude wrote ---
+      # Posting must not be the model's own final `gh` call — when that call doesn't
+      # happen the job still goes green with nothing published, which is undetectable.
+      - name: Post Claude's comment
+        if: "!cancelled() && inputs.comment_file != ''"
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          COMMENT_FILE: ${{ inputs.comment_file }}
+          REQUIRE_COMMENT: ${{ inputs.require_comment }}
+          IS_PR: ${{ github.event_name == 'pull_request_target' || github.event_name == 'pull_request_review_comment' }}
+          NUMBER: ${{ (github.event_name == 'pull_request_target' || github.event_name == 'pull_request_review_comment') && github.event.pull_request.number || github.event.issue.number }}
+          RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
+        run: |
+          set -euo pipefail
+          if [ ! -s "$COMMENT_FILE" ]; then
+            if [ "$REQUIRE_COMMENT" = "true" ]; then
+              echo "::error title=Claude produced no review::Expected the review at $COMMENT_FILE; it is missing or empty. Claude ran but published nothing — this is a harness failure, NOT 'the PR was clean' (a clean PR still gets an Approve comment). Log: $RUN_URL"
+              exit 1
+            fi
+            echo "No comment at $COMMENT_FILE — nothing to post (silence is a valid outcome for this job)."
+            exit 0
+          fi
+          if [ "$IS_PR" = "true" ]; then
+            gh pr comment "$NUMBER" --body-file "$COMMENT_FILE"
+          else
+            gh issue comment "$NUMBER" --body-file "$COMMENT_FILE"
+          fi
+          echo "✅ Posted $(wc -c < "$COMMENT_FILE") bytes to #$NUMBER"
+
       # --- Verify a result was actually produced ---
       # Don't trust step outcome (continue-on-error masks it; an install crash exits
       # before it's set). Check the real artifact from whichever attempt ran. !cancelled()
@@ -366,5 +409,5 @@ jobs:
           if [ "$ok" = "true" ]; then
             echo "✅ Claude $KIND completed (log: $EXEC_FILE)"
           else
-            echo "::warning title=Claude $KIND did not complete::No $KIND was posted to $TARGET after 2 attempts — the Claude action crashed or errored before finishing (often the upstream install-phase ENOENT flake). Re-run: $RUN_URL"
+            echo "::warning title=Claude $KIND did not complete::Claude did not finish the $KIND for $TARGET after 2 attempts — the action crashed or errored before completing (often the upstream install-phase ENOENT flake). Re-run: $RUN_URL"
           fi

.github/workflows/claude.yml

@@ -133,15 +133,25 @@ jobs:
       # Run the gaia-testing skill to produce real-world evidence before the review.
       # Gated inside claude-run.yml to same-repo PRs touching a testable surface — never fork code.
       generate_evidence: true
+      # Claude writes the review here; claude-run.yml posts it. A review is mandatory, so an
+      # empty/absent file fails the job rather than passing silently.
+      comment_file: /tmp/review.md
+      require_comment: true
       prompt: |
             Your review POLICY — correctness-first severity, the nit cap, skip rules, and
             length caps — lives in the review rubric. Read it from `/tmp/review-rubric.md`
             (fetched from the base branch) and follow it exactly. If that file is absent or
             empty, fall back to the inline policy in this prompt and note at the top of your
             review that the rubric was unavailable. Do NOT add a `custom_instructions:` input —
             it was removed from the action API and is ignored.
-            Then post one review comment in the TWO-part format defined under "Output format — TWO parts" below: a visible human summary on top (verdict + headline issues in plain words), with the per-severity issues, `file:line`, and ```suggestion blocks tucked into the collapsed `<details>` block.
-            Post the review via: `gh pr comment ${{ github.event.pull_request.number }} --body-file /tmp/review.md` (write body to /tmp/review.md first).
+            Then write one review comment in the TWO-part format defined under "Output format — TWO parts" below: a visible human summary on top (verdict + headline issues in plain words), with the per-severity issues, `file:line`, and ```suggestion blocks tucked into the collapsed `<details>` block.
+
+            HOW YOUR REVIEW SHIPS — write it to `/tmp/review.md`. That file IS the review: a
+            workflow step posts it after you exit. Do NOT run `gh pr comment` (or any other
+            `gh` posting command) yourself — that would double-post. Writing `/tmp/review.md`
+            is the ONE required output of this job; finishing without it fails the run, so
+            write it even when the PR is clean (a clean PR gets an Approve comment, not silence)
+            and write it BEFORE you run low on turns.
 
             ### Real-world evidence — a REQUIRED section, and your verdict must depend on it
             ALWAYS include a `### Real-world evidence` heading in the VISIBLE part of your comment (after
@@ -399,6 +409,10 @@ jobs:
     with:
       # Re-review fork-PR pushes too; same trade-off as pr-review (see file header).
       allowed_non_write_users: "*"
+      # Claude writes the re-review here; claude-run.yml posts it. require_comment stays false —
+      # REVIEW.md makes silence the correct default for a re-review, so no file means no comment.
+      comment_file: /tmp/rereview.md
+      require_comment: false
       prompt: |
         You are doing a LIGHTWEIGHT re-review of a push to an already-reviewed GAIA pull
         request — NOT a full review. The first pr-review already covered this PR in depth.
@@ -433,8 +447,10 @@ jobs:
         the PR. This is intentionally STRICTER than the full review's nit cap: a re-review
         posts ZERO nits (the rubric's nit budget is the ceiling for the first review only).
 
-        If you do comment, write it to /tmp/rereview.md first, then:
-          `gh pr comment ${{ github.event.pull_request.number }} --body-file /tmp/rereview.md`
+        If — and ONLY if — you have something worth flagging, write it to `/tmp/rereview.md`.
+        A workflow step posts that file after you exit; do NOT run `gh pr comment` yourself
+        (that would double-post). If you have nothing to flag, write NO file and exit — that
+        is the expected outcome for most pushes.
 
         ## Output format — TWO parts (mandatory)
         Structure the comment in two parts:

tests/unit/test_claude_review_posting.py

@@ -0,0 +1,77 @@
+# Copyright(C) 2025-2026 Advanced Micro Devices, Inc. All rights reserved.
+# SPDX-License-Identifier: MIT
+
+"""Guards the PR-review posting contract in .github/workflows/.
+
+Claude WRITES its review to a file; a workflow step POSTS it. When the model was
+asked to run `gh pr comment` itself, a skipped final tool call published nothing
+while the job stayed green — 50 PRs merged unreviewed before anyone noticed.
+These tests fail if that arrangement is reintroduced.
+"""
+
+import re
+from pathlib import Path
+
+import pytest
+
+yaml = pytest.importorskip("yaml")
+
+WORKFLOWS = Path(__file__).resolve().parents[2] / ".github" / "workflows"
+REVIEW_JOBS = ("pr-review", "pr-rereview")
+
+# `gh pr comment N --body-file X` / `gh issue comment ...` — the executable posting
+# form. A bare "do NOT run `gh pr comment`" prohibition has no --body-file and is fine.
+POSTING_CMD = re.compile(r"gh\s+(?:pr|issue)\s+comment\b[^\n]*--body-file")
+
+
+def _load(name):
+    return yaml.safe_load((WORKFLOWS / name).read_text(encoding="utf-8"))
+
+
+@pytest.fixture(scope="module")
+def claude():
+    return _load("claude.yml")
+
+
+@pytest.fixture(scope="module")
+def runner():
+    return _load("claude-run.yml")
+
+
+@pytest.mark.parametrize("job,require", [("pr-review", True), ("pr-rereview", False)])
+def test_review_jobs_delegate_posting_to_the_workflow(claude, job, require):
+    """Both review jobs hand a file to claude-run.yml instead of posting inline."""
+    with_ = claude["jobs"][job]["with"]
+    assert with_.get("comment_file"), f"{job} must set comment_file"
+    # pr-review: a review is mandatory, so no file is a failure. pr-rereview:
+    # REVIEW.md makes silence correct, so no file is a legitimate no-op.
+    assert with_.get("require_comment", False) is require
+
+
+@pytest.mark.parametrize("job", REVIEW_JOBS)
+def test_review_prompts_never_tell_the_model_to_post(claude, job):
+    """The model must not be handed a posting command — that path fails silently."""
+    prompt = claude["jobs"][job]["with"]["prompt"]
+    found = POSTING_CMD.search(prompt)
+    assert found is None, (
+        f"{job}'s prompt tells the model to post: {found.group(0)!r}. "
+        "Posting belongs to claude-run.yml's 'Post Claude's comment' step; "
+        "a model-issued post is invisible when it doesn't happen."
+    )
+
+
+def test_runner_posts_and_is_gated_on_comment_file(runner):
+    steps = runner["jobs"]["run"]["steps"]
+    post = next((s for s in steps if s.get("name") == "Post Claude's comment"), None)
+    assert post is not None, "claude-run.yml lost its posting step"
+    assert "inputs.comment_file != ''" in post["if"], (
+        "posting must be gated on comment_file so callers that still post inline "
+        "(issue-handler, pr-comment) don't double-post"
+    )
+    assert POSTING_CMD.search(post["run"]), "posting step no longer posts anything"
+
+
+def test_runner_declares_the_posting_inputs(runner):
+    inputs = runner[True]["workflow_call"]["inputs"]  # PyYAML reads `on:` as True
+    assert inputs["comment_file"]["default"] == ""
+    assert inputs["require_comment"]["default"] is False