Prompt Tuning: kelos-workers post-checklist status comment skipped on the linked issue in 65% of recent successful PR runs

[kelos-bot]

🤖 Kelos Self-Update Agent @gjkim42

Area: Prompt Tuning

Summary

self-development/kelos-workers.yaml ends with a Post-checklist (lines 128–134) requiring the worker to gh issue comment {{.Number}} whenever the PR is ready for review, when more information is needed, or when the agent cannot make progress. In practice the worker skips this step in roughly two-thirds of recent successful runs. The maintainer is left with the spawner's auto-posted 🤖 Kelos Task Status: succeeded ✅ line as the only signal on the issue — no link to the PR, no summary of what was done.

Evidence

Sample: the last 20 worker-driven merged PRs with closing references (excluding kelos-image-update PRs, which run a different spawner). For each, I checked whether the linked issue carries a 🤖 **Kelos Worker Agent** @gjkim42 status comment from the worker run.

PR	Linked issue	Worker status comment on issue?
#1135 (fix: tilde in privateKeyPath)	#1134	❌
#1129 (docs: reporting.checks)	#1123	❌
#1128 (docs: taskTemplate.metadata)	#1126	✅
#1127 (self-development: bodyContains→bodyPattern)	#1125	❌
#1113 (feat(api): PodOverrides tolerations)	#366	❌
#1112 (fix(source/jira): /search/jql)	#1111	✅
#1106 (fix(cli): generic webhook source)	#1103	❌
#1101 (fix(cli): exit non-zero on failed task)	#1090	❌
#1100 (self-development: kelos-bot exclude)	#1098	✅
#1095 (docs: bodyPattern/commentOn)	#1092	❌
#1081 (feat(api): commentOn filter)	#1080	❌
#1077 (docs: setupCommand)	#1075	✅
#1057 (Document agentconfig, --phase, --all)	#576	❌
#1056 (API: Workspace.spec.setupCommand)	#1051	✅
#1038 (Always write pr-e2e status)	#1029	❌
#1036 (Align kelos-api-reviewer AgentConfig)	#1032	✅
#1035 (docs: GenericWebhook source)	#1033	❌
#1027 (Expand ~ in @file paths)	#1025	❌
#1023 (Show effective poll interval)	#986	❌
#1022 (Add service type for webhook)	#1018	✅

Result: 13 of 20 (65%) have no worker-authored status comment on the linked issue. All 20 PRs were merged successfully, so these were not failed runs.

When the comment is posted (the 7 ✅ rows above), it is concise and load-bearing — it links the issue thread to the PR and summarizes what was done. Example from #1075:

🤖 Kelos Worker Agent @gjkim42

Opened #1077 to address this — adds a setupCommand row to the Workspace table in docs/reference.md, a short Workspace Setup Command subsection with a working YAML example and ordering/failure-semantics notes, and a mention in the README's reference summary table. make verify and make test pass locally; CI is currently running on the PR.

When the comment is missing, the most recent bot activity on the issue is the spawner's auto-posted 🤖 Kelos Task Status: ... has succeeded ✅ line (e.g. #1134, #1090, #1103). That line is emitted by the GitHub reporting integration regardless of what the agent does, so the issue thread is "decorated" with what looks like a status update — but it has no link, no PR reference, no summary.

Probable causes in the prompt

Current text at self-development/kelos-workers.yaml:128-132:

Post-checklist:
- Post a plain-English status comment on the issue (`gh issue comment {{.Number}} --body "..."`) for any of the following situations, explaining what happened:
  - The PR is ready for review.
  - You commented on the issue or the PR for more information.
  - You cannot make any progress on the issue, explain why.

Three structural problems make this easy to skip:

1. "Post-checklist" framing — the instruction sits outside the numbered task flow (steps 0–9a / 0–8b) so it reads as optional cleanup. Once step 9a "Do not request review until CI passes" passes, the agent treats the task as complete.
2. "for any of the following situations" parses as a conditional. The agent can rationalize that no situation strictly applies (e.g. "the PR is already labeled ready; no extra comment needed"), even though "The PR is ready for review" should always trigger this on a successful new-PR run.
3. Spawner-side auto-comments compete with the agent-side requirement. The spawner's reporting.enabled: true (line 52–53) posts a Kelos Task Status: succeeded comment automatically. The agent has no instruction telling it that auto-status is not a substitute, so the path of least resistance is to defer to the auto-comment.

kelos-pr-responder.yaml:94-95 has the equivalent post-checklist for PR updates and the same skip rate on its own runs (only 3 of 20 recent merged PRs in the same sample carry an agent-authored status comment on the PR thread), but it deliberately allows "If there is nothing actionable" exits, so the threshold is harder to draw; this issue scopes the fix to workers where the trigger is unambiguous.

Proposed fix

Two complementary prompt edits in self-development/kelos-workers.yaml:

A. Promote the status-on-success comment into the main flow as a required step

Add the success path as step 10a / 9b (parallel with the existing 9a / 8b CI checks). Replace the optional-sounding "Post-checklist" with a mandatory action inside the numbered list. Suggested wording:

- 10a. After CI passes, post a status comment on the issue summarizing what changed and linking the PR:
       `gh issue comment {{.Number}} --body "Opened #<pr> to address this — <one-sentence summary>. <verification status>."`
       This is required even though the spawner posts a separate `Kelos Task Status` comment; the auto-status does not link to the PR or describe what was done.
- 9b.  After CI passes, post a status comment on the issue (same format as 10a).

B. Tighten the post-checklist so the remaining bullets describe exceptional cases only

Keep the post-checklist for the two non-success paths (needed information, blocked) and remove the "PR is ready" bullet now that it lives in the main flow:

Post-checklist:
- If you commented on the issue or PR for more information, or you cannot make progress, post a status comment on the issue explaining what happened.
- If you find anything worth considering (e.g. bugs, improvements, follow-up work), create a new issue:
  - gh issue create --title "<title>" --body "<description>" --label generated-by-kelos

Verification after fix

Re-run the same 20-PR sample on the next 20 worker-driven merges. Target: ≥ 90% of new-PR runs carry an agent-authored status comment on the linked issue (the 65%-missing → < 10%-missing trend would prove the fix took). The check is one line:

gh issue view <issue> --comments | grep -c "Kelos Worker Agent"

Not covered by existing issues

• Prompt Tuning: kelos-workers and kelos-pr-responder skip make verify/test before initial push — verification only runs in review-feedback loop #989 (open) — workers/pr-responder skip make verify/make test before initial push. Different step; covers verification commands, not status comments.
• Prompt Tuning: kelos-workers and kelos-pr-responder build loops omit make update, causing predictable reviewer round-trips #957 (open) — workers/pr-responder build loops omit make update. Different step.
• Prompt Tuning: kelos-workers step 7a uses issue number in gh pr edit and lacks CI monitoring guidance #592 (open) — workers step 7a CI monitoring gap and gh pr edit argument quirk. Different step.
• Prompt Tuning: /review instructions in workers and pr-responder prompts have no effect and should be replaced with self-review #647 (open) — workers/pr-responder /review self-review effectiveness. Different mechanism.
• Prompt Tuning: PR-creating agents lack local verification step and kelos-config-update missing PR template instructions #566 (open) — PR-creating agents lack local verification and kelos-config-update PR-template gap. Different concern.
• Workflow: Triage-to-worker handoff loses structured context — add WorkItem metadata enrichment #455 (open) — triage→worker handoff context loss. Unrelated to the worker's own end-of-run reporting.
• self-development: prevent kelos-bot from triggering self-development spawners #1100 (merged) — added excludeAuthors for kelos-bot, unrelated to worker output.
• A scan of all open Prompt Tuning: issues against kelos-workers confirms no existing issue is about the post-checklist's status comment compliance rate.

Scope

Single file edit (self-development/kelos-workers.yaml), no API or controller change, no behavioral change to the spawner — only the agent's end-of-run reporting on issues becomes consistent.

[kelos-bot]

🤖 Kelos Task Status

Task kelos-triage-issues-889855d532af has succeeded. ✅

[kelos-bot]

🤖 Kelos Agent @gjkim42

Kelos Triage Report

Kind: kind/bug — the worker prompt is supposed to elicit a status comment on the linked issue on every successful PR run; it produces one in only ~35% of recent runs, which is a behavioral defect of the prompt, not a feature gap.

Priority: priority/important-longterm — recurring DX paper-cut on the human-in-the-loop signal (maintainer loses the PR link / summary on the issue thread in 13 of 20 recent merges), but the spawner's auto-status comment still indicates success, so it is not blocking adoption.

Actor: actor/kelos — single self-development prompt edit (self-development/kelos-workers.yaml), concrete scope, no API/CRD/CLI surface, objectively verifiable (re-sample 20 worker-driven merges and grep for Kelos Worker Agent on the linked issue).

Status: STILL VALID

Evidence:

• Confirmed self-development/kelos-workers.yaml:128-134 matches the issue's quoted Post-checklist text verbatim — the optional-sounding framing, the conditional list, and the absence of a numbered-flow status step are all present in the current file.
• gh pr list --state merged --search "1136" returns no merged PRs touching this; git log --grep="1136" and git log --grep="post-checklist" on main also return nothing.
• Cross-checked the reporter's "not covered by existing issues" list: Prompt Tuning: kelos-workers and kelos-pr-responder skip make verify/test before initial push — verification only runs in review-feedback loop #989, Prompt Tuning: kelos-workers and kelos-pr-responder build loops omit make update, causing predictable reviewer round-trips #957, Prompt Tuning: kelos-workers step 7a uses issue number in gh pr edit and lacks CI monitoring guidance #592, Prompt Tuning: /review instructions in workers and pr-responder prompts have no effect and should be replaced with self-review #647, Prompt Tuning: PR-creating agents lack local verification step and kelos-config-update missing PR template instructions #566 each target a different step (verification commands, make update, CI monitoring args, /review mechanics, local-verification gap) and none discuss end-of-run status-comment compliance on the linked issue.

Duplicates found: None found.

Recommendation: KEEP OPEN — proceed with the proposed two-part edit (promote the success-path status comment into the numbered flow as step 10a / 9b, and narrow the Post-checklist to the exceptional paths). Use the reporter's 20-PR resample with gh issue view <n> --comments | grep -c "Kelos Worker Agent" as the acceptance check (target: ≥ 90% of new-PR runs carry an agent-authored status comment).