feat(slack): ask for a thumbs up/down on completion replies in pilot channels - #153
feat(slack): ask for a thumbs up/down on completion replies in pilot channels#153anomalogravity[bot] wants to merge 1 commit into
Conversation
…channels Completion replies in the score pilot channels now carry a one-line ask, 'React 👍/👎 to score this.', right above the existing Task: footer, and both reactions are pre-seeded on the reply so a human taps once instead of opening the emoji picker. The ask only appears on terminal (succeeded/failed) transitions, and only in the channels named in the new SLACK_SCORE_PILOT_CHANNELS env var (chart value slackServer.scorePilotChannels); an empty value keeps the feature off everywhere. Reaction names (+1/-1) match the collector's lists, which ignore bot reactions, so pre-seeding does not skew the verdict ratio. reactions.add failures are logged and never block task reporting.
|
@greptile review |
Greptile SummaryThis PR adds a pilot-scoped Slack completion-rating prompt and pre-seeds positive and negative reactions, with chart configuration for selecting pilot channels.
Confidence Score: 4/5This PR should not merge until partial continuation failures stop persisting an incomplete completion reply and attaching score reactions to the wrong message. A failed final continuation in the progress-update path is ignored, after which reactions are applied to an earlier fragment and the terminal phase is persisted, preventing delivery from being retried. Files Needing Attention: internal/reporting/watcher.go
|
| log.Error(err, "Failed to post continuation message", "task", task.Name) | ||
| continue | ||
| } | ||
| lastReplyTS = replyTS | ||
| } | ||
| tr.preSeedScoreReactions(ctx, channel, lastReplyTS, askForScore) |
There was a problem hiding this comment.
Failed continuation persists completion
When the final ask-bearing continuation fails after an earlier continuation succeeds, this loop retains the earlier reply timestamp, seeds reactions on that fragment, and persists the terminal phase. The score ask and Task footer are never delivered, and later reporting cycles cannot retry the incomplete reply.
Knowledge Base Used: Deployment and operational interfaces
Prompt To Fix With AI
This is a comment left during a code review.
Path: internal/reporting/watcher.go
Line: 598-603
Comment:
**Failed continuation persists completion**
When the final ask-bearing continuation fails after an earlier continuation succeeds, this loop retains the earlier reply timestamp, seeds reactions on that fragment, and persists the terminal phase. The score ask and Task footer are never delivered, and later reporting cycles cannot retry the incomplete reply.
**Knowledge Base Used:** [Deployment and operational interfaces](https://app.greptile.com/anomalo/-/custom-context/knowledge-base/datagravity-ai/kelos/-/docs/deployment-and-operations.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.|
Closing in favor of doing this entirely on the dquality side — no Kelos change needed. Tracked in GRAV-715. The reaction pre-seeding and channel gating here can both be done from a job we own: The one thing that genuinely requires this PR is the ask text line, since Kelos composes the message body. We are choosing to drop it: a pre-seeded 👍/👎 pair is self-explanatory on its own, and it costs no extra sentence on every reply. The deciding factor is that this targets Nothing here is wasted if we change our minds: the branch stays, and the trade-off is written up in GRAV-715. What we give up is instant placement (reactions now appear on the next collector tick, within 5 minutes) and the text line. |
What type of PR is this?
/kind feature
What this PR does / why we need it:
Gravity's Slack replies end with a
Task:footer but no invitation to rate the answer. For the agent-effectiveness scorecard (GRAV-715), Slack verdicts only accumulate once the ask exists — so this ships the ask early, pilot-scoped.succeeded/failedtransitions) in the score pilot channels now carry a one-line ask,React 👍/👎 to score this., as a context block directly above the existingTask:footer. It never appears on the "working on your request" (accepted) message or on progress/activity updates.+1/-1) match the collector's positive/negative lists, which ignore bot reactions, so pre-seeding cannot skew the verdict ratio.reactions.addfailures are logged and never block task reporting — the reply itself is already posted, and a missing reaction only costs the one-tap convenience. AddedTestSlackTaskReporter_ScoreReactionFailureStillReportsfor that path.SLACK_SCORE_PILOT_CHANNELSenv var (chart valueslackServer.scorePilotChannels). Empty/unset disables the feature everywhere, so default deployments are unchanged.Which issue(s) this PR is related to:
N/A (Gravity Linear GRAV-715 — internal tracker, not a GH issue)
Special notes for your reviewer:
FormatSlackTransitionMessagekeeps its public signature (ask off); the ask is threaded through the newformatSlackTransitionMessage(..., askForScore)core so the split-message math reserves the extra block correctly and the ask rides on the last part (which carries the trailing blocks).slack_test.go(ask placement per phase, survives splitting,ParseScorePilotChannelsmatrix) andwatcher_test.go(pilot vs. non-pilot, nil map, failed phase, update path, reaction-failure resilience).Does this PR introduce a user-facing change?