[8] Report PR costs as work progresses

[tautvydasLiekis]

Kimchi now uploads PR cost reports to the user's own account while work is in progress, so the backend can show what each pull request cost from its own bills. Until now, PR costs stayed on the developer's machine. Reporting follows the existing SaaS telemetry setting, shows a one-time notice, and can be turned off with /pr-reporting off.

Depends on #1351 and the reporting API [7]. The dashboard is a separate change, [9] !948. Merge and deployment are separate steps.

What users get

• One notice in the first interactive session (after the first turn in Studio) says what is reported and how to turn it off.
• /pr-reporting on|off|status. The choice survives restart. off cancels delivery and deletes pending uploads; local history and reports the server already accepted remain. status shows the setting, pending reports, repositories waiting for their upload window, server limits and delivery problems. Typing /pr-reporting suggests the three arguments.
• status reads as a small table (PR reporting Enabled (SaaS default), Repositories 1 queued · 0 acknowledged). Delivery problems and server limits come first and make it a warning; history Kimchi could not report, such as requests without their original account or repository, is listed after the table as information.
• CI, print/JSON and benchmark runs never upload, and KIMCHI_PR_COST_REPORTING=0 turns uploads off for a process. They keep recording locally, so a later interactive session reports the same work.
• Reporting is part of the "Cost per PR" extension from [2] Keep work tied to its account and task #1347, so disabling that extension in /resources also stops uploads.

How it works

After each main-agent turn, at startup and every 30 seconds, Kimchi rebuilds one snapshot per account and repository from its local journals and uploads it at most every five minutes. Journals are read in the background. While they and the local queue are unchanged, a pass skips the rebuild; it rebuilds after an hour anyway so the upload window still moves. A PR opening, merging, closing or reopening, a /work link or unlink, a withdrawal and /pr-reporting on go out at once.

Each upload replaces the previous one for that repository under a higher revision. Revision 1 may have requests but no PR; revision 2 adds the discovered PR; revision 3 adds billing IDs. Replacing a report never counts a charge twice, and the server reads prices from its own bills.

flowchart TD
    A[Recorded request] --> B{Original account and repository known?}
    B -- No --> Local[Keep locally]
    B -- Yes --> C{PR ownership evidence}
    C -- No PR yet --> Unknown[Report as unlinked]
    C -- One PR --> E{Why?}
    E -- Its edits landed, or explicit choice --> Confirmed[Confirmed]
    E -- Same work or model guess --> Inferred[Inferred]
    C -- Several PRs --> Shared[Shared]
    Unknown --> Send[Upload when enabled and the account matches]
    Confirmed --> Send
    Inferred --> Send
    Shared --> Send

Loading

What is sent: repository and PR metadata, request and billing IDs, timestamps, coverage, the matching method and correction receipts. What stays local: prompts, plans, paths, work and session IDs, commit hashes and credentials. Health metrics use the existing telemetry setting and carry no user, request, repository or PR IDs.

Which work is sent: open PRs and their work, work from the last 32 days, and finished PRs for 32 days plus two days of clock grace. Older finished PRs are named in windowedPullRequestIds, so the server keeps their claims. The server freezes a PR's total 32 days after it finishes and keeps frozen totals indefinitely; explicit corrections can still change one until day 90.

Limits and failures:

• A snapshot over the server's limits (32,000 requests, 250 PRs, 8 MiB) is trimmed, unlinked requests first and open PRs last, and marked incomplete. A request with more than 8 billing IDs is sent without IDs, and the server finds its bills by the request's tag.
• A server limit (HTTP 429 PR_COST_LIMIT) waits about six hours. A snapshot limit also teaches a smaller cap that fits after one rejection; a producer limit holds only that repository; a contributor or organization limit pauses the account, while withdrawals still go out.
• HTTP 403 says that uploading needs an Owner or Member role and retries hourly, then up to every six hours. Other errors honour Retry-After or back off.
• A repository with missing history is held without blocking others. The local queue is limited to 24 MiB; a replacement that does not fit waits, and the snapshot already queued for that repository is still delivered.

Queue state is in ~/.config/kimchi/harness/pr-cost-reporting/state.json. The reporting docs describe the payload, limits and health metrics.

Evidence

Review fix on d97fe9c55 (merged up as 55671c371): argument completions, and the skipped-history count saved as its own state field instead of inside the error text, so status no longer shows it as a warning. 14 new or changed tests fail without the change; pnpm check, 252 reporting tests and the TUI and ACP pr-cost-reporting e2e tests pass.

12 October retest of this head's binary (55671c371) with real GLM-5.3 and GLM-5.3 Flash sessions and real bills, uploading to the backend from !55779 (16d4018ad, running locally): all 3 uploads were accepted, and the backend's row for PR #927 equals Kimchi's split to the nano-dollar ($0.037085120 confirmed, $0 inferred, 1 contributor).

On da5d1ce61, after the final review fixes, with fixtures for accounts, replies and bills; no paid inference or shared-service upload:

• CI: build, pr-checks, master-checks, TUI e2e in 4 shards, ACP e2e and MCP e2e pass on this head. A duplicate tui-e2e entry from a superseded workflow run shows as cancelled.
• Locally on this head, the full unit suite passes except 5 tests that need tools this machine lacks: a DAP js-debug adapter, a built validate-skill.mjs and a binary rebuilt for the smoke test.
• New tests: an unchanged pass is skipped, journals are read in the background and the upload window still moves; a repository whose only history is older than the upload window queues nothing; an acknowledged withdrawal is not sent again when history completeness changes; only interrupted appends are skipped in journals.
• Lab on a binary built from the stack top 95614e7c6 (#1389), which contains this PR: 73/73 default scenarios pass, including default-on reporting, startup backfill, replacement before a PR exists and opt-out.
• End to end with that binary, the backend from !55779 (local PostgreSQL and ClickHouse) and the console from !948: all three uploads were accepted. While PR test(e2e/tui): reduce nudge suppression suite to wiring smoke tests #731 was open, the backend returned $0.020899 spent so far; after the merge it returned $0.015330 confirmed and $0.005569 inferred, equal to the client's totals to the nano-dollar.
• A later live test on the same binary, backend and console used varied work: seven works across five PRs (merged, open, closed without merging, one work split over two PRs, follow-up after a merge, a /work link correction and bills that arrived late). All 7 uploads were accepted. Each request's assignment and price, the backend's totals and the console's figures matched the expected result.

The notice and /pr-reporting status from the 12 October run of this head:

 PR costs are reported to your account (repository/PR details and request/billing IDs). Turn off with /pr-reporting off.

 PR reporting   Enabled (SaaS default)
 Repositories   0 queued · 1 acknowledged

Risks and remaining work

• Not yet verified: deployed authentication, a real billed merged PR in the dashboard, and matching accuracy on representative work.
• No TUI or ACP test checks an actual upload; the fake server has no snapshot endpoint or provider identity. Uploads are covered by unit tests with HTTP fixtures, the private lab and an end-to-end run against the backend from !55779 (see Evidence).
• A pass that finds changed journals re-reads all of them. With a very long history (estimated at well over 100,000 recorded requests), capture runs out of its five-second budget and stops queuing new snapshots; queued ones still go out.
• A home directory used live by two machines, such as a bind-mounted dev container, starts a new producer at every switch. The other machine's earlier claims stay on the server until it has been idle for 40 days.
• Reporting is on by default for SaaS users who have telemetry on; the notice and /pr-reporting off are the opt-out.

[kimchi-review]

Kimchi Code Review

Property	Value
Commit	eee640f
Author	@tautvydasLiekis
Files changed	21
Review status	Completed
Comments	0
Duration	15s

Summary

🚫 Merge request too large to review

📏 Size: 21 file(s) changed, +2311 / −9 lines

🧠 Cognitive load: 5/5 (1 = trivial, 5 = extremely demanding)

An entire new subsystem (snapshot schema, durable queue, HTTP worker, extension wiring) lands at once, interleaved with provider-ID and cost-report refactors in existing modules. Subtle ACK/revision/durability semantics plus ~1000 lines of dense tests make the blast radius very hard to hold in mind.

✂️ Suggested split

1. Capture stable provider IDs in pull request status — Stores target-repo PR and repository IDs and exposes repository identity lookup.
   • src/extensions/pull-request-status/pull-requests.ts
   • src/extensions/pull-request-status/pull-requests.test.ts
2. Expose validated work cost report from source journals — Single validated inventory reused by local summaries and later reporting.
   • src/extensions/work-attribution/cost-sync.ts
   • src/extensions/work-attribution/cost-sync.test.ts
   • src/extensions/work-attribution/costs.test.ts
3. Add PR cost snapshot schema and builder — Allowlisted wire format with strict validation; private local data never crosses.
   • src/extensions/pr-cost-reporting/snapshot.ts
   • src/extensions/pr-cost-reporting/snapshot.test.ts
4. Add durable opt-in reporting queue with acknowledgements — Persistent consent, revisions, membership hashes and retry state under agent dir.
   • src/extensions/pr-cost-reporting/queue.ts
   • src/extensions/pr-cost-reporting/queue.test.ts
5. Add reporting worker, extension wiring and docs — Bounded authenticated delivery, /pr-reporting command, CLI registration, docs.
   • src/extensions/pr-cost-reporting/worker.ts
   • src/extensions/pr-cost-reporting/worker.test.ts
   • src/extensions/pr-cost-reporting/index.ts
   • src/extensions/pr-cost-reporting/index.test.ts
   • src/cli.ts
   • docs/pr-cost-reporting.md

👉 Please split this merge request into the smaller merge requests suggested above — the AI review will run automatically on each of them.

⚠️ Full review skipped — this merge request exceeds the reviewable size limit. See the split proposal above.

What to expect

Kimchi will analyze the changes in this pull request and post:

• A summary of the overall changes
• Inline comments on specific lines with findings categorized by issue type

The review typically completes within a few minutes. This comment will be updated once the review is ready.

Interact with Kimchi

• @getkimchi review — re-trigger a full review on the latest commit
• @getkimchi summary — regenerate the PR summary
• @getkimchi ignore — skip this PR (no review will be posted)
• Reply to any inline comment to ask follow-up questions or request clarification

Configuration

Reviews are configured by your organization admin.
Review instructions, excluded directories, and severity thresholds can be adjusted per repository in the Kimchi dashboard.

---

Powered by Kimchi — AI-powered code review by CAST AI

[kimchi-review]

🚫 Merge request too large to review

📏 Size: 21 file(s) changed, +2311 / −9 lines

🧠 Cognitive load: 5/5 (1 = trivial, 5 = extremely demanding)

An entire new subsystem (snapshot schema, durable queue, HTTP worker, extension wiring) lands at once, interleaved with provider-ID and cost-report refactors in existing modules. Subtle ACK/revision/durability semantics plus ~1000 lines of dense tests make the blast radius very hard to hold in mind.

✂️ Suggested split

1. Capture stable provider IDs in pull request status — Stores target-repo PR and repository IDs and exposes repository identity lookup.
   • src/extensions/pull-request-status/pull-requests.ts
   • src/extensions/pull-request-status/pull-requests.test.ts
2. Expose validated work cost report from source journals — Single validated inventory reused by local summaries and later reporting.
   • src/extensions/work-attribution/cost-sync.ts
   • src/extensions/work-attribution/cost-sync.test.ts
   • src/extensions/work-attribution/costs.test.ts
3. Add PR cost snapshot schema and builder — Allowlisted wire format with strict validation; private local data never crosses.
   • src/extensions/pr-cost-reporting/snapshot.ts
   • src/extensions/pr-cost-reporting/snapshot.test.ts
4. Add durable opt-in reporting queue with acknowledgements — Persistent consent, revisions, membership hashes and retry state under agent dir.
   • src/extensions/pr-cost-reporting/queue.ts
   • src/extensions/pr-cost-reporting/queue.test.ts
5. Add reporting worker, extension wiring and docs — Bounded authenticated delivery, /pr-reporting command, CLI registration, docs.
   • src/extensions/pr-cost-reporting/worker.ts
   • src/extensions/pr-cost-reporting/worker.test.ts
   • src/extensions/pr-cost-reporting/index.ts
   • src/extensions/pr-cost-reporting/index.test.ts
   • src/cli.ts
   • docs/pr-cost-reporting.md

👉 Please split this merge request into the smaller merge requests suggested above — the AI review will run automatically on each of them.

⚠️ Full review skipped — this merge request exceeds the reviewable size limit. See the split proposal above.