diff --git a/.github/workflows/tui-e2e.yml b/.github/workflows/tui-e2e.yml index 4571a4753..039f759be 100644 --- a/.github/workflows/tui-e2e.yml +++ b/.github/workflows/tui-e2e.yml @@ -47,6 +47,13 @@ jobs: go-version-file: tools/proxy-helper/go.mod cache-dependency-path: tools/proxy-helper/go.sum + # Fresh test homes otherwise download these before the editor accepts input. + - name: Install search tools + timeout-minutes: 3 + run: | + sudo apt-get update -qq + sudo apt-get install --yes --no-install-recommends ripgrep fd-find + - name: Build proxy-helper run: make copy-for-dev # Builds the binary and copies it to bin/ where the tests expect it working-directory: tools/proxy-helper diff --git a/docs/pr-cost-reporting.md b/docs/pr-cost-reporting.md new file mode 100644 index 000000000..592b590a4 --- /dev/null +++ b/docs/pr-cost-reporting.md @@ -0,0 +1,203 @@ +# PR cost reporting + +PR cost reporting follows the SaaS telemetry setting by default. When SaaS uploads are on, Kimchi reports existing work automatically, including requests made before a PR exists. An explicit `/pr-reporting on` or `off` overrides that default and survives restart. Existing saved choices are preserved. + +The first interactive main session shows one notice that PR costs are reported, with `/pr-reporting off` to disable it. Studio shows it as a notification after the first turn, because notifications sent before a session is registered are dropped. The notice is saved per installation and does not repeat after restart. It is skipped when reporting is off or the user already made an explicit choice. + +- `/pr-reporting on` enables uploads for the verified account and requests an immediate update. It also clears upload windows, retry deadlines and account pauses, so the next snapshot of every repository goes out at once. +- `/pr-reporting status` shows whether reporting is enabled and whether that follows the SaaS default or an explicit choice, whether this session uploads at all, queued and acknowledged repositories (and how many wait for their upload window), server limits, partially reported repositories, delivery problems and incomplete history coverage. Server limits and delivery problems come first and make the message a warning. Incomplete history alone is informational: requests without original account or repository evidence are skipped, and the other requests are still reported. Lines fit a 60-column terminal. +- `/pr-reporting off` cancels delivery and deletes pending payloads. Local work history and revision counters remain. Reports already accepted by the server remain there. + +Typing `/pr-reporting ` suggests `on`, `off` and `status`. + +For example: + +```text +Delivery problems +- PR cost reporting paused for this organization: + organization limit reached (requests 50,000 of 50,000). + Ask an admin to free space or wait. + Next try in about 5 h. +- owner/repo is partially reported: 1,200 requests left + out to fit the upload limits. + +PR reporting Enabled (SaaS default) +This session No uploads; local attribution continues + Reason: CI environment (GITHUB_ACTIONS) +Repositories 2 queued · 3 acknowledged + 1 waiting for the 5-minute upload window + +History coverage incomplete + 3,693 requests skipped without original account or + repository evidence. Other requests are still reported. +``` + +Only main sessions with work tracking run this extension. Child sessions contribute their existing request records through the local calculator; they do not start upload workers. Turning Cost per PR off in `/resources` turns off work tracking, PR/MR status and reporting together. + +The worker gathers the existing local inventory at startup, refreshes after each agent turn when reporting is on, and checks again every 30 seconds while Kimchi runs. A request can first appear as unlinked or unknown, then gain a PR and a price in later reports. Changed reports receive a higher `revision` and a new `generatedAt`; the server replaces the previous report, so repeated updates do not add the same charge again. A running pass or a saved retry deadline can delay an update. + +Every change replaces the queued snapshot on disk at once, but each repository uploads at most once every five minutes. These changes skip that window and go out on the next pass: a PR that opens, merges, closes or reopens; a new explicit `/work link` or `/work unlink` correction (it also sends the other changed repositories of the account, because a correction can move requests between them); an empty snapshot that withdraws a repository; and `/pr-reporting on`. Such a change stays urgent through later replacements until the server acknowledges it. Saved-plan and artifact confirmations, new requests, prices and refresh times wait for the window. A change still waiting when Kimchi exits stays queued and is sent by the next session. + +### Sessions that never upload + +CI jobs, one-shot runs and benchmarks would only add noise, so they keep recording and attributing work locally but skip the reporting pass and never upload. A later interactive session on the same machine builds and sends the same reports from those records. `/pr-reporting status` names the reason: + +- `KIMCHI_PR_COST_REPORTING=0` (also `false`, `off` or `no`) turns uploads off for the process, whatever the saved choice. +- `--print`/`-p` and `--mode json` runs, including piped input, which Pi runs in print mode. Interactive TUI, Studio (ACP) and RPC sessions upload. +- CI: any of `CI`, `GITHUB_ACTIONS`, `GITLAB_CI`, `BUILDKITE`, `JENKINS_URL`, `TEAMCITY_VERSION`, `CIRCLECI` or `TF_BUILD` set to a value other than empty, `0`, `false`, `off` or `no`. +- Benchmark runs, recognized by a positive `KIMCHI_INFRA_BREAKER_THRESHOLD`; only the benchmark harness and CI set it. + +The saved choice and the default-on notice are unaffected; nothing overrides the detection to force uploads. + +Teleport transfers conversation history but does not transfer the local accounting journals or account scope. On a fresh sandbox, normal continuation starts a new work on the first input. An explicitly selected work keeps its ID, but its missing scope remains unknown. Neither case establishes a complete cost across local and remote work. Remote-agent accounting is outside this reporting change. + +## What leaves the machine + +Each upload replaces one producer's inventory for one account and repository. It contains: + +- The Git provider, host, stable target repository ID, and optional repository name. +- Stable PR/MR IDs, numbers, links, state and merge or close times. +- Exact request attempt UUIDs, verified billing-row UUIDs and request start times. +- Allocation kind, referenced PR IDs and evidence method. An `unknown` request names the PRs it may belong to, so only those totals stay incomplete. An input whose native edits landed in the PR sends `native`; other session inputs send `session` and stay inferred. Continued plans send `explicit`; `/work link` corrections send `user-correction`. Model guesses send `model` and stay inferred. +- Request counts, missing-price counts, history completeness and the latest applicable cost refresh time. + +Optional fields explain later updates. `windowedPullRequestIds` lists finished PRs deliberately left out of this upload. A request's `correction` contains the validated link's UUID, revision, time and source (`work-command` or `producer-confirmation`). The source distinguishes an explicit `/work link` or `/work unlink` from a verified saved-plan or artifact confirmation. `coverage.trimmedRequests` counts requests left out to fit the upload limits; it appears only when some were, and then `historyComplete` is `false`. + +Kimchi does not upload prompts, plans, messages, local paths, work IDs, session IDs, commit hashes, credential fingerprints or prices. The backend reads its own billing rows. A request without a confirmed price remains in the inventory; it is not reported as free. Authentication determines the contributor; the body has no user field. + +An upload with one request still waiting for billing looks like this: + +```json +{ + "schemaVersion": 1, + "producerId": "11111111-1111-4111-8111-111111111111", + "revision": "3", + "generatedAt": "2026-10-05T12:00:00.000Z", + "repository": { "provider": "github", "host": "github.com", "id": "42", "name": "example/repo" }, + "pullRequests": [], + "requests": [{ + "requestId": "22222222-2222-4222-8222-222222222222", + "billingRecordIds": [], + "startedAt": "2026-10-05T11:59:00.000Z", + "allocation": { "kind": "unknown", "pullRequestIds": [], "method": "session" } + }], + "coverage": { "observedRequests": 1, "unpricedRequests": 1, "historyComplete": true } +} +``` + +## Matching and delivery + +```mermaid +flowchart TD + A[Original request and billing records] --> B{Complete readable inventory?} + B -- No --> Hold[Hold affected snapshots and show a local error] + B -- Yes --> C[Reuse the local cost calculator] + C --> D{One repository or several?} + D -- Several --> U[Unknown allocation in each repository] + D -- One --> E{What evidence exists?} + E -- Confirmed PR --> PR[PR allocation] + E -- Shared within repository --> Shared[Shared allocation] + E -- Model or session --> Inferred[Keep the inferred method] + E -- Missing or conflicting --> Unknown[Unlinked or unknown] + E -- Open PR or later request --> Outside[Unmerged or post-merge] + U --> Trim + PR --> Trim + Shared --> Trim + Inferred --> Trim + Unknown --> Trim + Outside --> Trim[Trim to the upload limits] + Trim --> Queue[Save and fsync a replacement snapshot] + Queue --> Consent{Reporting on, interactive session and account matches?} + Consent -- No --> Wait[Keep for an interactive session of the original account, or discard on opt-out] + Consent -- Yes --> Window{Urgent change or five minutes since the last upload?} + Window -- No --> Later[Keep only the newest snapshot queued] + Window -- Yes --> Send[Send bounded authenticated request] + Send --> Ack{Matching acknowledgement?} + Ack -- Yes --> Done[Remove only the acknowledged revision] + Ack -- PR_COST_LIMIT --> Quota[Trim, skip the repository or pause the account, then wait about six hours] + Ack -- No --> Retry[Keep payload and retry later] +``` + +GitHub IDs come from the PR's `id` and target `base.repo.id`. GitLab IDs come from the MR's `id` and `target_project_id`. A fork's source repository ID never replaces the target ID. Old merged links without these IDs are refreshed. + +The inventory comes from the original journals through the same billing validation and calculator used by local cost summaries. It does not concatenate per-work `costs.json` files. Two works contributing to one PR do not duplicate requests. Cross-repository shared requests are reported as unknown in each repository; the backend handles duplicate billing evidence across snapshots. A request that starts after several PRs merged keeps all candidate IDs as a shared claim. The backend compares its start time with each merge time and places it outside those PR totals. + +Records without their original account or repository are skipped; Kimchi never attaches them to today's login. Independent valid repositories can still report with `historyComplete: false`. Only records of work with repository evidence (a scoped request or a commit) count; a question asked outside Git has neither. Records that would fall outside the upload window anyway do not affect completeness. A request in a repository without a GitHub or GitLab identity (no such remote, or the local repository no longer exists) has no snapshot to go to and does not affect completeness either. The same applies to a self-hosted server that Kimchi cannot recognize as GitHub or GitLab, for example because its API cannot be reached without saved credentials; Kimchi checks it again a day later. A repository lookup that failed for another reason, such as a network error or rate limit, leaves recent history incomplete until it succeeds. Missing earlier requests or conflicting provider metadata hold the affected repository. A GitLab MR with no merge time leaves its requests unknown while other MRs keep reporting. An unparseable final journal line without a newline is ignored as an interrupted append, as is a cut-off record that later appends moved past (older versions start a new line after it). Other malformed records, including a cut-off final record, and unreadable journals still hold the scan. These cases appear in `/pr-reporting status`: skipped records as incomplete history coverage, held repositories and scans as delivery problems. `historyComplete` describes the captured inventory, not activity before tracking was installed or while it was disabled. + +## Local state and retries + +State lives under `getAgentDir()`: + +```text +~/.config/kimchi/harness/pr-cost-reporting/state.json +``` + +An isolated harness home uses its own directory. The state file contains consent, a producer UUID, account/repository revision bookkeeping, pending snapshots, upload times, the PR states and `/work` correction receipts the server last acknowledged, validated server limits, and how many requests the latest capture skipped for missing evidence. It also keeps a salted hash of the operating system's machine ID: a home restored or migrated onto another machine starts a new producer with empty bookkeeping, so two machines never replace each other's claims. The server deduplicates bills across producers, so copied history is not counted twice. Revision bookkeeping includes SHA-256 values of request UUIDs from acknowledged and possibly delivered snapshots, so a later incomplete scan cannot silently remove earlier claims. These hashes shrink to the new membership only when that exact revision is acknowledged. A lost or late acknowledgement leaves the earlier evidence intact. Opt-out retains these membership hashes and revision metadata, but removes pending bodies and raw request/billing IDs. Files are private to the local user. A lock protects concurrent writers; a synced temporary file and rename publish each update before HTTP starts. + +Only the newest pending snapshot for a repository is retained. Corrections and revocations with preserved request membership replace earlier allocations. When all requests move to another verified repository in the same account, an empty higher-revision snapshot withdraws the old group. The old membership remains protected while that withdrawal is pending, including after opt-out and restart. Missing source evidence alone cannot trigger that withdrawal. An acknowledgement for an older in-flight revision cannot delete a newer payload. Restarting resumes durable state instead of adding the same charges again. + +The existing reconciliation worker runs reporting after local cost lookup. Each pass reads the local history in the background and queues snapshots within five seconds, then delivers within another five seconds, with at most three upload attempts. While the journals and the local queue are unchanged since the last capture, a pass skips reading them; it captures again after an hour so the upload window still moves. Queued reports are delivered even when capture fails. Verification and upload reject redirects and cap response bodies at 64 KiB. Credentials and endpoint are checked again after awaited operations and immediately before dispatch. Opt-out from another harness process cancels an active upload through a local state-file watcher; if the watcher cannot start, each upload still rereads the state first. + +Repository lookups share the capture budget. A found identity is kept for a day, and so is a repository without a GitHub or GitLab identity. Other failed lookups wait at least 30 seconds before retry. Unchecked repositories go first, and each pass reserves time to queue the repositories it already knows. A slow or unavailable provider does not discard that progress. + +Failed attempts retain their payload and retry time. A failed attempt also starts the repository's five-minute window, so an ordinary change is not retried sooner than that. Three kinds of failure wait differently; transient failures and rejections share one attempt counter, and server limits leave it untouched: + +- Transient: timeouts, network errors, HTTP 408, 5xx and HTTP 429 without a limit detail. A `Retry-After` is honored between 30 seconds and six hours. Without one, the wait doubles from 30 seconds to 30 minutes with ±20% jitter. +- Rejection that retrying cannot fix, such as HTTP 404 from a backend without this endpoint: one hour, doubling to six hours, with ±20% jitter. HTTP 403 means the API key may not upload; `/pr-reporting status` says that an Owner or Member role is needed. +- Server limit: HTTP 429 whose JSON body carries a `google.rpc.ErrorInfo` with reason `PR_COST_LIMIT` and domain `ai-optimizer`. It is quota, not an outage: the next try waits six hours with ±20% jitter. Only the validated scope, limit name and integer `current`/`maximum` are saved; response text never is. What happens depends on the scope: + - `snapshot`: Kimchi learns a smaller cap for that dimension (`requests`, `pullRequests` or `bytes`) and trims later snapshots to it for a week. The cap is the smaller of the server's `current` scaled onto Kimchi's own measure and Kimchi's measure minus the server's overflow (`current` − `maximum`), so one rejection is enough whether the server measured the upload differently or counted records besides it. A limit trimming cannot meet, such as `billingIds` or `windowedPullRequests`, only waits. + - `producer`: that repository is skipped until the retry; other repositories keep uploading. + - `contributor` or `organization`: every repository of the account pauses. Empty withdrawal snapshots still go out, because they free space. One notice per pause says `PR cost reporting paused for this organization: limit reached. Ask an admin to free space or wait.` (or `for your account in this organization` for a contributor limit). An accepted upload that adds claims ends the pause. + - Any other scope waits on that repository alone. The server's `frozen` limits bound its queries and never reject an upload. + +The reason appears in `/pr-reporting status` until an upload is accepted. An unavailable or older backend therefore leaves a visible pending report. Nothing is sent to a different account to clear that queue. + +Uploads match the server's per-snapshot limits: 32,000 requests, 250 PRs and 8 MiB of JSON, measured with the longest envelope a later revision can have. A request with more than 8 billing IDs, the server's per-request limit, is sent without IDs; the server then finds its bills by the request's exact tag. A larger inventory is trimmed rather than held. Kimchi leaves out the oldest unlinked and candidate-less unknown requests first, then the requests of finished PRs, earliest finish first, and the oldest requests of open PRs only after everything else. The PR limit is met by removing whole PRs, finished ones first. A PR left without requests is removed rather than listed in `windowedPullRequestIds`, because the server would restore its earlier claims into the same snapshot. Correction receipts are never left out. If they alone exceed the limits, the snapshot is held and `/pr-reporting status` says so. A trimmed snapshot sets `historyComplete: false` and `trimmedRequests`, and each repository shows one notice, for example `PR costs for owner/repo are partially reported: limit reached. See /pr-reporting status.` The local queue is bounded to 24 MiB. No request, session, PR or user IDs are added to metric labels. + +## Upload window and saved totals + +The client sends each request from the last 32 days and each request that may belong to an open PR. A merged or closed PR stays in uploads with all its contributing requests for 32 days plus two days of clock-skew grace after its provider-reported finish time. Older requests of a long-lived work stay out, so snapshots stay small. A correction receipt is sent only while the server can still apply it, until five minutes before the PR's details expire at day 90. Missing finish metadata keeps the work eligible. The local source inventory distinguishes requests deliberately left outside this window from records that went missing; missing evidence still holds an affected upload. + +The client names intentionally omitted finished PRs in `windowedPullRequestIds`. The server keeps those PRs and their claims, including when its clock has not reached day 32 yet. Omitting a PR without naming it withdraws its uploaded claims; it does not silently change an already frozen assignment. + +At day 32 the server saves a frozen subtotal. It keeps detailed claims for 90 days after merge or close, then keeps only the small report. Frozen reports do not expire, and their original freeze date never changes. Missing prices and disputed ownership stay incomplete. Expiring one PR's details does not mark the rest of the repository's recorded history incomplete. + +Until day 90, an explicit `/work link`, `/work unlink` or verified producer confirmation can correct frozen ownership. The server saves the changed total before acknowledging an accepted correction and records `correctedAt`; `frozenAt` stays unchanged. Replayed corrections and ordinary uploads cannot move frozen costs. The dashboard shows both dates. + +For attempts without billing IDs, the server checks the exact request tag. A complete empty lookup settles the attempt at $0 once it is 24 hours old. Missing and zero prices are checked again about once a day after freezing, until day 90, so late bills can still count. The saved check time survives server restarts. Failed lookups preserve the last verified price and cannot settle an attempt. After day 90 the saved total and any missing prices remain as they are. + +```mermaid +flowchart TD + Update[Update for a finished PR] --> Age{Before day 90?} + Age -- No --> Final[Keep saved total and freeze date] + Age -- Yes --> Change{What changed?} + Change -- Explicit correction --> Correct[Update ownership and total; save correction date] + Change -- Late bill --> Price[Refresh price when daily check is due] + Change -- Named window omission --> Keep[Keep saved claims and total] + Change -- Ordinary or stale upload --> Preserve[Keep frozen ownership] + Correct --> Dates[Keep original freeze date] + Price --> Dates +``` + +The backend also caps what each producer, contributor and organization stores: 32 repositories per installation; 64 repository snapshots, 32 MiB, 128,000 requests, 1,000 PRs and 128,000 billing IDs per contributor; and organization totals that grow with the number of contributors. An upload that would cross one is rejected with that scope's limit, and existing reports stay readable. + +## Health metrics + +The existing telemetry setting controls these metrics separately from `/pr-reporting`. They use the normal telemetry flush; there is no extra history scan or model call. + +- `kimchi.pr_cost.matching.count` counts one final outcome per input: `explicit`, `inferred`, `session`, `unknown`, `limited` (too much history to compare) or `failed`. +- `kimchi.pr_cost.delivery.count` counts delivery attempts, including account verification: `success`, `failed` or `canceled`. +- `kimchi.pr_cost.pricing.unpriced` records the unpriced request count from the latest local cost report. +- `kimchi.pr_cost.queue.depth` records repository snapshots waiting for acknowledgement, including zero after the queue clears. +- `kimchi.pr_cost.snapshot.requests` and `kimchi.pr_cost.snapshot.bytes` record the largest request count and the largest serialized size among the snapshots queued in the latest pass that queued one, after trimming. They show how close clients come to the upload limits. +- `kimchi.pr_cost.reconciliation.age` records seconds since the worker last started a scan. It measures liveness, not lookup success. + +Only `client=pi` and the fixed outcome appear as labels. The metrics add no session, user, work, request, repository or PR identifiers. Turning telemetry off drops buffered health counts and stops retries; turning it back on starts a new counter stream. An already dispatched request cannot be recalled. + +`kimchi config telemetry off` changes a local consent version in the config file. A quick `off` then `on` from another terminal also discards older health batches, even when the running session did not check settings while telemetry was off. This version stays local and is not sent as a metric label. + +Counters are cumulative within their OTLP start time; gauges describe the latest observed state and can decrease. Repeated exports must keep the latest value for that stream. Backend health queries use raw samples rather than session-based productivity rollups. These metrics help find missing prices and stuck delivery; they do not establish matching accuracy or change any PR total. + +## Validation scope + +Colocated tests cover fork target IDs, legacy refresh, source validation, exact billing IDs, privacy, missing prices, inferred and cross-repository allocation, consent, account changes, replacement revisions, restart, late acknowledgements, retries, the upload window and its urgent changes, trimming, server limits, sessions that never upload, response limits and shutdown. HTTP tests use local fixtures; they do not establish that a shared backend deployment is available. diff --git a/docs/work-attribution-accuracy.md b/docs/work-attribution-accuracy.md index 042fb8af1..0606b1347 100644 --- a/docs/work-attribution-accuracy.md +++ b/docs/work-attribution-accuracy.md @@ -8,11 +8,11 @@ pnpm exec tsx src/extensions/work-attribution/accuracy-cli.ts report.json refere The command reads these two files and prints the differences. It makes no network or model calls, reads no credentials and writes nothing. Keep real receipts and labels outside the repository. -By default, Kimchi saves the report at `~/.config/kimchi/harness/work//costs.json`. It contains request rows, PR totals and all six unallocated buckets for that work and every work connected to it. Works are connected when they contribute to the same PR or when a work link, such as a `/work link` correction, moves requests between them. Connections are transitive: if work A shares a PR with work B, and B shares another PR with work C, A's report also includes C's requests and PR totals. Prepare reference entries for every request in that file. Do not add per-work reports together: connected works repeat the same requests. +By default, Kimchi saves the report at `~/.config/kimchi/harness/work//costs.json`. It contains request rows, PR totals and all six unallocated buckets for that work and every work connected to it. Kimchi refreshes it at most every five minutes per work, and at once when it is missing, so a report saved right after a price arrived can lag `/work` by a few minutes. Works are connected when they contribute to the same PR or when a work link, such as a `/work link` correction, moves requests between them. Connections are transitive: if work A shares a PR with work B, and B shares another PR with work C, A's report also includes C's requests and PR totals. Prepare reference entries for every request in that file. Do not add per-work reports together: connected works repeat the same requests. ## Prepare the reference before inspecting matcher output -Use provider or gateway receipts to list requests, their billing accounts and exact billed prices. Label which PR each request belongs to using the task history and Git evidence. Do not generate this reference from `work.json`, `costs.json`, matcher confidence or the allocator under test: that would repeat the same mistakes on both sides. +Use provider or gateway receipts to list requests, their billing accounts and exact billed prices. Label which PR each request belongs to using the task history and Git evidence. Do not generate this reference from the work summary, `costs.json`, matcher confidence or the allocator under test: that would repeat the same mistakes on both sides. The version 1 reference has one entry per request: diff --git a/docs/work-attribution.md b/docs/work-attribution.md index 521f371f0..3a0433154 100644 --- a/docs/work-attribution.md +++ b/docs/work-attribution.md @@ -2,19 +2,21 @@ Kimchi keeps a local record of the model requests, file edits, plans and commits that belong to the same work. A `workId` connects them, even when planning and implementation happen in different sessions or worktrees of the same repository. -The result is `~/.config/kimchi/harness/work//work.json` in the user's home directory. Kimchi finds GitHub pull requests and GitLab merge requests for recorded commits, then looks up their request costs in the background. `/work` shows one total split into confirmed and inferred spend, or an open PR's spend so far, plus the work's priced spend; `costs.json` keeps the calculation. Incomplete billing stays unknown. These files are not uploaded. +The result is a summary in `~/.config/kimchi/harness/work//` in the user's home directory: `work.json` and its row logs. Kimchi finds GitHub pull requests and GitLab merge requests for recorded commits, then looks up their request costs in the background. `/work` shows one total split into confirmed and inferred spend, or an open PR's spend so far, plus the work's own priced spend; `costs.json` keeps the calculation. Incomplete billing stays unknown. These files are not uploaded. ## Where the files live -**`work.json` is stored per user, outside the project.** All projects use the same `~/.config/kimchi/harness/` directory, with a separate folder for each work ID. `~` means the home directory of the user running Kimchi. +**The summary is stored per user, outside the project.** All projects use the same `~/.config/kimchi/harness/` directory, with a separate folder for each work ID. `~` means the home directory of the user running Kimchi. ```text # Project: editable plan ~/src/example/.kimchi/plans/add-search.md -# User: summary and retained plan versions +# User: summary, cost reports and retained plan versions ~/.config/kimchi/harness/work//work.json +~/.config/kimchi/harness/work//rows/..jsonl ~/.config/kimchi/harness/work//scope.json +~/.config/kimchi/harness/work//cost-totals.json ~/.config/kimchi/harness/work//costs.json ~/.config/kimchi/harness/work//plans/add-search-.md @@ -42,7 +44,7 @@ Deleting a worktree removes its editable plan but leaves the user-level summary flowchart LR A["Session A: plan"] -->|save native plan or write Markdown| P["Saved planning evidence"] P -->|name its path or paste the saved native plan| B["Fresh session B: implement"] - A -->|requests| W["Local work.json"] + A -->|requests| W["Local work summary"] B -->|requests and commits| W ``` @@ -51,6 +53,16 @@ flowchart LR 3. **Continue in a fresh session.** Name the saved file, for example `Implement docs/adr/search.md`, or paste a native plan with its work-ID header. Pasted text must match a retained version. If you enabled model matching, a separate model call can also recognize a paraphrased task. Without matching evidence, the new session keeps its own work ID, even on the same branch. 4. **Implement and commit.** Kimchi adds requests, edits and linked commits to the work's summary. A native plan still works after its original worktree is deleted: use its retained path or paste its saved text. +## Browse works with `/work` + +In the terminal, `/work` opens a list of works. The current work comes first, marked `●`, followed by up to 29 works with activity in the last 35 days, newest first. Each row names the work by its saved plan's title, or else by repository and branch, with a short work ID. It also shows the work's own priced spend and PR state, for example `$0.0623 · PR #731 open` or `$0.0123 known so far · 2 PRs`. Own spend covers the work's requests and requests a correction linked into it, not other works on the same PR. + +Amounts in the list are rounded; the details keep exact values. `known so far` means some requests still have no price, including requests made since the last cost check. A work whose requests have no price yet shows `cost unknown`, never `$0`. The header adds up the cost reports of the listed works. Connected works share one report, so its requests count once, including those of connected works that are not listed. + +The selected row shows what `/work` prints, plus its repository, branch, last activity and request count. Only the current work has live PR lookup lines; other works show their saved PR links. Enter prints the selected work's details into the conversation, and Esc closes the list. The list reads only each work's small `work.json` and `cost-totals.json`, so it opens quickly for long works and never waits for the network. Other works without a readable `work.json` are left out until an update rebuilds their summary. + +ACP, RPC and print mode keep printing the current work's details as text. So do `/work new`, `/work `, `/work matching on|off` and `/work link|unlink`. + ## How Kimchi matches things ### 1. Choose the work ID before the request @@ -107,7 +119,7 @@ Model matching is off by default. `/work matching on` allows saved task text to Matching calls run before the main reply on the same small models as permission checks, in order: `kimchi-dev/deepseek-v4-flash-0731`, `kimchi-dev/glm-5.3-flash`, `kimchi-dev/minimax-m3`. When one cannot answer (no credentials, a provider error or an unusable reply), the next one decides. When the catalog has none of them, the model selected when you submit the message decides. These calls use the model's normal authentication and provider connection. They have no tools and do not enter the chat history. No local model, extra model setting or `judge` role is required. With no saved task to compare, the first message needs no matching call. -While matching is enabled, the first user message is kept in `work//intent.json` after its account is verified. Later checks use that text and the latest retained native plan. Candidates must belong to the same Git repository, API endpoint, organization and authenticated user. Rotating a key for the same account keeps that identity. Missing user identity or legacy text without an account prevents automatic semantic continuation; old text is not assigned to whoever is logged in now. Enabled redaction applies before sending. Raw text and credentials are not copied into `work.json`. +While matching is enabled, the first user message is kept in `work//intent.json` after its account is verified. Later checks use that text and the latest retained native plan. Candidates must belong to the same Git repository, API endpoint, organization and authenticated user. Rotating a key for the same account keeps that identity. Missing user identity or legacy text without an account prevents automatic semantic continuation; old text is not assigned to whoever is logged in now. Enabled redaction applies before sending. Raw text and credentials are not copied into the work summary. The model checks the current task first. To find earlier work, it first reads the message alone, then compares each eligible saved task separately. Exactly one must match and every competitor must be clearly different. Each check is a separate paid request with its own ID, saved before dispatch. Matching requests stay with the work active before the decision; adopting another work never moves those earlier requests. @@ -137,7 +149,15 @@ If account verification is unavailable for a new work's first input, the work ke Continuing a verified plan or native artifact also checks its producing request. Kimchi confirms the requests from that input under the continued work, whether they were ordinary session work, inferred or unresolved. For example, `Plan docs/new-feature.md` starts as session work; continuing its saved plan later confirms the planning requests automatically. Other inputs in that conversation stay unchanged. -This needs one recorded producer, an unchanged retained plan version or native edit, and matching account/repository scope. Missing or conflicting producer records stay unresolved. An existing correction or revocation takes precedence. The resulting `workLinks` entry names the plan/artifact evidence, producing `requestId` and `segmentId`; no source rows are rewritten. Model guesses do not create these confirmations. +This needs one recorded producer, the accepted retained plan version or native edit, and matching account/repository scope. Requests the work made before its delayed first scope stay unscoped and do not prevent this; a request saved under another account or repository does. Missing or conflicting producer records stay unresolved. An existing correction or revocation takes precedence. The resulting `workLinks` entry names the plan/artifact evidence, producing `requestId` and `segmentId`; no source rows are rewritten. Model guesses do not create these confirmations. + +Kimchi saves each accepted continuation, including a reference to the current work. Plan receipts retain a hash of the accepted bytes; receipts also retain the original account, repository and accepting input, plus the producing request when it is known. Editing a plan later does not change that receipt. + +After restart and during ordinary background reconciliation, Kimchi retries accepted continuations whose producer could not yet be confirmed. It uses the original scoped records and makes no model or authentication calls for this repair. A restored source record can complete the evidence. A later identical plan cannot replace a known producer; without a pinned producer, the source record must predate acceptance. Missing evidence stays unresolved and is retried. Old plan receipts without an acceptance hash remain unresolved, even when they name a retained snapshot; older native-artifact receipts can use their exact saved transition ID. This never changes original request IDs or exact prices, and it cannot replace a correction or revocation. + +Repair shares the existing reconciliation worker and lease. PR discovery and billing run first. Each repair pass reads history once and checks its three-second budget. It skips receipts that are confirmed or can never be, because their producing input already has a confirmation or correction or was an explicit choice, and examines at most 25 others, rotating past slow or failed candidates. Failed or interrupted passes do not mark history complete. When a pass has examined every remaining receipt and confirmed none, later passes skip the read until a journal changes. A retained plan file restored by hand is therefore checked after the next journal write. Once no receipt can still be confirmed, later passes read only the journals changed since, looking for new receipts. + +A crash can leave the last journal line unfinished. Readers skip that line while it has no final newline. The next append starts a new line and leaves the unfinished bytes in place, because another process may be appending to the same journal. Readers skip a record that was cut off mid-value once later records follow it, including journals written by v1.7.1, so another session can still continue a plan. Other malformed lines and invalid complete records still block confirmation and historical repair until the journal is repaired; the warning names the journal and its first damaged line. A complete record of a type this version does not know, for example one written by a newer Kimchi sharing this history, is not damage. It blocks confirmation only for the work and requests it names, with a warning that names its journal; other works are still confirmed. If planning and implementation already ended up in separate works, open the implementing work and run: @@ -145,11 +165,11 @@ If planning and implementation already ended up in separate works, open the impl /work link ``` -The planning work's `work.json` lists `requests[].segment.id`. The command selects requests already recorded for that input, including its retries and side calls. Other inputs in the same conversation stay separate. Both works and the selected requests must have matching saved account and repository scope. Older records without that evidence cannot be repaired this way. +The planning work's request rows list `segment.id`. The command selects requests already recorded for that input, including its retries and side calls. Other inputs in the same conversation stay separate. Both works and the selected requests must have matching saved account and repository scope. Older records without that evidence cannot be repaired this way. If an uncertain input already belongs to the right work, use that current work ID as the source. This records your confirmation without changing its original request or work IDs. `/work unlink ` also revokes this confirmation. -The command saves a `work_link` revision in the implementing work. It names the selected requests and their intended work; original request and work IDs stay unchanged. The target work's `work.json` keeps these revisions in `workLinks` for cost calculations. Model decisions do not retroactively rewrite earlier requests. +The command saves a `work_link` revision in the implementing work. It names the selected requests and their intended work; original request and work IDs stay unchanged. The target work's summary keeps these revisions in its `workLinks` rows for cost calculations. Model decisions do not retroactively rewrite earlier requests. To withdraw it, run `/work unlink ` in the implementing work. A revoked or conflicting correction leaves the affected assignment unknown until corrected again. Linking those requests from another work replaces the earlier link with a newer revision, including a revoked automatic confirmation. Only the work with the newest revision can then revoke it. @@ -160,7 +180,7 @@ flowchart LR R["Model request with requestId"] --> A["Assistant returns tool calls"] A --> T["Native write or edit"] T --> F["fileTransitions: requestId,
toolCallId, repository and path"] - F --> W["Same work.json"] + F --> W["Same work summary"] ``` One request can produce edits in several repositories. Each edit keeps the ID of the request that produced its tool call, even if another request runs before the file write finishes. The request's `cwd` remains the session's working directory; each edit records the repository and worktree it actually changed. @@ -285,9 +305,31 @@ Kimchi tries credentials in this order: an environment token for the selected ho Before sending a covered model request, Kimchi saves and flushes its request ID, work ID, session ID, input segment and account/repository scope. Missing scope is saved as `null`. It then sends `X-Request-Id`. This also works with telemetry disabled. -Records with the same work ID go into the same `work.json`. Requests are deduplicated by request ID; commit records keep the contributing sessions. `fileTransitions` keeps native edits, `fileObservations` keeps candidate Bash/MCP changes, `continuations` explains why another session adopted the work, and `workLinks` keeps correction revisions. IDs identify records; array positions have no meaning. +Records with the same work ID go into the same summary. Requests are deduplicated by request ID; commit records keep the contributing sessions. `fileTransitions` keeps native edits, `fileObservations` keeps candidate Bash/MCP changes, `continuations` explains why an input continued the current or another saved work, and `workLinks` keeps correction revisions. IDs identify records; positions have no meaning. + +`work.json` is a small manifest written on one line. It holds what `/work` lists (latest activity, repository, branch, retained plan and every commit's PR links) and one entry per log. Each collection is an append-only log, `rows/..jsonl`, with one JSON row per line. The work's sessions are a log too, `rows/sessions..jsonl`, which gains a row when a session first contributes. A later line with the same key replaces an earlier one and keeps its position. Keys are the request, transition, observation and session IDs; a commit's key is its session, SHA, repository and worktree; a plan's is its session, path and retained copy; a continuation's is its session, source and evidence; and a work link's is the whole revision. Only the first `bytes` of each log, as `work.json` records them, are committed (formatted here): + +```json +{ + "version": 2, + "workId": "6f102360-39a9-4d6e-8ac7-984c97c0e313", + "logs": { + "requests": { "generation": 3, "bytes": 33441207, "rows": 15012 }, + "commits": { "generation": 1, "bytes": 41877, "rows": 38 }, + "sessions": { "generation": 1, "bytes": 106, "rows": 2 } + } +} +``` + +To read one collection by hand, take its committed bytes and keep each key's last row: + +```sh +cd ~/.config/kimchi/harness/work/ +g=$(jq .logs.requests.generation work.json); n=$(jq .logs.requests.bytes work.json) +head -c "$n" "rows/requests.$g.jsonl" | jq -sc 'reduce .[] as $r ({}; .[$r.requestId] = $r) | [.[]]' +``` -A request record describes an attempt. It does not prove a successful response or a charge. Covered HTTP replies add their status and safe response IDs. Billing lookups later add exact prices when available. `work.json` contains paths, request metadata, PR links and billing rows, with no prompts or file contents. Retained native plans contain the plan text and stay local. PR lookup sends repository and commit identifiers to the repository's GitHub or GitLab API. +A request record describes an attempt. It does not prove a successful response or a charge. Covered HTTP replies add their status and safe response IDs. Billing lookups later add exact prices when available. The summary contains paths, request metadata, PR links and billing rows, with no prompts or file contents. Retained native plans contain the plan text and stay local. PR lookup sends repository and commit identifiers to the repository's GitHub or GitLab API.
Local files and recovery @@ -296,8 +338,10 @@ All paths below are inside the agent directory. | File | What it is for | | --- | --- | -| `work//work.json` | Readable summary of sessions, request attempts, plans, file edits, continuation evidence, commits and PR links. | -| `work//costs.json` | Derived PR totals and each request's allocation, price and billing lookup status. Rebuilt from source records when missing, and rewritten only after a journal or refresh failure changes. | +| `work//work.json` | Manifest: what `/work` lists and the committed length of each row log. Replacing it commits new rows. | +| `work//rows/..jsonl` | Readable summary rows: request attempts, plans, file edits, continuation evidence, commits, PR links and, in `sessions`, the contributing sessions. A later line with the same key replaces an earlier one. | +| `work//cost-totals.json` | What `/work` prints: PR totals, and the work's own requests and priced spend. Rewritten when it changes. | +| `work//costs.json` | Derived PR totals and each request's allocation, price and billing lookup status, for all connected works. Refreshed at most every five minutes after a journal or refresh failure changes, and rebuilt from source records when missing. | | `work-attribution/billing-polls.json` | Latest polling times and the last refresh that failed before any billing page, only for requests whose billing window is still open. Replaced after checks, so unchanged results and failed refreshes do not grow the journal. | | `work//scope.json` | Original API endpoint, organization, user and Git common-directory identity. Credentials are not saved here. | | `work//intent.json` | Saved task text for opt-in model matching, bound to its original account and repository. | @@ -309,9 +353,11 @@ All paths below are inside the agent directory. | `work-attribution/pr-checks.json` | When each recorded repository and commit was last checked for a PR or MR, including checks whose unchanged result was not appended. A restart therefore keeps the lookup schedule. Kimchi replaces the file at most once per pass and only after a change. It drops entries for commits it no longer records and for checks more than a day old. Deleting it costs at most one extra check per commit. | - Each plan record has `path` for the editable file, `snapshotPath` for its retained version and `contentHash` to detect changes. Plans produced by an attributed tool also name its `requestId`. If saving the retained copy fails, Kimchi shows a warning and leaves the local plan usable. Naming or pasting that plan later then does not continue its work or confirm its producing input; `/work ` still selects it. -- Kimchi flushes source records before updating `work.json`. Writers merge under a lock and replace the summary atomically. Shutdown waits for queued summary writes. -- Recovery saves the time of its last successful scan plus each summary's size and modification time. It reads source logs changed since then and skips writing unchanged results. -- A missing or invalid known summary triggers a full replay of the source logs, including edit journals. A failed log read or summary write leaves the checkpoint unchanged. Older summaries remain readable when the new collections are absent. +- Kimchi flushes source records before updating the summary. Writers merge under a per-work lock, append only changed rows, sync them, then replace `work.json`, which commits them. Readers stop at each log's committed length, so an interrupted append is ignored and the next writer removes it. Shutdown waits for queued summary writes. +- When a log's superseded lines outweigh half its current rows and exceed 1 MiB, the writer copies each key's latest row into the next generation, commits it and deletes the old file. A reader that loaded the old manifest rereads it once. +- Recovery saves the time of its last successful scan plus each manifest's size and modification time. It reads source logs changed since then and skips writing unchanged results. +- A missing or invalid summary, a log shorter than its committed length, or a damaged row triggers a full replay of the work's source logs, including edit journals, into a new generation. A failed log read or summary write leaves the checkpoint unchanged. +- A `work.json` that is not a version 2 manifest, such as a `"version": 1` summary saved by v1.7.1 or earlier, is deleted the next time Kimchi lists or updates that work; no other file is touched. The summary is then rebuilt like a missing one, and `/work` leaves the work out until it is. `costs.json`, `cost-totals.json` and PR cost reports come from the source logs, so they do not change. Earlier version 2 manifests that list `sessions` themselves stay readable; the work's next update moves them into the sessions log.
@@ -391,7 +437,7 @@ An exact price does not confirm a model's task match. Inferred requests remain l A refresh that fails before any billing page arrives adds no evidence. This covers being offline, DNS or TLS errors, a failed API key check, an HTTP error such as 429 or 5xx on the first page, and Kimchi's pass deadline. The last known price stays usable and nothing is added to the journal. `billing-polls.json` keeps the failure, the request's `billingLookup` in `costs.json` shows it with its real timestamp, and `/work` says that the last refresh failed. Errors after the first page, including partial pagination, keep the full total unknown. Failure records saved by earlier versions also stay unknown until a complete lookup succeeds, because they do not record whether a page had arrived; their pass-deadline records marked as happening before any page are the exception. -Billing details appear on the matching `work.json` request. This excerpt uses example IDs; comments explain the fields and are not part of the stored JSON: +Billing details appear on the matching request row in `rows/requests..jsonl`. This excerpt uses example IDs; comments explain the fields and are not part of the stored JSON: ```jsonc { @@ -465,14 +511,14 @@ The footer and Ferment budgets have different token scopes. Ferment V2 counts pa ## Turning Cost per PR off -Cost per PR is listed in `/resources` like other built-in extensions and is on by default. Disabling it there, or with `kimchi resources disable extensions.cost-per-pr`, takes effect at the next start: Kimchi records no work, looks up no PRs and shows no `/work` or PR/MR status. Other extensions stop recording work at once. Records already saved stay on disk. +Cost per PR is listed in `/resources` like other built-in extensions and is on by default. Disabling it there, or with `kimchi resources disable extensions.cost-per-pr`, takes effect at the next start: Kimchi records no work, looks up no PRs, shows no `/work` or PR/MR status and uploads no PR cost reports. Other extensions stop recording work at once. Records already saved stay on disk. ## What still needs work - **Shared costs:** planning and requests without complete exclusive edit evidence stay shared when the work spans several PRs. - **Missing request identity:** older untagged attempts and requests whose billing tag had to be skipped stay unpriced. - **Stable repository IDs:** recover stable provider IDs for older links that lack them. -- **Historical repair:** old records without account or producer evidence stay unresolved. Automatic replay of newly available evidence is still separate work. +- **Historical repair:** accepted plan and artifact continuations are retried automatically, but plan receipts saved without an acceptance hash and records without account or producer evidence stay unresolved. - **Remote agents:** link their requests and returned commits to the local work. Remote sessions are outside this MVP.
diff --git a/src/cli.ts b/src/cli.ts index c014ff3fe..360b136f8 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -118,6 +118,7 @@ import piiRedactionExtension from "./extensions/pii-redaction/index.js" import plannotatorExtension from "./extensions/plannotator/index.js" import pluginPackageHooksAdapter from "./extensions/plugin-package-hook-adapter/index.js" import powershellGateExtension from "./extensions/powershell-gate.js" +import prCostReportingExtension from "./extensions/pr-cost-reporting/index.js" import { setPrintGate } from "./extensions/print-mode.js" import promptEnrichmentExtension from "./extensions/prompt-construction/prompt-enrichment.js" import promptSummaryExtension from "./extensions/prompt-summary.js" @@ -837,6 +838,7 @@ try { ...enabledExtensionFactories([ { id: COST_PER_PR_RESOURCE_ID, factory: createWorkAttributionExtension() }, { id: COST_PER_PR_RESOURCE_ID, factory: pullRequestStatusExtension }, + { id: COST_PER_PR_RESOURCE_ID, factory: prCostReportingExtension }, ] satisfies ManagedExtensionFactory[]), requestTimingExtension, llmResponseLogExtension, diff --git a/src/config.test.ts b/src/config.test.ts index dab387ff4..3bdbf0312 100644 --- a/src/config.test.ts +++ b/src/config.test.ts @@ -30,6 +30,7 @@ import { writeHideTips, writeSessionModeWizardSeenAt, writeStudioOnboardingSeenAt, + writeTelemetryEnabled, writeTeleportCompactHintEnabled, writeTuiWheelScrollLines, } from "./config.js" @@ -1032,6 +1033,21 @@ describe("readTelemetryConfig", () => { expect(config.metricsEndpoint).toBe("https://custom.example.com/metrics:ingest") }) + it("persists consent changes across off/on while repeated on keeps the current version", () => { + writeFileSync(configPath, JSON.stringify({ telemetry: { enabled: true } })) + expect(readTelemetryConfig(configPath).consentVersion).toBeUndefined() + writeTelemetryEnabled(false, configPath) + const disabled = readTelemetryConfig(configPath) + expect(disabled.enabled).toBe(false) + expect(disabled.consentVersion).toEqual(expect.any(String)) + writeTelemetryEnabled(true, configPath) + const enabled = readTelemetryConfig(configPath) + expect(enabled.enabled).toBe(true) + expect(enabled.consentVersion).not.toBe(disabled.consentVersion) + writeTelemetryEnabled(true, configPath) + expect(readTelemetryConfig(configPath).consentVersion).toBe(enabled.consentVersion) + }) + it("falls back to default metrics endpoint when absent", () => { writeFileSync( configPath, diff --git a/src/config.ts b/src/config.ts index 00f6c2f41..3da3a6f9f 100644 --- a/src/config.ts +++ b/src/config.ts @@ -1,3 +1,4 @@ +import { randomUUID } from "node:crypto" import { existsSync, readFileSync, statSync } from "node:fs" import { homedir } from "node:os" import { join, relative, resolve } from "node:path" @@ -95,6 +96,7 @@ export function buildSkillPathOptions(discoveredDirs: string[]): string[] { export interface TelemetryConfig { enabled: boolean + consentVersion?: string endpoint: string metricsEndpoint: string headers: Record @@ -524,6 +526,7 @@ export function readTelemetryConfig(configPath?: string): TelemetryConfig { const path = configPath ?? KIMCHI_CONFIG_PATH const envEnabled = process.env.KIMCHI_TELEMETRY_ENABLED let fileEnabled: boolean | undefined + let fileConsentVersion: string | undefined let fileEndpoint: string | undefined let fileMetricsEndpoint: string | undefined let fileHeaders: Record | undefined @@ -538,6 +541,7 @@ export function readTelemetryConfig(configPath?: string): TelemetryConfig { const t = parsed.telemetry if (t && typeof t === "object") { if (typeof t.enabled === "boolean") fileEnabled = t.enabled + if (typeof t.consentVersion === "string") fileConsentVersion = t.consentVersion if (typeof t.endpoint === "string" && t.endpoint.length > 0) fileEndpoint = t.endpoint if (typeof t.metricsEndpoint === "string" && t.metricsEndpoint.length > 0) fileMetricsEndpoint = t.metricsEndpoint if (t.headers && typeof t.headers === "object" && !Array.isArray(t.headers)) { @@ -579,6 +583,7 @@ export function readTelemetryConfig(configPath?: string): TelemetryConfig { return { enabled, + consentVersion: fileConsentVersion, endpoint: fileEndpoint ?? telemetryLogsUrl(region), metricsEndpoint: fileMetricsEndpoint ?? telemetryMetricsUrl(region), headers, @@ -1143,6 +1148,8 @@ export function writeTelemetryEnabled(enabled: boolean, configPath?: string): vo const path = configPath ?? KIMCHI_CONFIG_PATH updateConfigFile(path, (raw) => { const t = (raw.telemetry as Record | undefined) ?? {} + // A different process may turn telemetry off and on between export attempts. + if (t.enabled !== enabled) t.consentVersion = randomUUID() t.enabled = enabled raw.telemetry = t }) diff --git a/src/extensions/__mocks__/theme.ts b/src/extensions/__mocks__/theme.ts index 9758c9343..bc31348d3 100644 --- a/src/extensions/__mocks__/theme.ts +++ b/src/extensions/__mocks__/theme.ts @@ -1,11 +1,14 @@ import { Theme } from "@earendil-works/pi-coding-agent" import palette from "../../../themes/kimchi.json" with { type: "json" } +import lightPalette from "../../../themes/kimchi-light.json" with { type: "json" } // Use real ANSI styling so width and terminal-control checks exercise coloured output. -const colors = { - ...palette.colors, - ...Object.fromEntries( - Object.entries(palette.colors).map(([key, value]) => [key, Reflect.get(palette.vars, value) ?? value]), - ), +function paletteTheme({ colors, vars }: { colors: typeof palette.colors; vars: object }): Theme { + const resolved = { + ...colors, + ...Object.fromEntries(Object.entries(colors).map(([key, value]) => [key, Reflect.get(vars, value) ?? value])), + } + return new Theme(resolved, resolved, "truecolor") } -export const testTheme = new Theme(colors, colors, "truecolor") +export const testTheme = paletteTheme(palette) +export const lightTestTheme = paletteTheme(lightPalette) diff --git a/src/extensions/__mocks__/work-summary.ts b/src/extensions/__mocks__/work-summary.ts new file mode 100644 index 000000000..e6a2dbb4c --- /dev/null +++ b/src/extensions/__mocks__/work-summary.ts @@ -0,0 +1,34 @@ +import { join } from "node:path" +import { + COLLECTIONS, + type Collection, + ROW_KEYS, + type Row, + readWorkSummary, + updateWorkRows, + type WorkSummaryView, +} from "../work-attribution/row-log.js" + +/** A work's saved rows, folded from its logs; fails the test when the summary is missing or damaged. */ +export function savedWorkSummary(agentDir: string, workId: string): WorkSummaryView { + const summary = readWorkSummary(agentDir, workId) + if (!summary) throw new Error(`No readable work summary for ${workId}`) + return summary +} + +/** Saves rows as the summary of a work that has none yet, as the writer commits merged records. */ +export async function saveWorkSummary( + agentDir: string, + { workId, sessions, ...rows }: Pick & Partial>, +): Promise { + await updateWorkRows( + join(agentDir, "work", workId), + workId, + { check: () => true, assertLease: () => {} }, + async (summary) => { + for (const sessionId of sessions) summary.sessions.add(sessionId) + for (const collection of COLLECTIONS) + for (const row of rows[collection] ?? []) summary[collection].set(ROW_KEYS[collection](row), row) + }, + ) +} diff --git a/src/extensions/pr-cost-reporting/agreement.test.ts b/src/extensions/pr-cost-reporting/agreement.test.ts new file mode 100644 index 000000000..e166b7820 --- /dev/null +++ b/src/extensions/pr-cost-reporting/agreement.test.ts @@ -0,0 +1,135 @@ +import { readFileSync } from "node:fs" +import { afterEach, beforeEach, expect, it, vi } from "vitest" +import type { WorkPullRequest } from "../pull-request-status/pull-requests.js" +import { calculatePullRequestCosts, decimalNanos, usd } from "../work-attribution/costs.js" +import type { WorkRecord } from "../work-attribution/summary.js" +import { buildSnapshots } from "./snapshot.js" + +// The backend checks the same content in services/ai-optimizer/internal/services/prcosts/testdata/client-agreement.golden. +// Update both copies together: they pin where /work and the console must agree. +const account = { + apiUrl: "https://api.example", + organizationId: "11111111-1111-4111-8111-111111111111", + userId: "22222222-2222-4222-8222-222222222222", +} +const started = "2026-10-04T12:00:00.000Z" +const merged = "2026-10-04T13:00:00.000Z" +const repositories = new Map([ + ["/repo-a/.git", { provider: "github" as const, host: "github.com", id: "42", name: "example/a" }], + ["/repo-b/.git", { provider: "github" as const, host: "github.com", id: "43", name: "example/b" }], +]) +beforeEach(() => { + vi.useFakeTimers({ toFake: ["Date"] }) + vi.setSystemTime(new Date("2026-10-07T12:00:00Z")) +}) +afterEach(() => vi.useRealTimers()) + +const id = (prefix: string, n: number) => `${prefix}0000000-0000-4000-8000-00000000000${n}` +function pull(number: number, repository: "a" | "b"): WorkPullRequest { + return { + provider: "github", + id: String(number), + repositoryId: repository === "a" ? "42" : "43", + host: "github.com", + repository: `example/${repository}`, + number, + url: `https://github.com/example/${repository}/pull/${number}`, + state: "merged", + mergedAt: merged, + closedAt: merged, + checkedAt: merged, + headSha: "a".repeat(40), + mergeCommitSha: "b".repeat(40), + } +} +function work(n: number, pulls: [WorkPullRequest, "a" | "b"][]): WorkRecord[] { + return pulls.map(([pr, repository], index) => ({ + version: 1, + type: "commit", + workId: id("1", n), + sessionId: "session", + sha: String(n * 10 + index).padStart(40, "0"), + repository: `/repo-${repository}/.git`, + worktree: `/repo-${repository}`, + recordedAt: merged, + pullRequests: [pr], + })) +} +function request(n: number, workNumber: number, attribution: string, startedAt = started): WorkRecord { + return { + version: 1, + type: "request", + workId: id("1", workNumber), + sessionId: "session", + requestId: id("2", n), + startedAt, + recordedAt: startedAt, + scope: { account, repository: "/repo-a/.git" }, + segment: { id: `segment-${n}`, attribution, reason: attribution }, + } +} + +it("pins the PR totals that /work and the console must agree on", () => { + const records = [ + // 101: confirmed spend, a verified no-charge attempt and later post-merge work. + ...work(1, [[pull(101, "a"), "a"]]), + request(1, 1, "explicit"), + request(6, 1, "explicit"), + request(7, 1, "explicit", "2026-10-04T14:00:00.000Z"), + // 102: a model match is inferred spend. + ...work(2, [[pull(102, "a"), "a"]]), + request(2, 2, "inferred"), + // Unresolved matching with no candidate PR leaves every PR complete. + request(3, 3, "unknown"), + // 103: an explicit attempt with no bill yet stays on its PR, which stays incomplete until the bill arrives. + ...work(4, [[pull(103, "a"), "a"]]), + request(4, 4, "explicit"), + // 104 and 201: one attempt shared across repositories is counted once and leaves both incomplete. + ...work(5, [ + [pull(104, "a"), "a"], + [pull(201, "b"), "b"], + ]), + request(5, 5, "explicit"), + ] + const bills = [ + { request: 1, costUsd: "1.25" }, + { request: 2, costUsd: "2" }, + { request: 3, costUsd: "3" }, + { request: 5, costUsd: "5" }, + { request: 7, costUsd: "7" }, + ].map(({ request, costUsd }) => ({ + requestId: id("2", request), + billingRecordId: id("3", request), + costUsd, + account, + })) + const noCharge = new Map([[id("2", 6), account]]) + const report = calculatePullRequestCosts(records, bills, new Set(), noCharge) + const { snapshots, incomplete } = buildSnapshots(records, report, repositories) + expect(incomplete).toBe(false) + // Each attempt appears once in the request list, whichever bucket or PR it belongs to. + const organizationNanos = report.requests.reduce((sum, row) => sum + (decimalNanos(row.knownCostUsd) ?? 0n), 0n) + const fixture = { + actorId: account.userId, + snapshots: snapshots.map((snapshot, index) => ({ + schemaVersion: 1, + producerId: id("4", index + 1), + revision: "1", + generatedAt: "2026-10-07T12:00:00.000Z", + ...snapshot.content, + })), + bills: bills.map((bill) => ({ id: bill.billingRecordId, ownerId: account.userId, totalPrice: bill.costUsd })), + noChargeRequestIds: [...noCharge.keys()], + organizationKnownCostUsd: usd(organizationNanos), + pullRequests: report.pullRequests + .map((pr) => ({ + repositoryId: pr.pullRequest?.repositoryId, + id: pr.pullRequest?.id, + knownCostUsd: pr.knownCostUsd, + complete: pr.totalCostUsd !== null, + })) + .sort((a, b) => `${a.repositoryId}/${a.id}`.localeCompare(`${b.repositoryId}/${b.id}`)), + } + const committed = JSON.parse(readFileSync(new URL("./client-agreement.json", import.meta.url), "utf8")) + expect(fixture).toEqual(committed) +}) diff --git a/src/extensions/pr-cost-reporting/automation.test.ts b/src/extensions/pr-cost-reporting/automation.test.ts new file mode 100644 index 000000000..ed3cad534 --- /dev/null +++ b/src/extensions/pr-cost-reporting/automation.test.ts @@ -0,0 +1,43 @@ +import { describe, expect, it } from "vitest" +import { uploadSkipReason } from "./automation.js" + +describe("PR cost uploads from automated runs", () => { + it.each(["tui", "rpc"] as const)("lets an interactive %s session upload", (mode) => { + expect(uploadSkipReason(mode, {})).toBeUndefined() + }) + it.each([ + ["print", "non-interactive print mode"], + ["json", "non-interactive JSON mode"], + ] as const)("skips a one-shot %s run", (mode, reason) => { + expect(uploadSkipReason(mode, {})).toBe(reason) + }) + it.each([ + "CI", + "GITHUB_ACTIONS", + "GITLAB_CI", + "BUILDKITE", + "JENKINS_URL", + "TEAMCITY_VERSION", + "CIRCLECI", + "TF_BUILD", + ])("skips a CI job detected through %s", (name) => { + expect(uploadSkipReason("tui", { [name]: name === "JENKINS_URL" ? "https://jenkins.example/" : "true" })).toBe( + `CI environment (${name})`, + ) + }) + it.each(["", " ", "0", "false", "FALSE"])("treats CI=%j as a developer terminal", (value) => { + expect(uploadSkipReason("tui", { CI: value })).toBeUndefined() + }) + it.each(["0", "false", "off", "no"])("honors KIMCHI_PR_COST_REPORTING=%s in every mode", (value) => { + expect(uploadSkipReason("rpc", { KIMCHI_PR_COST_REPORTING: value })).toBe("KIMCHI_PR_COST_REPORTING=0") + }) + it("offers no way around CI detection", () => { + expect(uploadSkipReason("tui", { KIMCHI_PR_COST_REPORTING: "1", CI: "true" })).toBe("CI environment (CI)") + }) + it("skips benchmark runs, the only users of the infrastructure breaker", () => { + expect(uploadSkipReason("tui", { KIMCHI_INFRA_BREAKER_THRESHOLD: "3" })).toBe( + "benchmark run (KIMCHI_INFRA_BREAKER_THRESHOLD)", + ) + expect(uploadSkipReason("tui", { KIMCHI_INFRA_BREAKER_THRESHOLD: "0" })).toBeUndefined() + }) +}) diff --git a/src/extensions/pr-cost-reporting/automation.ts b/src/extensions/pr-cost-reporting/automation.ts new file mode 100644 index 000000000..8290c9ed3 --- /dev/null +++ b/src/extensions/pr-cost-reporting/automation.ts @@ -0,0 +1,30 @@ +import type { ExtensionContext } from "@earendil-works/pi-coding-agent" +import { INFRA_BREAKER_THRESHOLD_ENV, resolveInfrastructureBreakerThreshold } from "../../upstream-retry-patch.js" + +export const CI_VARIABLES = [ + "CI", + "GITHUB_ACTIONS", + "GITLAB_CI", + "BUILDKITE", + "JENKINS_URL", + "TEAMCITY_VERSION", + "CIRCLECI", + "TF_BUILD", +] + +const disabled = (value: string | undefined) => ["0", "false", "off", "no"].includes(value?.trim().toLowerCase() ?? "") + +/** + * Why this process must not upload PR costs, or undefined when it may. CI, one-shot print or JSON runs + * and benchmarks (the only callers of the infrastructure breaker) would report noise. Interactive TUI, + * ACP and RPC sessions still report. Local attribution runs either way. + */ +export function uploadSkipReason(mode: ExtensionContext["mode"], env = process.env): string | undefined { + if (disabled(env.KIMCHI_PR_COST_REPORTING)) return "KIMCHI_PR_COST_REPORTING=0" + if (mode === "print") return "non-interactive print mode" + if (mode === "json") return "non-interactive JSON mode" + const ci = CI_VARIABLES.find((name) => env[name]?.trim() && !disabled(env[name])) + if (ci) return `CI environment (${ci})` + if (resolveInfrastructureBreakerThreshold(env) > 0) return `benchmark run (${INFRA_BREAKER_THRESHOLD_ENV})` + return undefined +} diff --git a/src/extensions/pr-cost-reporting/client-agreement.json b/src/extensions/pr-cost-reporting/client-agreement.json new file mode 100644 index 000000000..d9f5137b2 --- /dev/null +++ b/src/extensions/pr-cost-reporting/client-agreement.json @@ -0,0 +1,223 @@ +{ + "actorId": "22222222-2222-4222-8222-222222222222", + "snapshots": [ + { + "schemaVersion": 1, + "producerId": "40000000-0000-4000-8000-000000000001", + "revision": "1", + "generatedAt": "2026-10-07T12:00:00.000Z", + "repository": { + "provider": "github", + "host": "github.com", + "id": "42", + "name": "example/a" + }, + "pullRequests": [ + { + "id": "101", + "number": 101, + "url": "https://github.com/example/a/pull/101", + "state": "merged", + "mergedAt": "2026-10-04T13:00:00.000Z" + }, + { + "id": "102", + "number": 102, + "url": "https://github.com/example/a/pull/102", + "state": "merged", + "mergedAt": "2026-10-04T13:00:00.000Z" + }, + { + "id": "103", + "number": 103, + "url": "https://github.com/example/a/pull/103", + "state": "merged", + "mergedAt": "2026-10-04T13:00:00.000Z" + }, + { + "id": "104", + "number": 104, + "url": "https://github.com/example/a/pull/104", + "state": "merged", + "mergedAt": "2026-10-04T13:00:00.000Z" + } + ], + "requests": [ + { + "requestId": "20000000-0000-4000-8000-000000000001", + "billingRecordIds": ["30000000-0000-4000-8000-000000000001"], + "startedAt": "2026-10-04T12:00:00.000Z", + "allocation": { + "kind": "pull-request", + "pullRequestIds": ["101"], + "method": "explicit" + } + }, + { + "requestId": "20000000-0000-4000-8000-000000000002", + "billingRecordIds": ["30000000-0000-4000-8000-000000000002"], + "startedAt": "2026-10-04T12:00:00.000Z", + "allocation": { + "kind": "pull-request", + "pullRequestIds": ["102"], + "method": "model" + } + }, + { + "requestId": "20000000-0000-4000-8000-000000000003", + "billingRecordIds": ["30000000-0000-4000-8000-000000000003"], + "startedAt": "2026-10-04T12:00:00.000Z", + "allocation": { + "kind": "unknown", + "pullRequestIds": [], + "method": "session" + } + }, + { + "requestId": "20000000-0000-4000-8000-000000000004", + "billingRecordIds": [], + "startedAt": "2026-10-04T12:00:00.000Z", + "allocation": { + "kind": "pull-request", + "pullRequestIds": ["103"], + "method": "explicit" + } + }, + { + "requestId": "20000000-0000-4000-8000-000000000005", + "billingRecordIds": ["30000000-0000-4000-8000-000000000005"], + "startedAt": "2026-10-04T12:00:00.000Z", + "allocation": { + "kind": "unknown", + "pullRequestIds": ["104"], + "method": "explicit" + } + }, + { + "requestId": "20000000-0000-4000-8000-000000000006", + "billingRecordIds": [], + "startedAt": "2026-10-04T12:00:00.000Z", + "allocation": { + "kind": "pull-request", + "pullRequestIds": ["101"], + "method": "explicit" + } + }, + { + "requestId": "20000000-0000-4000-8000-000000000007", + "billingRecordIds": ["30000000-0000-4000-8000-000000000007"], + "startedAt": "2026-10-04T14:00:00.000Z", + "allocation": { + "kind": "post-merge", + "pullRequestIds": ["101"], + "method": "explicit" + } + } + ], + "coverage": { + "observedRequests": 7, + "unpricedRequests": 1, + "historyComplete": true + } + }, + { + "schemaVersion": 1, + "producerId": "40000000-0000-4000-8000-000000000002", + "revision": "1", + "generatedAt": "2026-10-07T12:00:00.000Z", + "repository": { + "provider": "github", + "host": "github.com", + "id": "43", + "name": "example/b" + }, + "pullRequests": [ + { + "id": "201", + "number": 201, + "url": "https://github.com/example/b/pull/201", + "state": "merged", + "mergedAt": "2026-10-04T13:00:00.000Z" + } + ], + "requests": [ + { + "requestId": "20000000-0000-4000-8000-000000000005", + "billingRecordIds": ["30000000-0000-4000-8000-000000000005"], + "startedAt": "2026-10-04T12:00:00.000Z", + "allocation": { + "kind": "unknown", + "pullRequestIds": ["201"], + "method": "explicit" + } + } + ], + "coverage": { + "observedRequests": 1, + "unpricedRequests": 0, + "historyComplete": true + } + } + ], + "bills": [ + { + "id": "30000000-0000-4000-8000-000000000001", + "ownerId": "22222222-2222-4222-8222-222222222222", + "totalPrice": "1.25" + }, + { + "id": "30000000-0000-4000-8000-000000000002", + "ownerId": "22222222-2222-4222-8222-222222222222", + "totalPrice": "2" + }, + { + "id": "30000000-0000-4000-8000-000000000003", + "ownerId": "22222222-2222-4222-8222-222222222222", + "totalPrice": "3" + }, + { + "id": "30000000-0000-4000-8000-000000000005", + "ownerId": "22222222-2222-4222-8222-222222222222", + "totalPrice": "5" + }, + { + "id": "30000000-0000-4000-8000-000000000007", + "ownerId": "22222222-2222-4222-8222-222222222222", + "totalPrice": "7" + } + ], + "noChargeRequestIds": ["20000000-0000-4000-8000-000000000006"], + "organizationKnownCostUsd": "18.250000000", + "pullRequests": [ + { + "repositoryId": "42", + "id": "101", + "knownCostUsd": "1.250000000", + "complete": true + }, + { + "repositoryId": "42", + "id": "102", + "knownCostUsd": "2.000000000", + "complete": true + }, + { + "repositoryId": "42", + "id": "103", + "knownCostUsd": "0.000000000", + "complete": false + }, + { + "repositoryId": "42", + "id": "104", + "knownCostUsd": "0.000000000", + "complete": false + }, + { + "repositoryId": "43", + "id": "201", + "knownCostUsd": "0.000000000", + "complete": false + } + ] +} diff --git a/src/extensions/pr-cost-reporting/index.test.ts b/src/extensions/pr-cost-reporting/index.test.ts new file mode 100644 index 000000000..97926b12a --- /dev/null +++ b/src/extensions/pr-cost-reporting/index.test.ts @@ -0,0 +1,291 @@ +import { existsSync, mkdtempSync, rmSync, writeFileSync } from "node:fs" +import { tmpdir } from "node:os" +import { join } from "node:path" +import type { ExtensionContext, SessionStartEvent } from "@earendil-works/pi-coding-agent" +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest" +import { INFRA_BREAKER_THRESHOLD_ENV } from "../../upstream-retry-patch.js" +import { createCommandContext, createContext } from "../__mocks__/context.js" +import { createExtensionApi } from "../__mocks__/extension-api.js" +import * as supervisor from "../work-attribution/reconcile-supervisor.js" +import { WORK_CHANGED_EVENT, WORK_STATE_REQUEST_EVENT, type WorkStateRequest } from "../work-attribution.js" +import { CI_VARIABLES } from "./automation.js" +import reportingExtension from "./index.js" +import * as queue from "./queue.js" +import { queueSnapshots, readReportingState, setReportingEnabled } from "./queue.js" +import { reconcileReporting } from "./worker.js" + +vi.mock("../work-attribution/reconcile-supervisor.js", () => ({ + subscribeReportingReconciliation: vi.fn(), + requestWorkReconciliation: vi.fn(), +})) +vi.mock("./worker.js", () => ({ reconcileReporting: vi.fn() })) +const mode = vi.hoisted(() => ({ acp: false })) +vi.mock("../../modes/acp/state.js", () => ({ + get IS_ACP_MODE() { + return mode.acp + }, +})) +let directory: string +beforeEach(() => { + directory = mkdtempSync(join(tmpdir(), "kimchi-reporting-extension-")) + vi.stubEnv("PI_CODING_AGENT_DIR", directory) + vi.stubEnv("KIMCHI_TELEMETRY_ENABLED", "true") + // CI runners set these; each test opts into automation explicitly. + for (const name of [...CI_VARIABLES, "KIMCHI_PR_COST_REPORTING", INFRA_BREAKER_THRESHOLD_ENV]) vi.stubEnv(name, "") + vi.mocked(supervisor.subscribeReportingReconciliation).mockReturnValue(async () => {}) +}) +afterEach(() => { + vi.clearAllMocks() + vi.unstubAllEnvs() + rmSync(directory, { recursive: true, force: true }) +}) +/** An extension API whose work tracking reports `current()` as the tracked main session. */ +function trackedApi(current: () => ExtensionContext) { + const api = createExtensionApi() + api.api.events.on(WORK_STATE_REQUEST_EVENT, (value) => { + Object.assign(value as WorkStateRequest, { tracking: true, current: { workId: "test-work", ctx: current() } }) + }) + return api +} + +describe("optional PR reporting", () => { + it("does not break session startup or turn completion when reporting state is damaged", async () => { + await setReportingEnabled(directory, true) + writeFileSync(join(directory, "pr-cost-reporting", "state.json"), "{broken") + const ctx = createContext() + const api = trackedApi(() => ctx) + reportingExtension(api.api) + await expect( + api.getHandler("session_start")({ type: "session_start", reason: "new" }, ctx), + ).resolves.toBeUndefined() + await expect(Promise.resolve(api.getHandler("agent_end")({}, ctx))).resolves.toBeUndefined() + expect(supervisor.requestWorkReconciliation).not.toHaveBeenCalled() + await api.getHandler("session_shutdown")({}, ctx) + }) + it("shows the default-on notice once across launches and says how to turn it off", async () => { + const ctx = createContext() + const notices: unknown[] = [] + for (let launch = 0; launch < 2; launch++) { + const api = trackedApi(() => ctx) + reportingExtension(api.api) + await api.getHandler("session_start")({ type: "session_start", reason: "new" }, ctx) + await api.getHandler("agent_end")({}, ctx) + await api.getHandler("session_shutdown")({}, ctx) + notices.push(...api.getAppendedEntries("pr-cost-reporting-notice")) + } + expect(notices).toEqual([expect.stringContaining("/pr-reporting off")]) + // Consecutive info notifications replace one another in the TUI. + expect(ctx.ui.notify).not.toHaveBeenCalled() + }) + it("shows the default-on notice in Studio after the first turn, once the client knows the session", async () => { + mode.acp = true + try { + const ctx = createContext() + const api = trackedApi(() => ctx) + reportingExtension(api.api) + await api.getHandler("session_start")({ type: "session_start", reason: "new" }, ctx) + // Studio drops notifications for a session it has not registered yet. + expect(ctx.ui.notify).not.toHaveBeenCalled() + expect((await readReportingState(directory)).defaultNoticeShown).toBeUndefined() + await api.getHandler("agent_end")({}, ctx) + await api.getHandler("session_shutdown")({}, ctx) + expect(ctx.ui.notify).toHaveBeenCalledWith(expect.stringContaining("/pr-reporting off"), "info") + expect(api.getAppendedEntries("pr-cost-reporting-notice")).toEqual([]) + } finally { + mode.acp = false + } + }) + it("lets the turn finish while reporting state is still being read", async () => { + await setReportingEnabled(directory, true) + const state = await readReportingState(directory) + const ctx = createContext() + const api = trackedApi(() => ctx) + reportingExtension(api.api) + await api.getHandler("session_start")({ type: "session_start", reason: "new" }, ctx) + let release!: (value: queue.ReportingState) => void + const blocked = new Promise((resolve) => { + release = resolve + }) + const read = vi.spyOn(queue, "readReportingState").mockReturnValueOnce(blocked) + let finished = false + const dispatched = Promise.resolve(api.getHandler("agent_end")({}, ctx)).then(() => { + finished = true + }) + try { + await vi.waitFor(() => expect(finished).toBe(true), { timeout: 100 }) + expect(supervisor.requestWorkReconciliation).not.toHaveBeenCalled() + } finally { + release(state) + await dispatched + await api.getHandler("session_shutdown")({}, ctx) + read.mockRestore() + } + }) + it.each(["telemetry-off", "explicit-on", "explicit-off"])("skips the default notice for %s", async (choice) => { + if (choice === "telemetry-off") vi.stubEnv("KIMCHI_TELEMETRY_ENABLED", "false") + else await setReportingEnabled(directory, choice === "explicit-on") + const ctx = createContext() + const api = trackedApi(() => ctx) + reportingExtension(api.api) + await api.getHandler("session_start")({ type: "session_start", reason: "new" }, ctx) + await api.getHandler("agent_end")({}, ctx) + expect(ctx.ui.notify).not.toHaveBeenCalled() + expect(api.getAppendedEntries("pr-cost-reporting-notice")).toEqual([]) + if (choice !== "explicit-on") expect(supervisor.requestWorkReconciliation).not.toHaveBeenCalled() + await api.getHandler("session_shutdown")({}, ctx) + }) + it("cancels the old account context before switching to another project session", async () => { + let ctx = createContext({ cwd: "/first" }) + const api = trackedApi(() => ctx) + const stop = vi.fn(async () => {}) + vi.mocked(supervisor.subscribeReportingReconciliation).mockReturnValue(stop) + reportingExtension(api.api) + await api.getHandler("session_start")({ type: "session_start", reason: "new" }, ctx) + ctx = createContext({ cwd: "/other" }) + api.api.events.emit(WORK_CHANGED_EVENT, {}) + expect(stop).toHaveBeenCalledOnce() + expect(supervisor.subscribeReportingReconciliation).toHaveBeenCalledTimes(2) + await api.getHandler("session_shutdown")({}, ctx) + }) + it("stays inactive when work tracking is absent or this is a child", async () => { + const api = createExtensionApi() + reportingExtension(api.api) + await api.getHandler("session_start")({ type: "session_start", reason: "new" }, createContext()) + expect(supervisor.subscribeReportingReconciliation).not.toHaveBeenCalled() + expect(existsSync(join(directory, "pr-cost-reporting"))).toBe(false) + await api.getHandler("agent_end")({}, createContext()) + expect(supervisor.requestWorkReconciliation).not.toHaveBeenCalled() + }) + it("starts main reporting by default and refreshes immediately when enabled or a turn ends", async () => { + const ctx = createContext() + const api = trackedApi(() => ctx) + const stop = vi.fn(async () => {}) + vi.mocked(supervisor.subscribeReportingReconciliation).mockReturnValue(stop) + reportingExtension(api.api) + await api.getHandler("session_start")({ type: "session_start", reason: "new" }, ctx) + expect(supervisor.subscribeReportingReconciliation).toHaveBeenCalledOnce() + expect((await readReportingState(directory)).enabled).toBe(true) + const command = api.getRegisteredCommand("pr-reporting") + const commandCtx = createCommandContext() + await command.handler("on", commandCtx) + expect(supervisor.requestWorkReconciliation).toHaveBeenCalledOnce() + expect((await readReportingState(directory)).enabled).toBe(true) + expect(commandCtx.ui.notify).toHaveBeenCalledWith(expect.stringContaining("request and billing IDs"), "info") + await api.getHandler("agent_end")({}, ctx) + await vi.waitFor(() => expect(supervisor.requestWorkReconciliation).toHaveBeenCalledTimes(2)) + await command.handler("off", commandCtx) + expect((await readReportingState(directory)).enabled).toBe(false) + await api.getHandler("session_shutdown")({}, ctx) + expect(stop).toHaveBeenCalledOnce() + await api.getHandler("agent_end")({}, ctx) + expect(supervisor.requestWorkReconciliation).toHaveBeenCalledTimes(2) + }) + it.each([ + ["tui", true], + ["rpc", true], + ["print", false], + ["json", false], + ] as const)("keeps attributing in a %s session and reports only when interactive", async (mode, deliver) => { + const ctx = createContext({ mode }) + const api = trackedApi(() => ctx) + reportingExtension(api.api) + await api.getHandler("session_start")({ type: "session_start", reason: "new" }, ctx) + // A session that never uploads reads no history; the next interactive session builds the same reports. + const subscriptions = vi.mocked(supervisor.subscribeReportingReconciliation).mock.calls + expect(subscriptions).toHaveLength(deliver ? 1 : 0) + if (deliver) { + const signal = new AbortController().signal + await subscriptions[0][0](directory, signal, () => {}) + expect(reconcileReporting).toHaveBeenCalledWith(directory, ctx.cwd, signal, expect.any(Function)) + } + const commandCtx = { ...createCommandContext(), mode } + await api.getRegisteredCommand("pr-reporting").handler("status", commandCtx) + const [[text]] = vi.mocked(commandCtx.ui.notify).mock.calls + if (deliver) expect(text).not.toContain("This session") + else + expect(text).toContain( + [ + "This session No uploads; local attribution continues", + ` Reason: non-interactive ${mode === "json" ? "JSON" : "print"} mode`, + ].join("\n"), + ) + await api.getHandler("session_shutdown")({}, ctx) + }) + it("shows that a CI session does not upload while reporting stays on", async () => { + vi.stubEnv("GITHUB_ACTIONS", "true") + const api = createExtensionApi() + reportingExtension(api.api) + const commandCtx = createCommandContext() + await api.getRegisteredCommand("pr-reporting").handler("status", commandCtx) + expect(commandCtx.ui.notify).toHaveBeenCalledWith( + [ + "PR reporting Enabled (SaaS default)", + "This session No uploads; local attribution continues", + " Reason: CI environment (GITHUB_ACTIONS)", + "Repositories 0 queued · 0 acknowledged", + ].join("\n"), + "info", + ) + }) + it("suggests on, off and status after /pr-reporting and narrows them as the user types", async () => { + const api = createExtensionApi() + reportingExtension(api.api) + const { getArgumentCompletions } = api.getRegisteredCommand("pr-reporting") + const values = async (prefix: string) => + (await getArgumentCompletions?.(prefix))?.map((item) => [item.value, item.label]) ?? [] + expect(await getArgumentCompletions?.("")).toEqual( + ["on", "off", "status"].map((value) => ({ value, label: value, description: expect.any(String) })), + ) + expect(await values("o")).toEqual([ + ["on", "on"], + ["off", "off"], + ]) + expect(await values("OF")).toEqual([["off", "off"]]) + expect(await values(" st")).toEqual([["status", "status"]]) + expect(await values("x")).toEqual([]) + }) + it.each([false, true])("warns once that a repository is partially reported (Studio=%s)", async (acp) => { + mode.acp = acp + try { + await setReportingEnabled(directory, true) + await queueSnapshots(directory, [ + { + account: { + apiUrl: "https://api.example", + organizationId: "11111111-1111-4111-8111-111111111111", + userId: "22222222-2222-4222-8222-222222222222", + }, + content: { + repository: { provider: "github", host: "github.com", id: "42", name: "owner/repo" }, + pullRequests: [], + requests: [ + { + requestId: "33333333-3333-4333-8333-333333333333", + billingRecordIds: [], + startedAt: "2026-10-04T12:00:00Z", + allocation: { kind: "unlinked", pullRequestIds: [], method: "native" }, + }, + ], + coverage: { observedRequests: 1, unpricedRequests: 1, historyComplete: false, trimmedRequests: 3 }, + }, + }, + ]) + const text = "PR costs for owner/repo are partially reported: limit reached. See /pr-reporting status." + const ctx = createContext() + for (let launch = 0; launch < 2; launch++) { + const api = trackedApi(() => ctx) + reportingExtension(api.api) + await api.getHandler("session_start")({ type: "session_start", reason: "new" }, ctx) + // Studio drops notifications for a session it has not registered yet. + if (acp && launch === 0) expect(ctx.ui.notify).not.toHaveBeenCalled() + await api.getHandler("agent_end")({}, ctx) + await api.getHandler("session_shutdown")({}, ctx) + } + expect(vi.mocked(ctx.ui.notify).mock.calls.filter(([, level]) => level === "warning")).toEqual([ + [text, "warning"], + ]) + } finally { + mode.acp = false + } + }) +}) diff --git a/src/extensions/pr-cost-reporting/index.ts b/src/extensions/pr-cost-reporting/index.ts new file mode 100644 index 000000000..d74c71cc0 --- /dev/null +++ b/src/extensions/pr-cost-reporting/index.ts @@ -0,0 +1,141 @@ +import { type ExtensionAPI, type ExtensionContext, getAgentDir } from "@earendil-works/pi-coding-agent" +import { Text } from "@earendil-works/pi-tui" +import { IS_ACP_MODE } from "../../modes/acp/state.js" +import { + requestWorkReconciliation, + subscribeReportingReconciliation, +} from "../work-attribution/reconcile-supervisor.js" +import { WORK_CHANGED_EVENT, WORK_STATE_REQUEST_EVENT, type WorkStateRequest } from "../work-attribution.js" +import { uploadSkipReason } from "./automation.js" +import { + dueLimitNotices, + readReportingState, + setReportingEnabled, + takeLimitNotices, + takeReportingNotice, +} from "./queue.js" +import { limitNoticeTexts, statusText } from "./status.js" +import { reconcileReporting } from "./worker.js" + +const ARGUMENT_COMPLETIONS = [ + { value: "on", label: "on", description: "Report PR costs to your account" }, + { value: "off", label: "off", description: "Stop reporting and delete pending uploads" }, + { value: "status", label: "status", description: "Show the setting, queued reports and problems" }, +] + +export default function prCostReportingExtension(pi: ExtensionAPI): void { + // A custom entry preserves other startup notices and stays out of model context. + pi.registerEntryRenderer("pr-cost-reporting-notice", (entry, _options, theme) => + typeof entry.data === "string" ? new Text(theme.fg("dim", entry.data), 1, 0) : undefined, + ) + + let context: ExtensionContext | undefined + /** CI, print, JSON and benchmark runs keep local attribution but never upload. */ + let skipReason: string | undefined + let started = false + let activeKey = "" + let stop: (() => Promise) | undefined + let draining = Promise.resolve() + + async function reportingEnabled(): Promise { + try { + const state = await readReportingState(getAgentDir()) + const ctx = context + if (ctx?.hasUI && state.enabled && state.followsTelemetry && !state.defaultNoticeShown) + if (await takeReportingNotice(getAgentDir())) { + const text = + "PR costs are reported to your account (repository/PR details and request/billing IDs). Turn off with /pr-reporting off." + // Studio shows notifications but not custom entries. + if (IS_ACP_MODE) ctx.ui.notify(text, "info") + else pi.appendEntry("pr-cost-reporting-notice", text) + } + const due = dueLimitNotices(state) + if (ctx?.hasUI && (due.repositories.length || due.pauses.length)) + for (const text of limitNoticeTexts(await takeLimitNotices(getAgentDir()))) ctx.ui.notify(text, "warning") + return state.enabled + } catch { + return false + } + } + + function synchronize(ctx = context): void { + if (!started || !ctx) return + const request: WorkStateRequest = {} + pi.events.emit(WORK_STATE_REQUEST_EVENT, request) + const next = request.current?.ctx ?? ctx + const key = JSON.stringify([next.cwd, next.sessionManager.getSessionId()]) + if (activeKey && activeKey !== key && stop) { + draining = Promise.all([draining, stop()]).then(() => {}) + stop = undefined + } + context = next + activeKey = key + // Sessions that never upload (CI, print or JSON mode, benchmarks) skip the pass; the next interactive + // session builds the same reports from the same journals. + if (request.tracking && request.current && !skipReason) { + stop ??= subscribeReportingReconciliation(async (agentDir, signal, assertLease) => { + if (context) await reconcileReporting(agentDir, context.cwd, signal, assertLease) + }) + } else if (stop) { + draining = Promise.all([draining, stop()]).then(() => {}) + stop = undefined + } + } + pi.events.on(WORK_CHANGED_EVENT, () => synchronize()) + + pi.on("session_start", async (_event, ctx) => { + started = true + skipReason = uploadSkipReason(ctx.mode) + synchronize(ctx) + // Studio drops notifications for a session it has not registered yet, so it gets the notice after a turn. + if (stop && !IS_ACP_MODE) await reportingEnabled() + }) + + pi.on("agent_end", () => { + // Disk reads must not hold turn completion or the dialogs waiting for idle. + draining = draining.then(async () => { + if (started && stop && (await reportingEnabled())) requestWorkReconciliation() + }) + }) + + pi.on("session_shutdown", async () => { + started = false + await Promise.all([draining, stop?.()]) + stop = undefined + context = undefined + }) + + pi.registerCommand("pr-reporting", { + description: "PR cost reporting: on, off, or status (defaults to the SaaS telemetry setting)", + getArgumentCompletions: (prefix) => + ARGUMENT_COMPLETIONS.filter((item) => item.value.startsWith(prefix.trimStart().toLowerCase())), + handler: async (args, ctx) => { + const command = args.trim() || "status" + if (!["on", "off", "status"].includes(command)) { + ctx.ui.notify("Usage: /pr-reporting on|off|status", "info") + return + } + try { + if (command === "on") { + await setReportingEnabled(getAgentDir(), true) + if (started && stop) requestWorkReconciliation() + ctx.ui.notify( + "PR reporting on. Kimchi will send repository/PR metadata, request and billing IDs, timestamps, and matching evidence to your account. Prompts, plans, local work/session IDs, paths, and prices stay local. Delivery runs in the background.", + "info", + ) + } else if (command === "off") { + await setReportingEnabled(getAgentDir(), false) + ctx.ui.notify( + "PR reporting off. Pending uploads were deleted; local work history and revision counters were retained. Previously accepted reports remain on the server.", + "info", + ) + } else { + const status = statusText(await readReportingState(getAgentDir()), uploadSkipReason(ctx.mode)) + ctx.ui.notify(status.text, status.warning ? "warning" : "info") + } + } catch { + ctx.ui.notify("PR reporting state could not be saved or read. Existing reports were retained.", "warning") + } + }, + }) +} diff --git a/src/extensions/pr-cost-reporting/machine.test.ts b/src/extensions/pr-cost-reporting/machine.test.ts new file mode 100644 index 000000000..bfae230e3 --- /dev/null +++ b/src/extensions/pr-cost-reporting/machine.test.ts @@ -0,0 +1,23 @@ +import { describe, expect, it } from "vitest" +import { macMachineId, windowsMachineId } from "./machine.js" + +describe("machine identity", () => { + it("reads the hardware UUID from ioreg output", () => { + const output = `+-o J316sAP + { + "IOPlatformSerialNumber" = "ABC123" + "IOPlatformUUID" = "4C4C4544-0032-3910-8044-B4C04F4B4D32" + }` + expect(macMachineId(output)).toBe("4C4C4544-0032-3910-8044-B4C04F4B4D32") + expect(macMachineId("{}")).toBeUndefined() + }) + + it("reads MachineGuid from reg query output", () => { + const output = + "\r\nHKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\Cryptography\r\n MachineGuid REG_SZ 6c0f0d8a-0c5e-4d6b-9a4b-0f1b2c3d4e5f\r\n" + expect(windowsMachineId(output)).toBe("6c0f0d8a-0c5e-4d6b-9a4b-0f1b2c3d4e5f") + expect( + windowsMachineId("ERROR: The system was unable to find the specified registry key or value."), + ).toBeUndefined() + }) +}) diff --git a/src/extensions/pr-cost-reporting/machine.ts b/src/extensions/pr-cost-reporting/machine.ts new file mode 100644 index 000000000..52d675afa --- /dev/null +++ b/src/extensions/pr-cost-reporting/machine.ts @@ -0,0 +1,47 @@ +import { execFile } from "node:child_process" +import { createHash } from "node:crypto" +import { readFile } from "node:fs/promises" +import { promisify } from "node:util" + +const run = promisify(execFile) + +/** Reads the hardware UUID printed by `ioreg -rd1 -c IOPlatformExpertDevice`. */ +export function macMachineId(output: string): string | undefined { + const match = /"IOPlatformUUID"\s*=\s*"([^"]+)"/.exec(output) + return match ? match[1] : undefined +} + +/** Reads the value printed by `reg query HKLM\SOFTWARE\Microsoft\Cryptography /v MachineGuid`. */ +export function windowsMachineId(output: string): string | undefined { + const match = /MachineGuid\s+REG_SZ\s+(\S+)/.exec(output) + return match ? match[1] : undefined +} + +async function machineId(): Promise { + try { + if (process.platform === "darwin") + return macMachineId( + (await run("/usr/sbin/ioreg", ["-rd1", "-c", "IOPlatformExpertDevice"], { timeout: 2000 })).stdout, + ) + if (process.platform === "win32") + return windowsMachineId( + (await run("reg", ["query", "HKLM\\SOFTWARE\\Microsoft\\Cryptography", "/v", "MachineGuid"], { timeout: 2000 })) + .stdout, + ) + for (const path of ["/etc/machine-id", "/var/lib/dbus/machine-id"]) { + const id = (await readFile(path, "utf8").catch(() => "")).trim() + if (id) return id + } + } catch {} + return undefined +} + +let fingerprint: Promise | undefined + +/** A salted hash of the operating system's machine ID; the raw ID is never stored or sent. */ +export function machineFingerprint(): Promise { + fingerprint ??= machineId().then((id) => + id ? createHash("sha256").update(`kimchi-pr-cost-producer\0${id}`).digest("hex") : undefined, + ) + return fingerprint +} diff --git a/src/extensions/pr-cost-reporting/queue.test.ts b/src/extensions/pr-cost-reporting/queue.test.ts new file mode 100644 index 000000000..8c42137bb --- /dev/null +++ b/src/extensions/pr-cost-reporting/queue.test.ts @@ -0,0 +1,913 @@ +import { createHash } from "node:crypto" +import * as files from "node:fs/promises" +import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises" +import { tmpdir } from "node:os" +import { join } from "node:path" +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest" +import type { WorkPullRequest } from "../pull-request-status/pull-requests.js" +import * as health from "../telemetry/pr-cost.js" +import { calculatePullRequestCosts } from "../work-attribution/costs.js" +import type { WorkRecord } from "../work-attribution/summary.js" +import { machineFingerprint } from "./machine.js" +import { + acknowledgeSnapshot, + deferSnapshot, + dueLimitNotices, + learnedLimits, + limitSnapshot, + queueSnapshots, + readReportingState, + setReportingEnabled, + takeLimitNotices, + takeReportingNotice, +} from "./queue.js" +import { + accountKey, + buildSnapshots, + fitSnapshot, + type RepositorySnapshot, + repositoryKey, + SNAPSHOT_LIMITS, +} from "./snapshot.js" + +const account = { + apiUrl: "https://api.example", + organizationId: "11111111-1111-4111-8111-111111111111", + userId: "22222222-2222-4222-8222-222222222222", +} +const requestId = "33333333-3333-4333-8333-333333333333" +const otherRequestId = "44444444-4444-4444-8444-444444444444" +const billing = "77777777-7777-4777-8777-777777777777" +const snapshot = (requestIds: string[] = []): RepositorySnapshot => ({ + account, + content: { + repository: { provider: "github", host: "github.com", id: "42" }, + pullRequests: [], + requests: requestIds.map((requestId) => ({ + requestId, + startedAt: "2026-10-04T12:00:00Z", + billingRecordIds: [], + allocation: { kind: "unlinked", pullRequestIds: [], method: "native" }, + })), + coverage: { observedRequests: requestIds.length, unpricedRequests: requestIds.length, historyComplete: true }, + }, +}) +let directory: string +vi.mock("node:fs/promises", async (original) => ({ ...(await original()) })) +vi.mock("./machine.js", () => ({ machineFingerprint: vi.fn() })) +beforeEach(async () => { + directory = await mkdtemp(join(tmpdir(), "kimchi-reporting-")) + vi.stubEnv("KIMCHI_TELEMETRY_ENABLED", "true") + vi.mocked(machineFingerprint).mockResolvedValue(undefined) +}) +afterEach(async () => { + vi.restoreAllMocks() + vi.unstubAllEnvs() + await rm(directory, { recursive: true, force: true }) +}) + +describe("durable reporting queue", () => { + it.each([ + true, + false, + ])("reopens an old queue without resetting its explicit choice or request history (%s)", async (enabled) => { + const previous = snapshot([requestId]) + const key = `${accountKey(account)}:${repositoryKey(previous.content.repository)}` + const producerId = "55555555-5555-4555-8555-555555555555" + const legacy = { + version: 1, + enabled, + producerId, + entries: { + [key]: { + account, + repository: previous.content.repository, + revision: "7", + requestHashes: [createHash("sha256").update(requestId).digest("hex")], + pending: { + ...previous.content, + schemaVersion: 1, + producerId, + revision: "7", + generatedAt: "2026-10-04T12:00:00Z", + }, + pendingDigest: "old-digest", + attempts: 2, + retryAt: 100, + }, + }, + } + await files.mkdir(join(directory, "pr-cost-reporting"), { recursive: true }) + await writeFile(join(directory, "pr-cost-reporting/state.json"), JSON.stringify(legacy)) + vi.stubEnv("KIMCHI_TELEMETRY_ENABLED", String(!enabled)) + expect(await readReportingState(directory)).toEqual(legacy) + await setReportingEnabled(directory, true) + const next = await queueSnapshots(directory, [snapshot([requestId, otherRequestId])], true) + expect(next.producerId).toBe(producerId) + expect(next.entries[key].revision).toBe("8") + expect(next.entries[key].requestHashes).toHaveLength(2) + await acknowledgeSnapshot(directory, key, "7", { + status: "accepted", + revision: "7", + receivedAt: "2026-10-07T12:00:00Z", + }) + expect((await readReportingState(directory)).entries[key].pending?.revision).toBe("8") + }) + it("allows deliberate window expiry while still holding a missing source request", async () => { + const initial = await queueSnapshots(directory, [snapshot([requestId, otherRequestId])], true) + const [key] = Object.keys(initial.entries) + await acknowledgeSnapshot(directory, key, "1", { + status: "accepted", + revision: "1", + receivedAt: new Date().toISOString(), + }) + const missing = await queueSnapshots(directory, [snapshot([otherRequestId])], true) + expect(missing.entries[key].held).toBe(true) + const expired = snapshot([otherRequestId]) + expired.observedRequestIds = [requestId, otherRequestId] + const queued = await queueSnapshots(directory, [expired], true) + expect(queued.entries[key].held).toBeUndefined() + expect(queued.entries[key].pending?.requests.map((request) => request.requestId)).toEqual([otherRequestId]) + expect(queued.entries[key].requestHashes).toHaveLength(2) + await acknowledgeSnapshot(directory, key, "2", { + status: "accepted", + revision: "2", + receivedAt: new Date().toISOString(), + }) + expect((await readReportingState(directory)).entries[key].requestHashes).toHaveLength(1) + }) + it("queues nothing for a repository whose only history is older than the upload window", async () => { + await setReportingEnabled(directory, true) + const old = new Date(Date.now() - 120 * 24 * 60 * 60_000).toISOString() + const rows: WorkRecord[] = [ + { + version: 1, + type: "request", + workId: "55555555-5555-4555-8555-555555555555", + sessionId: "old-session", + requestId, + startedAt: old, + recordedAt: old, + scope: { account, repository: "/repo/.git" }, + }, + ] + const repositories = new Map([["/repo/.git", snapshot().content.repository]]) + const built = buildSnapshots(rows, calculatePullRequestCosts(rows, []), repositories) + expect(built.snapshots.map((value) => value.content.requests)).toEqual([[]]) + expect((await queueSnapshots(directory, built.snapshots, true)).entries).toEqual({}) + }) + it("claims the default-on notice once across concurrent sessions", async () => { + expect((await Promise.all([takeReportingNotice(directory), takeReportingNotice(directory)])).sort()).toEqual([ + false, + true, + ]) + expect(await takeReportingNotice(directory)).toBe(false) + }) + it("follows SaaS uploads by default after the queue is saved and reopened", async () => { + expect((await readReportingState(directory)).enabled).toBe(true) + await queueSnapshots(directory, [snapshot([requestId])]) + const original = await readReportingState(directory) + expect(Object.values(original.entries)[0].pending?.requests[0].requestId).toBe(requestId) + vi.stubEnv("KIMCHI_TELEMETRY_ENABLED", "false") + expect((await readReportingState(directory)).enabled).toBe(false) + vi.stubEnv("KIMCHI_TELEMETRY_ENABLED", "true") + expect(await readReportingState(directory)).toEqual(original) + }) + it.each([true, false])("preserves an explicit reporting choice (%s) when SaaS uploads change", async (enabled) => { + await setReportingEnabled(directory, enabled) + for (const telemetry of ["false", "true"]) { + vi.stubEnv("KIMCHI_TELEMETRY_ENABLED", telemetry) + expect((await readReportingState(directory)).enabled).toBe(enabled) + } + }) + it.each([ + { acknowledged: true, optOut: false }, + { acknowledged: true, optOut: true }, + { acknowledged: false, optOut: false }, + { acknowledged: false, optOut: true }, + ])("holds a pending migration after evidence loss ($acknowledged, opt-out=$optOut)", async ({ + acknowledged, + optOut, + }) => { + await setReportingEnabled(directory, true) + await queueSnapshots(directory, [snapshot([requestId])], true) + const [key, original] = Object.entries((await readReportingState(directory)).entries)[0] + if (acknowledged) + await acknowledgeSnapshot(directory, key, "1", { + status: "accepted", + revision: "1", + receivedAt: new Date().toISOString(), + }) + const moved = snapshot([requestId]) + moved.content.repository.id = "43" + await queueSnapshots(directory, [moved], true) + expect((await readReportingState(directory)).entries[key].pending?.requests).toEqual([]) + if (optOut) { + await setReportingEnabled(directory, false) + await acknowledgeSnapshot(directory, key, "2", { + status: "accepted", + revision: "2", + receivedAt: new Date().toISOString(), + }) + expect(JSON.stringify(await readReportingState(directory))).not.toContain(requestId) + await setReportingEnabled(directory, true) + } + // A restarted process sees only an independent repository; neither migration payload was acknowledged. + const healthy = snapshot([otherRequestId]) + healthy.content.repository.id = "44" + await queueSnapshots(directory, [healthy], false) + const restarted = await readReportingState(directory) + expect(restarted.entries[key]).toMatchObject({ held: true, requestHashes: original.requestHashes }) + expect(Object.values(restarted.entries).find((entry) => entry.repository.id === "43")?.held).toBe(true) + expect(Object.values(restarted.entries).find((entry) => entry.repository.id === "44")?.pending).toBeDefined() + // Restoring that same membership permits the correction, then its matching ACK releases the old hashes. + await queueSnapshots(directory, [moved, healthy], true) + const restored = (await readReportingState(directory)).entries[key] + expect(restored.held).toBeUndefined() + expect(restored.pending?.requests).toEqual([]) + await acknowledgeSnapshot(directory, key, restored.revision, { + status: acknowledged ? "accepted" : "unchanged", + revision: restored.revision, + receivedAt: new Date().toISOString(), + }) + expect((await readReportingState(directory)).entries[key].requestHashes).toEqual([]) + }) + it("retains every possibly delivered membership when replacement acknowledgements are lost or late", async () => { + await setReportingEnabled(directory, true) + await queueSnapshots(directory, [snapshot([requestId])], true) + const [key] = Object.keys((await readReportingState(directory)).entries) + const moved = snapshot([requestId]) + moved.content.repository.id = "43" + await queueSnapshots(directory, [snapshot([otherRequestId]), moved], true) + moved.content.requests.push(snapshot([otherRequestId]).content.requests[0]) + moved.content.coverage = { observedRequests: 2, unpricedRequests: 2, historyComplete: true } + await queueSnapshots(directory, [moved], true) + await acknowledgeSnapshot(directory, key, "2", { + status: "accepted", + revision: "2", + receivedAt: new Date().toISOString(), + }) + const queued = (await readReportingState(directory)).entries[key] + expect(queued.pending?.revision).toBe("3") + expect(queued.requestHashes).toHaveLength(2) + await queueSnapshots(directory, [], false) + expect((await readReportingState(directory)).entries[key].held).toBe(true) + }) + it("resends a same-membership correction after opt-out discarded an unacknowledged replacement", async () => { + await setReportingEnabled(directory, true) + const original = snapshot([requestId]) + await queueSnapshots(directory, [original], true) + const [key] = Object.keys((await readReportingState(directory)).entries) + await acknowledgeSnapshot(directory, key, "1", { + status: "accepted", + revision: "1", + receivedAt: new Date().toISOString(), + }) + const correction = snapshot([requestId]) + correction.content.requests[0].allocation.kind = "unknown" + await queueSnapshots(directory, [correction], true) + // Revision 2 may have reached the server before consent was revoked and its ACK was lost. + await setReportingEnabled(directory, false) + await setReportingEnabled(directory, true) + await queueSnapshots(directory, [original], true) + await acknowledgeSnapshot(directory, key, "2", { + status: "accepted", + revision: "2", + receivedAt: new Date().toISOString(), + }) + const current = (await readReportingState(directory)).entries[key] + expect(current.pending).toMatchObject({ revision: "3", requests: [{ allocation: { kind: "unlinked" } }] }) + expect(current.requestHashes).toHaveLength(1) + }) + it("does not shrink membership when a stale acknowledgement only establishes a higher server revision", async () => { + await setReportingEnabled(directory, true) + await queueSnapshots(directory, [snapshot([requestId])], true) + const [key, original] = Object.entries((await readReportingState(directory)).entries)[0] + const moved = snapshot([requestId]) + moved.content.repository.id = "43" + await queueSnapshots(directory, [moved], true) + await acknowledgeSnapshot(directory, key, "2", { + status: "stale", + revision: "8", + receivedAt: new Date().toISOString(), + }) + await queueSnapshots(directory, [], false) + expect((await readReportingState(directory)).entries[key]).toMatchObject({ + revision: "8", + requestHashes: original.requestHashes, + held: true, + }) + await queueSnapshots(directory, [moved], true) + expect((await readReportingState(directory)).entries[key].pending?.revision).toBe("9") + }) + it("holds an oversized repository without truncating it or blocking an independent snapshot", async () => { + await setReportingEnabled(directory, true) + const oversized = snapshot() + oversized.content.requests.push({ + requestId: "33333333-3333-4333-8333-333333333333", + startedAt: "2026-10-04T12:00:00Z", + billingRecordIds: Array.from( + { length: 9 }, + (_, index) => `00000000-0000-4000-8000-${String(index + 1).padStart(12, "0")}`, + ), + allocation: { kind: "unlinked", pullRequestIds: [], method: "native" }, + }) + oversized.content.coverage.observedRequests = 1 + const healthy = snapshot([otherRequestId]) + healthy.content.repository.id = "43" + await queueSnapshots(directory, [oversized, healthy]) + const state = await readReportingState(directory) + expect(Object.values(state.entries).map((entry) => entry.repository.id)).toEqual(["43"]) + expect(state.error).toContain("limits") + }) + it("does not rewrite or sync an unchanged queue", async () => { + await setReportingEnabled(directory, true) + await queueSnapshots(directory, [snapshot([requestId])]) + const path = join(directory, "pr-cost-reporting", "state.json") + const before = await files.stat(path) + await queueSnapshots(directory, [snapshot([requestId])]) + const after = await files.stat(path) + expect([after.ino, after.mtimeMs]).toEqual([before.ino, before.mtimeMs]) + }) + it("holds only the replacement that would overflow the local queue", async () => { + await setReportingEnabled(directory, true) + // Four full repositories pass the 24 MiB queue; the others are queued, and the last one waits for room. + const full = (repositoryId: string) => { + const value = snapshot(Array.from({ length: SNAPSHOT_LIMITS.requests }, () => crypto.randomUUID())) + value.content.repository.id = repositoryId + return value + } + await queueSnapshots(directory, ["51", "52", "53", "54"].map(full)) + const state = await readReportingState(directory) + expect(Object.values(state.entries).filter((entry) => entry.pending)).toHaveLength(3) + expect(state.error).toContain("until queued reports leave room in the local queue") + expect((await files.stat(join(directory, "pr-cost-reporting", "state.json"))).size).toBeLessThan(24 * 1024 * 1024) + }, 60_000) + it("reports an invalid replacement once, as invalid", async () => { + await setReportingEnabled(directory, true) + await queueSnapshots(directory, [snapshot([requestId])]) + const invalid = snapshot([requestId]) + invalid.content.requests[0].billingRecordIds = Array.from( + { length: 9 }, + (_, index) => `00000000-0000-4000-8000-${String(index + 1).padStart(12, "0")}`, + ) + await queueSnapshots(directory, [invalid]) + expect((await readReportingState(directory)).error).toBe( + "PR reporting held 1 snapshot(s) that are invalid or exceed the upload limits after trimming", + ) + }) + it("keeps the skipped request count of the latest capture until an explicit choice", async () => { + await setReportingEnabled(directory, true) + expect((await queueSnapshots(directory, [snapshot([requestId])], true, 3693)).skippedRequests).toBe(3693) + expect((await queueSnapshots(directory, [snapshot([requestId])], true)).skippedRequests).toBeUndefined() + await queueSnapshots(directory, [snapshot([requestId])], true, 2) + expect((await setReportingEnabled(directory, false)).skippedRequests).toBeUndefined() + const path = join(directory, "pr-cost-reporting", "state.json") + await writeFile(path, JSON.stringify({ ...JSON.parse(await readFile(path, "utf8")), skippedRequests: -1 })) + await expect(readReportingState(directory)).rejects.toThrow("unreadable") + }) + it("keeps the retry deadline when new source evidence replaces a rate-limited snapshot", async () => { + await setReportingEnabled(directory, true) + await queueSnapshots(directory, [snapshot([requestId])]) + const [key] = Object.keys((await readReportingState(directory)).entries) + const retryAt = Date.now() + 120000 + await deferSnapshot(directory, key, "1", retryAt, "PR reporting returned HTTP 429") + const changed = snapshot([requestId]) + changed.content.coverage.historyComplete = false + await queueSnapshots(directory, [changed]) + expect((await readReportingState(directory)).entries[key]).toMatchObject({ revision: "2", retryAt, attempts: 1 }) + }) + it("leaves the last durable revision unchanged when fsync fails", async () => { + await setReportingEnabled(directory, true) + const before = await readReportingState(directory) + const originalOpen = files.open + vi.spyOn(files, "open").mockImplementation(async (...args) => { + const file = await originalOpen(...args) + if (String(args[0]).endsWith(".tmp")) vi.spyOn(file, "sync").mockRejectedValue(new Error("disk unavailable")) + return file + }) + await expect(queueSnapshots(directory, [snapshot([requestId])])).rejects.toThrow("disk unavailable") + expect(await readReportingState(directory)).toEqual(before) + }) + it("replaces an accepted inventory with a higher-revision empty snapshot after local removal", async () => { + await setReportingEnabled(directory, true) + const original = snapshot() + original.content.requests.push({ + requestId: "33333333-3333-4333-8333-333333333333", + startedAt: "2026-10-04T12:00:00Z", + billingRecordIds: [], + allocation: { kind: "unlinked", pullRequestIds: [], method: "native" }, + }) + original.content.coverage = { observedRequests: 1, unpricedRequests: 1, historyComplete: true } + await queueSnapshots(directory, [original]) + const [key] = Object.keys((await readReportingState(directory)).entries) + await acknowledgeSnapshot(directory, key, "1", { + status: "accepted", + revision: "1", + receivedAt: new Date().toISOString(), + }) + const moved = structuredClone(original) + moved.content.repository.id = "43" + await queueSnapshots(directory, [moved], true) + expect((await readReportingState(directory)).entries[key].pending).toMatchObject({ + revision: "2", + requests: [], + pullRequests: [], + }) + }) + it("holds an acknowledged group's missing request after restart while another group progresses", async () => { + await setReportingEnabled(directory, true) + const original = snapshot() + original.content.requests.push({ + requestId: "33333333-3333-4333-8333-333333333333", + startedAt: "2026-10-04T12:00:00Z", + billingRecordIds: [], + allocation: { kind: "unlinked", pullRequestIds: [], method: "native" }, + }) + original.content.coverage = { observedRequests: 1, unpricedRequests: 1, historyComplete: true } + await queueSnapshots(directory, [original]) + const [key] = Object.keys((await readReportingState(directory)).entries) + await acknowledgeSnapshot(directory, key, "1", { + status: "accepted", + revision: "1", + receivedAt: new Date().toISOString(), + }) + const incomplete = snapshot() + incomplete.content.coverage.historyComplete = false + const healthy = snapshot([otherRequestId]) + healthy.content.repository.id = "44" + await queueSnapshots(directory, [incomplete, healthy]) + const restarted = await readReportingState(directory) + expect(restarted.entries[key]).toMatchObject({ revision: "1", held: true }) + expect(restarted.entries[key].pending).toBeUndefined() + expect(Object.values(restarted.entries).find((entry) => entry.repository.id === "44")?.pending).toBeDefined() + const corrected = structuredClone(original) + corrected.content.requests[0].allocation.kind = "unknown" + corrected.content.coverage.historyComplete = false + await queueSnapshots(directory, [corrected, healthy]) + expect((await readReportingState(directory)).entries[key].pending?.revision).toBe("2") + }) + it("fails closed on damaged durable state instead of resetting its producer or revisions", async () => { + await setReportingEnabled(directory, true) + await writeFile(join(directory, "pr-cost-reporting", "state.json"), "{broken") + await expect(queueSnapshots(directory, [snapshot()])).rejects.toThrow("unreadable") + expect(await readFile(join(directory, "pr-cost-reporting", "state.json"), "utf8")).toBe("{broken") + }) + it("defaults off while SaaS uploads are disabled and starts collecting when they are enabled", async () => { + vi.stubEnv("KIMCHI_TELEMETRY_ENABLED", "false") + await queueSnapshots(directory, [snapshot()]) + expect((await readReportingState(directory)).entries).toEqual({}) + vi.stubEnv("KIMCHI_TELEMETRY_ENABLED", "true") + await queueSnapshots(directory, [snapshot([requestId])]) + expect(Object.values((await readReportingState(directory)).entries)[0]?.pending?.requests[0].requestId).toBe( + requestId, + ) + }) + it("replaces pending snapshots, survives restart and ignores a late acknowledgement", async () => { + await setReportingEnabled(directory, true) + await queueSnapshots(directory, [snapshot([requestId])]) + const before = await readReportingState(directory) + const [key, first] = Object.entries(before.entries)[0] + expect(first.pending?.revision).toBe("1") + const next = snapshot([requestId]) + next.content.coverage.historyComplete = false + await queueSnapshots(directory, [next]) + await acknowledgeSnapshot(directory, key, "1", { + status: "accepted", + revision: "1", + receivedAt: new Date().toISOString(), + }) + const after = await readReportingState(directory) + expect(after.producerId).toBe(before.producerId) + expect(after.entries[key].pending?.revision).toBe("2") + expect( + JSON.parse(await readFile(join(directory, "pr-cost-reporting", "state.json"), "utf8")).entries[key].pending + .revision, + ).toBe("2") + }) + it("deduplicates concurrent captures and deletes payloads on opt-out without resetting revisions", async () => { + await setReportingEnabled(directory, true) + await Promise.all([ + queueSnapshots(directory, [snapshot([requestId])]), + queueSnapshots(directory, [snapshot([requestId])]), + ]) + const [key] = Object.keys((await readReportingState(directory)).entries) + expect((await readReportingState(directory)).entries[key].revision).toBe("1") + await setReportingEnabled(directory, false) + expect((await readReportingState(directory)).entries[key].pending).toBeUndefined() + await setReportingEnabled(directory, true) + await queueSnapshots(directory, [snapshot([requestId])]) + expect((await readReportingState(directory)).entries[key].pending?.revision).toBe("2") + }) + it("separates accounts and advances beyond a server revision without acknowledging newer content", async () => { + await setReportingEnabled(directory, true) + const other = snapshot([requestId]) + other.account = { ...account, userId: "33333333-3333-4333-8333-333333333333" } + await queueSnapshots(directory, [snapshot([requestId]), other]) + const [key] = Object.keys((await readReportingState(directory)).entries) + await acknowledgeSnapshot(directory, key, "1", { + status: "stale", + revision: "8", + receivedAt: new Date().toISOString(), + }) + await queueSnapshots(directory, [snapshot([requestId]), other]) + const state = await readReportingState(directory) + expect(Object.keys(state.entries)).toHaveLength(2) + expect(state.entries[key].pending?.revision).toBe("9") + }) +}) + +describe("producer identity across machines", () => { + const original = "a".repeat(64) + const copy = "b".repeat(64) + async function savedOn(machine: string | undefined) { + vi.mocked(machineFingerprint).mockResolvedValue(machine) + await setReportingEnabled(directory, true) + const saved = await queueSnapshots(directory, [snapshot([requestId])]) + expect(saved.machine).toBe(machine) + return saved + } + + it("starts a new producer when the state was copied from another machine", async () => { + const saved = await savedOn(original) + vi.mocked(machineFingerprint).mockResolvedValue(copy) + const copied = await readReportingState(directory) + expect(copied.producerId).not.toBe(saved.producerId) + expect(copied.entries).toEqual({}) + const next = await queueSnapshots(directory, [snapshot([requestId])]) + expect(next.machine).toBe(copy) + expect(Object.values(next.entries).map((entry) => [entry.revision, entry.pending?.producerId])).toEqual([ + ["1", next.producerId], + ]) + expect((await readReportingState(directory)).producerId).toBe(next.producerId) + }) + + it("keeps the producer on the same machine and records the machine for older state", async () => { + const legacy = await savedOn(undefined) + vi.mocked(machineFingerprint).mockResolvedValue(original) + expect(await readReportingState(directory)).toEqual(legacy) + const next = await queueSnapshots(directory, [snapshot([requestId, otherRequestId])]) + expect([next.producerId, next.machine]).toEqual([legacy.producerId, original]) + expect(Object.values(next.entries)[0].revision).toBe("2") + }) + + it("keeps the producer when the machine cannot be identified", async () => { + const saved = await savedOn(original) + vi.mocked(machineFingerprint).mockResolvedValue(undefined) + expect(await readReportingState(directory)).toEqual(saved) + }) +}) + +describe("unusable provider metadata for one PR", () => { + const at = (minutes: number) => new Date(Date.UTC(2026, 9, 2, 10, minutes)).toISOString() + const mergeRequest = (number: number, overrides: Partial = {}): WorkPullRequest => ({ + provider: "gitlab", + id: String(9000 + number), + repositoryId: "42", + host: "gitlab.com", + repository: "team/repo", + number, + url: `https://gitlab.com/team/repo/-/merge_requests/${number}`, + state: "open", + headSha: "a".repeat(40), + mergeCommitSha: null, + mergedAt: null, + closedAt: null, + checkedAt: at(15), + ...overrides, + }) + const request = (id: string, workId: string, minutes: number): WorkRecord => ({ + version: 1, + type: "request", + workId, + sessionId: workId, + requestId: id, + recordedAt: at(minutes), + scope: { account, repository: "/repo/.git" }, + }) + const commit = (workId: string, sha: string, pullRequests: WorkPullRequest[]): WorkRecord => ({ + version: 1, + type: "commit", + workId, + sessionId: workId, + sha: sha.repeat(40), + repository: "/repo/.git", + worktree: "/repo", + recordedAt: at(20), + pullRequests, + }) + async function scan(rows: WorkRecord[]) { + const built = buildSnapshots( + rows, + calculatePullRequestCosts(rows, []), + new Map([["/repo/.git", { provider: "gitlab" as const, host: "gitlab.com", id: "42", name: "team/repo" }]]), + ) + return queueSnapshots(directory, built.snapshots, !built.incomplete) + } + + it("keeps reporting other MRs in a repository after GitLab marks one MR merged without merged_at", async () => { + const first = await scan([request(requestId, "work-a", 10), commit("work-a", "a", [mergeRequest(7)])]) + const [key] = Object.keys(first.entries) + await acknowledgeSnapshot(directory, key, "1", { status: "accepted", revision: "1", receivedAt: at(16) }) + // GitLab can report state "merged" while omitting merged_at; meanwhile unrelated work lands on MR !8. + const next = await scan([ + request(requestId, "work-a", 10), + commit("work-a", "a", [mergeRequest(7, { state: "merged", checkedAt: at(40) })]), + request(otherRequestId, "work-b", 41), + commit("work-b", "b", [mergeRequest(8, { checkedAt: at(42) })]), + ]) + expect(next.entries[key].held).toBeUndefined() + expect(next.entries[key].pending?.requests.map((row) => row.requestId).sort()).toEqual( + [requestId, otherRequestId].sort(), + ) + }) +}) + +describe("snapshot size health", () => { + it("reports the largest snapshot queued in a pass", async () => { + await setReportingEnabled(directory, true) + const metric = vi.spyOn(health, "trackPRCostMetric").mockImplementation(() => {}) + const other = snapshot([requestId, otherRequestId, billing]) + other.content.repository = { ...other.content.repository, id: "43" } + const state = await queueSnapshots(directory, [snapshot([requestId]), other]) + const bytes = Math.max( + ...Object.values(state.entries).map((entry) => Buffer.byteLength(JSON.stringify(entry.pending))), + ) + expect(metric).toHaveBeenCalledWith({ kind: "snapshotRequests", value: 3 }) + expect(metric).toHaveBeenCalledWith({ kind: "snapshotBytes", value: bytes }) + metric.mockClear() + await queueSnapshots(directory, [snapshot([requestId]), other]) + expect(metric.mock.calls.filter(([value]) => value.kind.startsWith("snapshot"))).toEqual([]) + }) +}) + +describe("upload window", () => { + const key = `${accountKey(account)}:${repositoryKey(snapshot().content.repository)}` + const at = "2026-10-04T12:00:00.000Z" + const pull = (state: "open" | "merged" | "closed", requestIds = [requestId]) => { + const value = snapshot(requestIds) + value.content.pullRequests = [ + { + id: "101", + number: 1, + url: "https://github.com/example/repo/pull/1", + state, + ...(state === "merged" ? { mergedAt: at } : {}), + ...(state === "closed" ? { closedAt: at } : {}), + }, + ] + value.content.requests[0].allocation = { kind: "pull-request", pullRequestIds: ["101"], method: "native" } + return value + } + const corrected = (source: "work-command" | "producer-confirmation", revision = 1) => { + const value = snapshot([requestId]) + value.content.requests[0].correction = { id: otherRequestId, revision, recordedAt: at, source } + return value + } + const queue = async (...values: RepositorySnapshot[]) => (await queueSnapshots(directory, values, true)).entries + const accept = async (id = key) => { + const entry = (await readReportingState(directory)).entries[id] + await acknowledgeSnapshot(directory, id, entry.revision, { + status: "accepted", + revision: entry.revision, + receivedAt: new Date().toISOString(), + }) + } + + it("lets a PR opening, merge or close skip the window, but not ordinary changes", async () => { + await setReportingEnabled(directory, true) + expect((await queue(snapshot([requestId])))[key].urgent).toBeUndefined() + await accept() + expect((await queue(pull("open")))[key].urgent).toBe(true) + await accept() + expect((await readReportingState(directory)).entries[key].markers).toEqual(["pull-request:101:open"]) + expect((await queue(pull("open", [requestId, otherRequestId])))[key].urgent).toBeUndefined() + // A merge stays urgent through later replacements until the server acknowledges it. + expect((await queue(pull("merged", [requestId, otherRequestId])))[key].urgent).toBe(true) + expect((await queue(pull("merged")))[key].urgent).toBe(true) + await accept() + expect((await queue(pull("merged", [requestId, otherRequestId])))[key].urgent).toBeUndefined() + expect((await queue(pull("closed", [requestId, otherRequestId])))[key].urgent).toBe(true) + }) + it("lets an explicit /work correction skip the window, but not a producer confirmation", async () => { + await setReportingEnabled(directory, true) + await queue(snapshot([requestId])) + await accept() + expect((await queue(corrected("producer-confirmation")))[key].urgent).toBeUndefined() + expect((await queue(corrected("work-command")))[key].urgent).toBe(true) + await accept() + expect((await queue(corrected("work-command", 2)))[key].urgent).toBe(true) + }) + it("sends a withdrawal and the receiving repository of a correction at once", async () => { + await setReportingEnabled(directory, true) + const other = snapshot([otherRequestId]) + other.content.repository = { ...other.content.repository, id: "43" } + const otherKey = `${accountKey(account)}:${repositoryKey(other.content.repository)}` + await queue(snapshot([requestId]), other) + await accept() + await accept(otherKey) + // /work link moves the request to repository 43; repository 42 is withdrawn. + const moved = snapshot([otherRequestId, requestId]) + moved.content.repository = other.content.repository + moved.content.requests[1].correction = { id: billing, revision: 1, recordedAt: at, source: "work-command" } + const entries = await queue(moved) + expect([entries[key].pending?.requests, entries[key].urgent]).toEqual([[], true]) + expect(entries[otherKey].urgent).toBe(true) + }) + it("does not send an acknowledged withdrawal again when history completeness changes", async () => { + await setReportingEnabled(directory, true) + const moved = snapshot([requestId]) + moved.content.repository = { ...moved.content.repository, id: "43" } + await queue(snapshot([requestId])) + await accept() + await queue(moved) + await accept() + // A repository lookup that fails and later recovers flips the completeness of the captures in between. + for (const complete of [false, true]) await queueSnapshots(directory, [moved], complete) + const { entries } = await readReportingState(directory) + expect([entries[key].revision, entries[key].requestHashes, entries[key].pending]).toEqual(["2", [], undefined]) + }) + it("sends other changed repositories of the account at once only when a correction first appears", async () => { + await setReportingEnabled(directory, true) + const other = (requestIds: string[]) => { + const value = snapshot(requestIds) + value.content.repository = { ...value.content.repository, id: "43" } + return value + } + const otherKey = `${accountKey(account)}:${repositoryKey(other([]).content.repository)}` + const third = "66666666-6666-4666-8666-666666666666" + await queue(snapshot([requestId]), other([otherRequestId])) + await accept() + await accept(otherKey) + const correction = corrected("work-command") + expect((await queue(correction, other([otherRequestId, third])))[otherKey].urgent).toBe(true) + await accept(otherKey) + // The receipt is still unacknowledged, yet a later ordinary change elsewhere waits for its window. + expect((await queue(correction, other([otherRequestId])))[otherKey].urgent).toBeUndefined() + }) + it("restarts the window and the retry deadline after /pr-reporting on", async () => { + await setReportingEnabled(directory, true) + await queue(snapshot([requestId])) + await deferSnapshot(directory, key, "1", Date.now() + 3_600_000, "PR reporting returned HTTP 404") + expect((await readReportingState(directory)).entries[key]).toMatchObject({ + attempts: 1, + uploadedAt: expect.any(Number), + }) + const state = await setReportingEnabled(directory, true) + expect(state.entries[key]).toMatchObject({ attempts: 0, retryAt: 0, pending: { revision: "1" } }) + expect(state.entries[key].uploadedAt).toBeUndefined() + }) +}) + +describe("server limits", () => { + const key = `${accountKey(account)}:${repositoryKey(snapshot().content.repository)}` + const requests = (count: number) => + Array.from({ length: count }, (_, index) => `66666666-6666-4666-8666-${String(index).padStart(12, "0")}`) + + it("pauses every repository of the account for an organization limit and keeps the notice once per pause", async () => { + await setReportingEnabled(directory, true) + await queueSnapshots(directory, [snapshot([requestId])]) + const limit = { scope: "organization" as const, limit: "requests" as const, current: 50000, maximum: 50000, at: 1 } + const first = await limitSnapshot(directory, key, "1", limit, 1000) + expect(first.paused).toEqual({ [accountKey(account)]: { limit, retryAt: 1000 } }) + expect(first.entries[key]).toMatchObject({ retryAt: 0, attempts: 0 }) + expect(first.entries[key].limit).toBeUndefined() + expect(dueLimitNotices(first)).toEqual({ repositories: [], pauses: [limit] }) + expect(await takeLimitNotices(directory)).toEqual({ repositories: [], pauses: [limit] }) + const again = await limitSnapshot(directory, key, "1", { ...limit, at: 2 }, 2000) + expect(again.paused?.[accountKey(account)]).toMatchObject({ retryAt: 2000, noticeShown: true }) + expect(await takeLimitNotices(directory)).toEqual({ repositories: [], pauses: [] }) + }) + it("lifts a pause only when an upload that adds claims is accepted", async () => { + await setReportingEnabled(directory, true) + const other = snapshot([otherRequestId]) + other.content.repository = { ...other.content.repository, id: "43" } + const otherKey = `${accountKey(account)}:${repositoryKey(other.content.repository)}` + await queueSnapshots(directory, [snapshot([requestId]), other], true) + await limitSnapshot(directory, key, "1", { scope: "contributor", at: 1 }, Date.now() + 3_600_000) + // Repository 43 is withdrawn: an empty snapshot frees space but proves nothing about the limit. + const withdrawn = snapshot([otherRequestId, requestId]) + await queueSnapshots(directory, [withdrawn], true) + const ack = { status: "accepted" as const, receivedAt: new Date().toISOString() } + const state = await acknowledgeSnapshot(directory, otherKey, "2", { ...ack, revision: "2" }) + expect(state.paused).toBeDefined() + expect((await acknowledgeSnapshot(directory, key, "2", { ...ack, revision: "2" })).paused).toBeUndefined() + }) + it("waits on its own counter for a producer limit and clears it once an upload is accepted", async () => { + await setReportingEnabled(directory, true) + await queueSnapshots(directory, [snapshot([requestId])]) + const limit = { scope: "producer" as const, limit: "repositories" as const, current: 101, maximum: 100, at: 1 } + const state = await limitSnapshot(directory, key, "1", limit, 5000) + expect(state.entries[key]).toMatchObject({ limit, attempts: 0, retryAt: 5000 }) + expect(state.entries[key].learned).toBeUndefined() + expect(dueLimitNotices(state).repositories).toEqual([snapshot().content.repository]) + const accepted = await acknowledgeSnapshot(directory, key, "1", { + status: "accepted", + revision: "1", + receivedAt: new Date().toISOString(), + }) + expect(accepted.entries[key].limit).toBeUndefined() + }) + it.each([ + // 6 sent, 12 counted: the server adds 6, so only 4 fit, not the proportional 5. + ["requests", 12, 10, 6, { requests: 4 }], + ["requests", undefined, undefined, 6, { requests: 5 }], + ["pullRequests", 3, 2, 3, undefined], + ["windowedPullRequests", 2001, 2000, 3, undefined], + ] as const)("learns a smaller %s limit from a snapshot rejection (%s of %s)", async (name, current, maximum, size, learned) => { + await setReportingEnabled(directory, true) + await queueSnapshots(directory, [snapshot(requests(size))]) + const at = Date.now() + const limit = { scope: "snapshot" as const, limit: name, current, maximum, at } + const state = await limitSnapshot(directory, key, "1", limit, at + 1000) + expect(state.entries[key].limit).toEqual(limit) + if (!learned) { + // No PRs were sent, and windowed IDs never exceed the server's limit, so sending fewer cannot help. + expect(state.entries[key].learned).toBeUndefined() + return + } + expect(state.entries[key].learned).toEqual({ ...learned, until: at + 7 * 24 * 60 * 60_000 }) + expect(learnedLimits(state, at).get(key)).toEqual({ ...SNAPSHOT_LIMITS, ...learned }) + expect(learnedLimits(state, at + 7 * 24 * 60 * 60_000).size).toBe(0) + }) + it("learns in one rejection a cap that fits when the server counts 26,000 retained requests besides the upload", async () => { + await setReportingEnabled(directory, true) + const retained = 26_000 + const upload = snapshot(requests(15_800)) + await queueSnapshots(directory, [upload]) + const at = Date.now() + const current = upload.content.requests.length + retained + const limit = { scope: "snapshot" as const, limit: "requests" as const, current, maximum: 32_000, at } + const limits = learnedLimits(await limitSnapshot(directory, key, "1", limit, at + 1000), at).get(key) + expect(limits).toEqual({ ...SNAPSHOT_LIMITS, requests: 6_000 }) + // The next upload is trimmed to the learned cap, and with the retained requests it fits the server's. + const next = fitSnapshot(upload.content, limits ?? SNAPSHOT_LIMITS, () => true) + expect((next?.content.requests.length ?? current) + retained).toBeLessThanOrEqual(32_000) + }) + it("learns bytes from the server's measure, so a different encoding still converges", async () => { + await setReportingEnabled(directory, true) + await queueSnapshots(directory, [snapshot(requests(4))]) + const pending = (await readReportingState(directory)).entries[key].pending + const bytes = Buffer.byteLength(JSON.stringify(pending)) + const state = await limitSnapshot( + directory, + key, + "1", + { scope: "snapshot", limit: "bytes", current: bytes * 2, maximum: bytes, at: 1 }, + 1000, + ) + expect(state.entries[key].learned?.bytes).toBe(Math.floor(bytes / 2)) + }) + it("lets /pr-reporting on retry an account pause and a repository limit at once", async () => { + await setReportingEnabled(directory, true) + await queueSnapshots(directory, [snapshot([requestId])]) + await limitSnapshot(directory, key, "1", { scope: "producer", at: 1 }, Date.now() + 3_600_000) + await limitSnapshot(directory, key, "1", { scope: "organization", at: 1 }, Date.now() + 3_600_000) + const state = await setReportingEnabled(directory, true) + expect(state.paused).toBeUndefined() + expect(state.entries[key]).toMatchObject({ retryAt: 0, limit: { scope: "producer" } }) + }) + it("ignores a limit for a revision that was already replaced", async () => { + await setReportingEnabled(directory, true) + await queueSnapshots(directory, [snapshot([requestId])]) + await queueSnapshots(directory, [snapshot([requestId, otherRequestId])]) + const state = await limitSnapshot(directory, key, "1", { scope: "producer", at: 1 }, 1000) + expect(state.entries[key].limit).toBeUndefined() + }) + it("shows a repository's partial-report notice once across concurrent sessions", async () => { + await setReportingEnabled(directory, true) + const trimmed = snapshot([requestId]) + trimmed.content.coverage = { ...trimmed.content.coverage, historyComplete: false, trimmedRequests: 4 } + const state = await queueSnapshots(directory, [trimmed]) + expect(state.entries[key].trimmed).toBe(4) + const taken = await Promise.all([takeLimitNotices(directory), takeLimitNotices(directory)]) + expect(taken.flatMap((notices) => notices.repositories)).toEqual([trimmed.content.repository]) + expect((await queueSnapshots(directory, [snapshot([requestId, otherRequestId])])).entries[key]).toMatchObject({ + limitNoticeShown: true, + }) + expect((await readReportingState(directory)).entries[key].trimmed).toBeUndefined() + }) + it.each([ + ["markers", "pull-request:101:open"], + ["uploadedAt", "yesterday"], + ["limit", { scope: "everywhere", at: 1 }], + ["learned", { requests: 0, until: 1 }], + ["urgent", false], + ])("fails closed on an invalid %s field", async (field, value) => { + await setReportingEnabled(directory, true) + await queueSnapshots(directory, [snapshot([requestId])]) + const path = join(directory, "pr-cost-reporting", "state.json") + const saved = JSON.parse(await readFile(path, "utf8")) + saved.entries[key][field] = value + await writeFile(path, JSON.stringify(saved)) + await expect(readReportingState(directory)).rejects.toThrow("unreadable") + }) + it("fails closed on an invalid account pause", async () => { + await setReportingEnabled(directory, true) + const path = join(directory, "pr-cost-reporting", "state.json") + const saved = JSON.parse(await readFile(path, "utf8")) + saved.paused = { [accountKey(account)]: { limit: { scope: "organization", at: 1 }, retryAt: "soon" } } + await writeFile(path, JSON.stringify(saved)) + await expect(readReportingState(directory)).rejects.toThrow("unreadable") + }) +}) diff --git a/src/extensions/pr-cost-reporting/queue.ts b/src/extensions/pr-cost-reporting/queue.ts new file mode 100644 index 000000000..1e56e738e --- /dev/null +++ b/src/extensions/pr-cost-reporting/queue.ts @@ -0,0 +1,710 @@ +import { createHash, randomUUID } from "node:crypto" +import { mkdir, open, stat } from "node:fs/promises" +import { join } from "node:path" +import { lock } from "proper-lockfile" +import { writeFileDurably } from "../../config/json.js" +import { readTelemetryConfig } from "../../config.js" +import { isWorkId } from "../../shared/work-id.js" +import { trackPRCostMetric } from "../telemetry/pr-cost.js" +import { isWorkAccount, type WorkAccount } from "../work-attribution/scope.js" +import { object, SHA256_HEX } from "../work-attribution/summary.js" +import { machineFingerprint } from "./machine.js" +import { + accountKey, + MAX_REVISION, + type ReportingRepository, + type RepositorySnapshot, + repositoryKey, + revision, + SNAPSHOT_LIMITS, + type SnapshotContent, + type SnapshotLimits, + validateSnapshot, + validTime, + type WireSnapshot, +} from "./snapshot.js" + +export interface SnapshotAck { + status: "accepted" | "unchanged" | "stale" + revision: string + receivedAt: string +} + +/** Scopes and limit names an upload rejection can carry; the server's frozen limits only bound its queries. */ +export const LIMIT_SCOPES = ["snapshot", "producer", "contributor", "organization"] as const +export const LIMIT_NAMES = [ + "bytes", + "requests", + "pullRequests", + "windowedPullRequests", + "snapshots", + "repositories", + "billingIds", +] as const +/** Contributor and organization limits apply to every repository of the account. */ +export const ACCOUNT_SCOPES: readonly string[] = ["contributor", "organization"] + +/** A validated PR_COST_LIMIT rejection. Free-form server text never enters durable state. */ +export interface ServerLimit { + scope?: (typeof LIMIT_SCOPES)[number] + limit?: (typeof LIMIT_NAMES)[number] + current?: number + maximum?: number + /** When the server reported it. */ + at: number +} + +export interface AccountPause { + limit: ServerLimit + retryAt: number + noticeShown?: true +} + +/** Ordinary changes upload at most once in this window per repository. */ +export const UPLOAD_INTERVAL_MS = 5 * 60_000 +/** A limit learned from the server is retried at the normal limits after a week. */ +const LEARNED_LIMIT_MS = 7 * 24 * 60 * 60_000 +/** The local queue's size limit; a replacement that would cross it waits instead of blocking every repository. */ +const MAX_STATE_BYTES = 24 * 1024 * 1024 +/** Room for JSON punctuation the per-entry estimate leaves out. */ +const STATE_SIZE_MARGIN = 64 * 1024 + +export interface PendingRepository { + account: WorkAccount + repository: ReportingRepository + revision: string + /** Membership that may still exist on the server; only a matching ACK can shrink it. */ + requestHashes: string[] + held?: boolean + acceptedDigest?: string + pendingDigest?: string + pending?: WireSnapshot + attempts: number + retryAt: number + lastError?: string + lastAcknowledgedAt?: string + /** Start of the last upload attempt; ordinary changes wait UPLOAD_INTERVAL_MS after it. */ + uploadedAt?: number + /** The pending snapshot changes a PR state, carries a new explicit correction or withdraws claims. */ + urgent?: true + /** PR states and explicit corrections the server acknowledged; a change to them uploads at once. */ + markers?: string[] + /** The last PR_COST_LIMIT rejection of this repository, until an upload is accepted. */ + limit?: ServerLimit + /** Smaller snapshot limits learned from a snapshot-scope rejection, used until `until`. */ + learned?: Partial & { until: number } + /** Requests left out of the latest queued snapshot to fit the limits. */ + trimmed?: number + /** The one-time partial-report notice was shown for this repository. */ + limitNoticeShown?: true +} + +export interface ReportingState { + version: 1 + enabled: boolean + /** New queues follow SaaS uploads until an explicit /pr-reporting choice is saved. */ + followsTelemetry?: true + defaultNoticeShown?: true + producerId: string + /** Hashed machine ID that owns producerId; a home copied to another machine starts a new producer. */ + machine?: string + entries: Record + /** Account-wide limit pauses, keyed by account. */ + paused?: Record + error?: string + /** Requests the latest capture left out without original account or repository evidence; others still report. */ + skippedRequests?: number +} + +export function reportingDirectory(agentDir: string): string { + return join(agentDir, "pr-cost-reporting") +} + +const statePath = (agentDir: string) => join(reportingDirectory(agentDir), "state.json") +const requestHash = (requestId: string) => createHash("sha256").update(requestId).digest("hex") +const count = (value: unknown) => Number.isSafeInteger(value) && Number(value) >= 0 + +function validLimit(value: unknown): value is ServerLimit { + return ( + object(value) && + (value.scope === undefined || LIMIT_SCOPES.some((scope) => scope === value.scope)) && + (value.limit === undefined || LIMIT_NAMES.some((name) => name === value.limit)) && + (value.current === undefined || count(value.current)) && + (value.maximum === undefined || count(value.maximum)) && + Number.isFinite(value.at) + ) +} + +function validEntryExtras(entry: PendingRepository): boolean { + const learned = entry.learned + return ( + (entry.uploadedAt === undefined || Number.isFinite(entry.uploadedAt)) && + (entry.urgent === undefined || entry.urgent === true) && + (entry.markers === undefined || + (Array.isArray(entry.markers) && + entry.markers.every((marker) => typeof marker === "string" && marker.length <= 256))) && + (entry.limit === undefined || validLimit(entry.limit)) && + (learned === undefined || + (object(learned) && + Number.isFinite(learned.until) && + [learned.requests, learned.pullRequests, learned.bytes].every( + (value) => value === undefined || (count(value) && value > 0), + ))) && + (entry.trimmed === undefined || count(entry.trimmed)) && + (entry.limitNoticeShown === undefined || entry.limitNoticeShown === true) + ) +} + +/** An ordinary change waits out the window after the last upload attempt; an urgent one, or a clock set back, does not. */ +export function inUploadWindow(entry: PendingRepository, now: number): boolean { + return ( + !entry.urgent && + entry.uploadedAt !== undefined && + entry.uploadedAt <= now && + now - entry.uploadedAt < UPLOAD_INTERVAL_MS + ) +} + +/** What the server must hear about at once: PR states and explicit `/work` corrections. */ +export function snapshotMarkers(content: SnapshotContent): string[] { + const markers = new Set(content.pullRequests.map((pr) => `pull-request:${pr.id}:${pr.state}`)) + for (const request of content.requests) + if (request.correction?.source === "work-command") + markers.add(`correction:${request.correction.id}:${request.correction.revision}`) + return [...markers].sort() +} + +function empty(): ReportingState { + return { + version: 1, + enabled: readTelemetryConfig().enabled, + followsTelemetry: true, + producerId: randomUUID(), + entries: {}, + } +} + +export async function readReportingState(agentDir: string): Promise { + return (await loadReportingState(agentDir)).state +} + +/** Changes with every saved update, since each one replaces the file; empty before the first. */ +export async function reportingStateVersion(agentDir: string): Promise { + const saved = await stat(statePath(agentDir)).catch(() => undefined) + return saved ? JSON.stringify([saved.ino, saved.size, saved.mtimeMs]) : "" +} + +/** The validated state and, when the file exists, its text, so an unchanged update can skip the write. */ +async function loadReportingState(agentDir: string): Promise<{ state: ReportingState; text?: string }> { + try { + const file = await open(statePath(agentDir), "r") + let value: ReportingState + let text: string + try { + if ((await file.stat()).size > MAX_STATE_BYTES) throw new Error("PR reporting state exceeds its local size limit") + text = await file.readFile("utf8") + value = JSON.parse(text) + } finally { + await file.close() + } + if ( + value?.version !== 1 || + typeof value.enabled !== "boolean" || + (value.followsTelemetry !== undefined && value.followsTelemetry !== true) || + (value.defaultNoticeShown !== undefined && value.defaultNoticeShown !== true) || + !isWorkId(value.producerId) || + (value.machine !== undefined && !SHA256_HEX.test(value.machine)) || + (value.skippedRequests !== undefined && !count(value.skippedRequests)) || + !value.entries || + typeof value.entries !== "object" || + Array.isArray(value.entries) + ) + throw new Error("Invalid PR reporting state") + for (const [key, entry] of Object.entries(value.entries)) { + if ( + !isWorkAccount(entry.account) || + !revision(entry.revision) || + !entry.repository || + !Number.isSafeInteger(entry.attempts) || + entry.attempts < 0 || + !Number.isFinite(entry.retryAt) || + !Array.isArray(entry.requestHashes) || + entry.requestHashes.some((hash) => typeof hash !== "string" || !SHA256_HEX.test(hash)) || + key !== `${accountKey(entry.account)}:${repositoryKey(entry.repository)}` || + !validEntryExtras(entry) + ) + throw new Error("Invalid PR reporting state") + if (entry.pending) { + validateSnapshot(entry.pending) + if ( + entry.pending.producerId !== value.producerId || + entry.pending.revision !== entry.revision || + repositoryKey(entry.pending.repository) !== repositoryKey(entry.repository) + ) + throw new Error("Invalid PR reporting state") + } + } + if ( + value.paused !== undefined && + (!object(value.paused) || + Object.values(value.paused).some( + (pause) => + !object(pause) || + !validLimit(pause.limit) || + !Number.isFinite(pause.retryAt) || + (pause.noticeShown !== undefined && pause.noticeShown !== true), + )) + ) + throw new Error("Invalid PR reporting state") + if (value.followsTelemetry) value.enabled = readTelemetryConfig().enabled + const machine = await machineFingerprint() + // Two machines sharing a producer would keep replacing each other's claims; the server deduplicates bills across producers. + if (machine && value.machine && value.machine !== machine) { + value.producerId = randomUUID() + value.machine = machine + value.entries = {} + value.paused = undefined + } + trackPRCostMetric({ + kind: "queueDepth", + value: Object.values(value.entries).filter((entry) => entry.pending).length, + }) + return { state: value, text } + } catch (error) { + if (object(error) && error.code === "ENOENT") { + trackPRCostMetric({ kind: "queueDepth", value: 0 }) + return { state: empty() } + } + throw new Error("PR reporting state is unreadable; queued reports were not replaced", { cause: error }) + } +} + +async function update(agentDir: string, mutate: (state: ReportingState) => void): Promise { + const directory = reportingDirectory(agentDir) + await mkdir(directory, { recursive: true, mode: 0o700 }) + let compromised: Error | undefined + const release = await lock(directory, { + retries: { retries: 20, minTimeout: 10, maxTimeout: 50 }, + stale: 5000, + update: 1000, + onCompromised: (error) => { + compromised = error + }, + }) + try { + const { state, text } = await loadReportingState(agentDir) + state.machine ??= await machineFingerprint() + mutate(state) + const body = `${JSON.stringify(state)}\n` + // Passes run every 30 seconds; an unchanged queue is not rewritten or synced. + if (body === text) return state + if (Buffer.byteLength(body) > MAX_STATE_BYTES) throw new Error("PR reporting queue exceeds its local size limit") + await writeFileDurably(statePath(agentDir), body, () => { + if (compromised) throw compromised + }) + + const parent = await open(directory, "r") + try { + await parent.sync() + } finally { + await parent.close() + } + trackPRCostMetric({ + kind: "queueDepth", + value: Object.values(state.entries).filter((entry) => entry.pending).length, + }) + return state + } finally { + if (!compromised) await release() + } +} + +/** An explicit choice also restarts delivery: the next snapshot of every repository goes out at once. */ +export function setReportingEnabled(agentDir: string, enabled: boolean): Promise { + return update(agentDir, (state) => { + state.enabled = enabled + state.followsTelemetry = undefined + state.error = undefined + state.skippedRequests = undefined + state.paused = undefined + for (const entry of Object.values(state.entries)) { + entry.uploadedAt = undefined + entry.attempts = 0 + entry.retryAt = 0 + if (enabled) continue + // The discarded replacement may have arrived even if its acknowledgement did not. + if (entry.pending) entry.acceptedDigest = undefined + entry.pending = undefined + entry.pendingDigest = undefined + entry.urgent = undefined + entry.lastError = undefined + entry.held = undefined + } + }) +} + +/** Claim the installation notice under the queue lock so concurrent sessions show it once. */ +export async function takeReportingNotice(agentDir: string): Promise { + let show = false + await update(agentDir, (state) => { + if (state.enabled && state.followsTelemetry && !state.defaultNoticeShown) { + state.defaultNoticeShown = true + show = true + } + }) + return show +} + +export function recordReportingError(agentDir: string, error: string): Promise { + return update(agentDir, (state) => { + if (state.enabled) state.error = error + }) +} + +/** + * Replaces a repository's entire inventory. The fsynced rename completes before delivery can begin. + * Every change replaces the pending snapshot; whether it may skip the upload window is decided here. + */ +export async function queueSnapshots( + agentDir: string, + snapshots: RepositorySnapshot[], + completeInventory = false, + skippedRequests = 0, +): Promise { + let largest: { requests: number; bytes: number } | undefined + const state = await update(agentDir, (state) => { + if (!state.enabled) return + const now = Date.now() + const invalid = new Set() + const current = new Map() + for (const snapshot of snapshots) { + const key = `${accountKey(snapshot.account)}:${repositoryKey(snapshot.content.repository)}` + try { + const next = BigInt(state.entries[key]?.revision ?? "0") + 1n + validateSnapshot({ + schemaVersion: 1, + producerId: state.producerId, + revision: String(next), + generatedAt: new Date().toISOString(), + ...snapshot.content, + }) + current.set(key, snapshot) + } catch { + invalid.add(key) + } + } + + const reported = new Map>() + for (const snapshot of current.values()) { + if (snapshot.incomplete) continue + const key = accountKey(snapshot.account) + const members = reported.get(key) ?? new Set() + for (const requestId of snapshot.observedRequestIds ?? + snapshot.content.requests.map((request) => request.requestId)) + if (isWorkId(requestId)) members.add(requestHash(requestId)) + reported.set(key, members) + } + + let held = 0 + const withdrawals = new Set() + for (const [key, entry] of Object.entries(state.entries)) { + if (entry.learned && entry.learned.until <= now) entry.learned = undefined + // An acknowledged withdrawal left no claims; rebuilding it would only send it again. + if (!current.has(key) && (entry.requestHashes.length || entry.pending)) { + if ( + !invalid.has(key) && + entry.requestHashes.every((member) => reported.get(accountKey(entry.account))?.has(member)) + ) { + current.set(key, { + account: entry.account, + content: { + repository: entry.repository, + pullRequests: [], + requests: [], + coverage: { observedRequests: 0, unpricedRequests: 0, historyComplete: completeInventory }, + }, + }) + withdrawals.add(key) + } else { + entry.held = true + // An invalid replacement is reported as invalid, not also as missing evidence. + if (!invalid.has(key)) held++ + } + } + } + + const changes: [string, RepositorySnapshot, string, number][] = [] + for (const [key, snapshot] of current) { + const text = JSON.stringify(snapshot.content) + const digest = createHash("sha256").update(text).digest("hex") + const entry = state.entries[key] + if ( + entry && + (snapshot.incomplete || + entry.requestHashes.some((member) => !reported.get(accountKey(entry.account))?.has(member))) + ) { + entry.held = true + held++ + continue + } + if (entry) entry.held = undefined + if (entry?.pendingDigest === digest || (!entry?.pending && entry?.acceptedDigest === digest)) continue + // History that is all outside the upload window has nothing to send and no earlier upload to replace. + if (!entry && !snapshot.content.requests.length && !snapshot.content.pullRequests.length) continue + changes.push([key, snapshot, digest, text.length]) + } + + // An explicit correction can move requests to another repository; when one first appears, every + // changed repository of that account goes out at once, not only the one that carries the receipt. + const corrected = new Set() + for (const [key, snapshot] of changes) { + const entry = state.entries[key] + const seen = new Set([...(entry?.markers ?? []), ...(entry?.pending ? snapshotMarkers(entry.pending) : [])]) + if (snapshotMarkers(snapshot.content).some((marker) => marker.startsWith("correction:") && !seen.has(marker))) + corrected.add(accountKey(snapshot.account)) + } + + // Smaller replacements first; one that would push the queue past its size limit waits for space. The snapshot it + // would replace stays deliverable: acknowledging queued snapshots is what frees that space. + let size = changes.length ? Buffer.byteLength(JSON.stringify(state)) : 0 + let tooLarge = 0 + changes.sort((a, b) => a[3] - b[3]) + for (const [key, snapshot, digest] of changes) { + const entry = state.entries[key] + const next = BigInt(entry?.revision ?? "0") + 1n + if (next > MAX_REVISION) throw new Error("PR reporting revision limit reached") + const pending: WireSnapshot = { + schemaVersion: 1, + producerId: state.producerId, + revision: String(next), + generatedAt: new Date().toISOString(), + ...snapshot.content, + } + + // Unacknowledged PR states and corrections stay urgent through replacements until the server has them. + const acknowledged = new Set(entry?.markers) + const urgent = + entry?.urgent || + withdrawals.has(key) || + corrected.has(accountKey(snapshot.account)) || + snapshotMarkers(snapshot.content).some((marker) => !acknowledged.has(marker)) + const replacement: PendingRepository = { + ...entry, + account: snapshot.account, + repository: snapshot.content.repository, + revision: String(next), + requestHashes: [ + ...new Set([ + ...(entry?.requestHashes ?? []), + ...snapshot.content.requests.map((request) => request.requestId).map(requestHash), + ]), + ].sort(), + held: undefined, + pendingDigest: digest, + pending, + urgent: urgent || undefined, + trimmed: snapshot.content.coverage.trimmedRequests || undefined, + attempts: entry?.attempts ?? 0, + retryAt: entry?.retryAt ?? 0, + } + + const grown = + Buffer.byteLength(JSON.stringify({ [key]: replacement })) - + (entry ? Buffer.byteLength(JSON.stringify({ [key]: entry })) : 0) + if (size + grown > MAX_STATE_BYTES - STATE_SIZE_MARGIN) { + tooLarge++ + continue + } + size += grown + state.entries[key] = replacement + largest = { + requests: Math.max(largest?.requests ?? 0, pending.requests.length), + bytes: Math.max(largest?.bytes ?? 0, Buffer.byteLength(JSON.stringify(pending))), + } + } + state.error = + [ + ...(held ? [`PR reporting held ${held} repository snapshot(s) because earlier evidence is missing`] : []), + ...(invalid.size + ? [ + `PR reporting held ${invalid.size} snapshot(s) that are invalid or exceed the upload limits after trimming`, + ] + : []), + ...(tooLarge + ? [`PR reporting held ${tooLarge} snapshot(s) until queued reports leave room in the local queue`] + : []), + ].join(". ") || undefined + state.skippedRequests = skippedRequests || undefined + }) + if (largest) { + trackPRCostMetric({ kind: "snapshotRequests", value: largest.requests }) + trackPRCostMetric({ kind: "snapshotBytes", value: largest.bytes }) + } + return state +} + +export function acknowledgeSnapshot( + agentDir: string, + key: string, + sentRevision: string, + ack: SnapshotAck, +): Promise { + return update(agentDir, (state) => { + const entry = state.entries[key] + if (!state.enabled || !entry?.pending || entry.pending.revision !== sentRevision) return + if ( + !ack || + !revision(ack.revision) || + BigInt(ack.revision) < BigInt(sentRevision) || + !["accepted", "unchanged", "stale"].includes(ack.status) || + !validTime(ack.receivedAt) || + (ack.status === "stale" ? BigInt(ack.revision) <= BigInt(sentRevision) : ack.revision !== sentRevision) + ) + throw new Error("Invalid PR reporting acknowledgement") + if (ack.status !== "stale") { + entry.requestHashes = entry.pending.requests.map((request) => requestHash(request.requestId)).sort() + entry.acceptedDigest = entry.pendingDigest + entry.lastAcknowledgedAt = ack.receivedAt + entry.markers = snapshotMarkers(entry.pending) + entry.limit = undefined + // Only an upload that adds claims proves an account limit no longer blocks; withdrawals pass it. + if (state.paused && (entry.pending.requests.length || entry.pending.pullRequests.length)) { + delete state.paused[accountKey(entry.account)] + if (!Object.keys(state.paused).length) state.paused = undefined + } + } else entry.acceptedDigest = undefined + entry.revision = ack.revision + entry.pending = undefined + entry.pendingDigest = undefined + entry.urgent = undefined + entry.lastError = undefined + entry.attempts = 0 + entry.retryAt = 0 + entry.uploadedAt = Date.now() + }) +} + +export function deferSnapshot( + agentDir: string, + key: string, + sentRevision: string, + retryAt: number, + message: string, +): Promise { + return update(agentDir, (state) => { + const entry = state.entries[key] + if (!state.enabled || entry?.pending?.revision !== sentRevision) return + entry.attempts++ + entry.retryAt = retryAt + entry.lastError = message + entry.uploadedAt = Date.now() + }) +} + +/** + * A smaller cap for the rejected dimension, learned in one step. The server's overflow may scale with the + * upload (another encoding of the same body) or add a constant (records it counts besides the upload), so + * the cap is the smaller of the proportional and the subtractive target. A subtractive target of zero or + * less cannot be met by any upload, so only the proportional one applies then. The client never sends more + * than the server's 2,000 windowed PR IDs, so that limit teaches nothing. + */ +function learnedLimit(pending: WireSnapshot, limit: ServerLimit): number | undefined { + if (limit.limit !== "requests" && limit.limit !== "pullRequests" && limit.limit !== "bytes") return undefined + const measured = { + requests: pending.requests.length, + pullRequests: pending.pullRequests.length, + bytes: Buffer.byteLength(JSON.stringify(pending)), + }[limit.limit] + const { current, maximum } = limit + let target = Math.floor(measured * 0.9) + if (current !== undefined && maximum !== undefined && current > maximum) { + const subtractive = measured - (current - maximum) + target = Math.floor((measured * maximum) / current) + if (subtractive > 0) target = Math.min(target, subtractive) + } + + const learned = Math.min(target, measured - 1) + return learned > 0 ? learned : undefined +} + +/** + * A PR_COST_LIMIT rejection is quota, not an outage. Account-wide limits pause every repository of the + * account; a repository limit waits on its own counter, and a snapshot limit also teaches a smaller cap. + */ +export function limitSnapshot( + agentDir: string, + key: string, + sentRevision: string, + limit: ServerLimit, + retryAt: number, +): Promise { + return update(agentDir, (state) => { + const entry = state.entries[key] + if (!state.enabled || !entry?.pending || entry.pending.revision !== sentRevision) return + entry.uploadedAt = Date.now() + entry.lastError = undefined + if (limit.scope && ACCOUNT_SCOPES.includes(limit.scope)) { + const account = accountKey(entry.account) + const previous = state.paused?.[account] + state.paused = { + ...state.paused, + [account]: { limit, retryAt, ...(previous?.noticeShown ? { noticeShown: true } : {}) }, + } + return + } + entry.limit = limit + entry.retryAt = retryAt + const learned = limit.scope === "snapshot" ? learnedLimit(entry.pending, limit) : undefined + if (learned && limit.limit) + entry.learned = { + ...(entry.learned && entry.learned.until > limit.at ? entry.learned : {}), + [limit.limit]: learned, + until: limit.at + LEARNED_LIMIT_MS, + } + }) +} + +/** Limits a repository's server rejections taught, by entry key, for buildSnapshots. */ +export function learnedLimits(state: ReportingState, now = Date.now()): Map { + const limits = new Map() + for (const [key, entry] of Object.entries(state.entries)) { + const learned = entry.learned + if (!learned || learned.until <= now) continue + limits.set(key, { + requests: Math.min(SNAPSHOT_LIMITS.requests, learned.requests ?? Number.POSITIVE_INFINITY), + pullRequests: Math.min(SNAPSHOT_LIMITS.pullRequests, learned.pullRequests ?? Number.POSITIVE_INFINITY), + bytes: Math.min(SNAPSHOT_LIMITS.bytes, learned.bytes ?? Number.POSITIVE_INFINITY), + }) + } + return limits +} + +export interface LimitNotices { + repositories: ReportingRepository[] + pauses: ServerLimit[] +} + +/** Repositories and accounts whose one-time limit notice is still due. */ +export function dueLimitNotices(state: ReportingState): LimitNotices { + if (!state.enabled) return { repositories: [], pauses: [] } + return { + repositories: Object.values(state.entries) + .filter((entry) => !entry.limitNoticeShown && (entry.trimmed || entry.limit)) + .map((entry) => entry.repository), + pauses: Object.values(state.paused ?? {}) + .filter((pause) => !pause.noticeShown) + .map((pause) => pause.limit), + } +} + +/** Claim limit notices under the queue lock so concurrent sessions show each once. */ +export async function takeLimitNotices(agentDir: string): Promise { + let notices: LimitNotices = { repositories: [], pauses: [] } + await update(agentDir, (state) => { + notices = dueLimitNotices(state) + if (!state.enabled) return + for (const entry of Object.values(state.entries)) if (entry.trimmed || entry.limit) entry.limitNoticeShown = true + for (const pause of Object.values(state.paused ?? {})) pause.noticeShown = true + }) + return notices +} diff --git a/src/extensions/pr-cost-reporting/snapshot.test.ts b/src/extensions/pr-cost-reporting/snapshot.test.ts new file mode 100644 index 000000000..e584a5786 --- /dev/null +++ b/src/extensions/pr-cost-reporting/snapshot.test.ts @@ -0,0 +1,998 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest" +import type { WorkPullRequest } from "../pull-request-status/pull-requests.js" +import { calculatePullRequestCosts } from "../work-attribution/costs.js" +import type { WorkRecord } from "../work-attribution/summary.js" +import { + accountKey, + buildSnapshots, + fitSnapshot, + type RepositoryIdentity, + repositoryKey, + SNAPSHOT_LIMITS, + type SnapshotContent, + validateSnapshot, + type WireSnapshot, + wireBytes, +} from "./snapshot.js" + +const account = { + apiUrl: "https://api.example", + organizationId: "11111111-1111-4111-8111-111111111111", + userId: "22222222-2222-4222-8222-222222222222", +} +const requestId = "33333333-3333-4333-8333-333333333333" +const billingId = "44444444-4444-4444-8444-444444444444" +const at = "2026-10-04T12:00:00.000Z" +beforeEach(() => { + vi.useFakeTimers({ toFake: ["Date"] }) + vi.setSystemTime(new Date("2026-10-07T12:00:00Z")) +}) +afterEach(() => vi.useRealTimers()) +const pull = (number = 1, repository = "example/repo", repositoryId = "42"): WorkPullRequest => ({ + provider: "github", + id: String(100 + number), + repositoryId, + host: "github.com", + repository, + number, + url: `https://github.com/${repository}/pull/${number}`, + state: "merged", + mergedAt: "2026-10-04T13:00:00.000Z", + closedAt: null, + checkedAt: at, + headSha: "a".repeat(40), + mergeCommitSha: null, +}) +function records(pulls = [pull()], fields: Record = {}): WorkRecord[] { + return [ + { + version: 1, + type: "request", + workId: "private-work", + sessionId: "private-session", + requestId, + startedAt: at, + recordedAt: at, + scope: { account, repository: "/private/repo/.git" }, + prompt: "secret text", + ...fields, + }, + { + version: 1, + type: "commit", + workId: "private-work", + sessionId: "private-session", + sha: "a".repeat(40), + repository: "/private/repo/.git", + worktree: "/private/repo", + recordedAt: at, + pullRequests: pulls, + }, + ] +} +function build(rows = records(), priced = true) { + return buildSnapshots( + rows, + calculatePullRequestCosts( + rows, + priced ? [{ requestId, billingRecordId: billingId, costUsd: "1.234567891", account }] : [], + ), + new Map(), + ) +} +function wire(): WireSnapshot { + return { schemaVersion: 1, producerId: requestId, revision: "1", generatedAt: at, ...build().snapshots[0].content } +} + +describe("allowlisted repository snapshots", () => { + it.each([ + "open", + "merged", + "closed", + ] as const)("uploads old work while a PR is %s or within its 32-day grace", (state) => { + const pr = { ...pull(), state, mergedAt: state === "merged" ? at : null, closedAt: state === "closed" ? at : null } + const rows = records([pr], { startedAt: "2026-01-01T00:00:00Z", recordedAt: "2026-01-01T00:00:00Z" }) + const content = build(rows).snapshots[0].content + expect(content.requests).toHaveLength(1) + expect(content.pullRequests[0]).toMatchObject({ id: pr.id, state }) + if (state === "closed") expect(content.pullRequests[0]).toHaveProperty("closedAt", at) + }) + it.each([ + "merged", + "closed", + ] as const)("omits a %s PR after 32 days plus two days of clock grace without treating its evidence as lost", (state) => { + const finished = "2026-09-03T11:59:59Z" + const pr = { + ...pull(), + state, + mergedAt: state === "merged" ? finished : null, + closedAt: state === "closed" ? finished : null, + } + const rows = records([pr], { startedAt: "2026-08-01T00:00:00Z", recordedAt: "2026-08-01T00:00:00Z" }) + const snapshot = build(rows).snapshots[0] + expect(snapshot.content.requests).toEqual([]) + expect(snapshot.content.pullRequests).toEqual([]) + expect(snapshot.content).toHaveProperty("windowedPullRequestIds", [pr.id]) + expect(snapshot).toHaveProperty("observedRequestIds", [requestId]) + expect(snapshot.content.coverage).toMatchObject({ observedRequests: 0, unpricedRequests: 0, historyComplete: true }) + }) + it.each([ + "merged", + "closed", + ] as const)("keeps a %s PR when the client clock is three hours ahead of its 32-day cutoff", (state) => { + const finished = "2026-09-05T12:00:00Z" + vi.setSystemTime(new Date("2026-10-07T15:00:00Z")) + const pr = { + ...pull(), + state, + mergedAt: state === "merged" ? finished : null, + closedAt: state === "closed" ? finished : null, + } + const rows = records([pr], { startedAt: "2026-08-01T00:00:00Z", recordedAt: "2026-08-01T00:00:00Z" }) + const content = build(rows).snapshots[0].content + expect(content.requests.map((request) => request.requestId)).toEqual([requestId]) + expect(content.pullRequests.map((pull) => pull.id)).toEqual([pr.id]) + }) + it("uploads a long-lived work's recent requests without its older unlinked history", () => { + const rows = records([], { startedAt: "2026-01-01T00:00:00Z" }) + rows.push({ ...rows[0], requestId: billingId, startedAt: at }) + const report = calculatePullRequestCosts(rows, []) + const result = buildSnapshots( + rows, + report, + new Map([["/private/repo/.git", { provider: "github", host: "github.com", id: "42" }]]), + ) + expect(result.snapshots[0].content.requests.map((request) => request.requestId)).toEqual([billingId]) + }) + it("drops old unlinked work but keeps a closed PR with an unknown close time", () => { + const rows = records([], { startedAt: "2026-01-01T00:00:00Z" }) + const report = calculatePullRequestCosts(rows, []) + const result = buildSnapshots( + rows, + report, + new Map([["/private/repo/.git", { provider: "github", host: "github.com", id: "42" }]]), + ) + expect(result.snapshots[0].content.requests).toEqual([]) + expect( + build(records([{ ...pull(), state: "closed", mergedAt: null }], { startedAt: "2026-01-01T00:00:00Z" })) + .snapshots[0].content.requests, + ).toHaveLength(1) + }) + it("contains provider and billing evidence only, without prices or private local data", () => { + const content = build().snapshots[0].content + // Without an input record or native proof, the request is inferred and uploads as session grouping. + expect(content.requests[0]).toEqual({ + requestId, + billingRecordIds: [billingId], + startedAt: at, + allocation: { kind: "pull-request", pullRequestIds: ["101"], method: "session" }, + }) + for (const privateValue of [ + "private-work", + "private-session", + "/private", + "secret text", + "1.234567891", + "a".repeat(40), + account.userId, + ]) + expect(JSON.stringify(content)).not.toContain(privateValue) + expect(() => validateSnapshot(wire())).not.toThrow() + }) + it("keeps missing prices in the complete inventory", () => { + const content = build(records(), false).snapshots[0].content + expect(content.requests[0].billingRecordIds).toEqual([]) + expect(content.coverage).toMatchObject({ observedRequests: 1, unpricedRequests: 1 }) + }) + it("exports cross-repository shared work as unknown with each repository's candidate", () => { + const result = build(records([pull(), pull(2, "example/other", "43")])) + expect( + result.snapshots.map(({ content }) => [content.repository.id, content.requests[0].allocation.pullRequestIds]), + ).toEqual([ + ["42", ["101"]], + ["43", ["102"]], + ]) + for (const { content } of result.snapshots) { + expect(content.requests[0].allocation.kind).toBe("unknown") + expect(() => + validateSnapshot({ schemaVersion: 1, producerId: requestId, revision: "1", generatedAt: at, ...content }), + ).not.toThrow() + } + }) + it("keeps a PR complete beside an unknown request that cannot belong to it", () => { + const unresolved = "55555555-5555-4555-8555-555555555555" + const other: WorkRecord = { + ...records()[0], + workId: "other-work", + sessionId: "other-session", + requestId: unresolved, + segment: { id: "segment", attribution: "unknown", reason: "model-uncertain" }, + } + const rows = [...records(), other] + const report = calculatePullRequestCosts( + rows, + [{ requestId, billingRecordId: billingId, costUsd: "1.25", account }], + new Set(), + new Map([[unresolved, account]]), + ) + const repositories = new Map([ + ["/private/repo/.git", { provider: "github" as const, host: "github.com", id: "42" }], + ]) + const content = buildSnapshots(rows, report, repositories).snapshots[0].content + // The server leaves only an unknown request's candidate PRs incomplete. + expect(content.requests.find((row) => row.requestId === unresolved)?.allocation).toMatchObject({ + kind: "unknown", + pullRequestIds: [], + }) + expect(report.pullRequests[0].totalCostUsd).toBe("1.250000000") + }) + it("ignores unscoped history older than the upload window", () => { + const legacy: WorkRecord = { + version: 1, + type: "request", + workId: "legacy-work", + sessionId: "legacy-session", + requestId: "55555555-5555-4555-8555-555555555555", + recordedAt: "2026-03-01T00:00:00Z", + } + const result = build([legacy, ...records()]) + expect(result).toMatchObject({ skippedRequests: 0, incomplete: false }) + expect(result.snapshots[0].content.coverage.historyComplete).toBe(true) + }) + it("reports a verified no-charge attempt as priced", () => { + const unbilled = "55555555-5555-4555-8555-555555555555" + const rows = [...records(), { ...records()[0], requestId: unbilled }] + const report = calculatePullRequestCosts( + rows, + [{ requestId, billingRecordId: billingId, costUsd: "1.5", account }], + new Set(), + new Map([[unbilled, account]]), + ) + const content = buildSnapshots(rows, report, new Map()).snapshots[0].content + expect(report.pullRequests[0].totalCostUsd).toBe("1.500000000") + expect(content.coverage).toMatchObject({ observedRequests: 2, unpricedRequests: 0 }) + }) + it.each([1, 2, 3])("retains all %s post-merge PR candidates in a valid snapshot", (count) => { + const rows = records( + Array.from({ length: count }, (_, index) => pull(index + 1)), + { + startedAt: "2026-10-04T14:00:00.000Z", + }, + ) + const report = calculatePullRequestCosts(rows, [{ requestId, billingRecordId: billingId, costUsd: "1", account }]) + expect(report.requests[0].allocation).toBe("post-merge") + const content = buildSnapshots(rows, report, new Map()).snapshots[0].content + expect(content.requests[0].allocation).toEqual({ + kind: count === 1 ? "post-merge" : "shared", + pullRequestIds: Array.from({ length: count }, (_, index) => String(101 + index)), + method: "session", + }) + expect(content.requests[0].billingRecordIds).toEqual([billingId]) + expect(() => + validateSnapshot({ schemaVersion: 1, producerId: requestId, revision: "1", generatedAt: at, ...content }), + ).not.toThrow() + }) + it.each(["open", "merged"] as const)("uploads a model-matched request on a %s PR as a model guess", (state) => { + const pr = { ...pull(), state, mergedAt: state === "merged" ? pull().mergedAt : null } + const rows = records([pr], { segment: { id: "segment", attribution: "inferred", reason: "model-same" } }) + const report = calculatePullRequestCosts(rows, [{ requestId, billingRecordId: billingId, costUsd: "1", account }]) + expect(buildSnapshots(rows, report, new Map()).snapshots[0].content.requests[0].allocation).toMatchObject({ + method: "model", + }) + }) + it("keeps semantic and session assignments visibly inferred", () => { + for (const [attribution, expected] of [ + ["inferred", "model"], + ["session", "session"], + ]) { + const result = build( + records([pull()], { segment: { id: "private-segment", attribution, reason: "secret reason" } }), + ) + expect(result.snapshots[0].content.requests[0].allocation.method).toBe(expected) + } + }) + it("does not stamp missing accounts with today's account or export foreign billing IDs", () => { + expect(build(records([pull()], { scope: undefined })).snapshots).toEqual([]) + const rows = records() + const report = calculatePullRequestCosts(rows, [ + { requestId, billingRecordId: billingId, costUsd: "1", account: { ...account, userId: requestId } }, + ]) + expect(buildSnapshots(rows, report, new Map()).snapshots[0].content.requests[0].billingRecordIds).toEqual([]) + }) + it("keeps billing rows with an unverified contributor out of the upload", () => { + const rows = records() + const report = calculatePullRequestCosts(rows, [{ requestId, billingRecordId: billingId, costUsd: "1" }]) + expect(buildSnapshots(rows, report, new Map()).snapshots[0].content.requests[0].billingRecordIds).toEqual([]) + }) + it("matches the server's per-snapshot limits", () => { + expect(SNAPSHOT_LIMITS).toEqual({ requests: 32_000, pullRequests: 250, bytes: 8 * 1024 * 1024 }) + }) + it.each([ + "requests", + "pullRequests", + "bills", + "bytes", + ])("rejects a payload past the %s limit at the boundary", (field) => { + const value = wire() + const bills = Array.from( + { length: 8 }, + (_, index) => `00000000-0000-4000-8000-${String(index + 1).padStart(12, "0")}`, + ) + if (field === "requests") { + value.requests = Array.from({ length: 32_001 }, () => value.requests[0]) + value.coverage.observedRequests = 32_001 + } + if (field === "pullRequests") value.pullRequests = Array.from({ length: 251 }, () => value.pullRequests[0]) + if (field === "bills") value.requests[0].billingRecordIds = [...bills, "00000000-0000-4000-8000-000000000009"] + if (field === "bytes") { + // Within every count limit, but 30,000 requests with eight bills each exceed 8 MiB. + value.requests = Array.from({ length: 30_000 }, (_, index) => ({ + requestId: `10000000-0000-4000-8000-${String(index).padStart(12, "0")}`, + billingRecordIds: bills, + startedAt: at, + allocation: { kind: "unlinked", pullRequestIds: [], method: "session" }, + })) + value.coverage = { observedRequests: 30_000, unpricedRequests: 0, historyComplete: true } + expect(Buffer.byteLength(JSON.stringify(value))).toBeGreaterThan(SNAPSHOT_LIMITS.bytes) + } + expect(() => validateSnapshot(value)).toThrow() + }) + it.each([ + [0, true, true], + [3, false, true], + [3, true, false], + [-1, false, false], + [1.5, false, false], + ])("accepts %s trimmed requests with complete history=%s: %s", (trimmed, historyComplete, valid) => { + const value = wire() + value.coverage = { ...value.coverage, historyComplete, trimmedRequests: trimmed } + if (valid) expect(() => validateSnapshot(value)).not.toThrow() + else expect(() => validateSnapshot(value)).toThrow() + }) + it("rejects unexpected fields in a recovered pending payload", () => { + const value = wire() + Object.assign(value.requests[0], { prompt: "must never upload" }) + expect(() => validateSnapshot(value)).toThrow() + }) + it.each([ + { id: "not-a-uuid" }, + { revision: 0 }, + { revision: 2147483648 }, + { recordedAt: "yesterday" }, + { source: "model" }, + { localPath: "/private/plan.md" }, + ])("rejects an invalid or private correction receipt: %j", (fields) => { + const value = wire() + value.requests[0].correction = { id: billingId, revision: 1, recordedAt: at, source: "work-command" } + Object.assign(value.requests[0].correction, fields) + expect(() => validateSnapshot(value)).toThrow() + }) + it.each([ + ["101"], + ["102", "102"], + ["not-a-provider-id"], + ])("rejects an overlapping or invalid window marker: %j", (...ids) => { + const value = wire() + value.windowedPullRequestIds = ids + expect(() => validateSnapshot(value)).toThrow() + }) + it("keeps refresh timestamps scoped to requests in the reported account and repository", () => { + const rows = records() + const report = calculatePullRequestCosts(rows, [{ requestId, billingRecordId: billingId, costUsd: "1", account }]) + const refreshed = new Map([ + [requestId, at], + ["other-account-request", "2026-10-05T13:00:00Z"], + ]) + expect(buildSnapshots(rows, report, new Map(), refreshed).snapshots[0].content.coverage.lastCostRefreshAt).toBe(at) + }) +}) + +describe("local and reported confidence", () => { + it("sends native evidence for the editing input and session evidence for a separate input", () => { + const segment = { id: billingId, attribution: "session", reason: "matching-disabled" } + const [request, commit] = records([pull()], { segment }) + const other = "55555555-5555-4555-8555-555555555555" + const transition: WorkRecord = { + ...request, + type: "file_transition", + transitionId: "edit", + toolCallId: "tool", + cwd: "/private/repo", + repository: "/private/repo/.git", + worktree: "/private/repo", + path: "code.ts", + baseline: "c".repeat(40), + baselineFile: { blob: "d".repeat(40), mode: "100644" }, + before: { blob: "d".repeat(40), mode: "100644" }, + after: { blob: "e".repeat(40), mode: "100644" }, + cursor: { bytes: 100, digest: "f".repeat(64) }, + } + const rows = [ + request, + { ...request, requestId: other, segment: { ...segment, id: other } }, + transition, + { + ...commit, + source: "native-file-transition", + paths: ["code.ts"], + transitionIds: ["edit"], + fileMatches: [{ path: "code.ts", worktree: "/private/repo", method: "file-chain", transitionIds: ["edit"] }], + }, + ] + const report = calculatePullRequestCosts(rows, [ + { requestId, billingRecordId: billingId, costUsd: "0.1", account }, + { requestId: other, billingRecordId: other, costUsd: "0.000000001", account }, + ]) + expect(report.pullRequests[0]).toMatchObject({ + totalCostUsd: "0.100000001", + explicit: { requestIds: [requestId] }, + inferred: { requestIds: [other] }, + }) + expect(buildSnapshots(rows, report, new Map()).snapshots[0].content.requests).toMatchObject([ + { requestId, allocation: { kind: "pull-request", method: "native" } }, + { requestId: other, allocation: { kind: "pull-request", method: "session" } }, + ]) + }) + it.each([ + "saved-plan", + "pasted-plan", + "named-artifact", + "work-command", + ])("labels a %s confirmation with its actual source", (source) => { + const rows: WorkRecord[] = records([pull()], { + segment: { id: billingId, attribution: "session", reason: "test" }, + }).map((row) => ({ ...row, workId: requestId })) + rows.push({ + ...rows[0], + type: "work_link", + linkId: "55555555-5555-4555-8555-555555555555", + revision: 1, + status: "active", + sourceWorkId: requestId, + targetWorkId: requestId, + requestIds: [requestId], + evidence: { source, requestId, segmentId: billingId }, + }) + const result = build(rows) + expect(result.snapshots[0].content.requests[0]).toHaveProperty("correction", { + id: "55555555-5555-4555-8555-555555555555", + revision: 1, + recordedAt: at, + source: source === "work-command" ? "work-command" : "producer-confirmation", + }) + expect(result.snapshots[0].content.requests[0].allocation).toEqual({ + kind: "pull-request", + pullRequestIds: ["101"], + method: source === "work-command" ? "user-correction" : "explicit", + }) + }) + it.each(["active", "revoked"])("uploads an explicit %s correction after the normal window until day 90", (status) => { + const finished = "2026-08-28T12:00:00.000Z" + const rows: WorkRecord[] = records([{ ...pull(), mergedAt: finished }], { + startedAt: "2026-08-27T12:00:00.000Z", + recordedAt: "2026-08-27T12:00:00.000Z", + segment: { id: billingId, attribution: "session", reason: "test" }, + }).map((row) => ({ ...row, workId: requestId })) + rows.push({ + ...rows[0], + type: "work_link", + recordedAt: at, + linkId: billingId, + revision: 2, + status, + sourceWorkId: requestId, + targetWorkId: requestId, + requestIds: [requestId], + evidence: { source: "work-command", segmentId: billingId }, + }) + const result = build(rows) + expect(result.snapshots[0].content.requests).toHaveLength(1) + expect(result.snapshots[0].content.requests[0]).toHaveProperty("correction", { + id: billingId, + revision: 2, + recordedAt: at, + source: "work-command", + }) + expect(result.snapshots[0].content.pullRequests.map((pr) => pr.id)).toEqual(["101"]) + expect(() => + validateSnapshot({ + schemaVersion: 1, + producerId: requestId, + revision: "2", + generatedAt: at, + ...result.snapshots[0].content, + }), + ).not.toThrow() + vi.setSystemTime(new Date("2026-11-26T12:00:00.000Z")) + expect(build(rows).snapshots[0].content.requests).toEqual([]) + }) + it.each([ + "active", + "revoked", + ])("uploads an explicit %s correction from another work after the normal window until day 90", (status) => { + const planning = "66666666-6666-4666-8666-666666666666" + const implementing = "77777777-7777-4777-8777-777777777777" + const finished = "2026-08-28T12:00:00.000Z" + const [plan, commit] = records([{ ...pull(), mergedAt: finished }], { + startedAt: "2026-08-27T12:00:00.000Z", + recordedAt: "2026-08-27T12:00:00.000Z", + segment: { id: billingId, attribution: "unknown", reason: "unresolved-reference" }, + }) + const rows: WorkRecord[] = [ + { ...plan, workId: planning }, + { + ...plan, + workId: implementing, + requestId: "88888888-8888-4888-8888-888888888888", + segment: { id: implementing, attribution: "explicit", reason: "work-command" }, + }, + { ...commit, workId: implementing }, + { + ...plan, + type: "work_link", + workId: implementing, + recordedAt: at, + linkId: "99999999-9999-4999-8999-999999999999", + revision: 2, + status, + sourceWorkId: planning, + targetWorkId: implementing, + requestIds: [requestId], + evidence: { source: "work-command", segmentId: billingId }, + }, + ] + const content = build(rows).snapshots[0].content + expect(content.requests.find((request) => request.requestId === requestId)).toHaveProperty("correction", { + id: "99999999-9999-4999-8999-999999999999", + revision: 2, + recordedAt: at, + source: "work-command", + }) + expect(content.pullRequests.map((pr) => pr.id)).toEqual(["101"]) + expect(content.windowedPullRequestIds).toBeUndefined() + }) + it("does not send a correction when copies disagree about whether the same revision was revoked", () => { + const rows = records().map((row) => ({ ...row, workId: requestId })) + const link: WorkRecord = { + ...rows[0], + type: "work_link", + linkId: billingId, + revision: 2, + sourceWorkId: requestId, + targetWorkId: requestId, + requestIds: [requestId], + evidence: { source: "work-command" }, + } + const result = build([...rows, { ...link, status: "active" }, { ...link, status: "revoked" }]) + expect(result.snapshots[0].content.requests[0].allocation.kind).toBe("unknown") + expect(result.snapshots[0].content.requests[0].correction).toBeUndefined() + }) + it.each([ + ["an explicit segment", { id: billingId, attribution: "explicit", reason: "work-command" }, false, "session"], + ["a /work link", { id: billingId, attribution: "session", reason: "test" }, true, "session"], + [ + "a model match with a /work link", + { id: billingId, attribution: "inferred", reason: "model-same" }, + true, + "model", + ], + ])("never reports an inferred PR assignment carrying %s as confirmed", (_case, segment, linked, method) => { + const rows: WorkRecord[] = records([pull()], { segment }).map((row) => ({ ...row, workId: requestId })) + if (linked) + rows.push({ + ...rows[0], + type: "work_link", + linkId: "55555555-5555-4555-8555-555555555555", + revision: 1, + status: "active", + sourceWorkId: requestId, + targetWorkId: requestId, + requestIds: [requestId], + evidence: { source: "work-command", requestId, segmentId: billingId }, + }) + const report = calculatePullRequestCosts(rows, [{ requestId, billingRecordId: billingId, costUsd: "1", account }]) + // Narrowing a request to one PR can leave it inferred despite such evidence; it must not reach the server as confirmed. + report.requests[0].allocation = "inferred" + const [request] = buildSnapshots(rows, report, new Map()).snapshots[0].content.requests + expect(request.allocation).toEqual({ kind: "pull-request", pullRequestIds: ["101"], method }) + if (linked) expect(request.correction).toMatchObject({ source: "work-command" }) + }) + // The server counts only native, explicit and user-correction methods as confirmed (explicit) PR spend. + const confirmedMethods = ["native", "explicit", "user-correction"] + it.each([ + "session", + "explicit", + "inferred", + ] as const)("reports a %s request with the confidence it has in the local PR total", (attribution) => { + const rows = records([pull()], { segment: { id: "segment", attribution, reason: "test" } }) + const report = calculatePullRequestCosts(rows, [{ requestId, billingRecordId: billingId, costUsd: "1", account }]) + const confirmedLocally = report.requests + .filter((request) => request.allocation === "pull-request") + .map((request) => request.requestId) + const confirmedOnServer = buildSnapshots(rows, report, new Map()) + .snapshots[0].content.requests.filter( + (request) => request.allocation.kind === "pull-request" && confirmedMethods.includes(request.allocation.method), + ) + .map((request) => request.requestId) + expect(confirmedOnServer).toEqual(confirmedLocally) + }) +}) + +describe("conflicting PR metadata", () => { + it("holds only the repository whose provider metadata conflicts", () => { + const other = "55555555-5555-4555-8555-555555555555" + const healthy = records([pull(2, "example/other", "43")], { requestId: other }).map((row) => ({ + ...row, + workId: "other-work", + })) + const rows = [...records(), ...healthy] + const report = calculatePullRequestCosts(rows, []) + const bad = report.pullRequests.find((row) => row.pullRequest?.id === "101") + const request = report.requests.find((row) => row.requestId === requestId) + if (!bad?.pullRequest || !request) throw new Error("Fixture has no first PR") + report.pullRequests.push({ ...bad, key: "contradictory", pullRequest: { ...bad.pullRequest, number: 9 } }) + request.pullRequestIds.push("contradictory") + const result = buildSnapshots(rows, report, new Map()) + expect(result.incomplete).toBe(true) + expect(result.snapshots.map((snapshot) => snapshot.content.repository.id)).toEqual(["43"]) + }) + it("keeps reporting when a PR has links from before and after a repository rename", () => { + const beforeRename = { ...pull(1), state: "open", mergedAt: null, checkedAt: "2026-10-04T12:30:00.000Z" } + const afterRename = { ...pull(1, "example/renamed"), checkedAt: "2026-10-04T13:30:00.000Z" } + const otherRequest = "55555555-5555-4555-8555-555555555555" + const [request, commit] = records() + const rows: WorkRecord[] = [ + request, + { ...commit, pullRequests: [beforeRename, afterRename] }, + { + version: 1, + type: "request", + workId: "other-work", + sessionId: "other-session", + requestId: otherRequest, + startedAt: at, + recordedAt: at, + scope: { account, repository: "/private/other/.git" }, + }, + { + version: 1, + type: "commit", + workId: "other-work", + sessionId: "other-session", + sha: "b".repeat(40), + repository: "/private/other/.git", + worktree: "/private/other", + recordedAt: at, + pullRequests: [pull(2, "example/other", "43")], + }, + ] + const report = calculatePullRequestCosts(rows, []) + let snapshots: ReturnType["snapshots"] = [] + expect(() => { + snapshots = buildSnapshots(rows, report, new Map()).snapshots + }).not.toThrow() + expect(snapshots.map((snapshot) => snapshot.content.repository.id).sort()).toEqual(["42", "43"]) + }) +}) + +describe("per-request upload window and evidence labels", () => { + const work = "55555555-5555-4555-8555-555555555551" + const planned = "33333333-3333-4333-8333-333333333331" + const later = "33333333-3333-4333-8333-333333333332" + const segment = "66666666-6666-4666-8666-666666666661" + const scope = { account, repository: "/private/repo/.git" } + const merged = (id: string, mergedAt: string | null, number = 1): WorkPullRequest => ({ + ...pull(number), + id, + state: mergedAt ? "merged" : "open", + mergedAt, + checkedAt: mergedAt ?? at, + }) + const request = (requestId: string, startedAt: string, attribution = "session", reason = "test"): WorkRecord => ({ + version: 1, + type: "request", + workId: work, + sessionId: "session", + requestId, + startedAt, + recordedAt: startedAt, + scope, + segment: { id: segment, attribution, reason }, + }) + const commit = (pullRequests: WorkPullRequest[], sha = "b"): WorkRecord => ({ + version: 1, + type: "commit", + workId: work, + sessionId: "session", + sha: sha.repeat(40), + repository: "/private/repo/.git", + worktree: "/private/repo", + recordedAt: "2026-07-01T00:00:00.000Z", + pullRequests, + }) + const correction = (recordedAt: string): WorkRecord => ({ + version: 1, + type: "work_link", + workId: work, + sessionId: "session", + recordedAt, + linkId: "77777777-7777-4777-8777-777777777771", + revision: 1, + status: "active", + sourceWorkId: work, + targetWorkId: work, + requestIds: [planned], + scope, + evidence: { source: "work-command", segmentId: segment }, + }) + const bill = (requestId: string, n: number, billed = account) => ({ + requestId, + billingRecordId: `44444444-4444-4444-8444-44444444444${n}`, + costUsd: "1", + account: billed, + }) + const snapshot = (rows: WorkRecord[], bills: ReturnType[]) => + buildSnapshots(rows, calculatePullRequestCosts(rows, bills), new Map()) + + it("keeps an old merged PR's earlier claims when a recent post-merge request names it", () => { + // The server replaces a listed PR's whole inventory, so the pre-merge request must come along. + const rows = [ + request(planned, "2026-08-01T00:00:00.000Z"), + commit([merged("101", "2026-08-02T00:00:00.000Z")]), + request(later, "2026-10-06T00:00:00.000Z"), + ] + const { content } = snapshot(rows, [bill(planned, 1), bill(later, 2)]).snapshots[0] + expect(content.requests.map((row) => [row.requestId, row.allocation.kind])).toEqual([ + [planned, "pull-request"], + [later, "post-merge"], + ]) + expect(content.pullRequests.map((pr) => pr.id)).toEqual(["101"]) + expect(content.windowedPullRequestIds).toBeUndefined() + }) + it("sends a recent request without a PR the server settled past its detail window", () => { + const rows = [ + request(planned, "2026-06-30T00:00:00.000Z"), + commit([merged("101", "2026-07-01T00:00:00.000Z")]), + request(later, "2026-10-06T00:00:00.000Z"), + ] + const { content } = snapshot(rows, [bill(planned, 1), bill(later, 2)]).snapshots[0] + expect(content.requests.map((row) => [row.requestId, row.allocation])).toEqual([ + [later, { kind: "unknown", pullRequestIds: [], method: "session" }], + ]) + expect(content.pullRequests).toEqual([]) + expect(content.windowedPullRequestIds).toBeUndefined() + }) + it.each([ + ["past day 90", "2026-07-08T12:00:00.000Z"], + ["within the server's last five minutes", "2026-07-09T12:03:00.000Z"], + ])("never sends a correction receipt %s", (_case, mergedAt) => { + const rows = [ + request(planned, "2026-07-08T11:00:00.000Z"), + commit([merged("101", mergedAt)]), + correction("2026-10-01T12:00:00.000Z"), + request(later, "2026-10-06T12:00:00.000Z"), + ] + const requests = snapshot(rows, [bill(planned, 1), bill(later, 2)]).snapshots.flatMap( + (item) => item.content.requests, + ) + expect(requests.every((row) => row.correction === undefined)).toBe(true) + }) + it.each([ + ["a model match whose bill has not arrived yet", false, "2026-10-06T10:00:00.000Z", "inferred", "model"], + ["a model match that started after the merge", true, "2026-10-06T14:00:00.000Z", "inferred", "model"], + ["an unresolved input", true, "2026-10-06T10:00:00.000Z", "unknown", "session"], + ])("labels %s without claiming recorded evidence", (_case, priced, startedAt, attribution, method) => { + const rows = [request(planned, startedAt, attribution), commit([merged("101", "2026-10-06T12:00:00.000Z")])] + const { content } = snapshot(rows, priced ? [bill(planned, 1)] : []).snapshots[0] + expect(content.requests[0].allocation).toMatchObject({ pullRequestIds: ["101"], method }) + }) + it("keeps repository history complete beside a question asked outside Git", () => { + const rows = [ + request(planned, "2026-10-06T10:00:00.000Z"), + commit([merged("101", "2026-10-06T12:00:00.000Z")]), + { + version: 1, + type: "request", + workId: "55555555-5555-4555-8555-555555555559", + sessionId: "home-directory", + requestId: later, + startedAt: "2026-10-07T09:00:00.000Z", + recordedAt: "2026-10-07T09:00:00.000Z", + segment: { id: segment, attribution: "session", reason: "matching-disabled" }, + } satisfies WorkRecord, + ] + const built = snapshot(rows, [bill(planned, 1), bill(later, 2)]) + expect(built).toMatchObject({ incomplete: false, skippedRequests: 0 }) + expect(built.snapshots[0].content.coverage.historyComplete).toBe(true) + }) + it("sends a request with more than 8 bills without IDs, so the server finds them by its tag", () => { + const rows = records() + const bills = Array.from({ length: 9 }, (_, index) => ({ + requestId, + billingRecordId: `00000000-0000-4000-8000-${String(index + 1).padStart(12, "0")}`, + costUsd: "0.100000000", + account, + })) + const { content } = buildSnapshots(rows, calculatePullRequestCosts(rows, bills), new Map()).snapshots[0] + expect(content.requests.map((request) => request.billingRecordIds)).toEqual([[]]) + expect(() => + validateSnapshot({ schemaVersion: 1, producerId: requestId, revision: "1", generatedAt: at, ...content }), + ).not.toThrow() + }) + it("never sends another account's billing IDs for a post-merge request", () => { + const other = { ...account, organizationId: "99999999-9999-4999-8999-999999999999" } + const rows = [ + request(planned, "2026-10-06T10:00:00.000Z"), + request(later, "2026-10-06T14:00:00.000Z"), + commit([merged("101", "2026-10-06T12:00:00.000Z")]), + ] + const { content } = snapshot(rows, [bill(planned, 1), bill(later, 3, other)]).snapshots[0] + expect(content.requests.find((row) => row.requestId === later)).toMatchObject({ + allocation: { kind: "post-merge" }, + billingRecordIds: [], + }) + }) +}) + +describe("repositories without a provider identity", () => { + const scratch = "/src/scratch/.git" + const scratchRequestId = "55555555-5555-4555-8555-555555555555" + // One request in a local repository without a GitHub or GitLab remote, beside a reported PR repository. + const scan = (startedAt: string, identities: [string, RepositoryIdentity][]) => { + const rows: WorkRecord[] = [ + ...records(), + { + version: 1, + type: "request", + workId: "scratch-work", + sessionId: "scratch-session", + requestId: scratchRequestId, + startedAt, + recordedAt: startedAt, + scope: { account, repository: scratch }, + }, + ] + return buildSnapshots(rows, calculatePullRequestCosts(rows, []), new Map(identities)) + } + + it("does not let an out-of-window request there mark other repositories incomplete", () => { + const built = scan("2026-03-20T00:00:00.000Z", []) + expect(built).toMatchObject({ incomplete: false, skippedRequests: 0 }) + expect(built.snapshots.map(({ content }) => [content.repository.id, content.coverage.historyComplete])).toEqual([ + ["42", true], + ]) + }) + it("does not let a recent request in an unsupported repository mark them incomplete either", () => { + const built = scan("2026-10-06T12:00:00.000Z", [[scratch, "unsupported"]]) + expect(built).toMatchObject({ incomplete: false, skippedRequests: 0 }) + expect(built.snapshots[0].content.coverage.historyComplete).toBe(true) + expect(JSON.stringify(built.snapshots)).not.toContain(scratchRequestId) + }) + it("still marks history incomplete while a recent request's repository is not identified yet", () => { + const built = scan("2026-10-06T12:00:00.000Z", []) + expect(built).toMatchObject({ incomplete: true, skippedRequests: 1 }) + expect(built.snapshots[0].content.coverage.historyComplete).toBe(false) + }) +}) + +describe("trimming a snapshot to the upload limits", () => { + const id = (n: number) => `55555555-5555-4555-8555-${String(n).padStart(12, "0")}` + const pr = (number: number, mergedAt?: string) => ({ + id: String(200 + number), + number, + url: `https://github.com/example/repo/pull/${number}`, + ...(mergedAt ? { state: "merged" as const, mergedAt } : { state: "open" as const }), + }) + const request = (n: number, startedAt: string, pullRequestIds: string[], correction = false) => ({ + requestId: id(n), + billingRecordIds: [], + startedAt, + ...(correction + ? { correction: { id: billingId, revision: 1, recordedAt: at, source: "work-command" as const } } + : {}), + allocation: { + kind: pullRequestIds.length ? ("pull-request" as const) : ("unlinked" as const), + pullRequestIds, + method: "native" as const, + }, + }) + const content = (): SnapshotContent => ({ + repository: { provider: "github", host: "github.com", id: "42" }, + pullRequests: [pr(1), pr(2, "2026-10-01T00:00:00.000Z"), pr(3, "2026-10-03T00:00:00.000Z")], + requests: [ + request(1, "2026-10-05T10:00:00.000Z", []), + request(2, "2026-10-05T09:00:00.000Z", []), + request(3, "2026-09-30T00:00:00.000Z", ["202"]), + request(4, "2026-10-02T00:00:00.000Z", ["203"]), + request(5, "2026-09-01T00:00:00.000Z", ["201"]), + request(6, "2026-08-01T00:00:00.000Z", [], true), + ], + coverage: { observedRequests: 6, unpricedRequests: 6, historyComplete: true }, + }) + const trim = ( + limits: Partial, + unpriced: (row: SnapshotContent["requests"][number]) => boolean = () => true, + ) => fitSnapshot(content(), { ...SNAPSHOT_LIMITS, ...limits }, unpriced) + + it("leaves a snapshot within the limits untouched", () => { + const original = content() + expect(fitSnapshot(original, SNAPSHOT_LIMITS, () => true)).toEqual({ content: original, trimmed: 0 }) + }) + it.each([ + [5, [1, 3, 4, 5, 6], ["201", "202", "203"]], + [4, [3, 4, 5, 6], ["201", "202", "203"]], + [3, [4, 5, 6], ["201", "203"]], + [2, [5, 6], ["201"]], + [1, [6], []], + ])("keeps %s requests: unlinked work first, then the earliest finished PR, open PRs last, corrections never", (requests, ids, pulls) => { + const result = trim({ requests }) + expect(result?.content.requests.map((row) => row.requestId)).toEqual(ids.map(id)) + expect(result?.content.pullRequests.map((row) => row.id)).toEqual(pulls) + expect(result?.content.coverage).toEqual({ + observedRequests: ids.length, + unpricedRequests: ids.length, + historyComplete: false, + trimmedRequests: 6 - ids.length, + }) + expect(result?.trimmed).toBe(6 - ids.length) + expect(result?.content.windowedPullRequestIds).toBeUndefined() + if (result) + expect(() => + validateSnapshot({ + schemaVersion: 1, + producerId: requestId, + revision: "2", + generatedAt: at, + ...result.content, + }), + ).not.toThrow() + }) + it("holds instead of dropping an explicit correction", () => { + expect(trim({ requests: 0 })).toBeUndefined() + }) + it("meets the PR limit by removing whole finished PRs before open ones, without touching unrelated requests", () => { + const result = trim({ pullRequests: 1 }) + expect(result?.content.pullRequests.map((row) => row.id)).toEqual(["201"]) + expect(result?.content.requests.map((row) => row.requestId)).toEqual([1, 2, 5, 6].map(id)) + expect(result?.trimmed).toBe(2) + }) + it("keeps a PR retained only for a revocation receipt", () => { + const value = content() + value.pullRequests.push(pr(4, "2026-09-02T00:00:00.000Z")) + const result = fitSnapshot(value, { ...SNAPSHOT_LIMITS, pullRequests: 3 }, () => true) + expect(result?.content.pullRequests.map((row) => row.id)).toEqual(["201", "203", "204"]) + }) + it("meets the byte limit, counting the longest envelope a later revision can have", () => { + const bytes = wireBytes(content()) - 1 + const result = trim({ bytes }) + expect(result?.trimmed).toBe(1) + expect(result?.content.requests.map((row) => row.requestId)).not.toContain(id(2)) + expect(wireBytes(result?.content ?? content())).toBeLessThanOrEqual(bytes) + expect(trim({ bytes: 100 })).toBeUndefined() + }) + it("recounts missing prices among the kept requests", () => { + expect(trim({ requests: 2 }, (row) => row.requestId === id(5))?.content.coverage.unpricedRequests).toBe(1) + }) + it("applies a smaller limit the server taught while keeping the full inventory for membership checks", () => { + const later = "55555555-5555-4555-8555-555555555555" + const rows = records() + rows.push({ + ...rows[0], + requestId: later, + startedAt: "2026-10-04T12:30:00.000Z", + recordedAt: "2026-10-04T12:30:00.000Z", + }) + const key = `${accountKey(account)}:${repositoryKey({ provider: "github", host: "github.com", id: "42" })}` + const [snapshot] = buildSnapshots( + rows, + calculatePullRequestCosts(rows, []), + new Map(), + new Map(), + new Map([[key, { ...SNAPSHOT_LIMITS, requests: 1 }]]), + ).snapshots + expect(snapshot.content.requests.map((row) => row.requestId)).toEqual([later]) + expect(snapshot.content.pullRequests.map((row) => row.id)).toEqual(["101"]) + expect(snapshot.content.coverage).toMatchObject({ observedRequests: 1, historyComplete: false, trimmedRequests: 1 }) + expect(snapshot.observedRequestIds).toEqual([later, requestId].sort()) + }) +}) diff --git a/src/extensions/pr-cost-reporting/snapshot.ts b/src/extensions/pr-cost-reporting/snapshot.ts new file mode 100644 index 000000000..c6740c8d2 --- /dev/null +++ b/src/extensions/pr-cost-reporting/snapshot.ts @@ -0,0 +1,792 @@ +import { isWorkId } from "../../shared/work-id.js" +import { plainURL } from "../../utils/url.js" +import { providerId } from "../pull-request-status/provider-records.js" +import { type PullRequestCostReport, type RequestCostAllocation, time } from "../work-attribution/costs.js" +import { requestWorkLinks } from "../work-attribution/links.js" +import { isWorkScope, sameWorkAccount, type WorkAccount } from "../work-attribution/scope.js" +import { object, type WorkRecord } from "../work-attribution/summary.js" + +export interface ReportingRepository { + provider: "github" | "gitlab" + host: string + id: string + name?: string +} + +/** A local repository's provider identity, or `unsupported` when it has none (no GitHub or GitLab remote, or gone). */ +export type RepositoryIdentity = ReportingRepository | "unsupported" + +interface CorrectionReceipt { + id: string + revision: number + recordedAt: string + source: "work-command" | "producer-confirmation" +} + +export interface SnapshotContent { + repository: ReportingRepository + windowedPullRequestIds?: string[] + pullRequests: { + id: string + number: number + url: string + state: "open" | "closed" | "merged" + mergedAt?: string + closedAt?: string + }[] + requests: { + requestId: string + billingRecordIds: string[] + startedAt: string + correction?: CorrectionReceipt + allocation: { + kind: "pull-request" | "shared" | "unlinked" | "unmerged" | "post-merge" | "unknown" + pullRequestIds: string[] + method: "native" | "explicit" | "user-correction" | "model" | "session" + } + }[] + coverage: { + observedRequests: number + unpricedRequests: number + historyComplete: boolean + lastCostRefreshAt?: string + /** Requests left out to fit the upload limits; only present when some were. */ + trimmedRequests?: number + } +} + +type SnapshotRequest = SnapshotContent["requests"][number] + +export interface RepositorySnapshot { + account: WorkAccount + content: SnapshotContent + /** Complete local inventory before windowing. Lets the queue tell windowed-out requests from lost evidence. */ + observedRequestIds?: string[] + /** Missing provider evidence holds prior claims for this group; never enters the wire body. */ + incomplete?: boolean +} + +export interface WireSnapshot extends SnapshotContent { + schemaVersion: 1 + producerId: string + revision: string + generatedAt: string +} + +export interface SnapshotLimits { + requests: number + pullRequests: number + bytes: number +} + +/** The server's per-snapshot limits. A larger inventory is trimmed rather than held. */ +export const SNAPSHOT_LIMITS: SnapshotLimits = { requests: 32_000, pullRequests: 250, bytes: 8 * 1024 * 1024 } +const UPLOAD_WINDOW_MS = 32 * 24 * 60 * 60 * 1000 +const DETAIL_WINDOW_MS = 90 * 24 * 60 * 60 * 1000 +/** The server stops accepting corrections five minutes before a PR's total becomes final on day 90. */ +const CORRECTION_MARGIN_MS = 5 * 60_000 +const FINISHED_PR_GRACE_MS = 2 * 24 * 60 * 60 * 1000 +const MAX_BILLING_IDS = 8 +export const MAX_REVISION = 9223372036854775807n + +export function revision(value: unknown): value is string { + return typeof value === "string" && /^[1-9]\d{0,18}$/.test(value) && BigInt(value) <= MAX_REVISION +} + +export function repositoryKey(repository: ReportingRepository): string { + return JSON.stringify([repository.provider, repository.host, repository.id]) +} + +export function accountKey(account: WorkAccount): string { + return JSON.stringify([account.apiUrl, account.organizationId, account.userId]) +} + +/** Protobuf timestamps start at year 1. */ +export function validTime(value: unknown): value is string { + return typeof value === "string" && Number(value.slice(0, 4)) >= 1 && time(value) !== undefined +} + +function onlyKeys(value: object, allowed: string[]): boolean { + return Object.keys(value).every((key) => allowed.includes(key)) +} + +/** When a PR merged or closed, as the server reads it, in either the local or the wire shape. An open PR has not. */ +function finishedAt(pr: { state: string; mergedAt?: string | null; closedAt?: string | null } | null | undefined) { + const value = pr?.state === "merged" ? pr.mergedAt : pr?.state === "closed" ? pr.closedAt : undefined + return value ? Date.parse(value) : undefined +} + +/** A final boundary check also applies to pending payloads recovered from disk. */ +export function validateSnapshot(snapshot: WireSnapshot): void { + const fail = () => { + throw new Error("PR reporting snapshot is invalid or exceeds the upload limits") + } + + const { repository: repo, pullRequests, requests, coverage } = snapshot + if ( + !onlyKeys(snapshot, [ + "schemaVersion", + "producerId", + "revision", + "generatedAt", + "repository", + "pullRequests", + "requests", + "coverage", + "windowedPullRequestIds", + ]) + ) + fail() + if ( + snapshot.schemaVersion !== 1 || + !isWorkId(snapshot.producerId) || + !revision(snapshot.revision) || + !validTime(snapshot.generatedAt) || + !repo || + !["github", "gitlab"].includes(repo.provider) || + !providerId(repo.id) || + typeof repo.host !== "string" || + repo.host.length > 253 || + !Array.isArray(pullRequests) || + !Array.isArray(requests) || + requests.length > SNAPSHOT_LIMITS.requests || + pullRequests.length > SNAPSHOT_LIMITS.pullRequests || + !coverage || + coverage.observedRequests !== requests.length || + !Number.isSafeInteger(coverage.unpricedRequests) || + coverage.unpricedRequests < 0 || + coverage.unpricedRequests > requests.length || + typeof coverage.historyComplete !== "boolean" || + (coverage.lastCostRefreshAt !== undefined && !validTime(coverage.lastCostRefreshAt)) || + (coverage.trimmedRequests !== undefined && + (!Number.isSafeInteger(coverage.trimmedRequests) || + coverage.trimmedRequests < 0 || + (coverage.trimmedRequests > 0 && coverage.historyComplete))) + ) + fail() + const host = plainURL(`https://${repo.host}`) + if ( + !onlyKeys(repo, ["provider", "host", "id", "name"]) || + !onlyKeys(coverage, [ + "observedRequests", + "unpricedRequests", + "historyComplete", + "lastCostRefreshAt", + "trimmedRequests", + ]) + ) + fail() + if ( + host?.host !== repo.host || + host.pathname !== "/" || + (repo.name !== undefined && (typeof repo.name !== "string" || repo.name.length > 256)) + ) + fail() + const pulls = new Set() + for (const pr of pullRequests) { + if (!onlyKeys(pr, ["id", "number", "url", "state", "mergedAt", "closedAt"])) fail() + if ( + !providerId(pr.id) || + pulls.has(pr.id) || + !Number.isSafeInteger(pr.number) || + pr.number < 1 || + pr.number > 2147483647 || + !["open", "closed", "merged"].includes(pr.state) || + typeof pr.url !== "string" || + pr.url.length > 2048 || + (pr.state === "merged") !== (pr.mergedAt !== undefined) || + (pr.mergedAt !== undefined && !validTime(pr.mergedAt)) || + (pr.closedAt !== undefined && (pr.state !== "closed" || !validTime(pr.closedAt))) + ) + fail() + if (plainURL(pr.url)?.host !== repo.host) fail() + pulls.add(pr.id) + } + + const windowed = snapshot.windowedPullRequestIds + if ( + windowed !== undefined && + (!Array.isArray(windowed) || + windowed.length > 2000 || + new Set(windowed).size !== windowed.length || + windowed.some((id) => !providerId(id) || pulls.has(id))) + ) + fail() + const seen = new Set() + for (const request of requests) { + const allocation = request.allocation + if ( + !onlyKeys(request, ["requestId", "billingRecordIds", "startedAt", "allocation", "correction"]) || + !allocation || + !onlyKeys(allocation, ["kind", "pullRequestIds", "method"]) + ) + fail() + if ( + !isWorkId(request.requestId) || + seen.has(request.requestId) || + !validTime(request.startedAt) || + !Array.isArray(request.billingRecordIds) || + request.billingRecordIds.length > MAX_BILLING_IDS || + request.billingRecordIds.some((id) => !isWorkId(id)) || + new Set(request.billingRecordIds).size !== request.billingRecordIds.length || + !allocation || + !["native", "explicit", "user-correction", "model", "session"].includes(allocation.method) || + !Array.isArray(allocation.pullRequestIds) || + new Set(allocation.pullRequestIds).size !== allocation.pullRequestIds.length || + allocation.pullRequestIds.some((id) => !pulls.has(id)) + ) + fail() + const correction = request.correction + if ( + correction !== undefined && + (!correction || + !onlyKeys(correction, ["id", "revision", "recordedAt", "source"]) || + !isWorkId(correction.id) || + !Number.isSafeInteger(correction.revision) || + correction.revision < 1 || + correction.revision > 2147483647 || + !validTime(correction.recordedAt) || + !["work-command", "producer-confirmation"].includes(correction.source)) + ) + fail() + const count = allocation.pullRequestIds.length + if ( + ["pull-request", "unmerged", "post-merge"].includes(allocation.kind) + ? count !== 1 + : allocation.kind === "shared" + ? count < 2 + : allocation.kind === "unlinked" + ? count !== 0 + : allocation.kind !== "unknown" + ) + fail() + seen.add(request.requestId) + } + if (Buffer.byteLength(JSON.stringify(snapshot)) > SNAPSHOT_LIMITS.bytes) fail() +} + +/** Serialized size with the longest envelope, so a later revision cannot cross the byte limit. */ +export function wireBytes(content: SnapshotContent): number { + return Buffer.byteLength( + JSON.stringify({ + schemaVersion: 1, + producerId: "00000000-0000-4000-8000-000000000000", + revision: String(MAX_REVISION), + generatedAt: new Date(0).toISOString(), + ...content, + }), + ) +} + +/** + * Leaves out the requests the server can best do without until the snapshot fits: unlinked and + * candidate-less unknown requests first, then requests of finished PRs (earliest finish first), and + * requests of open PRs only after everything else. Correction receipts are never left out. A PR left + * without requests is removed rather than named as windowed, because the server would restore its + * old claims into the same snapshot. Returns undefined when the kept requests alone exceed the limits. + */ +export function fitSnapshot( + content: SnapshotContent, + limits: SnapshotLimits, + unpriced: (request: SnapshotRequest) => boolean, +): { content: SnapshotContent; trimmed: number } | undefined { + if ( + content.requests.length <= limits.requests && + content.pullRequests.length <= limits.pullRequests && + wireBytes(content) <= limits.bytes + ) + return { content, trimmed: 0 } + const size = (value: unknown) => Buffer.byteLength(JSON.stringify(value)) + const total = content.requests.length + // Everything but the two lists; the final counts never have more digits than these. + const base = wireBytes({ + ...content, + pullRequests: [], + requests: [], + coverage: { + ...content.coverage, + observedRequests: total, + unpricedRequests: total, + historyComplete: false, + trimmedRequests: total, + }, + }) + + const finishTimes = new Map(content.pullRequests.map((pr) => [pr.id, finishedAt(pr)])) + const pullSize = new Map(content.pullRequests.map((pr) => [pr.id, size(pr)])) + const requestSize = new Map(content.requests.map((request) => [request.requestId, size(request)])) + const references = new Map() + const byPull = new Map() + const locked = new Set() + for (const request of content.requests) + for (const id of request.allocation.pullRequestIds) { + references.set(id, (references.get(id) ?? 0) + 1) + const list = byPull.get(id) + if (list) list.push(request) + else byPull.set(id, [request]) + if (request.correction) locked.add(id) + } + let requestCount = total + let requestBytes = [...requestSize.values()].reduce((sum, value) => sum + value, 0) + let pullCount = content.pullRequests.length + let pullBytes = [...pullSize.values()].reduce((sum, value) => sum + value, 0) + const bytes = () => base + requestBytes + Math.max(0, requestCount - 1) + pullBytes + Math.max(0, pullCount - 1) + const dropped = new Set() + const removed = new Set() + const drop = (request: SnapshotRequest) => { + if (dropped.has(request.requestId)) return + dropped.add(request.requestId) + requestCount-- + requestBytes -= requestSize.get(request.requestId) ?? 0 + for (const id of request.allocation.pullRequestIds) { + const left = (references.get(id) ?? 0) - 1 + references.set(id, left) + if (left) continue + removed.add(id) + pullCount-- + pullBytes -= pullSize.get(id) ?? 0 + } + } + // Only whole PRs satisfy the PR limit. Removing them first also frees requests and bytes. + if (pullCount > limits.pullRequests) { + const latest = new Map( + [...byPull].map(([id, list]) => [ + id, + list.reduce((max, request) => Math.max(max, Date.parse(request.startedAt)), 0), + ]), + ) + + // Finished PRs by finish time, then open PRs with the oldest latest activity. A PR without + // requests stays: buildSnapshots keeps one only for a revocation receipt. + const removable = content.pullRequests + .filter((pr) => !locked.has(pr.id) && byPull.has(pr.id)) + .map((pr) => ({ id: pr.id, finished: finishTimes.get(pr.id) })) + .sort( + (a, b) => + Number(a.finished === undefined) - Number(b.finished === undefined) || + (a.finished ?? latest.get(a.id) ?? 0) - (b.finished ?? latest.get(b.id) ?? 0) || + a.id.localeCompare(b.id), + ) + for (const { id } of removable) { + if (pullCount <= limits.pullRequests) break + for (const request of byPull.get(id) ?? []) drop(request) + } + } + + const rank = new Map() + for (const request of content.requests) { + const ids = request.allocation.pullRequestIds + const started = Date.parse(request.startedAt) + const finished = ids.map((id) => finishTimes.get(id)) + rank.set( + request.requestId, + !ids.length + ? [0, 0, started] + : finished.some((value) => value === undefined) + ? [2, 0, started] + : [1, Math.max(...finished.map(Number)), started], + ) + } + + const order = content.requests + .filter((request) => !request.correction && !dropped.has(request.requestId)) + .sort((a, b) => { + const [x, y] = [rank.get(a.requestId) ?? [0, 0, 0], rank.get(b.requestId) ?? [0, 0, 0]] + return x[0] - y[0] || x[1] - y[1] || x[2] - y[2] || a.requestId.localeCompare(b.requestId) + }) + for (const request of order) { + if (requestCount <= limits.requests && bytes() <= limits.bytes) break + drop(request) + } + if (requestCount > limits.requests || pullCount > limits.pullRequests || bytes() > limits.bytes) return undefined + const requests = content.requests.filter((request) => !dropped.has(request.requestId)) + return { + trimmed: dropped.size, + content: { + ...content, + pullRequests: content.pullRequests.filter((pr) => !removed.has(pr.id)), + requests, + coverage: { + ...content.coverage, + observedRequests: requests.length, + unpricedRequests: requests.filter(unpriced).length, + historyComplete: false, + ...(requests.length ? {} : { lastCostRefreshAt: undefined }), + trimmedRequests: dropped.size, + }, + }, + } +} + +function method( + request: RequestCostAllocation, + links: Map, +): SnapshotContent["requests"][number]["allocation"]["method"] { + // An inferred local assignment stays inferred on the server, whatever evidence its segment or a link carries. + if (request.allocation === "inferred") return request.segment?.attribution === "inferred" ? "model" : "session" + if (request.linkIds?.length) + return request.linkIds.some((id) => { + const evidence = links.get(id)?.evidence + return object(evidence) && evidence.source === "work-command" + }) + ? "user-correction" + : "explicit" + if (request.segment?.attribution === "explicit") return "explicit" + // Only a confirmed PR allocation rests on recorded evidence; anything else is a model guess or session grouping. + if (request.allocation === "pull-request") return "native" + return request.segment?.attribution === "inferred" ? "model" : "session" +} + +/** Construct fields individually: local identities, paths, prompts and prices never cross this boundary. */ +export function buildSnapshots( + records: WorkRecord[], + report: PullRequestCostReport, + /** Identities by Git directory; a directory left out has not been looked up successfully yet. */ + repositories: Map, + costRefreshes = new Map(), + /** Stricter limits a repository's server rejection taught, by account and repository key. */ + limits = new Map(), +): { snapshots: RepositorySnapshot[]; incomplete: boolean; skippedRequests: number } { + const groups = new Map() + let incomplete = false + let skippedRequests = 0 + const original = new Map() + const links = new Map() + const conflictingLinkStatus = new Set() + const conflicts = new Set() + const cutoff = Date.now() - UPLOAD_WINDOW_MS + // Works with repository evidence. Questions asked outside Git have neither scope nor commits. + const repositoryWorks = new Set() + for (const row of records) { + if ((row.type === "request" && isWorkScope(row.scope)) || row.type === "commit") + repositoryWorks.add(String(row.workId)) + if (row.type === "request" && typeof row.requestId === "string") + original.set(row.requestId, [...(original.get(row.requestId) ?? []), row]) + if (row.type === "work_link" && typeof row.linkId === "string" && typeof row.revision === "number") { + const previous = links.get(row.linkId) + if (!previous || Number(previous.revision) < row.revision) { + links.set(row.linkId, row) + conflictingLinkStatus.delete(row.linkId) + } else if (previous.revision === row.revision && previous.status !== row.status) { + conflictingLinkStatus.add(row.linkId) + } + } + } + + // Reuse the native proof validator. Revocations are explicit user decisions; + // validate their original scope and evidence before sending the receipt. + const verified = requestWorkLinks( + records.map((row) => (row.type === "work_link" && row.status === "revoked" ? { ...row, status: "active" } : row)), + ) + + const byRequest = new Map(report.requests.map((request) => [request.requestId, request])) + const corrections = new Map() + for (const request of report.requests) { + const proof = verified.get(request.requestId) + if (!proof || proof.unresolved || [...proof.linkIds].some((id) => conflictingLinkStatus.has(id))) continue + const rows = [...proof.linkIds] + .flatMap((id) => { + const row = links.get(id) + return row ? [row] : [] + }) + .filter((row) => isWorkId(row.linkId) && Number.isSafeInteger(row.revision) && validTime(row.recordedAt)) + .sort( + (a, b) => + Date.parse(String(b.recordedAt)) - Date.parse(String(a.recordedAt)) || + Number(b.revision) - Number(a.revision), + ) + const row = rows[0] + if (!row || !validTime(row.recordedAt)) continue + // A correction for PRs whose details the server is about to delete can no longer apply. + const account = request.account + const candidates = report.pullRequests.filter( + (pr) => account && pr.account && sameWorkAccount(pr.account, account) && request.pullRequestIds.includes(pr.key), + ) + if ( + candidates.length && + candidates.every((pr) => { + const finished = finishedAt(pr.pullRequest) + return finished !== undefined && Date.now() >= finished + DETAIL_WINDOW_MS - CORRECTION_MARGIN_MS + }) + ) + continue + const evidence = row.evidence + const manual = row.status === "revoked" || (object(evidence) && evidence.source === "work-command") + corrections.set(request.requestId, { + id: String(row.linkId), + revision: Number(row.revision), + recordedAt: row.recordedAt, + source: manual ? "work-command" : "producer-confirmation", + }) + } + + // Send recent attempts, attempts of open or recently finished PRs and corrections the server can still apply. + // Older finished history stays on the server through exact windowed PR IDs, so long-lived works stay small. + const recentPulls = new Set( + report.pullRequests + .filter((pr) => { + const finished = finishedAt(pr.pullRequest) + return finished === undefined || finished >= cutoff - FINISHED_PR_GRACE_MS + }) + .map((pr) => pr.key), + ) + + // The server keeps a finished PR's details until day 90; after that only its frozen total remains. + const detailed = new Set( + report.pullRequests + .filter((pr) => { + const finished = finishedAt(pr.pullRequest) + return finished === undefined || Date.now() < finished + DETAIL_WINDOW_MS + }) + .map((pr) => pr.key), + ) + + const current = (request: RequestCostAllocation) => + !request.startedAt || + Date.parse(request.startedAt) >= cutoff || + request.pullRequestIds.some((key) => recentPulls.has(key)) + const included = new Set( + report.requests + .filter((request) => { + const account = request.account + const receipt = corrections.get(request.requestId) + if (!account) return false + if (current(request)) return true + return ( + receipt !== undefined && + report.pullRequests.some((pr) => { + if (!pr.account || !sameWorkAccount(pr.account, account) || !request.pullRequestIds.includes(pr.key)) + return false + const finished = finishedAt(pr.pullRequest) + return ( + finished !== undefined && + Date.now() < finished + DETAIL_WINDOW_MS - CORRECTION_MARGIN_MS && + (Date.parse(receipt.recordedAt) >= cutoff || + Date.parse(receipt.recordedAt) >= finished + UPLOAD_WINDOW_MS) + ) + }) + ) + }) + .map((request) => request.requestId), + ) + + // The server replaces a listed PR's whole inventory. When an included request, such as a post-merge one, names a + // finished PR outside the window, send every request of that PR too, or its earlier claims would be withdrawn. + const requestsByPull = new Map() + for (const request of report.requests) + for (const key of request.pullRequestIds) { + const requests = requestsByPull.get(key) + if (requests) requests.push(request) + else requestsByPull.set(key, [request]) + } + const listed = new Set() + const pending = report.requests.filter((request) => included.has(request.requestId)) + for (let index = 0; index < pending.length; index++) + for (const key of pending[index].pullRequestIds) { + if (recentPulls.has(key) || listed.has(key) || !detailed.has(key)) continue + listed.add(key) + for (const other of requestsByPull.get(key) ?? []) + if (other.account && !included.has(other.requestId)) { + included.add(other.requestId) + pending.push(other) + } + } + const unpriced = new Set( + report.requests.filter((request) => request.priceStatus !== "priced").map((request) => request.requestId), + ) + + // A verified no-charge attempt is priced at zero and has no billing records to send. + const noCharge = new Set( + report.requests + .filter((request) => request.priceStatus === "priced" && !request.billingRecordIds.length) + .map((request) => request.requestId), + ) + for (const request of report.requests) { + if (!request.account || !isWorkId(request.requestId) || !request.startedAt) { + // Unscoped attempts of repository work that would be sent leave its history incomplete. + // Questions asked outside Git and older history never count. + if ( + current(request) && + [...request.workIds, ...(request.linkedWorkIds ?? [])].some((workId) => repositoryWorks.has(workId)) + ) { + incomplete = true + skippedRequests++ + } + continue + } + + const account = request.account + const candidates = report.pullRequests.filter( + (pr) => pr.account && sameWorkAccount(pr.account, account) && request.pullRequestIds.includes(pr.key), + ) + + const destinations = new Map() + let missing = false + for (const candidate of candidates) { + const pr = candidate.pullRequest + // Missing merge metadata leaves this request unknown; its repository can still report. + if (!pr) continue + if (!pr.id || !pr.repositoryId) { + missing = true + continue + } + + const repo: ReportingRepository = { + provider: pr.provider ?? "github", + host: pr.host, + id: pr.repositoryId, + name: pr.repository, + } + destinations.set(repositoryKey(repo), repo) + } + if (!destinations.size || missing) + for (const row of original.get(request.requestId) ?? []) { + if (!isWorkScope(row.scope) || !sameWorkAccount(row.scope.account, account)) { + missing = true + continue + } + + const repo = repositories.get(row.scope.repository) + // A repository without a provider identity has no snapshot to report to; nothing is missing from others. + if (repo === "unsupported") continue + if (repo) destinations.set(repositoryKey(repo), repo) + else missing = true + } + if (!destinations.size) { + // Like unscoped attempts, only requests that would be sent leave the history incomplete. + if (missing && current(request)) { + incomplete = true + skippedRequests++ + } + continue + } + if (missing) incomplete = true + for (const [repoKey, repo] of destinations) { + const key = `${accountKey(account)}:${repoKey}` + let group = groups.get(key) + if (!group) { + group = { + account, + content: { + repository: repo, + pullRequests: [], + requests: [], + coverage: { observedRequests: 0, unpricedRequests: 0, historyComplete: true }, + }, + } + groups.set(key, group) + } + if (missing) group.incomplete = true + const pullRequestIds: string[] = [] + for (const candidate of candidates) { + const pr = candidate.pullRequest + // A request naming a PR past its detail window is sent without it: the server already settled that PR, + // and its final total no longer changes. + if ( + !pr?.id || + pr.repositoryId !== repo.id || + pr.host !== repo.host || + (pr.provider ?? "github") !== repo.provider || + !detailed.has(candidate.key) + ) + continue + const metadata = { + id: pr.id, + number: pr.number, + url: pr.url, + state: pr.state, + ...(pr.mergedAt ? { mergedAt: pr.mergedAt } : {}), + ...(pr.state === "closed" && pr.closedAt ? { closedAt: pr.closedAt } : {}), + } + + const existing = group.content.pullRequests.find((other) => other.id === pr.id) + if (existing && JSON.stringify(existing) !== JSON.stringify(metadata)) { + conflicts.add(key) + incomplete = true + } + if (!existing) group.content.pullRequests.push(metadata) + pullRequestIds.push(pr.id) + } + + let kind: SnapshotContent["requests"][number]["allocation"]["kind"] = + request.allocation === "inferred" ? (pullRequestIds.length > 1 ? "shared" : "pull-request") : request.allocation + // The backend applies each candidate's merge cutoff to a shared claim. + if (kind === "post-merge" && pullRequestIds.length > 1) kind = "shared" + if ( + missing || + destinations.size > 1 || + (kind !== "unlinked" && pullRequestIds.length !== request.pullRequestIds.length) || + (!pullRequestIds.length && kind !== "unlinked") + ) + kind = "unknown" + const billingAccountVerified = + request.reason !== "work-account-mismatch" && request.reason !== "work-account-unverified" + // The server accepts 8 bill IDs per request; it finds a larger set through the request's exact billing tag. + const bills = + billingAccountVerified && request.billingRecordIds.length <= MAX_BILLING_IDS + ? [...request.billingRecordIds] + : [] + group.content.requests.push({ + requestId: request.requestId, + billingRecordIds: bills, + startedAt: request.startedAt, + ...(corrections.has(request.requestId) ? { correction: corrections.get(request.requestId) } : {}), + allocation: { + kind, + pullRequestIds: kind === "unlinked" ? [] : [...new Set(pullRequestIds)].sort(), + method: method(request, links), + }, + }) + + const refreshed = billingAccountVerified ? costRefreshes.get(request.requestId) : undefined + if ( + refreshed && + (!group.content.coverage.lastCostRefreshAt || + Date.parse(refreshed) > Date.parse(group.content.coverage.lastCostRefreshAt)) + ) + group.content.coverage.lastCostRefreshAt = refreshed + } + } + + const isUnpriced = (request: SnapshotRequest) => + unpriced.has(request.requestId) || (!request.billingRecordIds.length && !noCharge.has(request.requestId)) + const snapshots = [...groups.entries()] + .filter(([key]) => !conflicts.has(key)) + .sort(([a], [b]) => a.localeCompare(b)) + .map(([key, group]) => { + group.observedRequestIds = group.content.requests.map((request) => request.requestId).sort() + const retained = group.content.requests.filter((request) => included.has(request.requestId)) + const pulls = new Set(retained.flatMap((request) => request.allocation.pullRequestIds)) + // Revocations have no exclusive PR claim, but retain the affected PR's metadata. + for (const request of retained) + if (request.correction) { + const source = byRequest.get(request.requestId) + for (const pr of report.pullRequests) + if (pr.pullRequest?.id && source?.pullRequestIds.includes(pr.key)) pulls.add(pr.pullRequest.id) + } + const omitted: string[] = [] + group.content.requests = retained + group.content.pullRequests = group.content.pullRequests.filter((pr) => { + const finished = finishedAt(pr) + const keep = pulls.has(pr.id) || finished === undefined || finished >= cutoff - FINISHED_PR_GRACE_MS + if (!keep && finished !== undefined && Date.now() < finished + DETAIL_WINDOW_MS) omitted.push(pr.id) + return keep + }) + if (omitted.length) group.content.windowedPullRequestIds = omitted.sort() + group.content.coverage.observedRequests = retained.length + group.content.coverage.unpricedRequests = retained.filter(isUnpriced).length + if (!retained.length) group.content.coverage.lastCostRefreshAt = undefined + group.content.requests.sort((a, b) => a.requestId.localeCompare(b.requestId)) + group.content.pullRequests.sort((a, b) => a.id.localeCompare(b.id)) + if (incomplete) group.content.coverage.historyComplete = false + // A learned limit that corrections alone exceed falls back to the normal limits; a snapshot + // that cannot fit even those stays whole and the queue holds it with a visible error. + const learned = limits.get(key) + const fitted = + (learned && fitSnapshot(group.content, learned, isUnpriced)) || + fitSnapshot(group.content, SNAPSHOT_LIMITS, isUnpriced) + if (fitted) group.content = fitted.content + return group + }) + return { snapshots, incomplete, skippedRequests } +} diff --git a/src/extensions/pr-cost-reporting/status.test.ts b/src/extensions/pr-cost-reporting/status.test.ts new file mode 100644 index 000000000..37da8c0d7 --- /dev/null +++ b/src/extensions/pr-cost-reporting/status.test.ts @@ -0,0 +1,227 @@ +import { describe, expect, it } from "vitest" +import type { PendingRepository, ReportingState } from "./queue.js" +import { accountKey, repositoryKey } from "./snapshot.js" +import { limitNoticeTexts, statusText } from "./status.js" + +const now = Date.UTC(2026, 9, 8, 12) +const account = { + apiUrl: "https://api.example", + organizationId: "11111111-1111-4111-8111-111111111111", + userId: "22222222-2222-4222-8222-222222222222", +} +function entry(id: string, fields: Partial = {}): [string, PendingRepository] { + const repository = { provider: "github" as const, host: "github.com", id, name: `example/${id}` } + return [ + `${accountKey(account)}:${repositoryKey(repository)}`, + { account, repository, revision: "1", requestHashes: [], attempts: 0, retryAt: 0, ...fields }, + ] +} +const pending = { + schemaVersion: 1 as const, + producerId: "33333333-3333-4333-8333-333333333333", + revision: "2", + generatedAt: new Date(now).toISOString(), + repository: { provider: "github" as const, host: "github.com", id: "1" }, + pullRequests: [], + requests: [], + coverage: { observedRequests: 0, unpricedRequests: 0, historyComplete: true }, +} +const state = (entries: [string, PendingRepository][], fields: Partial = {}): ReportingState => ({ + version: 1, + enabled: true, + followsTelemetry: true, + producerId: "44444444-4444-4444-8444-444444444444", + entries: Object.fromEntries(entries), + ...fields, +}) + +describe("/pr-reporting status", () => { + it("states the setting plainly and counts repositories on one line", () => { + expect(statusText(state([]), undefined, now)).toEqual({ + text: ["PR reporting Enabled (SaaS default)", "Repositories 0 queued · 0 acknowledged"].join("\n"), + warning: false, + }) + }) + it("states an explicit opt-out with the repositories acknowledged before it", () => { + const acknowledged = { lastAcknowledgedAt: new Date(now).toISOString() } + expect( + statusText( + state([entry("1", acknowledged), entry("2", acknowledged)], { enabled: false, followsTelemetry: undefined }), + undefined, + now, + ), + ).toEqual({ + text: ["PR reporting Disabled (explicit choice)", "Repositories 0 queued · 2 acknowledged"].join("\n"), + warning: false, + }) + }) + it("reports incomplete history as information, because the other requests are still reported", () => { + expect( + statusText( + state([entry("1", { pending })], { followsTelemetry: undefined, skippedRequests: 3693 }), + undefined, + now, + ), + ).toEqual({ + text: [ + "PR reporting Enabled (explicit choice)", + "Repositories 1 queued · 0 acknowledged", + "", + "History coverage incomplete", + " 3,693 requests skipped without original account or", + " repository evidence. Other requests are still reported.", + ].join("\n"), + warning: false, + }) + }) + it("leads with delivery problems, each shown once, and warns", () => { + expect( + statusText( + state( + [ + entry("1", { pending, lastError: "PR reporting returned HTTP 503" }), + entry("2", { pending, lastError: "PR reporting returned HTTP 503" }), + ], + { error: "PR reporting held 1 repository snapshot(s) because earlier evidence is missing" }, + ), + undefined, + now, + ), + ).toEqual({ + text: [ + "Delivery problems", + "- PR reporting held 1 repository snapshot(s) because", + " earlier evidence is missing", + "- PR reporting returned HTTP 503", + "", + "PR reporting Enabled (SaaS default)", + "Repositories 2 queued · 0 acknowledged", + ].join("\n"), + warning: true, + }) + }) + it("explains a session that does not upload and repositories waiting for their upload window", () => { + const status = statusText( + state([ + entry("1", { pending, uploadedAt: now - 60_000, lastAcknowledgedAt: new Date(now).toISOString() }), + entry("2", { pending, uploadedAt: now - 60_000, urgent: true }), + ]), + "CI environment (GITHUB_ACTIONS)", + now, + ) + expect(status).toEqual({ + text: [ + "PR reporting Enabled (SaaS default)", + "This session No uploads; local attribution continues", + " Reason: CI environment (GITHUB_ACTIONS)", + "Repositories 2 queued · 1 acknowledged", + " 1 waiting for the 5-minute upload window", + ].join("\n"), + warning: false, + }) + }) + it("shows account pauses, repository limits and trimmed repositories with their next attempt", () => { + const limit = { + scope: "organization" as const, + limit: "requests" as const, + current: 50000, + maximum: 50000, + at: now, + } + const text = statusText( + state( + [ + entry("1", { + pending, + retryAt: now + 6 * 3_600_000, + limit: { scope: "producer", limit: "repositories", current: 101, maximum: 100, at: now }, + }), + entry("2", { + pending, + retryAt: now + 5 * 3_600_000, + limit: { scope: "snapshot", limit: "requests", current: 12, maximum: 10, at: now }, + learned: { requests: 9, until: now + 1 }, + }), + entry("3", { trimmed: 1200 }), + ], + { paused: { [accountKey(account)]: { limit, retryAt: now + 30 * 60_000 } } }, + ), + undefined, + now, + ) + expect(text).toEqual({ + text: [ + "Delivery problems", + "- PR cost reporting paused for this organization:", + " organization limit reached (requests 50,000 of 50,000).", + " Ask an admin to free space or wait.", + " Next try in about 30 min.", + "- example/1: producer limit reached (repositories 101 of", + " 100). Not reported; next try in about 6 h.", + "- example/2: snapshot limit reached (requests 12 of 10).", + " Sending a smaller snapshot in about 5 h.", + "- example/3 is partially reported: 1,200 requests left out", + " to fit the upload limits.", + "", + "PR reporting Enabled (SaaS default)", + "Repositories 2 queued · 0 acknowledged", + ].join("\n"), + warning: true, + }) + }) + it("fits a 60-column terminal with every section shown", () => { + const { text } = statusText( + state( + [ + entry("1", { + pending, + retryAt: now + 3_600_000, + uploadedAt: now - 60_000, + limit: { scope: "contributor", limit: "windowedPullRequests", current: 1000, maximum: 1000, at: now }, + }), + entry("2", { trimmed: 31_999, lastAcknowledgedAt: new Date(now).toISOString() }), + ], + { + paused: { [accountKey(account)]: { limit: { scope: "contributor", at: now }, retryAt: now } }, + error: + "PR reporting held 2 snapshot(s) until queued reports leave room in the local queue. Some queued reports are waiting for their original account endpoint and credentials", + skippedRequests: 123_456, + }, + ), + "benchmark run (KIMCHI_INFRA_BREAKER_THRESHOLD)", + now, + ) + // The TUI pads notifications by one column on each side. + expect(text.split("\n").filter((line) => line.length > 58)).toEqual([]) + expect(text).toContain( + [ + "This session No uploads; local attribution continues", + " Reason: benchmark run", + " (KIMCHI_INFRA_BREAKER_THRESHOLD)", + ].join("\n"), + ) + }) +}) + +describe("one-time limit notices", () => { + it("names the repository, or its provider identity, and tells account pauses apart", () => { + expect( + limitNoticeTexts({ + repositories: [ + { provider: "github", host: "github.com", id: "1", name: "owner/repo" }, + { provider: "gitlab", host: "gitlab.example", id: "7" }, + ], + pauses: [ + { scope: "organization", limit: "requests", at: now }, + { scope: "organization", limit: "bytes", at: now }, + { scope: "contributor", at: now }, + ], + }), + ).toEqual([ + "PR cost reporting paused for this organization: limit reached. Ask an admin to free space or wait.", + "PR cost reporting paused for your account in this organization: limit reached. Ask an admin to free space or wait.", + "PR costs for owner/repo are partially reported: limit reached. See /pr-reporting status.", + "PR costs for gitlab.example repository 7 are partially reported: limit reached. See /pr-reporting status.", + ]) + }) +}) diff --git a/src/extensions/pr-cost-reporting/status.ts b/src/extensions/pr-cost-reporting/status.ts new file mode 100644 index 000000000..a8a1163ed --- /dev/null +++ b/src/extensions/pr-cost-reporting/status.ts @@ -0,0 +1,131 @@ +import { wrapTextWithAnsi } from "@earendil-works/pi-tui" +import { + inUploadWindow, + type LimitNotices, + type PendingRepository, + type ReportingState, + type ServerLimit, +} from "./queue.js" +import type { ReportingRepository } from "./snapshot.js" + +const name = (repository: ReportingRepository) => repository.name ?? `${repository.host} repository ${repository.id}` +const formatCount = (count: number) => count.toLocaleString("en-US") +const plural = (count: number, noun: string) => `${formatCount(count)} ${noun}${count === 1 ? "" : "s"}` + +function inAbout(ms: number): string { + if (ms <= 0) return "at the next upload" + const minutes = Math.max(1, Math.round(ms / 60_000)) + return `in about ${minutes < 60 ? `${minutes} min` : `${Math.round(minutes / 60)} h`}` +} + +function reached(limit: ServerLimit): string { + const values = + limit.current !== undefined && limit.maximum !== undefined + ? ` ${formatCount(limit.current)} of ${formatCount(limit.maximum)}` + : "" + return `${limit.scope ?? "server"} limit reached${limit.limit ? ` (${limit.limit}${values})` : ""}` +} + +/** A snapshot limit in a dimension trimming can shrink is retried with a smaller snapshot. */ +function shrinking({ limit, learned }: PendingRepository): boolean { + return ( + limit?.scope === "snapshot" && + (limit.limit === "requests" || limit.limit === "pullRequests" || limit.limit === "bytes") && + learned?.[limit.limit] !== undefined + ) +} + +const owner = (limit: ServerLimit) => + limit.scope === "contributor" ? "your account in this organization" : "this organization" +const paused = (limit: ServerLimit) => + `PR cost reporting paused for ${owner(limit)}: limit reached. Ask an admin to free space or wait.` + +/** One-time notices: an account pause, or a repository that is only partly reported. */ +export function limitNoticeTexts(notices: LimitNotices): string[] { + return [ + ...new Set(notices.pauses.map(paused)), + ...notices.repositories.map( + (repository) => + `PR costs for ${name(repository)} are partially reported: limit reached. See /pr-reporting status.`, + ), + ] +} + +/** Lines fit a 60-column TUI, which pads notifications by one column on each side. */ +const WIDTH = 58 +/** "Repositories" and a three-space gutter. */ +const LABEL_WIDTH = 15 + +/** A label and its values, each starting a line of the value column. */ +const row = (label: string, ...values: string[]) => + values + .flatMap((value) => wrapTextWithAnsi(value, WIDTH - LABEL_WIDTH)) + .map((line, index) => `${(index ? "" : label).padEnd(LABEL_WIDTH)}${line}`) +/** Section text wrapped two columns in; `first` starts its first line. */ +const wrap = (text: string, first: string) => + wrapTextWithAnsi(text, WIDTH - 2).map((line, index) => `${index ? " " : first}${line}`) + +/** + * The `/pr-reporting status` text. `skipReason` describes this session, not the saved choice. Server limits and + * delivery problems come first and make it a warning; skipped history alone is information, because the other + * requests still report. + */ +export function statusText( + state: ReportingState, + skipReason: string | undefined, + now = Date.now(), +): { text: string; warning: boolean } { + const entries = Object.values(state.entries) + const pending = entries.filter((entry) => entry.pending) + const waiting = pending.filter((entry) => inUploadWindow(entry, now)).length + const acknowledged = entries.filter((entry) => entry.lastAcknowledgedAt).length + const problems = [ + // The advice is long, so the retry time keeps a line of its own. + ...Object.values(state.paused ?? {}).map( + (pause) => + `PR cost reporting paused for ${owner(pause.limit)}: ${reached(pause.limit)}. Ask an admin to free space or wait.\nNext try ${inAbout(pause.retryAt - now)}.`, + ), + ...entries.flatMap((entry) => { + const lines: string[] = [] + if (entry.limit) + lines.push( + `${name(entry.repository)}: ${reached(entry.limit)}. ${shrinking(entry) ? "Sending a smaller snapshot" : "Not reported; next try"} ${inAbout(entry.retryAt - now)}.`, + ) + if (entry.trimmed) + lines.push( + `${name(entry.repository)} is partially reported: ${plural(entry.trimmed, "request")} left out to fit the upload limits.`, + ) + return lines + }), + ...new Set( + [state.error, ...pending.map((entry) => entry.lastError)].filter((value): value is string => value !== undefined), + ), + ] + const sections = [ + ...(problems.length ? [["Delivery problems", ...problems.flatMap((problem) => wrap(problem, "- "))]] : []), + [ + ...row( + "PR reporting", + `${state.enabled ? "Enabled" : "Disabled"} (${state.followsTelemetry ? "SaaS default" : "explicit choice"})`, + ), + ...(skipReason ? row("This session", "No uploads; local attribution continues", `Reason: ${skipReason}`) : []), + ...row( + "Repositories", + `${formatCount(pending.length)} queued · ${formatCount(acknowledged)} acknowledged`, + ...(waiting ? [`${formatCount(waiting)} waiting for the 5-minute upload window`] : []), + ), + ], + ...(state.skippedRequests + ? [ + [ + "History coverage incomplete", + ...wrap( + `${plural(state.skippedRequests, "request")} skipped without original account or repository evidence. Other requests are still reported.`, + " ", + ), + ], + ] + : []), + ] + return { text: sections.map((lines) => lines.join("\n")).join("\n\n"), warning: problems.length > 0 } +} diff --git a/src/extensions/pr-cost-reporting/worker.test.ts b/src/extensions/pr-cost-reporting/worker.test.ts new file mode 100644 index 000000000..08af8fd45 --- /dev/null +++ b/src/extensions/pr-cost-reporting/worker.test.ts @@ -0,0 +1,1054 @@ +import * as fs from "node:fs" +import { appendFile, mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises" +import { tmpdir } from "node:os" +import { join } from "node:path" +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest" +import { createContext } from "../__mocks__/context.js" +import * as providerApi from "../pull-request-status/provider-api.js" +import { LookupError } from "../pull-request-status/provider-records.js" +import * as health from "../telemetry/pr-cost.js" +import { requestTagSelector } from "../work-attribution/billing-source.js" +import { readWorkCostReport } from "../work-attribution/cost-sync.js" +import * as costs from "../work-attribution/costs.js" +import * as summary from "../work-attribution/summary.js" +import { flushWorkSummaries } from "../work-attribution/summary.js" +import { appendWorkRecord } from "../work-attribution.js" +import { queueSnapshots, readReportingState, setReportingEnabled, UPLOAD_INTERVAL_MS } from "./queue.js" +import { buildSnapshots, type RepositorySnapshot, type WireSnapshot } from "./snapshot.js" +import { statusText } from "./status.js" +import { deliverSnapshots, LIMIT_RETRY_MS, reconcileReporting, serverLimit } from "./worker.js" + +const config = vi.hoisted(() => ({ key: "test-key", endpoint: "https://api.example" })) +vi.mock("node:fs", async (original) => ({ ...(await original()) })) +vi.mock("../work-attribution/costs.js", async (original) => ({ ...(await original()) })) +vi.mock("../work-attribution/summary.js", async (original) => ({ ...(await original()) })) +vi.mock("../../config.js", () => ({ + loadConfig: () => ({ apiKey: config.key }), + resolveEndpoints: () => ({ platformApiUrl: config.endpoint }), + readTelemetryConfig: () => ({ enabled: true }), +})) +const org = "11111111-1111-4111-8111-111111111111" +const user = "22222222-2222-4222-8222-222222222222" +const requestId = "33333333-3333-4333-8333-333333333333" +const content: RepositorySnapshot = { + account: { apiUrl: "https://api.example", organizationId: org, userId: user }, + content: { + repository: { provider: "github", host: "github.com", id: "42" }, + pullRequests: [], + requests: [ + { + requestId, + billingRecordIds: [], + startedAt: "2026-10-04T12:00:00Z", + allocation: { kind: "unlinked", pullRequestIds: [], method: "native" }, + }, + ], + coverage: { observedRequests: 1, unpricedRequests: 1, historyComplete: true }, + }, +} +let directory: string +const http = vi.fn() +beforeEach(async () => { + directory = await mkdtemp(join(tmpdir(), "kimchi-reporting-http-")) + config.key = "test-key" + config.endpoint = "https://api.example" + http + .mockReset() + .mockImplementation(async (input) => + String(input).endsWith("api-keys:verify") + ? Response.json({ organizationId: org, userId: user }) + : Response.json({ status: "accepted", revision: "1", receivedAt: new Date().toISOString() }), + ) + vi.stubGlobal("fetch", http) + await setReportingEnabled(directory, true) + await queueSnapshots(directory, [content]) +}) +afterEach(async () => { + await flushWorkSummaries() + vi.unstubAllEnvs() + vi.unstubAllGlobals() + vi.restoreAllMocks() + await rm(directory, { recursive: true, force: true }) +}) +const deliver = () => deliverSnapshots(directory, "/project", new AbortController().signal, () => {}) +async function seedLinked(priced = true) { + const at = "2026-10-04T12:00:00.000Z" + const common = { + version: 1, + workId: "55555555-5555-4555-8555-555555555555", + sessionId: "private-session", + recordedAt: at, + } + const source = { + apiUrl: "https://api.example", + gatewayUrl: "https://gateway.example/v1/chat/completions", + credentialHash: "a".repeat(64), + } + const rows = [ + { + ...common, + type: "request", + requestId, + startedAt: at, + scope: { account: content.account, repository: "/private/repo/.git" }, + }, + { + ...common, + type: "request_dispatch", + requestId, + dispatchedAt: at, + billingSource: source, + billingSelector: requestTagSelector(requestId, at), + }, + ...(priced + ? [ + { + ...common, + type: "request_cost", + requestId, + billingSource: source, + billingSelector: requestTagSelector(requestId, at), + billingRows: [{ id: "44444444-4444-4444-8444-444444444444", costUsd: "0.123456789" }], + billingLookup: { status: "priced", checkedAt: at, organizationId: org, userId: user }, + }, + ] + : []), + { + ...common, + type: "commit", + repository: "/private/repo/.git", + worktree: "/private/repo", + sha: "b".repeat(40), + pullRequests: [ + { + provider: "github", + id: "101", + repositoryId: "42", + host: "github.com", + repository: "example/repo", + number: 1, + url: "https://github.com/example/repo/pull/1", + state: "merged", + mergedAt: "2026-10-04T13:00:00.000Z", + closedAt: null, + checkedAt: at, + headSha: "b".repeat(40), + mergeCommitSha: null, + }, + ], + }, + ] + await mkdir(join(directory, "work-attribution"), { recursive: true }) + await writeFile( + join(directory, "work-attribution", "source.jsonl"), + `${rows.map((row) => JSON.stringify(row)).join("\n")}\n`, + ) +} + +describe("account-fenced reporting delivery", () => { + it("reports existing work before a PR exists, then replaces it when the PR is discovered", async () => { + await rm(join(directory, "pr-cost-reporting", "state.json")) + await seedLinked() + const path = join(directory, "work-attribution", "source.jsonl") + const linked = await readFile(path, "utf8") + await writeFile( + path, + `${linked + .trim() + .split("\n") + .filter((line) => JSON.parse(line).type !== "commit") + .join("\n")}\n`, + ) + vi.spyOn(providerApi, "lookupRepositoryIdentity").mockResolvedValue({ + ...content.content.repository, + name: "example/repo", + }) + const sent: WireSnapshot[] = [] + http.mockImplementation(async (input, init) => { + if (String(input).endsWith("api-keys:verify")) return Response.json({ organizationId: org, userId: user }) + const payload: WireSnapshot = JSON.parse(String(init?.body)) + sent.push(payload) + return Response.json({ status: "accepted", revision: payload.revision, receivedAt: new Date().toISOString() }) + }) + await reconcileReporting(directory, "/project", new AbortController().signal, () => {}) + expect(sent).toHaveLength(1) + expect(sent[0]).toMatchObject({ + revision: "1", + pullRequests: [], + requests: [{ requestId, allocation: { kind: "unlinked", pullRequestIds: [] } }], + }) + await writeFile(path, linked) + await reconcileReporting(directory, "/project", new AbortController().signal, () => {}) + expect(sent).toHaveLength(2) + expect(sent[1]).toMatchObject({ + producerId: sent[0].producerId, + revision: "2", + pullRequests: [{ id: "101" }], + requests: [{ requestId, allocation: { pullRequestIds: ["101"] } }], + }) + expect(sent[1].requests).toHaveLength(1) + expect(sent[1].requests[0].billingRecordIds).toEqual(sent[0].requests[0].billingRecordIds) + expect(Date.parse(sent[1].generatedAt)).toBeGreaterThanOrEqual(Date.parse(sent[0].generatedAt)) + await reconcileReporting(directory, "/project", new AbortController().signal, () => {}) + expect(sent).toHaveLength(2) + }) + it.each([ + "success", + "failed", + "canceled", + ] as const)("records one %s delivery outcome and the remaining queue", async (outcome) => { + const metric = vi.spyOn(health, "trackPRCostMetric").mockImplementation(() => {}) + const abort = new AbortController() + http.mockImplementation(async (input) => { + if (String(input).endsWith("api-keys:verify")) return Response.json({ organizationId: org, userId: user }) + if (outcome === "canceled") { + abort.abort() + throw abort.signal.reason + } + return outcome === "failed" + ? new Response(null, { status: 503 }) + : Response.json({ status: "accepted", revision: "1", receivedAt: new Date().toISOString() }) + }) + await deliverSnapshots(directory, "/project", abort.signal, () => {}) + await readReportingState(directory) + expect(metric.mock.calls.map(([value]) => value).filter((value) => value.kind === "delivery")).toEqual([ + { kind: "delivery", outcome }, + ]) + expect(metric).toHaveBeenLastCalledWith({ kind: "queueDepth", value: outcome === "success" ? 0 : 1 }) + }) + it("uploads every same-repository post-merge candidate without holding the repository", async () => { + await seedLinked() + const path = join(directory, "work-attribution", "source.jsonl") + const rows = (await readFile(path, "utf8")) + .trim() + .split("\n") + .map((line) => JSON.parse(line)) + rows.find((row) => row.type === "request").startedAt = "2026-10-04T14:00:00.000Z" + const commit = rows.find((row) => row.type === "commit") + commit.pullRequests.push({ + ...commit.pullRequests[0], + id: "102", + number: 2, + url: "https://github.com/example/repo/pull/2", + }) + await writeFile(path, `${rows.map((row) => JSON.stringify(row)).join("\n")}\n`) + const sent: WireSnapshot[] = [] + http.mockImplementation(async (input, init) => { + if (String(input).endsWith("api-keys:verify")) return Response.json({ organizationId: org, userId: user }) + const payload: WireSnapshot = JSON.parse(String(init?.body)) + sent.push(payload) + return Response.json({ status: "accepted", revision: payload.revision, receivedAt: new Date().toISOString() }) + }) + await reconcileReporting(directory, "/project", new AbortController().signal, () => {}) + expect(sent).toHaveLength(1) + expect(sent[0].requests[0]).toMatchObject({ + billingRecordIds: ["44444444-4444-4444-8444-444444444444"], + allocation: { kind: "shared", pullRequestIds: ["101", "102"] }, + }) + expect((await readReportingState(directory)).error).toBeUndefined() + }) + it("holds an undelivered repository withdrawal when the source ledger disappears after queueing", async () => { + await seedLinked() + http.mockImplementation(async (input, init) => { + if (String(input).endsWith("api-keys:verify")) return Response.json({ organizationId: org, userId: user }) + const payload = JSON.parse(String(init?.body)) + return Response.json({ status: "accepted", revision: payload.revision, receivedAt: new Date().toISOString() }) + }) + await reconcileReporting(directory, "/project", new AbortController().signal, () => {}) + const path = join(directory, "work-attribution", "source.jsonl") + await writeFile(path, (await readFile(path, "utf8")).replaceAll('"repositoryId":"42"', '"repositoryId":"43"')) + const source = readWorkCostReport(directory) + const built = buildSnapshots(source.records, source.report, new Map(), source.costRefreshes) + await queueSnapshots(directory, built.snapshots, !built.incomplete) + await rm(path) + http.mockClear() + await reconcileReporting(directory, "/project", new AbortController().signal, () => {}) + expect(http).not.toHaveBeenCalled() + expect(Object.values((await readReportingState(directory)).entries).every((entry) => entry.held)).toBe(true) + }) + it("reports valid scoped work beside unrelated unscoped legacy history with incomplete coverage", async () => { + await seedLinked() + const sourcePath = join(directory, "work-attribution", "source.jsonl") + await writeFile( + sourcePath, + // The legacy work has repository evidence, so its unscoped request leaves that history incomplete. + `${await readFile(sourcePath, "utf8")}${JSON.stringify({ version: 1, type: "request", workId: "66666666-6666-4666-8666-666666666666", sessionId: "legacy-session", requestId: "77777777-7777-4777-8777-777777777777", recordedAt: "2026-10-01T10:00:00Z" })}\n${JSON.stringify({ version: 1, type: "commit", workId: "66666666-6666-4666-8666-666666666666", sessionId: "legacy-session", sha: "c".repeat(40), repository: "/project/.git", worktree: "/project", recordedAt: "2026-10-01T10:00:00Z", pullRequests: [] })}\n`, + ) + let sent: Record | undefined + http.mockImplementation(async (input, init) => { + if (String(input).endsWith("api-keys:verify")) return Response.json({ organizationId: org, userId: user }) + sent = JSON.parse(String(init?.body)) + return Response.json({ status: "accepted", revision: sent?.revision, receivedAt: new Date().toISOString() }) + }) + await reconcileReporting(directory, "/project", new AbortController().signal, () => {}) + expect(sent).toMatchObject({ coverage: { observedRequests: 1, historyComplete: false } }) + expect(JSON.stringify(sent)).not.toContain("77777777-7777-4777-8777-777777777777") + // Skipped history is coverage, not a delivery problem. + const { error, skippedRequests } = await readReportingState(directory) + expect(error).toBeUndefined() + expect(skippedRequests).toBe(1) + }) + it("holds a broken known group while uploading an independent healthy repository", async () => { + await seedLinked() + const sourcePath = join(directory, "work-attribution", "source.jsonl") + const healthyId = "77777777-7777-4777-8777-777777777777" + const healthy = (await readFile(sourcePath, "utf8")) + .replaceAll(requestId, healthyId) + .replaceAll('"repositoryId":"42"', '"repositoryId":"43"') + const broken = { + version: 1, + type: "request", + workId: "66666666-6666-4666-8666-666666666666", + sessionId: "broken", + requestId, + recordedAt: "2026-10-01T10:00:00Z", + } + await writeFile(sourcePath, `${healthy}${JSON.stringify(broken)}\n`) + const sent: string[] = [] + http.mockImplementation(async (input, init) => { + if (String(input).endsWith("api-keys:verify")) return Response.json({ organizationId: org, userId: user }) + const payload = JSON.parse(String(init?.body)) + sent.push(payload.repository.id) + return Response.json({ status: "accepted", revision: payload.revision, receivedAt: new Date().toISOString() }) + }) + await reconcileReporting(directory, "/project", new AbortController().signal, () => {}) + expect(sent).toEqual(["43"]) + const held = Object.values((await readReportingState(directory)).entries).find( + (entry) => entry.repository.id === "42", + ) + expect(held).toMatchObject({ held: true, revision: "1", pending: { requests: [{ requestId }] } }) + }) + it.each([ + true, + false, + ])("builds a full source snapshot with validated billing evidence (priced=%s)", async (priced) => { + await seedLinked(priced) + let sent: Record | undefined + http.mockImplementation(async (input, init) => { + if (String(input).endsWith("api-keys:verify")) return Response.json({ organizationId: org, userId: user }) + sent = JSON.parse(String(init?.body)) + return Response.json({ status: "accepted", revision: sent?.revision, receivedAt: new Date().toISOString() }) + }) + await reconcileReporting(directory, "/project", new AbortController().signal, () => {}) + expect(sent).toMatchObject({ + coverage: { observedRequests: 1, unpricedRequests: priced ? 0 : 1, historyComplete: true }, + requests: [{ requestId, billingRecordIds: priced ? ["44444444-4444-4444-8444-444444444444"] : [] }], + }) + for (const privateValue of ["private-session", "/private", "0.123456789", "a".repeat(64)]) + expect(JSON.stringify(sent)).not.toContain(privateValue) + http.mockClear() + await reconcileReporting(directory, "/project", new AbortController().signal, () => {}) + expect(http).not.toHaveBeenCalled() + }) + it("does not look up or send with a different configured endpoint", async () => { + config.endpoint = "https://other.example" + await deliver() + expect(http).not.toHaveBeenCalled() + }) + it("does not follow a redirect or delete the durable attempt", async () => { + http.mockImplementation(async (input) => + String(input).endsWith("api-keys:verify") + ? Response.json({ organizationId: org, userId: user }) + : new Response(null, { status: 302, headers: { Location: "https://elsewhere.example" } }), + ) + await deliver() + expect(http).toHaveBeenCalledTimes(2) + expect(Object.values((await readReportingState(directory)).entries)[0].pending).toBeDefined() + }) + it("retains a newer snapshot when the older in-flight snapshot is acknowledged", async () => { + http.mockImplementation(async (input) => { + if (String(input).endsWith("api-keys:verify")) return Response.json({ organizationId: org, userId: user }) + const next = structuredClone(content) + next.content.requests[0].allocation.kind = "unknown" + await queueSnapshots(directory, [next]) + return Response.json({ status: "accepted", revision: "1", receivedAt: new Date().toISOString() }) + }) + await deliver() + expect(Object.values((await readReportingState(directory)).entries)[0].pending?.revision).toBe("2") + }) + it("bounds an unfinished response body and keeps the snapshot for retry", async () => { + vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout"] }) + try { + http.mockImplementation(async (input) => + String(input).endsWith("api-keys:verify") + ? Response.json({ organizationId: org, userId: user }) + : new Response(new ReadableStream({ start() {} })), + ) + const running = deliver() + await vi.waitFor(() => expect(http).toHaveBeenCalledTimes(2)) + await vi.advanceTimersByTimeAsync(5001) + await running + expect(Object.values((await readReportingState(directory)).entries)[0].pending).toBeDefined() + } finally { + vi.useRealTimers() + } + }) + it("persists retry timing when the complete reporting pass times out during upload", async () => { + await seedLinked() + http.mockImplementation(async (input, init) => { + if (String(input).endsWith("api-keys:verify")) return Response.json({ organizationId: org, userId: user }) + return new Promise((_resolve, reject) => + init?.signal?.addEventListener("abort", () => reject(init.signal?.reason), { once: true }), + ) + }) + await reconcileReporting(directory, "/project", new AbortController().signal, () => {}) + const entry = Object.values((await readReportingState(directory)).entries)[0] + expect(entry.attempts).toBe(1) + expect(entry.retryAt).toBeGreaterThan(Date.now()) + }, 8000) + it("persists before dispatch and sends the allowlisted body without user or account fields", async () => { + http.mockImplementation(async (input, init) => { + expect(init?.redirect).toBe("error") + if (String(input).endsWith("api-keys:verify")) return Response.json({ organizationId: org, userId: user }) + const persisted = JSON.parse(await readFile(join(directory, "pr-cost-reporting", "state.json"), "utf8")) + expect(Object.values(persisted.entries)).toHaveLength(1) + expect(String(input)).toBe(`https://api.example/ai-optimizer/v1beta/organizations/${org}/pr-cost-snapshots`) + const payload = JSON.parse(String(init?.body)) + expect(payload.requests).toHaveLength(1) + expect(payload.userId).toBeUndefined() + expect(payload.account).toBeUndefined() + return Response.json({ status: "accepted", revision: payload.revision, receivedAt: new Date().toISOString() }) + }) + await deliver() + expect(http).toHaveBeenCalledTimes(2) + expect(Object.values((await readReportingState(directory)).entries)[0].pending).toBeUndefined() + }) + it.each([ + "user", + "organization", + "key-during-verify", + "endpoint-during-verify", + ])("does not upload after %s changes", async (field) => { + http.mockImplementation(async () => { + if (field === "key-during-verify") config.key = "new-key" + if (field === "endpoint-during-verify") config.endpoint = "https://other.example" + return Response.json({ + organizationId: field === "organization" ? requestId : org, + userId: field === "user" ? requestId : user, + }) + }) + await deliver() + expect(http).toHaveBeenCalledOnce() + expect(Object.values((await readReportingState(directory)).entries)[0].pending).toBeDefined() + }) + it("persists Retry-After and does not retry before its deadline", async () => { + http.mockImplementation(async (input) => + String(input).endsWith("api-keys:verify") + ? Response.json({ organizationId: org, userId: user }) + : new Response("private response text", { status: 429, headers: { "Retry-After": "120" } }), + ) + await deliver() + const entry = Object.values((await readReportingState(directory)).entries)[0] + expect(entry.retryAt).toBeGreaterThan(Date.now() + 119000) + expect(entry.lastError).not.toContain("private response") + http.mockClear() + await deliver() + expect(http).not.toHaveBeenCalled() + }) + // Other 4xx responses, such as an endpoint that is not deployed yet, cannot be fixed by retrying. + it.each([404, 400, 403])("backs off for hours after a permanent HTTP %s rejection", async (status) => { + vi.spyOn(Math, "random").mockReturnValue(0.5) + http.mockImplementation(async (input) => + String(input).endsWith("api-keys:verify") + ? Response.json({ organizationId: org, userId: user }) + : new Response(null, { status }), + ) + const delays: number[] = [] + for (let attempt = 0; attempt < 4; attempt++) { + const before = Date.now() + await deliver() + const entry = Object.values((await readReportingState(directory)).entries)[0] + delays.push(Math.round((entry.retryAt - before) / 60_000)) + vi.spyOn(Date, "now").mockReturnValue(entry.retryAt + 1) + } + expect(delays).toEqual([60, 120, 240, 360]) + }) + it.each([ + [0, 48], + [0.999, 72], + ])("spreads a rejection's retry by 20%% so rejected clients do not return together (random %s)", async (random, minutes) => { + vi.spyOn(Math, "random").mockReturnValue(random) + http.mockImplementation(async (input) => + String(input).endsWith("api-keys:verify") + ? Response.json({ organizationId: org, userId: user }) + : new Response(null, { status: 404 }), + ) + const before = Date.now() + await deliver() + const entry = Object.values((await readReportingState(directory)).entries)[0] + expect(Math.round((entry.retryAt - before) / 60_000)).toBe(minutes) + }) + it.each([408, 503])("keeps retrying HTTP %s within a minute", async (status) => { + http.mockImplementation(async (input) => + String(input).endsWith("api-keys:verify") + ? Response.json({ organizationId: org, userId: user }) + : new Response(null, { status }), + ) + await deliver() + expect(Object.values((await readReportingState(directory)).entries)[0].retryAt).toBeLessThanOrEqual( + Date.now() + 60_000, + ) + }) + it("retries a 429 without a PR_COST_LIMIT detail like an outage, backing off with jitter", async () => { + vi.spyOn(Math, "random").mockReturnValue(0.5) + http.mockImplementation(async (input) => + String(input).endsWith("api-keys:verify") + ? Response.json({ organizationId: org, userId: user }) + : Response.json({ code: 8, message: "busy" }, { status: 429 }), + ) + const delays: number[] = [] + for (let attempt = 0; attempt < 4; attempt++) { + const before = Date.now() + await deliver() + const entry = Object.values((await readReportingState(directory)).entries)[0] + delays.push(Math.round((entry.retryAt - before) / 1000)) + // Past both the retry deadline and this repository's upload window. + vi.spyOn(Date, "now").mockReturnValue(Math.max(entry.retryAt, (entry.uploadedAt ?? 0) + UPLOAD_INTERVAL_MS) + 1) + } + expect(delays).toEqual([30, 60, 120, 240]) + expect(Object.values((await readReportingState(directory)).entries)[0]).toMatchObject({ + attempts: 4, + lastError: "PR reporting returned HTTP 429", + }) + expect(Object.values((await readReportingState(directory)).entries)[0].limit).toBeUndefined() + }) + it("bounds a Retry-After longer than six hours", async () => { + http.mockImplementation(async (input) => + String(input).endsWith("api-keys:verify") + ? Response.json({ organizationId: org, userId: user }) + : new Response(null, { status: 503, headers: { "Retry-After": "172800" } }), + ) + const before = Date.now() + await deliver() + const { retryAt } = Object.values((await readReportingState(directory)).entries)[0] + expect(retryAt - before).toBeGreaterThanOrEqual(LIMIT_RETRY_MS) + expect(retryAt - Date.now()).toBeLessThanOrEqual(LIMIT_RETRY_MS) + }) + it("cancels an in-flight upload when another instance opts out and discards its payload", async () => { + let sent: (() => void) | undefined + const sending = new Promise((resolve) => { + sent = resolve + }) + http.mockImplementation(async (input, init) => { + if (String(input).endsWith("api-keys:verify")) return Response.json({ organizationId: org, userId: user }) + sent?.() + return new Promise((_resolve, reject) => + init?.signal?.addEventListener("abort", () => reject(init.signal?.reason), { once: true }), + ) + }) + const running = deliver() + await sending + await setReportingEnabled(directory, false) + await running + const state = await readReportingState(directory) + expect(Object.values(state.entries)[0].pending).toBeUndefined() + expect(Object.values(state.entries)[0].requestHashes).toHaveLength(1) + expect(JSON.stringify(state)).not.toContain(requestId) + }) + it("keeps a malformed acknowledgement pending and never records response content", async () => { + http.mockImplementation(async (input) => + String(input).endsWith("api-keys:verify") + ? Response.json({ organizationId: org, userId: user }) + : Response.json({ + status: "accepted", + revision: "999", + receivedAt: new Date().toISOString(), + privateText: "secret", + }), + ) + await deliver() + const state = await readReportingState(directory) + expect(Object.values(state.entries)[0].pending).toBeDefined() + expect(JSON.stringify(state)).not.toContain("secret") + }) + it("explains a role that cannot upload and retries it slowly", async () => { + http.mockImplementation(async () => Response.json({ message: "permission denied" }, { status: 403 })) + const before = Date.now() + await deliverSnapshots(directory, "/project", new AbortController().signal, () => {}) + const state = await readReportingState(directory) + const [entry] = Object.values(state.entries) + expect(entry.lastError).toBe( + "PR reporting is not allowed for this API key: uploading PR costs needs an Owner or Member role (HTTP 403)", + ) + expect(entry.retryAt - before).toBeGreaterThanOrEqual(0.8 * 60 * 60 * 1000) + }) + it("still delivers when the opt-out watcher cannot start", async () => { + vi.spyOn(fs, "watch").mockImplementation(() => { + throw Object.assign(new Error("inotify watch limit reached"), { code: "ENOSPC" }) + }) + respond(accepted) + await deliverSnapshots(directory, "/project", new AbortController().signal, () => {}) + expect(posts()).toHaveLength(1) + expect(Object.values((await readReportingState(directory)).entries)[0].pending).toBeUndefined() + }) + it("treats a record type this version does not know as incomplete history", async () => { + await rm(join(directory, "pr-cost-reporting", "state.json")) + await seedLinked() + // A newer Kimchi sharing this history may write records this version cannot price or attribute. + await writeFile( + join(directory, "work-attribution", "newer.jsonl"), + `${JSON.stringify({ version: 1, type: "future_record", workId: "55555555-5555-4555-8555-555555555555", sessionId: "newer", recordedAt: "2026-10-04T12:00:00.000Z" })}\n`, + ) + respond(accepted) + await reconcileReporting(directory, "/project", new AbortController().signal, () => {}) + expect(posts()).toEqual([]) + expect((await readReportingState(directory)).error).toContain("could not capture a complete inventory") + }) + it("delivers the queued report but withdraws nothing when a ledger cannot be read", async () => { + await mkdir(join(directory, "work-attribution"), { recursive: true }) + await writeFile(join(directory, "work-attribution", "source.jsonl"), "{broken\n") + respond(accepted) + await reconcileReporting(directory, "/project", new AbortController().signal, () => {}) + // The report queued from the last complete capture still goes out; the unreadable ledger queues nothing new. + expect(posts().map((payload) => [payload.revision, payload.requests.length])).toEqual([["1", 1]]) + expect((await readReportingState(directory)).error).toContain("could not capture a complete inventory") + }) + it("still delivers queued snapshots whose larger replacements do not fit the local queue", async () => { + const ids = new Map(["51", "52", "53"].map((id) => [id, Array.from({ length: 32_000 }, () => crypto.randomUUID())])) + const full = (bills: number) => + [...ids].map(([repositoryId, requestIds]) => { + const value = inventory(requestIds, repositoryId) + for (const request of value.content.requests) + request.billingRecordIds = Array.from({ length: bills }, () => crypto.randomUUID()) + return value + }) + await queueSnapshots(directory, [content, ...full(0)]) + // Two bills per request make every replacement too large to queue beside the snapshots already waiting. + expect((await queueSnapshots(directory, [content, ...full(2)])).error).toContain( + "until queued reports leave room in the local queue", + ) + respond(accepted) + // A 24 MiB queue can outlast one 5-second pass on a slow CI runner; this checks for a stall, not speed. + await deliverSnapshots(directory, "/project", new AbortController().signal, () => {}, Date.now() + 50_000) + // Delivering what is already queued is what frees the room; nothing may stall it. + expect(posts().map((payload) => [payload.repository.id, payload.revision])).toEqual([ + ["42", "1"], + ["51", "1"], + ["52", "1"], + ]) + }, 60_000) +}) + +const second = "55555555-5555-4555-8555-555555555551" +const third = "55555555-5555-4555-8555-555555555552" +const posts = () => + http.mock.calls + .filter(([url]) => String(url).endsWith("/pr-cost-snapshots")) + .map(([, init]): WireSnapshot => JSON.parse(String(init?.body))) +const respond = (reply: (payload: WireSnapshot) => Response) => + http.mockImplementation(async (input, init) => + String(input).endsWith("api-keys:verify") + ? Response.json({ organizationId: org, userId: user }) + : reply(JSON.parse(String(init?.body))), + ) +const accepted = (payload: WireSnapshot) => + Response.json({ status: "accepted", revision: payload.revision, receivedAt: new Date().toISOString() }) +const inventory = (requestIds: string[], repositoryId = "42"): RepositorySnapshot => ({ + account: content.account, + content: { + ...content.content, + repository: { ...content.content.repository, id: repositoryId }, + requests: requestIds.map((id) => ({ ...content.content.requests[0], requestId: id })), + coverage: { observedRequests: requestIds.length, unpricedRequests: requestIds.length, historyComplete: true }, + }, +}) +const entryFor = async (repositoryId: string) => + Object.values((await readReportingState(directory)).entries).find((entry) => entry.repository.id === repositoryId) + +describe("upload window", () => { + it("uploads rapid ordinary changes once per five-minute window, sending the newest", async () => { + respond(accepted) + await deliver() + await queueSnapshots(directory, [inventory([requestId, second])]) + await deliver() + await queueSnapshots(directory, [inventory([requestId, second, third])]) + await deliver() + expect(posts().map((payload) => payload.revision)).toEqual(["1"]) + vi.spyOn(Date, "now").mockReturnValue(Date.now() + UPLOAD_INTERVAL_MS) + await deliver() + expect(posts().map((payload) => [payload.revision, payload.requests.length])).toEqual([ + ["1", 1], + ["3", 3], + ]) + }) + it("sends a PR opening at once inside the window", async () => { + respond(accepted) + await deliver() + const linked = inventory([requestId]) + linked.content.pullRequests = [ + { id: "101", number: 1, url: "https://github.com/example/repo/pull/1", state: "open" }, + ] + linked.content.requests[0].allocation = { kind: "pull-request", pullRequestIds: ["101"], method: "native" } + await queueSnapshots(directory, [linked]) + await deliver() + expect(posts().map((payload) => payload.revision)).toEqual(["1", "2"]) + }) + it("sends the next snapshot at once after /pr-reporting on", async () => { + respond(accepted) + await deliver() + await queueSnapshots(directory, [inventory([requestId, second])]) + await deliver() + expect(posts()).toHaveLength(1) + await setReportingEnabled(directory, true) + await deliver() + expect(posts().map((payload) => payload.revision)).toEqual(["1", "2"]) + }) +}) + +const limitDetail = (scope: string, limit: string, current: string, maximum: string) => ({ + "@type": "type.googleapis.com/google.rpc.ErrorInfo", + reason: "PR_COST_LIMIT", + domain: "ai-optimizer", + metadata: { scope, limit, current, maximum }, +}) +/** Status items wrap with a two-column indent; reading them as prose keeps these checks about content. */ +const unwrapped = (text: string) => text.replaceAll("\n ", " ") +const limited = (scope: string, limit: string, current: string, maximum: string) => + Response.json( + { + code: 8, + message: "PR cost reporting resource limit exceeded", + details: [limitDetail(scope, limit, current, maximum)], + }, + { status: 429 }, + ) + +describe("server limits", () => { + it("treats a producer limit as quota: about six hours on its own counter, while another repository uploads", async () => { + vi.spyOn(Math, "random").mockReturnValue(0.5) + await queueSnapshots(directory, [content, inventory([second], "43")]) + respond((payload) => + payload.repository.id === "42" ? limited("producer", "repositories", "101", "100") : accepted(payload), + ) + const before = Date.now() + await deliver() + const entry = await entryFor("42") + expect(entry).toMatchObject({ + attempts: 0, + limit: { scope: "producer", limit: "repositories", current: 101, maximum: 100 }, + }) + expect((entry?.retryAt ?? 0) - before).toBeGreaterThanOrEqual(LIMIT_RETRY_MS) + expect((entry?.retryAt ?? 0) - Date.now()).toBeLessThanOrEqual(LIMIT_RETRY_MS) + expect((await entryFor("43"))?.pending).toBeUndefined() + const state = await readReportingState(directory) + expect(JSON.stringify(state)).not.toContain("resource limit exceeded") + expect(unwrapped(statusText(state, undefined).text)).toContain( + "github.com repository 42: producer limit reached (repositories 101 of 100). Not reported; next try in about 6 h.", + ) + http.mockClear() + await deliver() + expect(posts()).toEqual([]) + }) + it.each([ + ["organization", "this organization"], + ["contributor", "your account in this organization"], + ])("pauses the whole account for a %s limit but still sends a withdrawal", async (scope, owner) => { + await queueSnapshots(directory, [content, inventory([second], "43")]) + respond((payload) => (payload.requests.length ? limited(scope, "requests", "50000", "50000") : accepted(payload))) + await deliver() + expect(posts()).toHaveLength(1) + const state = await readReportingState(directory) + expect(Object.values(state.paused ?? {})).toMatchObject([{ limit: { scope, limit: "requests" } }]) + expect(unwrapped(statusText(state, undefined).text)).toContain( + `PR cost reporting paused for ${owner}: ${scope} limit reached (requests 50,000 of 50,000). Ask an admin to free space or wait. Next try in about`, + ) + // The request moves to repository 43, so repository 42 is withdrawn; the withdrawal frees space. + http.mockClear() + await queueSnapshots(directory, [inventory([second, requestId], "43")], true) + await deliver() + expect(posts().map((payload) => [payload.repository.id, payload.requests.length])).toEqual([["42", 0]]) + expect((await readReportingState(directory)).paused).toBeDefined() + }) + it("does not pause the account for a limit the server reports only on queries", async () => { + await queueSnapshots(directory, [content, inventory([second], "43")]) + respond((payload) => + payload.repository.id === "42" ? limited("frozen", "reports", "250001", "250000") : accepted(payload), + ) + await deliver() + // Frozen totals never reject an upload; an unknown scope waits on this repository alone. + const state = await readReportingState(directory) + expect(state.paused).toBeUndefined() + expect(await entryFor("42")).toMatchObject({ limit: { current: 250001, maximum: 250000 } }) + expect((await entryFor("42"))?.limit?.scope).toBeUndefined() + expect((await entryFor("43"))?.pending).toBeUndefined() + }) + it("trims to a server snapshot limit and sends the smaller snapshot after the wait", async () => { + await rm(join(directory, "pr-cost-reporting", "state.json")) + await seedLinked() + const path = join(directory, "work-attribution", "source.jsonl") + const rows = (await readFile(path, "utf8")) + .trim() + .split("\n") + .map((line) => JSON.parse(line)) + const later = "66666666-6666-4666-8666-666666666666" + rows.push({ + ...rows.find((row) => row.type === "request"), + requestId: later, + startedAt: "2026-10-04T12:30:00.000Z", + recordedAt: "2026-10-04T12:30:00.000Z", + }) + await writeFile(path, `${rows.map((row) => JSON.stringify(row)).join("\n")}\n`) + let full = true + respond((payload) => (full ? limited("snapshot", "requests", "2", "1") : accepted(payload))) + const run = () => reconcileReporting(directory, "/project", new AbortController().signal, () => {}) + await run() + expect(posts().map((payload) => payload.requests.length)).toEqual([2]) + expect(await entryFor("42")).toMatchObject({ + limit: { scope: "snapshot", limit: "requests" }, + learned: { requests: 1 }, + }) + // The next pass queues the smaller snapshot at once, but it waits out the limit like any other. + await run() + const waiting = await entryFor("42") + expect(waiting).toMatchObject({ trimmed: 1, pending: { requests: [{ requestId: later }] } }) + expect(posts()).toHaveLength(1) + full = false + vi.spyOn(Date, "now").mockReturnValue((waiting?.retryAt ?? 0) + 1) + await run() + expect(posts().at(-1)).toMatchObject({ + requests: [{ requestId: later }], + coverage: { observedRequests: 1, historyComplete: false, trimmedRequests: 1 }, + }) + const sent = await entryFor("42") + expect(sent?.limit).toBeUndefined() + expect(sent?.learned?.requests).toBe(1) + }) + it("retries an oversized 429 body as an outage without reading past 64 KiB", async () => { + respond( + () => + new Response( + JSON.stringify({ details: [limitDetail("organization", "bytes", "1", "1")], padding: "x".repeat(70 * 1024) }), + { status: 429 }, + ), + ) + await deliver() + const entry = await entryFor("42") + expect(entry).toMatchObject({ attempts: 1, lastError: "PR reporting returned HTTP 429" }) + expect(entry?.limit).toBeUndefined() + expect((await readReportingState(directory)).paused).toBeUndefined() + expect(entry?.retryAt).toBeLessThanOrEqual(Date.now() + 36_000) + }) + it("reads only a PR_COST_LIMIT ErrorInfo from the ai-optimizer domain", () => { + const detail = limitDetail("producer", "snapshots", "100", "100") + expect(serverLimit({ details: [detail] }, 5)).toEqual({ + scope: "producer", + limit: "snapshots", + current: 100, + maximum: 100, + at: 5, + }) + expect(serverLimit({ details: [limitDetail("snapshot", "windowedPullRequests", "2001", "2000")] }, 5)).toEqual({ + scope: "snapshot", + limit: "windowedPullRequests", + current: 2001, + maximum: 2000, + at: 5, + }) + expect(serverLimit({ details: [{ ...detail, domain: "elsewhere" }] }, 5)).toBeUndefined() + expect(serverLimit({ details: [{ ...detail, reason: "QUOTA" }] }, 5)).toBeUndefined() + expect(serverLimit({ code: 8, message: "busy" }, 5)).toBeUndefined() + // Unknown values never reach durable state; the rejection still counts as a limit. + expect(serverLimit({ details: [limitDetail("galaxy", "planets", "1.5", "-1")] }, 5)).toEqual({ at: 5 }) + }) +}) + +describe("repository identity for work without a PR", () => { + it.each([ + ["unsupported", "This repository has no supported GitHub or GitLab remote.", true], + ["retry", "GitHub lookup timed out. Kimchi will retry.", false], + ] as const)("leaves other repositories complete only when a lookup fails as %s", async (kind, message, complete) => { + await rm(join(directory, "pr-cost-reporting", "state.json")) + await seedLinked() + const path = join(directory, "work-attribution", "source.jsonl") + const startedAt = new Date().toISOString() + await writeFile( + path, + `${await readFile(path, "utf8")}${JSON.stringify({ + version: 1, + type: "request", + workId: "77777777-7777-4777-8777-777777777777", + sessionId: "scratch-session", + requestId: "66666666-6666-4666-8666-666666666666", + startedAt, + recordedAt: startedAt, + scope: { account: content.account, repository: "/scratch/.git" }, + })}\n`, + ) + vi.spyOn(providerApi, "lookupRepositoryIdentity").mockRejectedValue(new LookupError(message, kind)) + respond(accepted) + await reconcileReporting(directory, "/project", new AbortController().signal, () => {}) + expect(posts().map((payload) => [payload.repository.id, payload.coverage.historyComplete])).toEqual([ + ["42", complete], + ]) + const { error, skippedRequests } = await readReportingState(directory) + expect(error).toBeUndefined() + expect(skippedRequests).toBe(complete ? undefined : 1) + }) + it("delivers a known repository while another lookup times out, then retries that lookup after cooldown", async () => { + await seedLinked() + const path = join(directory, "work-attribution", "source.jsonl") + const source = await readFile(path, "utf8") + await writeFile( + path, + `${source}${JSON.stringify({ + version: 1, + type: "request", + workId: "77777777-7777-4777-8777-777777777777", + sessionId: "other-session", + requestId: "66666666-6666-4666-8666-666666666666", + startedAt: "2026-10-04T12:00:00.000Z", + recordedAt: "2026-10-04T12:00:00.000Z", + scope: { account: content.account, repository: "/slow/.git" }, + })}\n`, + ) + const lookup = vi.spyOn(providerApi, "lookupRepositoryIdentity").mockImplementation( + (_path, signal) => + new Promise((_resolve, reject) => { + signal.addEventListener("abort", () => reject(signal.reason), { once: true }) + }), + ) + const run = () => reconcileReporting(directory, "/project", new AbortController().signal, () => {}) + await run() + expect(http.mock.calls.filter(([url]) => String(url).endsWith("pr-cost-snapshots"))).toHaveLength(1) + await run() + expect(lookup).toHaveBeenCalledOnce() + const next = Date.now() + 31_000 + vi.spyOn(Date, "now").mockReturnValue(next) + lookup.mockResolvedValue({ provider: "github", host: "github.com", name: "team/slow", id: "99" }) + await run() + expect(lookup).toHaveBeenCalledTimes(2) + expect( + Object.values((await readReportingState(directory)).entries).some((entry) => entry.repository.id === "99"), + ).toBe(true) + }) + it("keeps a repository identity for a day, then refreshes it", async () => { + await seedLinked() + const path = join(directory, "work-attribution", "source.jsonl") + const rows = (await readFile(path, "utf8")) + .trim() + .split("\n") + .filter((line) => JSON.parse(line).type !== "commit") + await writeFile(path, `${rows.join("\n")}\n`) + const lookup = vi.spyOn(providerApi, "lookupRepositoryIdentity").mockResolvedValue({ + provider: "github", + host: "github.com", + name: "team/first", + id: "98", + }) + const run = () => reconcileReporting(directory, "/project", new AbortController().signal, () => {}) + await run() + await run() + expect(lookup).toHaveBeenCalledOnce() + const clock = vi.spyOn(Date, "now") + const start = Date.now() + // Passes run every 30 seconds; an hour of them asks the provider nothing. + clock.mockReturnValue(start + 60 * 60_000) + await run() + expect(lookup).toHaveBeenCalledOnce() + clock.mockReturnValue(start + 24 * 60 * 60_000) + lookup.mockResolvedValue({ provider: "github", host: "github.com", name: "team/moved", id: "99" }) + await run() + expect(lookup).toHaveBeenCalledTimes(2) + }) + it("queues pre-PR work from a dozen repositories without re-fetching each repository's identity every pass", async () => { + await rm(join(directory, "pr-cost-reporting", "state.json")) + const rows = Array.from({ length: 12 }, (_, index) => ({ + version: 1, + type: "request", + workId: `55555555-5555-4555-8555-${String(index).padStart(12, "0")}`, + sessionId: `session-${index}`, + requestId: `66666666-6666-4666-8666-${String(index).padStart(12, "0")}`, + recordedAt: "2026-10-04T12:00:00.000Z", + startedAt: "2026-10-04T12:00:00.000Z", + scope: { account: content.account, repository: `/src/repo-${index}/.git` }, + })) + await mkdir(join(directory, "work-attribution"), { recursive: true }) + await writeFile( + join(directory, "work-attribution", "source.jsonl"), + `${rows.map((row) => JSON.stringify(row)).join("\n")}\n`, + ) + let lookups = 0 + // A realistic lookup: Git remote read, CLI token lookup and one provider API round-trip. + vi.spyOn(providerApi, "lookupRepositoryIdentity").mockImplementation(async (path) => { + lookups++ + await new Promise((resolve) => setTimeout(resolve, 450)) + return { + provider: "github", + host: "github.com", + name: `team/${path.split("/")[2]}`, + id: String(1000 + Number(path.split("-")[1].split("/")[0])), + } + }) + for (let pass = 0; pass < 3; pass++) + await reconcileReporting(directory, "/project", new AbortController().signal, () => {}) + const queued = Object.values((await readReportingState(directory)).entries).filter( + (entry) => entry.repository.provider === "github" && Number(entry.repository.id) >= 1000, + ) + expect(queued).toHaveLength(12) + // One lookup per repository, plus at most one abandoned when a pass deadline interrupts it. + expect(lookups).toBeLessThanOrEqual(13) + }, 30_000) +}) + +describe("idle reporting passes", () => { + it("skips an unchanged pass, reads journals in the background and still moves the window", async () => { + await seedLinked() + respond(accepted) + const run = () => reconcileReporting(directory, "/project", new AbortController().signal, () => {}) + await run() + // The acknowledgement changed the queue, so the second pass captures once more. + await run() + const calculate = vi.spyOn(costs, "calculatePullRequestCosts") + const blockingRead = vi.spyOn(summary, "readWorkRecords") + await run() + expect(calculate).not.toHaveBeenCalled() + // Without new records, the upload window still moves within the hour. + vi.spyOn(Date, "now").mockReturnValue(Date.now() + 60 * 60_000) + await run() + expect(calculate).toHaveBeenCalledOnce() + await appendFile( + join(directory, "work-attribution", "source.jsonl"), + `${JSON.stringify({ version: 1, type: "work", workId: "55555555-5555-4555-8555-555555555555", sessionId: "private-session", recordedAt: new Date().toISOString() })}\n`, + ) + await run() + expect(calculate).toHaveBeenCalledTimes(2) + expect(blockingRead).not.toHaveBeenCalled() + expect(posts()).toHaveLength(1) + }) +}) + +describe("journals with a torn final append", () => { + it("keeps reporting after the crashed session appends another record", async () => { + await rm(join(directory, "pr-cost-reporting", "state.json")) + await seedLinked() + vi.stubEnv("PI_CODING_AGENT_DIR", directory) + await writeFile(join(directory, "work-attribution", "crashed.jsonl"), '{"type":"request","requestId":"0b8f') + appendWorkRecord( + createContext({ cwd: "/project", sessionManager: { getSessionId: () => "crashed" } }), + { type: "work" }, + "55555555-5555-4555-8555-555555555555", + ) + await reconcileReporting(directory, "/project", new AbortController().signal, () => {}) + const sent = http.mock.calls.filter(([url]) => String(url).endsWith("/pr-cost-snapshots")) + expect(sent).toHaveLength(1) + expect( + JSON.parse(String(sent[0][1]?.body)).requests.map((request: { requestId: string }) => request.requestId), + ).toEqual([requestId]) + expect((await readReportingState(directory)).error).toBeUndefined() + }) + it("keeps reporting recorded work when another session's journal ends with a partial line", async () => { + await rm(join(directory, "pr-cost-reporting", "state.json")) + await seedLinked() + // A process died during its final append; that record was never complete or reported. + await writeFile(join(directory, "work-attribution", "crashed.jsonl"), '{"type":"request","requestId":"0b8f') + const sent: WireSnapshot[] = [] + http.mockImplementation(async (input, init) => { + if (String(input).endsWith("api-keys:verify")) return Response.json({ organizationId: org, userId: user }) + const payload: WireSnapshot = JSON.parse(String(init?.body)) + sent.push(payload) + return Response.json({ status: "accepted", revision: payload.revision, receivedAt: new Date().toISOString() }) + }) + await reconcileReporting(directory, "/project", new AbortController().signal, () => {}) + expect(sent.map((payload) => payload.requests.map((request) => request.requestId))).toEqual([[requestId]]) + }) +}) diff --git a/src/extensions/pr-cost-reporting/worker.ts b/src/extensions/pr-cost-reporting/worker.ts new file mode 100644 index 000000000..685e69d0a --- /dev/null +++ b/src/extensions/pr-cost-reporting/worker.ts @@ -0,0 +1,383 @@ +import { watch } from "node:fs" +import { verifyApiKey } from "../../api/organizations.js" +import { loadConfig } from "../../config.js" +import { boundedResponse, parseRetryAfterMs } from "../../utils/http.js" +import { plainURL } from "../../utils/url.js" +import { lookupRepositoryIdentity } from "../pull-request-status/provider-api.js" +import { LookupError } from "../pull-request-status/provider-records.js" +import { trackPRCostMetric } from "../telemetry/pr-cost.js" +import { readWorkCostReport } from "../work-attribution/cost-sync.js" +import { isWorkAccount, isWorkScope, platformApiUrl, sameWorkAccount } from "../work-attribution/scope.js" +import { object, readWorkRecordsAsync, workJournalFingerprint } from "../work-attribution/summary.js" +import { + ACCOUNT_SCOPES, + acknowledgeSnapshot, + deferSnapshot, + inUploadWindow, + LIMIT_NAMES, + LIMIT_SCOPES, + learnedLimits, + limitSnapshot, + queueSnapshots, + type ReportingState, + readReportingState, + recordReportingError, + reportingDirectory, + reportingStateVersion, + type ServerLimit, + type SnapshotAck, +} from "./queue.js" +import { accountKey, buildSnapshots, type ReportingRepository, type RepositoryIdentity } from "./snapshot.js" + +const PASS_MS = 5000 +const RESPONSE_BYTES = 64 * 1024 +const MINUTE_MS = 60_000 +const HOUR_MS = 60 * MINUTE_MS +/** Provider repository IDs are stable, so a found identity, or the lack of a supported remote, is kept a day. */ +const REPOSITORY_CACHE_MS = 24 * HOUR_MS +/** A limit rejection waits about six hours; the server frees space on its own schedule. */ +export const LIMIT_RETRY_MS = 6 * HOUR_MS +/** ±20%, so that clients rejected together do not come back together. */ +const jitter = (ms: number) => Math.round(ms * (0.8 + 0.4 * Math.random())) +/** + * Transient failures (timeouts, 408, 5xx and 429 without a limit) honor a bounded Retry-After, or back + * off from 30 seconds to 30 minutes. Other 4xx responses, such as an endpoint not deployed yet, cannot + * be fixed by retrying and wait one to six hours. + */ +export function retryDelay(status: number | undefined, attempts: number, retryAfterMs: number | null): number { + if (status !== undefined && status >= 400 && status < 500 && status !== 408 && status !== 429) + return jitter(Math.min(LIMIT_RETRY_MS, HOUR_MS * 2 ** attempts)) + if (retryAfterMs !== null) return Math.min(Math.max(retryAfterMs, 30_000), LIMIT_RETRY_MS) + return jitter(Math.min(30 * MINUTE_MS, 30_000 * 2 ** attempts)) +} + +/** Reads the PR_COST_LIMIT ErrorInfo of a 429 body; anything else is not a limit. */ +export function serverLimit(body: unknown, at = Date.now()): ServerLimit | undefined { + if (!object(body) || !Array.isArray(body.details)) return undefined + const info = body.details.find( + (detail) => + object(detail) && + detail["@type"] === "type.googleapis.com/google.rpc.ErrorInfo" && + detail.reason === "PR_COST_LIMIT" && + detail.domain === "ai-optimizer", + ) + if (!object(info)) return undefined + const metadata = object(info.metadata) ? info.metadata : {} + const number = (value: unknown) => + typeof value === "string" && /^\d{1,15}$/.test(value) ? { value: Number(value) } : undefined + const scope = LIMIT_SCOPES.find((value) => value === metadata.scope) + const limit = LIMIT_NAMES.find((value) => value === metadata.limit) + const current = number(metadata.current) + const maximum = number(metadata.maximum) + return { + ...(scope ? { scope } : {}), + ...(limit ? { limit } : {}), + ...(current ? { current: current.value } : {}), + ...(maximum ? { maximum: maximum.value } : {}), + at, + } +} + +/** A failed lookup keeps its kind: a repository without a GitHub or GitLab identity is not missing evidence. */ +const repositoryCache = new Map() + +function safeEndpoint(value: string): boolean { + const url = plainURL(value, ["https:", "http:"]) + return url?.protocol === "https:" || ["localhost", "127.0.0.1", "[::1]"].includes(url?.hostname ?? "") +} + +/** Delivery uses a fresh account verification and a credential fence for every bounded attempt. */ +export async function deliverSnapshots( + agentDir: string, + cwd: string, + signal: AbortSignal, + assertLease: () => void, + deadline = Date.now() + PASS_MS, +): Promise { + const controller = new AbortController() + const timer = setTimeout(() => controller.abort(), Math.max(1, deadline - Date.now())) + timer.unref() + const combined = AbortSignal.any([signal, controller.signal]) + // Seeing an opt-out from another process mid-upload is an optimisation: each upload rereads the state anyway. + // Without a watcher, for example when inotify watches run out, delivery still works. + // One durable write raises several events on Linux and the state can reach 24 MiB, so one reread runs at a time; + // events seen during it cause one more. + let watcher: ReturnType | undefined + let rereading = false + let changedAgain = false + const reread = () => { + if (rereading) { + changedAgain = true + return + } + rereading = true + void readReportingState(agentDir) + .then( + (state) => { + if (!state.enabled) controller.abort() + }, + () => controller.abort(), + ) + .finally(() => { + rereading = false + if (!changedAgain || !watcher) return + changedAgain = false + reread() + }) + } + try { + watcher = watch(reportingDirectory(agentDir), { persistent: false }, reread) + } catch {} + try { + const state = await readReportingState(agentDir) + if (!state.enabled) return + const configuredKey = loadConfig({ cwd }).apiKey + const configuredEndpoint = platformApiUrl(cwd) + if ( + Object.values(state.entries).some( + (entry) => entry.pending && (!configuredKey || entry.account.apiUrl !== configuredEndpoint), + ) + ) + await recordReportingError( + agentDir, + `${state.error ? `${state.error}. ` : ""}Some queued reports are waiting for their original account endpoint and credentials`, + ) + const now = Date.now() + // Withdrawals free space, so an account-wide limit only holds snapshots that add claims. + const paused = new Set( + Object.entries(state.paused ?? {}) + .filter(([, pause]) => pause.retryAt > now) + .map(([account]) => account), + ) + + const due = Object.entries(state.entries) + .filter(([, entry]) => entry.pending && !entry.held && entry.retryAt <= now && !inUploadWindow(entry, now)) + .sort(([, a], [, b]) => Number(!a.urgent) - Number(!b.urgent) || a.retryAt - b.retryAt) + let attempts = 0 + for (const [id, entry] of due) { + const snapshot = entry.pending + if (!snapshot) continue + if (paused.has(accountKey(entry.account)) && (snapshot.requests.length || snapshot.pullRequests.length)) continue + if (combined.aborted || attempts >= 3) break + const key = loadConfig({ cwd }).apiKey + const apiUrl = platformApiUrl(cwd) + if (!key || !safeEndpoint(apiUrl) || apiUrl !== entry.account.apiUrl) continue + attempts++ + let status: number | undefined + let retryAfterMs: number | null = null + let limit: ServerLimit | undefined + let errorMessage = "PR reporting request unavailable" + const assertCurrent = () => { + assertLease() + combined.throwIfAborted() + if (loadConfig({ cwd }).apiKey !== key || platformApiUrl(cwd) !== apiUrl) + throw new Error("PR reporting credentials changed") + } + + const fetchBounded: typeof fetch = async (input, init) => { + assertCurrent() + const current = await readReportingState(agentDir) + assertCurrent() + if ( + !current.enabled || + current.entries[id]?.held || + current.entries[id]?.pending?.revision !== snapshot.revision + ) + throw new Error("PR reporting snapshot changed") + const response = await fetch(input, { ...init, redirect: "error", signal: combined }) + assertCurrent() + status = response.status + retryAfterMs = parseRetryAfterMs(response) + if (!response.ok) + errorMessage = + response.status === 403 + ? "PR reporting is not allowed for this API key: uploading PR costs needs an Owner or Member role (HTTP 403)" + : `PR reporting returned HTTP ${response.status}` + // Verify, acknowledgement and error bodies are small; neither response text nor headers enter durable state. + const bounded = await boundedResponse( + response, + RESPONSE_BYTES, + combined, + "PR reporting response exceeds the size limit", + ) + assertCurrent() + return bounded + } + try { + const verified = { + apiUrl, + ...(await verifyApiKey(key, { + endpoint: apiUrl, + fetch: fetchBounded, + signal: combined, + retry: { maxRetries: 0 }, + })), + } + assertCurrent() + if (!isWorkAccount(verified) || !sameWorkAccount(entry.account, verified)) { + errorMessage = "PR reporting is waiting for the original account" + throw new Error(errorMessage) + } + + const response = await fetchBounded( + `${apiUrl}/ai-optimizer/v1beta/organizations/${entry.account.organizationId}/pr-cost-snapshots`, + { + method: "POST", + headers: { Authorization: `Bearer ${key}`, "Content-Type": "application/json", Accept: "application/json" }, + body: JSON.stringify(snapshot), + }, + ) + assertCurrent() + if (response.status === 429) limit = serverLimit(await response.json().catch(() => undefined)) + if (!response.ok) throw new Error(errorMessage) + const ack: SnapshotAck = await response.json() + assertCurrent() + await acknowledgeSnapshot(agentDir, id, snapshot.revision, ack) + trackPRCostMetric({ kind: "delivery", outcome: "success" }) + } catch { + trackPRCostMetric({ kind: "delivery", outcome: combined.aborted ? "canceled" : "failed" }) + if (signal.aborted) return + if (limit) { + await limitSnapshot(agentDir, id, snapshot.revision, limit, Date.now() + jitter(LIMIT_RETRY_MS)) + if (limit.scope && ACCOUNT_SCOPES.includes(limit.scope)) paused.add(accountKey(entry.account)) + } else + await deferSnapshot( + agentDir, + id, + snapshot.revision, + Date.now() + retryDelay(status, entry.attempts, retryAfterMs), + errorMessage, + ) + } + } + } finally { + watcher?.close() + watcher = undefined + clearTimeout(timer) + } +} + +/** Runs after local pricing, under the existing reconciliation lease, in sessions that may upload. */ +export async function reconcileReporting( + agentDir: string, + cwd: string, + signal: AbortSignal, + assertLease: () => void, +): Promise { + const state = await readReportingState(agentDir) + if (!state.enabled) return + try { + await captureSnapshots(agentDir, state, signal, assertLease) + } catch { + if (!signal.aborted) + await recordReportingError( + agentDir, + "PR reporting could not capture a complete inventory; previous reports were retained", + ) + } + // Reports already queued go out with their own time even when this pass could not capture a new inventory. + if (signal.aborted) return + try { + await deliverSnapshots(agentDir, cwd, signal, assertLease, Date.now() + PASS_MS) + } catch { + if (!signal.aborted) await recordReportingError(agentDir, "PR reporting could not deliver queued reports") + } +} + +/** Unchanged journals and queue give the same snapshots; only the moving upload window needs another capture. */ +const CAPTURE_REFRESH_MS = HOUR_MS +/** What the last capture that settled every repository identity read and left behind, and when it started. */ +let lastCapture: { inputs: string; at: number } | undefined + +/** Queues every repository's snapshot, unless the journals and queue have not changed since the last capture. */ +async function captureSnapshots( + agentDir: string, + state: ReportingState, + signal: AbortSignal, + assertLease: () => void, +): Promise { + const deadline = Date.now() + PASS_MS + const bounded = AbortSignal.any([signal, AbortSignal.timeout(PASS_MS)]) + const check = () => { + assertLease() + bounded.throwIfAborted() + if (Date.now() >= deadline) throw new Error("PR reporting pass time limit exceeded") + } + check() + // Fingerprint before reading: an append during the read only makes the next pass capture again. + const journals = await workJournalFingerprint(agentDir) + const inputs = async () => JSON.stringify([agentDir, journals, await reportingStateVersion(agentDir)]) + if (lastCapture?.inputs === (await inputs()) && Date.now() - lastCapture.at < CAPTURE_REFRESH_MS) return + const startedAt = Date.now() + let historyComplete = true + const records = await readWorkRecordsAsync(agentDir, bounded, () => { + historyComplete = false + }) + check() + if (!historyComplete) throw new Error("PR reporting is waiting for readable source records") + const { report, costRefreshes } = readWorkCostReport(agentDir, records) + const repositories = new Map() + const needed = new Set() + const requestRepositories = new Map>() + for (const row of records) + if (row.type === "request" && typeof row.requestId === "string" && isWorkScope(row.scope)) { + const values = requestRepositories.get(row.requestId) ?? new Set() + values.add(row.scope.repository) + requestRepositories.set(row.requestId, values) + } + for (const request of report.requests) { + check() + const linked = report.pullRequests.filter( + (pr) => + request.account && + pr.account && + sameWorkAccount(request.account, pr.account) && + request.pullRequestIds.includes(pr.key), + ) + if (linked.length && linked.every((pr) => pr.pullRequest?.id && pr.pullRequest.repositoryId)) continue + for (const repository of requestRepositories.get(request.requestId) ?? []) needed.add(repository) + } + for (const [key, cached] of repositoryCache) + if (Date.now() - cached.checkedAt >= REPOSITORY_CACHE_MS) repositoryCache.delete(key) + const cacheKey = (repository: string) => JSON.stringify([agentDir, repository]) + // One credential lookup per host and pass, not one per repository. + const tokens: Parameters[2] = new Map() + for (const repository of needed) { + const cached = repositoryCache.get(cacheKey(repository)) + if (cached?.value) repositories.set(repository, cached.value) + else if (cached?.unsupported) repositories.set(repository, "unsupported") + } + for (const repository of [...needed].sort( + (a, b) => (repositoryCache.get(cacheKey(a))?.checkedAt ?? 0) - (repositoryCache.get(cacheKey(b))?.checkedAt ?? 0), + )) { + const key = cacheKey(repository) + const cached = repositoryCache.get(key) + if (cached?.value || cached?.unsupported || (cached && Date.now() - cached.checkedAt < 30_000)) continue + // Reserve time to persist and deliver known groups even when discovery is slow. + if (Date.now() >= deadline - 2000) break + check() + try { + const lookupMs = Math.min(3000, deadline - Date.now() - 1000) + const value = await lookupRepositoryIdentity( + repository, + AbortSignal.any([bounded, AbortSignal.timeout(lookupMs)]), + tokens, + ) + repositoryCache.set(key, { checkedAt: Date.now(), value }) + repositories.set(repository, value) + } catch (error) { + // Transient failures (network, timeouts, rate limits) retry and leave the history incomplete meanwhile. + const unsupported = error instanceof LookupError && error.kind === "unsupported" + repositoryCache.set(key, { checkedAt: Date.now(), ...(unsupported ? { unsupported: true } : {}) }) + if (unsupported) repositories.set(repository, "unsupported") + check() + } + } + + const built = buildSnapshots(records, report, repositories, costRefreshes, learnedLimits(state)) + check() + await queueSnapshots(agentDir, built.snapshots, !built.incomplete, built.skippedRequests) + // A repository lookup that failed or ran out of time is retried by the next pass. + if ([...needed].every((repository) => repositories.has(repository))) + lastCapture = { inputs: await inputs(), at: startedAt } +} diff --git a/src/extensions/pull-request-status/index.test.ts b/src/extensions/pull-request-status/index.test.ts index b3a5a71d0..d33b4771c 100644 --- a/src/extensions/pull-request-status/index.test.ts +++ b/src/extensions/pull-request-status/index.test.ts @@ -80,6 +80,10 @@ function contribution(workId: string): discovery.WorkPullRequestUpdate { pullRequests: [pr], } } +/** Outside the terminal's work panel, `/work` prints its details as text. */ +function textCommandContext() { + return { ...createCommandContext(), ...ctx, mode: "rpc" as const } +} beforeEach(() => { directory = mkdtempSync(join(tmpdir(), "kimchi-pr-status-")) vi.stubEnv("PI_CODING_AGENT_DIR", directory) @@ -197,7 +201,7 @@ describe("PR status extension", () => { }) await Promise.resolve() expect(ctx.ui.notify).toHaveBeenCalledTimes(warningCount) - const commandCtx = { ...createCommandContext(), ...ctx } + const commandCtx = textCommandContext() await api.getRegisteredCommand("work").handler("", commandCtx) expect(ctx.ui.notify).toHaveBeenLastCalledWith(expect.stringContaining(linked.pullRequests[0].url), "info") await api.getRegisteredCommand("work").handler("new", commandCtx) @@ -326,7 +330,7 @@ describe("PR status extension", () => { const update = vi.mocked(supervisor.subscribePullRequestReconciliation).mock.calls[0][0].onPullRequest update({ ...contribution(getWorkId(ctx)), pullRequests: [mr] }) expect(ctx.ui.setStatus).toHaveBeenLastCalledWith("work-pr", "MR !7 open") - await work.getRegisteredCommand("work").handler("", { ...createCommandContext(), ...ctx }) + await work.getRegisteredCommand("work").handler("", textCommandContext()) expect(ctx.ui.notify).toHaveBeenLastCalledWith(expect.stringContaining(`MR !7 open: ${mr.url}`), "info") }) @@ -456,7 +460,7 @@ describe("PR status extension", () => { update(contribution(getWorkId(ctx))) expect(ctx.ui.setStatus).toHaveBeenCalledWith("work-pr", "PR #7 open") expect(ctx.ui.setStatus).toHaveBeenCalledWith("work-pr-url", pr.url) - const commandCtx = { ...createCommandContext(), ...ctx } + const commandCtx = textCommandContext() await work.getRegisteredCommand("work").handler("", commandCtx) expect(ctx.ui.notify).toHaveBeenLastCalledWith(expect.stringContaining(pr.url), "info") await work.getRegisteredCommand("work").handler("new", commandCtx) @@ -560,7 +564,17 @@ describe("PR status with the real reconciliation supervisor", () => { sha, recordedAt: pr.checkedAt, pullRequests: [ - { ...pr, url, number: index + 1, state: "merged", headSha: sha, mergeCommitSha: sha, mergedAt: pr.checkedAt }, + { + ...pr, + url, + id: String(index + 1), + repositoryId: "17", + number: index + 1, + state: "merged", + headSha: sha, + mergeCommitSha: sha, + mergedAt: pr.checkedAt, + }, ], } }) @@ -582,6 +596,14 @@ describe("PR status with the real reconciliation supervisor", () => { await pass(2 * rows.length) // Startup can clear the footer more than once under load; it ends on the current work's PR alone. const calls = vi.mocked(ctx.ui.setStatus).mock.calls + await vi.waitFor( + () => + expect(calls.slice(-2)).toEqual([ + ["work-pr-url", "https://github.com/example/repo/pull/1"], + ["work-pr", "PR #1 merged"], + ]), + { timeout: 25_000 }, + ) expect(calls.slice(-2)).toEqual([ ["work-pr-url", "https://github.com/example/repo/pull/1"], ["work-pr", "PR #1 merged"], @@ -641,7 +663,7 @@ describe("expected lookup failures", () => { await Promise.resolve() expect(ctx.ui.setStatus).toHaveBeenLastCalledWith("work-pr", "PR/MR waiting") expect(ctx.ui.notify).not.toHaveBeenCalled() - await api.getRegisteredCommand("work").handler("", { ...createCommandContext(), ...ctx }) + await api.getRegisteredCommand("work").handler("", textCommandContext()) const shown = vi.mocked(ctx.ui.notify).mock.calls.at(-1)?.[0] expect(shown).toContain("PR/MR lookup: 1 commit waiting") expect(shown).toContain("PR/MR lookup: GitHub is offline.") @@ -670,7 +692,7 @@ describe("expected lookup failures", () => { update({ ...commit, recordedAt, prLookup: { status: "pending", checkedAt } }) expect(ctx.ui.setStatus).not.toHaveBeenCalledWith("work-pr", "PR/MR waiting") expect(ctx.ui.notify).not.toHaveBeenCalled() - await api.getRegisteredCommand("work").handler("", { ...createCommandContext(), ...ctx }) + await api.getRegisteredCommand("work").handler("", textCommandContext()) const shown = vi.mocked(ctx.ui.notify).mock.calls.at(-1)?.[0] expect(shown).toContain("PR/MR lookup: 2 commits without a PR after 32 days, no longer checked") // Later PRs add other /work lines, such as costs still waiting for billing; only lookup lines matter here. diff --git a/src/extensions/pull-request-status/index.ts b/src/extensions/pull-request-status/index.ts index 21c7bbae5..c5c8973da 100644 --- a/src/extensions/pull-request-status/index.ts +++ b/src/extensions/pull-request-status/index.ts @@ -11,15 +11,10 @@ import { type WorkStateRequest, } from "../work-attribution.js" import { currentBranch, lookupBranchPullRequest } from "./branch-status.js" -import { mergePullRequestLinks } from "./links.js" +import { mergePullRequestLinks, pullRequestLabel } from "./links.js" import { lookupFailureReason } from "./provider-records.js" import { LOOKUP_WINDOW_MS, type WorkPullRequest, type WorkPullRequestUpdate } from "./pull-requests.js" -/** `PR #7 open` or `MR !7 merged`; the status line shows the state only while there is room. */ -function requestStatus(pr: WorkPullRequest): string { - return `${pr.provider === "gitlab" ? "MR !" : "PR #"}${pr.number} ${pr.state}` -} - /** A branch's PR is asked for again only after a branch change, a push or PR creation, or this long. */ const BRANCH_REFRESH_MS = 5 * 60_000 const PUBLISH_COMMAND = /\b(?:git\s+push|gh\s+pr\s+create|glab\s+mr\s+create)\b/ @@ -129,11 +124,11 @@ export default function pullRequestStatusExtension(pi: ExtensionAPI): void { function renderWork(): void { const { links, pending, errors } = details() if (errors.length) footer("PR/MR check /work") - else if (links.length === 1 && !pending) footer(requestStatus(links[0]), links[0].url) + else if (links.length === 1 && !pending) footer(pullRequestLabel(links[0]), links[0].url) else if (links.length) footer(`PRs/MRs ${links.length} linked${pending ? `, ${pending} waiting` : ""}`) else if (pending) footer("PR/MR waiting") // Orientation only: the branch PR's cost is tracked once this work records a commit. - else footer(branchPull && requestStatus(branchPull), branchPull?.url) + else footer(branchPull && pullRequestLabel(branchPull), branchPull?.url) } function receive(update: WorkPullRequestUpdate): void { @@ -232,7 +227,7 @@ export default function pullRequestStatusExtension(pi: ExtensionAPI): void { const result = branch ? await lookupBranchPullRequest(ctx.cwd, signal) : undefined if (!current()) return const pr = result?.pullRequest - footer(pr ? requestStatus(pr) : undefined, pr?.url) + footer(pr ? pullRequestLabel(pr) : undefined, pr?.url) }) .catch((error) => { if (!current()) return @@ -291,7 +286,7 @@ export default function pullRequestStatusExtension(pi: ExtensionAPI): void { if (typeof request.workId !== "string" || !Array.isArray(request.lines)) return const { links, pending, lookupErrors } = details(request.workId) request.lines.push( - ...links.map((pr) => `${pr.provider === "gitlab" ? "MR !" : "PR #"}${pr.number} ${pr.state}: ${pr.url}`), + ...links.map((pr) => `${pullRequestLabel(pr)}: ${pr.url}`), ...(pending ? [`PR/MR lookup: ${pending} commit${pending === 1 ? "" : "s"} waiting`] : []), ...lookupErrors.map((error) => `PR/MR lookup: ${error}`), ) diff --git a/src/extensions/pull-request-status/links.ts b/src/extensions/pull-request-status/links.ts index 57a6de4d6..94b83ee87 100644 --- a/src/extensions/pull-request-status/links.ts +++ b/src/extensions/pull-request-status/links.ts @@ -1,3 +1,5 @@ +import type { WorkPullRequest } from "./pull-requests.js" + interface PullRequestObservation { provider?: unknown host?: unknown @@ -6,6 +8,11 @@ interface PullRequestObservation { checkedAt?: unknown } +/** `PR #7 open` or `MR !7 merged`, as `/work` and the footer name a link; the footer drops the state when space runs out. */ +export function pullRequestLabel(pr: Pick): string { + return `${pr.provider === "gitlab" ? "MR !" : "PR #"}${pr.number} ${pr.state}` +} + export function pullRequestKey(pr: PullRequestObservation): string { const id = typeof pr.id === "string" && typeof pr.host === "string" ? pr.id : pr.url return JSON.stringify([pr.provider ?? "github", pr.host, id]) diff --git a/src/extensions/pull-request-status/provider-api.ts b/src/extensions/pull-request-status/provider-api.ts index b39c53987..78f9b88af 100644 --- a/src/extensions/pull-request-status/provider-api.ts +++ b/src/extensions/pull-request-status/provider-api.ts @@ -6,7 +6,7 @@ import { isAbsolute, join } from "node:path" import { parse as parseYaml, stringify as stringifyYaml } from "yaml" import { readGitToken } from "../../config.js" import { object } from "../work-attribution/summary.js" -import { httpsURL, LookupError, label, type Repository, repositoryPath } from "./provider-records.js" +import { httpsURL, LookupError, label, providerId, type Repository, repositoryPath } from "./provider-records.js" const COMMAND_TIMEOUT_MS = 5000 @@ -510,3 +510,18 @@ export async function repositoryIdentity( } throw new LookupError("This repository has no supported GitHub or GitLab API.", "unsupported") } + +/** A captured request's repository identity gets at most this long; the caller's signal usually ends it sooner. */ +const IDENTITY_BUDGET_MS = 10_000 + +/** Provider identity for captured unlinked requests; credentials stay inside discovery. */ +export async function lookupRepositoryIdentity( + cwd: string, + signal: AbortSignal, + tokens: Parameters[3] = new Map(), +): Promise<{ provider: "github" | "gitlab"; host: string; name: string; id: string }> { + const repository = await repositoryIdentity(cwd, signal, Date.now() + IDENTITY_BUDGET_MS, tokens) + const id = providerId(repository.id) + if (!id) throw new LookupError("The Git provider returned no stable repository ID.", "invalid") + return { provider: repository.provider, host: repository.host, name: repository.name, id } +} diff --git a/src/extensions/pull-request-status/provider-records.ts b/src/extensions/pull-request-status/provider-records.ts index 8256fb595..982ba478f 100644 --- a/src/extensions/pull-request-status/provider-records.ts +++ b/src/extensions/pull-request-status/provider-records.ts @@ -131,6 +131,7 @@ export function pullRequest( } } +/** Saved links that still pass provider validation; malformed entries are skipped. */ export function storedPullRequests(value: unknown): WorkPullRequest[] { if (!Array.isArray(value)) return [] return value.flatMap((item) => { diff --git a/src/extensions/pull-request-status/pull-requests.test.ts b/src/extensions/pull-request-status/pull-requests.test.ts index 2869c463a..8545b7f00 100644 --- a/src/extensions/pull-request-status/pull-requests.test.ts +++ b/src/extensions/pull-request-status/pull-requests.test.ts @@ -90,6 +90,8 @@ function seed(overrides: Record = {}, transitions = false): voi } function pull(number = 7, overrides: Record = {}) { return { + id: 98000 + number, + base: { repo: { id: 42 } }, html_url: `https://github.com/team/repo/pull/${number}`, number, state: "open", @@ -109,6 +111,7 @@ function mr(number = 7, overrides: Record = {}) { sha, source_branch: "feature", source_project_id: 42, + target_project_id: 42, merge_commit_sha: null, ...overrides, } @@ -153,6 +156,7 @@ function replies(handler: (url: URL, options: RequestInit) => unknown = () => [] const gitlab = /^\/api\/v4\/projects\/([^/]+)$/.exec(url.pathname) const value = github ? { + id: 42, full_name: github[1], html_url: `https://${url.host === "api.github.com" ? "github.com" : url.host}/${github[1]}`, } @@ -201,6 +205,29 @@ afterEach(async () => { }) describe("native provider credentials", () => { + it.each(providers)("keeps the target %s repository and global PR IDs for fork contributions", async (provider) => { + seed() + useProvider(provider) + replies(() => + provider === "github" + ? [pull(7, { id: 98731, base: { repo: { id: 42 } }, head: { sha, repo: { id: 999 } } })] + : [mr(7, { target_project_id: 42, source_project_id: 999 })], + ) + await lookup() + expect(saved().at(-1).pullRequests[0]).toMatchObject({ id: "98731", repositoryId: "42" }) + expect(readWorkPullRequestUpdates(agentDir)[0].pullRequests[0]).toMatchObject({ id: "98731", repositoryId: "42" }) + }) + it("refreshes merged legacy links to obtain stable IDs", async () => { + seed({ + prLookup: { status: "linked", checkedAt: "2026-10-01T12:00:00Z" }, + pullRequests: [stored("github", { state: "merged", mergedAt: "2026-10-01T11:00:00Z" })], + }) + replies(() => [ + pull(7, { id: 98731, base: { repo: { id: 42 } }, state: "closed", merged_at: "2026-10-01T11:00:00Z" }), + ]) + await lookup() + expect(saved().at(-1).pullRequests[0]).toMatchObject({ id: "98731", repositoryId: "42" }) + }) it.each(providers)("matches %s with a saved token and no CLI", async (provider) => { seed() useProvider(provider) diff --git a/src/extensions/pull-request-status/pull-requests.ts b/src/extensions/pull-request-status/pull-requests.ts index bb3b90e19..69f3f41cb 100644 --- a/src/extensions/pull-request-status/pull-requests.ts +++ b/src/extensions/pull-request-status/pull-requests.ts @@ -266,7 +266,7 @@ async function scan( const known = mergePullRequests(...commits.map((commit) => commit.pullRequests)) if ( known.length && - known.every((pr) => pr.state === "merged") && + known.every((pr) => pr.state === "merged" && pr.id && pr.repositoryId) && commits.every( (commit) => commit.prLookup?.status === "linked" && JSON.stringify(commit.pullRequests) === JSON.stringify(known), @@ -311,7 +311,10 @@ async function scan( let refreshedKnown = false let refreshFailure: LookupError | undefined for (const previous of known.sort((a, b) => Date.parse(a.checkedAt) - Date.parse(b.checkedAt))) { - if (previous.state === "merged" || found.some((item) => pullRequestKey(item) === pullRequestKey(previous))) + if ( + (previous.state === "merged" && previous.id && previous.repositoryId) || + found.some((item) => pullRequestKey(item) === pullRequestKey(previous)) + ) continue try { let request = refreshed.get(previous.url) diff --git a/src/extensions/telemetry/accumulator.ts b/src/extensions/telemetry/accumulator.ts index 7cd791de1..d6ee6b91b 100644 --- a/src/extensions/telemetry/accumulator.ts +++ b/src/extensions/telemetry/accumulator.ts @@ -1,4 +1,5 @@ import { computeLineChanges, computeWriteLines, inferLanguage, type ToolArgs } from "./helpers.js" +import type { PRCostMetric } from "./pr-cost.js" import type { MetricData } from "./transport.js" // --------------------------------------------------------------------------- @@ -15,6 +16,17 @@ export interface CumulativeState { toolUsage: Record toolDurationMs: Record sessionStartNano: string + prCost?: { + startTimeUnixNano: string + consentVersion?: string + matching: Partial["outcome"], number>> + delivery: Partial["outcome"], number>> + unpriced?: number + queueDepth?: number + snapshotRequests?: number + snapshotBytes?: number + reconciliationStartedAt?: number + } } export function createCumulativeState(): CumulativeState { @@ -79,6 +91,23 @@ export function handleEditCumulativeMetrics(state: CumulativeState, toolName: st export function collectMetrics(state: CumulativeState): MetricData[] { const out: MetricData[] = [] + if (state.prCost) { + const common = { scope: "aggregate" as const, startTimeUnixNano: state.prCost.startTimeUnixNano } + for (const [kind, counts] of Object.entries({ matching: state.prCost.matching, delivery: state.prCost.delivery })) + for (const [decision, value] of Object.entries(counts)) + out.push({ name: `kimchi.pr_cost.${kind}.count`, type: "Sum", value, attrs: { decision }, ...common }) + for (const [name, value] of Object.entries({ + "pricing.unpriced": state.prCost.unpriced, + "queue.depth": state.prCost.queueDepth, + "snapshot.requests": state.prCost.snapshotRequests, + "snapshot.bytes": state.prCost.snapshotBytes, + "reconciliation.age": + state.prCost.reconciliationStartedAt === undefined + ? undefined + : Math.max(0, Date.now() - state.prCost.reconciliationStartedAt) / 1000, + })) + if (value !== undefined) out.push({ name: `kimchi.pr_cost.${name}`, type: "Gauge", value, attrs: {}, ...common }) + } for (const [model, t] of Object.entries(state.tokensByModel)) { for (const [type, val] of Object.entries(t) as [string, number][]) { diff --git a/src/extensions/telemetry/pr-cost.test.ts b/src/extensions/telemetry/pr-cost.test.ts new file mode 100644 index 000000000..b7ec01b5f --- /dev/null +++ b/src/extensions/telemetry/pr-cost.test.ts @@ -0,0 +1,117 @@ +import { afterEach, beforeEach, expect, it, vi } from "vitest" +import { collectMetrics, createCumulativeState } from "./accumulator.js" +import { trackPRCostMetric } from "./pr-cost.js" +import type { TelemetryContext } from "./session-context.js" + +const fixture = vi.hoisted(() => ({ + enabled: true, + context: undefined as Pick | undefined, +})) +vi.mock("./index.js", () => ({ _getTelemetryCtx: () => fixture.context })) +vi.mock("../../config.js", () => ({ readTelemetryConfig: () => ({ enabled: fixture.enabled }) })) + +beforeEach(() => { + fixture.enabled = true + fixture.context = { + config: { enabled: true, metricsEndpoint: "http://localhost/metrics", endpoint: "", headers: {}, apiKey: "" }, + cumulative: createCumulativeState(), + } +}) +afterEach(() => vi.useRealTimers()) + +it("counts decisions and attempts while replacing queue and price gauges", () => { + trackPRCostMetric({ kind: "matching", outcome: "explicit" }) + trackPRCostMetric({ kind: "matching", outcome: "explicit" }) + trackPRCostMetric({ kind: "matching", outcome: "unknown" }) + trackPRCostMetric({ kind: "delivery", outcome: "failed" }) + trackPRCostMetric({ kind: "delivery", outcome: "success" }) + trackPRCostMetric({ kind: "unpriced", value: 4 }) + trackPRCostMetric({ kind: "unpriced", value: 1 }) + trackPRCostMetric({ kind: "queueDepth", value: 2 }) + trackPRCostMetric({ kind: "queueDepth", value: 0 }) + expect(fixture.context?.cumulative.prCost).toEqual({ + startTimeUnixNano: expect.any(String), + matching: { explicit: 2, unknown: 1 }, + delivery: { failed: 1, success: 1 }, + unpriced: 1, + queueDepth: 0, + }) + const state = fixture.context?.cumulative + expect(state).toBeDefined() + if (!state) throw new Error("Missing test accumulator") + const metrics = collectMetrics(state) + expect(metrics).toHaveLength(6) + expect(metrics.every((metric) => metric.scope === "aggregate")).toBe(true) + expect(metrics.filter((metric) => metric.type === "Gauge").map((metric) => metric.value)).toEqual([1, 0]) + expect(collectMetrics(state)).toEqual(metrics) +}) + +it("does not collect while disabled and clears prior values when consent is revoked", () => { + vi.useFakeTimers() + vi.setSystemTime(100_000) + const context = fixture.context + if (!context) throw new Error("Missing test context") + trackPRCostMetric({ kind: "matching", outcome: "explicit" }) + const before = collectMetrics(context.cumulative)[0].startTimeUnixNano + fixture.enabled = false + trackPRCostMetric({ kind: "delivery", outcome: "failed" }) + expect(fixture.context?.cumulative.prCost).toBeUndefined() + fixture.enabled = true + vi.setSystemTime(105_000) + trackPRCostMetric({ kind: "matching", outcome: "session" }) + expect(fixture.context?.cumulative.prCost?.matching).toEqual({ session: 1 }) + const after = collectMetrics(context.cumulative)[0].startTimeUnixNano + expect(after).not.toEqual(before) +}) + +it("measures elapsed time since reconciliation rather than claiming lookup success", () => { + vi.useFakeTimers() + vi.setSystemTime(100_000) + trackPRCostMetric({ kind: "reconciliation" }) + vi.setSystemTime(135_000) + const state = fixture.context?.cumulative + if (!state) throw new Error("Missing test accumulator") + expect(collectMetrics(state)).toContainEqual({ + name: "kimchi.pr_cost.reconciliation.age", + type: "Gauge", + value: 35, + attrs: {}, + scope: "aggregate", + startTimeUnixNano: "100000000000", + }) +}) + +it("ignores invalid gauge values and works without a telemetry instance", () => { + for (const value of [-1, Number.NaN, Number.POSITIVE_INFINITY, 1.5]) trackPRCostMetric({ kind: "queueDepth", value }) + expect(fixture.context?.cumulative.prCost?.queueDepth).toBeUndefined() + fixture.context = undefined + expect(() => trackPRCostMetric({ kind: "matching", outcome: "explicit" })).not.toThrow() +}) + +it("reports the largest queued snapshot's requests and bytes as client-only gauges", () => { + trackPRCostMetric({ kind: "snapshotRequests", value: 32_000 }) + trackPRCostMetric({ kind: "snapshotBytes", value: 8 * 1024 * 1024 }) + trackPRCostMetric({ kind: "snapshotRequests", value: 120 }) + for (const value of [-1, 1.5, Number.NaN]) trackPRCostMetric({ kind: "snapshotBytes", value }) + const state = fixture.context?.cumulative + if (!state) throw new Error("Missing test accumulator") + const gauges = collectMetrics(state).filter((metric) => metric.name.startsWith("kimchi.pr_cost.snapshot.")) + expect(gauges).toEqual([ + { + name: "kimchi.pr_cost.snapshot.requests", + type: "Gauge", + value: 120, + attrs: {}, + scope: "aggregate", + startTimeUnixNano: expect.any(String), + }, + { + name: "kimchi.pr_cost.snapshot.bytes", + type: "Gauge", + value: 8 * 1024 * 1024, + attrs: {}, + scope: "aggregate", + startTimeUnixNano: expect.any(String), + }, + ]) +}) diff --git a/src/extensions/telemetry/pr-cost.ts b/src/extensions/telemetry/pr-cost.ts new file mode 100644 index 000000000..453c0ab61 --- /dev/null +++ b/src/extensions/telemetry/pr-cost.ts @@ -0,0 +1,35 @@ +import { readTelemetryConfig } from "../../config.js" +import type { WorkSegment } from "../work-attribution.js" +import { nowNano } from "./helpers.js" +import { _getTelemetryCtx } from "./index.js" + +export type PRCostMetric = + | { kind: "matching"; outcome: WorkSegment["attribution"] | "failed" | "limited" } + | { kind: "delivery"; outcome: "success" | "failed" | "canceled" } + /** snapshotRequests and snapshotBytes describe the largest snapshot queued in a reporting pass. */ + | { kind: "unpriced" | "queueDepth" | "snapshotRequests" | "snapshotBytes"; value: number } + | { kind: "reconciliation" } + +/** Accept counts and fixed outcomes only; identifiers and error text never enter metric labels. */ +export function trackPRCostMetric(metric: PRCostMetric): void { + const ctx = _getTelemetryCtx() + if (!ctx) return + const consent = readTelemetryConfig() + if (!ctx.config.enabled || !ctx.config.metricsEndpoint || !consent.enabled) { + ctx.cumulative.prCost = undefined + return + } + if (ctx.cumulative.prCost?.consentVersion !== consent.consentVersion) ctx.cumulative.prCost = undefined + ctx.cumulative.prCost ??= { + matching: {}, + delivery: {}, + startTimeUnixNano: nowNano(), + consentVersion: consent.consentVersion, + } + const state = ctx.cumulative.prCost + if (metric.kind === "matching" || metric.kind === "delivery") { + const counts: Partial> = state[metric.kind] + counts[metric.outcome] = (counts[metric.outcome] ?? 0) + 1 + } else if (metric.kind === "reconciliation") state.reconciliationStartedAt = Date.now() + else if (Number.isSafeInteger(metric.value) && metric.value >= 0) state[metric.kind] = metric.value +} diff --git a/src/extensions/telemetry/session-context.test.ts b/src/extensions/telemetry/session-context.test.ts index 6a2470921..1b58d3818 100644 --- a/src/extensions/telemetry/session-context.test.ts +++ b/src/extensions/telemetry/session-context.test.ts @@ -1,9 +1,15 @@ +import { mkdtempSync, rmSync } from "node:fs" +import { tmpdir } from "node:os" +import { join } from "node:path" import { afterEach, beforeEach, describe, expect, it, vi } from "vitest" import type { TelemetryConfig } from "../../config.js" +import * as config from "../../config.js" import * as osMetadata from "../../utils/os-metadata.js" import { createContext } from "../__mocks__/context.js" import { PARENT_SESSION_ID_ENV_KEY } from "../agents/manager/constants.js" import { setTelemetryFermentV2Context } from "./ferment-v2-context.js" +import * as telemetry from "./index.js" +import { trackPRCostMetric } from "./pr-cost.js" import { _resetSharedAccumulators, TelemetryContext } from "./session-context.js" vi.mock("../../api/me.js", () => ({ @@ -41,12 +47,174 @@ describe("SessionContext", () => { globalThis.fetch = originalFetch _resetSharedAccumulators() vi.restoreAllMocks() + vi.unstubAllEnvs() // session.parent_id simulation env — never leak the subagent worker flag // or the parent session id into sibling tests. Reflect.deleteProperty(process.env, "KIMCHI_SUBAGENT") Reflect.deleteProperty(process.env, PARENT_SESSION_ID_ENV_KEY) }) + it("flushes PR health without adding user or session labels", async () => { + vi.stubEnv("KIMCHI_TELEMETRY_ENABLED", "true") + const ctx = new TelemetryContext(makeConfig()) + ctx.setPiSessionId("private-session") + ctx.userId = "private-user" + ctx.cumulative.prCost = { matching: { explicit: 1 }, delivery: {}, queueDepth: 0, startTimeUnixNano: "1000000000" } + ctx.flushMetrics() + await Promise.allSettled([...ctx.inFlight]) + const [, options] = vi.mocked(globalThis.fetch).mock.calls[0] + const body = JSON.parse(String(options?.body)) + const serialized = JSON.stringify(body) + expect(serialized).not.toContain("private-session") + expect(serialized).not.toContain("private-user") + expect(serialized).not.toContain("user.account_uuid") + expect(serialized).not.toContain("session.id") + expect(body.resourceMetrics[0].scopeMetrics[0].metrics).toHaveLength(2) + }) + + it("drops buffered PR health when telemetry is turned off before flush", async () => { + const ctx = new TelemetryContext(makeConfig()) + ctx.cumulative.prCost = { matching: { explicit: 1 }, delivery: {}, startTimeUnixNano: "1000000000" } + vi.stubEnv("KIMCHI_TELEMETRY_ENABLED", "false") + ctx.flushMetrics() + await Promise.allSettled([...ctx.inFlight]) + expect(globalThis.fetch).not.toHaveBeenCalled() + expect(ctx.cumulative.prCost).toBeUndefined() + }) + + it("rechecks PR health consent after pending identity lookup", async () => { + vi.stubEnv("KIMCHI_TELEMETRY_ENABLED", "true") + const ctx = new TelemetryContext(makeConfig()) + let release!: () => void + ctx.userEmailReady = new Promise((resolve) => { + release = resolve + }) + ctx.cumulative.prCost = { matching: { explicit: 1 }, delivery: {}, startTimeUnixNano: "1000000000" } + ctx.flushMetrics() + vi.stubEnv("KIMCHI_TELEMETRY_ENABLED", "false") + release() + await Promise.allSettled([...ctx.inFlight]) + expect(globalThis.fetch).not.toHaveBeenCalled() + expect(ctx.cumulative.prCost).toBeUndefined() + }) + + it("does not retry a PR health batch after telemetry is turned off", async () => { + vi.stubEnv("KIMCHI_TELEMETRY_ENABLED", "true") + vi.spyOn(Math, "random").mockReturnValue(0) + vi.mocked(globalThis.fetch).mockImplementationOnce(async () => { + vi.stubEnv("KIMCHI_TELEMETRY_ENABLED", "false") + return new Response(null, { status: 503 }) + }) + const ctx = new TelemetryContext(makeConfig()) + ctx.cumulative.prCost = { matching: { explicit: 1 }, delivery: {}, startTimeUnixNano: "1000000000" } + ctx.flushMetrics() + await Promise.allSettled([...ctx.inFlight]) + expect(globalThis.fetch).toHaveBeenCalledOnce() + }) + + it("does not restore a discarded batch if telemetry is re-enabled before identity resolves", async () => { + vi.stubEnv("KIMCHI_TELEMETRY_ENABLED", "true") + const ctx = new TelemetryContext(makeConfig()) + let release!: () => void + ctx.userEmailReady = new Promise((resolve) => { + release = resolve + }) + ctx.cumulative.prCost = { matching: { explicit: 4 }, delivery: {}, startTimeUnixNano: "1000000000" } + ctx.flushMetrics() + vi.stubEnv("KIMCHI_TELEMETRY_ENABLED", "false") + ctx.flushMetrics() + vi.stubEnv("KIMCHI_TELEMETRY_ENABLED", "true") + ctx.cumulative.prCost = { matching: { session: 1 }, delivery: {}, startTimeUnixNano: "2000000000" } + ctx.flushMetrics() + release() + await Promise.allSettled([...ctx.inFlight]) + expect(globalThis.fetch).toHaveBeenCalledOnce() + const body = JSON.parse(String(vi.mocked(globalThis.fetch).mock.calls[0][1]?.body)) + expect(body.resourceMetrics[0].scopeMetrics[0].metrics[0].sum.dataPoints[0].startTimeUnixNano).toBe("2000000000") + }) + + it("does not retry discarded health counts after telemetry is re-enabled", async () => { + vi.stubEnv("KIMCHI_TELEMETRY_ENABLED", "true") + vi.spyOn(Math, "random").mockReturnValue(0) + const ctx = new TelemetryContext(makeConfig()) + ctx.cumulative.prCost = { matching: { explicit: 4 }, delivery: {}, startTimeUnixNano: "1000000000" } + vi.mocked(globalThis.fetch).mockImplementationOnce(async () => { + vi.stubEnv("KIMCHI_TELEMETRY_ENABLED", "false") + ctx.flushMetrics() + vi.stubEnv("KIMCHI_TELEMETRY_ENABLED", "true") + ctx.cumulative.prCost = { matching: { session: 1 }, delivery: {}, startTimeUnixNano: "2000000000" } + return new Response(null, { status: 503 }) + }) + ctx.flushMetrics() + await Promise.allSettled([...ctx.inFlight]) + expect(globalThis.fetch).toHaveBeenCalledOnce() + + ctx.flushMetrics() + await Promise.allSettled([...ctx.inFlight]) + expect(globalThis.fetch).toHaveBeenCalledTimes(2) + const body = JSON.parse(String(vi.mocked(globalThis.fetch).mock.calls[1][1]?.body)) + expect(body.resourceMetrics[0].scopeMetrics[0].metrics[0].sum.dataPoints[0].startTimeUnixNano).toBe("2000000000") + }) + + it.each([ + "buffered", + "identity", + "identity-with-fresh-counts", + "retry", + ])("discards %s health after persisted off/on between flushes", async (stage) => { + const directory = mkdtempSync(join(tmpdir(), "kimchi-health-consent-")) + const path = join(directory, "config.json") + const readConfig = config.readTelemetryConfig + vi.stubEnv("KIMCHI_TELEMETRY_ENABLED", undefined) + vi.spyOn(config, "readTelemetryConfig").mockImplementation(() => readConfig(path)) + vi.spyOn(Math, "random").mockReturnValue(0) + try { + config.writeTelemetryEnabled(true, path) + const ctx = new TelemetryContext(makeConfig()) + vi.spyOn(telemetry, "_getTelemetryCtx").mockReturnValue(ctx) + trackPRCostMetric({ kind: "matching", outcome: "explicit" }) + const toggle = () => { + config.writeTelemetryEnabled(false, path) + config.writeTelemetryEnabled(true, path) + } + let release!: () => void + if (stage.startsWith("identity")) + ctx.userEmailReady = new Promise((resolve) => { + release = resolve + }) + if (stage === "retry") + vi.mocked(globalThis.fetch).mockImplementationOnce(async () => { + toggle() + return new Response(null, { status: 503 }) + }) + if (stage === "buffered") toggle() + ctx.flushMetrics() + if (stage.startsWith("identity")) { + toggle() + if (stage === "identity-with-fresh-counts") trackPRCostMetric({ kind: "matching", outcome: "session" }) + release() + } + await Promise.allSettled([...ctx.inFlight]) + const previousAttempts = stage === "retry" ? 1 : 0 + expect(globalThis.fetch).toHaveBeenCalledTimes(previousAttempts) + + if (stage !== "identity-with-fresh-counts") trackPRCostMetric({ kind: "matching", outcome: "session" }) + ctx.flushMetrics() + await Promise.allSettled([...ctx.inFlight]) + expect(globalThis.fetch).toHaveBeenCalledTimes(previousAttempts + 1) + const body = JSON.parse(String(vi.mocked(globalThis.fetch).mock.calls.at(-1)?.[1]?.body)) + const metrics = body.resourceMetrics[0].scopeMetrics[0].metrics + expect(metrics).toHaveLength(1) + expect(metrics[0].sum.dataPoints[0].asInt).toBe("1") + expect(metrics[0].sum.dataPoints[0].attributes).toContainEqual({ + key: "decision", + value: { stringValue: "session" }, + }) + } finally { + rmSync(directory, { recursive: true, force: true }) + } + }) + it("emit appends source and session_type to every event", async () => { const { getActiveFerment } = await import("../ferment/index.js") vi.mocked(getActiveFerment).mockReturnValue(undefined) diff --git a/src/extensions/telemetry/session-context.ts b/src/extensions/telemetry/session-context.ts index 0dc664f8b..10b7f7dbc 100644 --- a/src/extensions/telemetry/session-context.ts +++ b/src/extensions/telemetry/session-context.ts @@ -1,7 +1,7 @@ import crypto from "node:crypto" import type { ExtensionContext } from "@earendil-works/pi-coding-agent" import { getMe } from "../../api/me.js" -import type { TelemetryConfig } from "../../config.js" +import { readTelemetryConfig, type TelemetryConfig } from "../../config.js" import { IS_ACP_MODE } from "../../modes/acp/state.js" import { getOsMetadata } from "../../utils/os-metadata.js" import { getVersion } from "../../utils.js" @@ -327,23 +327,38 @@ export class TelemetryContext { } flushMetrics(): void { + const consent = readTelemetryConfig() + if (!this.config.enabled || !consent.enabled || this.cumulative.prCost?.consentVersion !== consent.consentVersion) + this.cumulative.prCost = undefined + const prCost = this.cumulative.prCost const metrics = collectMetrics(this.cumulative) if (metrics.length > 0) { this.track( - this.userEmailReady.then(() => - sendMetrics( + this.userEmailReady.then(() => { + const currentConsent = readTelemetryConfig() + const allowPRCostMetrics = + this.config.enabled && currentConsent.enabled && prCost?.consentVersion === currentConsent.consentVersion + if (!allowPRCostMetrics && prCost === this.cumulative.prCost) this.cumulative.prCost = undefined + return sendMetrics( this.config, this.resolveSessionId(), - metrics.map((m) => ({ - ...m, - attrs: { - ...m.attrs, - "user.account_uuid": this.userId ?? "", - }, - })), + metrics + .filter((m) => m.scope !== "aggregate" || (allowPRCostMetrics && prCost === this.cumulative.prCost)) + .map((m) => ({ + ...m, + attrs: { + ...m.attrs, + ...(m.scope === "aggregate" ? {} : { "user.account_uuid": this.userId ?? "" }), + }, + })), this.sessionStartNano, - ), - ), + undefined, + (current) => + this.config.enabled && + prCost === this.cumulative.prCost && + prCost?.consentVersion === current.consentVersion, + ) + }), ) } } diff --git a/src/extensions/telemetry/transport.test.ts b/src/extensions/telemetry/transport.test.ts index b72758f9c..54dbcf144 100644 --- a/src/extensions/telemetry/transport.test.ts +++ b/src/extensions/telemetry/transport.test.ts @@ -208,6 +208,23 @@ describe("sendLogBatch", () => { }) describe("sendMetrics", () => { + it("omits session identity from aggregate health metrics", async () => { + const config = makeConfig() + const health = { + name: "kimchi.pr_cost.queue.depth", + type: "Gauge" as const, + value: 2, + attrs: {}, + scope: "aggregate" as const, + } + await sendMetrics(config, "private-session", [health], BASE_NS) + const body = JSON.parse(vi.mocked(globalThis.fetch).mock.calls[0][1]?.body as string) + const point = body.resourceMetrics[0].scopeMetrics[0].metrics[0].gauge.dataPoints[0] + expect(point.asInt).toBe("2") + expect(point.attributes).toEqual([{ key: "client", value: { stringValue: "pi" } }]) + expect(JSON.stringify(body)).not.toContain("private-session") + }) + let originalFetch: typeof globalThis.fetch beforeEach(() => { diff --git a/src/extensions/telemetry/transport.ts b/src/extensions/telemetry/transport.ts index d79b8098c..998d665fd 100644 --- a/src/extensions/telemetry/transport.ts +++ b/src/extensions/telemetry/transport.ts @@ -1,6 +1,6 @@ import * as fs from "node:fs" import * as path from "node:path" -import type { TelemetryConfig } from "../../config.js" +import { readTelemetryConfig, type TelemetryConfig } from "../../config.js" import { fetchWithRetry } from "../../utils/http.js" import { getVersion } from "../../utils.js" import { nowNano, strAttr } from "./helpers.js" @@ -52,6 +52,10 @@ export interface MetricData { type: "Sum" | "Gauge" value: number attrs: Record + /** Operational health has no session or actor dimensions. */ + scope?: "aggregate" + /** A counter reset starts a new stream even if the user resumes the same session. */ + startTimeUnixNano?: string } // --------------------------------------------------------------------------- @@ -152,6 +156,7 @@ export async function sendMetrics( metrics: MetricData[], sessionStartNano: string, userEmail?: string, + canSendAggregate?: (consent: TelemetryConfig) => boolean, ): Promise { if (!config.enabled || !config.metricsEndpoint || metrics.length === 0) return const now = nowNano() @@ -168,10 +173,10 @@ export async function sendMetrics( dataPoints: [ { timeUnixNano: now, - startTimeUnixNano: sessionStartNano, + startTimeUnixNano: m.startTimeUnixNano ?? sessionStartNano, ...(Number.isInteger(m.value) ? { asInt: String(m.value) } : { asDouble: m.value }), attributes: [ - strAttr("session.id", sessionId), + ...(m.scope === "aggregate" ? [] : [strAttr("session.id", sessionId)]), strAttr("client", "pi"), ...Object.entries(m.attrs).map(([k, v]) => strAttr(k, String(v))), ], @@ -189,6 +194,8 @@ export async function sendMetrics( for (const metric of metrics) { logEventToFile(metric.name, { value: metric.value, ...metric.attrs }) } + const aggregate = metrics.some((metric) => metric.scope === "aggregate") + const consent = new AbortController() try { const response = await fetchWithRetry( config.metricsEndpoint, @@ -197,7 +204,19 @@ export async function sendMetrics( headers: { "Content-Type": "application/json", ...config.headers }, body: JSON.stringify(payload), }, - { timeoutMs: 10_000, retry: { maxRetries: 3 } }, + { + timeoutMs: 10_000, + retry: { maxRetries: 3 }, + signal: consent.signal, + fetchImpl: (url, init) => { + if (aggregate) { + const current = readTelemetryConfig() + if (!current.enabled || canSendAggregate?.(current) === false) consent.abort() + } + consent.signal.throwIfAborted() + return globalThis.fetch(url, init) + }, + }, ) if (!response.ok) { logEventToFile("telemetry.response.error", { diff --git a/src/extensions/tips/general-tips.test.ts b/src/extensions/tips/general-tips.test.ts index 8e2f9ecf9..b6d46073b 100644 --- a/src/extensions/tips/general-tips.test.ts +++ b/src/extensions/tips/general-tips.test.ts @@ -18,6 +18,7 @@ describe("GENERAL_TIPS", () => { expect(messages).toContain("Use `/agents` to manage agents or display running agents sessions") expect(messages).toContain("Use `/processes` to inspect running Bash scripts and output.") expect(messages).toContain("Tag requests in Analytics: `/tags add key:value` (e.g. project:myapp).") + expect(messages).toContain("Run `/work` to see your work's cost and PRs.") expect(messages).toContain("Resume the latest session with `kimchi --continue`.") expect(messages).toContain("Name a branch with `/branch `; resume it with `-r `.") expect(messages).toContain("Use `kimchi --verbose` when output looks off.") diff --git a/src/extensions/tips/general-tips.ts b/src/extensions/tips/general-tips.ts index 860d93f26..6f5642858 100644 --- a/src/extensions/tips/general-tips.ts +++ b/src/extensions/tips/general-tips.ts @@ -41,6 +41,11 @@ export const GENERAL_TIPS = [ scope: "general", message: "Tag requests in Analytics: `/tags add key:value` (e.g. project:myapp).", }, + { + id: "work-costs", + scope: "general", + message: "Run `/work` to see your work's cost and PRs.", + }, { id: "continue-session", scope: "general", diff --git a/src/extensions/work-attribution.input-boundaries.test.ts b/src/extensions/work-attribution.input-boundaries.test.ts index 05500e326..4ec2c0798 100644 --- a/src/extensions/work-attribution.input-boundaries.test.ts +++ b/src/extensions/work-attribution.input-boundaries.test.ts @@ -15,6 +15,7 @@ import { createContext } from "./__mocks__/context.js" import { createExtensionApi } from "./__mocks__/extension-api.js" import { createModel, createModelRegistry } from "./__mocks__/model-registry.js" import { createWorkScopeSnapshot } from "./__mocks__/work-scope.js" +import * as health from "./telemetry/pr-cost.js" import * as continuation from "./work-attribution/continuation.js" import * as supervisor from "./work-attribution/reconcile-supervisor.js" import * as scope from "./work-attribution/scope.js" @@ -85,8 +86,15 @@ describe("input submitted while the agent is still streaming", () => { const planned = "11111111-1111-4111-8111-111111111111" vi.spyOn(continuation, "findWorkContinuation").mockImplementation(async (_ctx, text) => text.includes("plan.md") - ? { workId: planned, source: "saved-plan", evidence: { path: "/plans/plan.md", contentHash: "a".repeat(64) } } - : undefined, + ? { + match: { + workId: planned, + source: "saved-plan", + evidence: { path: "/plans/plan.md", contentHash: "a".repeat(64) }, + }, + owned: true, + } + : { owned: false }, ) const api = createExtensionApi() createWorkAttributionExtension()(api.api) @@ -163,8 +171,8 @@ describe("queued input delivery", () => { .spyOn(continuation, "findWorkContinuation") .mockImplementation(async (_ctx, text) => text.includes("plan.md") - ? { workId: planned, source: "saved-plan", evidence: { path: "/plans/plan.md" } } - : undefined, + ? { match: { workId: planned, source: "saved-plan", evidence: { path: "/plans/plan.md" } }, owned: true } + : { owned: false }, ) const api = createExtensionApi() createWorkAttributionExtension()(api.api) @@ -327,9 +335,12 @@ describe("fresh-session guard after scope recovery", () => { const other = "22222222-2222-4222-8222-222222222222" vi.spyOn(scope, "readWorkScope").mockImplementation((workId) => (workId === legacy ? undefined : captured.scope)) const find = vi.spyOn(continuation, "findWorkContinuation").mockResolvedValue({ - workId: other, - source: "named-artifact", - evidence: { path: join(dir, "docs/adr.md"), transitionId: randomUUID() }, + match: { + workId: other, + source: "named-artifact", + evidence: { path: join(dir, "docs/adr.md"), transitionId: randomUUID() }, + }, + owned: true, }) const manager = SessionManager.inMemory(dir) manager.appendCustomEntry("work_identity", { workId: legacy }) @@ -374,7 +385,64 @@ it("leaves an input unresolved without a warning when matching history exceeds i const api = createExtensionApi() createWorkAttributionExtension()(api.api) const ctx = createContext({ cwd: dir, model: createModel("chat"), modelRegistry: createModelRegistry() }) + const metric = vi.spyOn(health, "trackPRCostMetric") await api.getHandler("input")({ type: "input", text: "Explain closures", source: "interactive" }, ctx) expect(getWorkSegment(ctx)).toMatchObject({ attribution: "unknown" }) expect(ctx.ui.notify).not.toHaveBeenCalled() + expect(metric).toHaveBeenCalledWith({ kind: "matching", outcome: "limited" }) +}) + +describe("journal problems during live plan continuation", () => { + const planned = "11111111-1111-4111-8111-111111111111" + /** Writes one other journal, then continues a saved plan of the planned work in a fresh session. */ + async function continuePlan(name: string, content: string) { + vi.spyOn(continuation, "findWorkContinuation").mockResolvedValue({ + match: { + workId: planned, + source: "saved-plan", + evidence: { path: "/plans/plan.md", contentHash: "a".repeat(64) }, + }, + owned: true, + }) + const journal = join(dir, "work-attribution", name) + mkdirSync(join(dir, "work-attribution"), { recursive: true }) + writeFileSync(journal, content) + const api = createExtensionApi() + createWorkAttributionExtension()(api.api) + const ctx = createContext({ cwd: dir, model: createModel("chat"), modelRegistry: createModelRegistry() }) + await api.getHandler("input")( + { type: "input", text: "Implement /plans/plan.md", source: "interactive" }, + ctx, + ) + expect(getWorkId(ctx)).toBe(planned) + return { ctx, journal } + } + const newer = (workId: string) => + `${JSON.stringify({ version: 1, type: "work_checkpoint", workId, sessionId: "newer" })}\n` + + it("adopts the plan without surfacing an attribution failure for an unrelated torn append", async () => { + const { ctx } = await continuePlan("crashed.jsonl", '{"type":"request","requestId":"0b8f') + expect(ctx.ui.notify).not.toHaveBeenCalledWith(expect.stringContaining("Incomplete work history"), "warning") + }) + + it("adopts the plan without a warning when an unknown record type names only another work", async () => { + const { ctx } = await continuePlan("newer.jsonl", newer("22222222-2222-4222-8222-222222222222")) + expect(ctx.ui.notify).not.toHaveBeenCalledWith(expect.anything(), "warning") + }) + + it.each([ + ["a damaged line", "damaged.jsonl", '{"type":"request"}}\n', "Incomplete work history contains invalid records"], + [ + "an unknown record type for this work", + "newer.jsonl", + newer(planned), + "Work history contains unknown record types", + ], + ])("adopts the plan, remembers its input and names the journal with %s in its warning", async (_kind, name, content, problem) => { + const remember = vi.spyOn(semantic, "rememberWorkIntent").mockResolvedValue() + const { ctx, journal } = await continuePlan(name, content) + expect(ctx.ui.notify).toHaveBeenCalledWith(`Work attribution unavailable: ${problem} in ${journal}:1`, "warning") + // The blocked confirmation must not also drop the adopted work's matching memory. + expect(remember).toHaveBeenCalledWith(dir, planned, "Implement /plans/plan.md") + }) }) diff --git a/src/extensions/work-attribution.test.ts b/src/extensions/work-attribution.test.ts index a2df9fce4..972dcc055 100644 --- a/src/extensions/work-attribution.test.ts +++ b/src/extensions/work-attribution.test.ts @@ -1,13 +1,14 @@ import { execFileSync } from "node:child_process" import { randomUUID } from "node:crypto" -import { appendFileSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, writeFileSync } from "node:fs" +import { appendFileSync, mkdtempSync, readdirSync, readFileSync, rmSync, writeFileSync } from "node:fs" import { tmpdir } from "node:os" -import { join } from "node:path" +import { dirname, join } from "node:path" import type { AssistantMessage } from "@earendil-works/pi-ai" import { complete, getModel } from "@earendil-works/pi-ai/compat" import { type BeforeProviderHeadersEvent, createLocalBashOperations, + type ExtensionUIContext, findCutPoint, type InputEvent, type SessionBeforeCompactEvent, @@ -16,13 +17,17 @@ import { type SessionStartEvent, sessionEntryToContextMessages, } from "@earendil-works/pi-coding-agent" +import { type Component, stripTerminalSequences, type TUI } from "@earendil-works/pi-tui" import { afterEach, beforeEach, describe, expect, it, vi } from "vitest" import { installGlobalFetchInstrumentation } from "../http/instrument-fetch.js" import { readPlanWorkId, savePlanMarkdown } from "../shared/planning/plan-markdown.js" import { createCommandContext, createContext } from "./__mocks__/context.js" import { createExtensionApi } from "./__mocks__/extension-api.js" +import { testTheme } from "./__mocks__/theme.js" import { createWorkScopeSnapshot } from "./__mocks__/work-scope.js" +import { savedWorkSummary } from "./__mocks__/work-summary.js" import requestTimingExtension from "./request-timing.js" +import { copySessionFileAndAddHandoffNote, removeTempDir } from "./teleport/provisioning/handoff-note.js" import * as billingSource from "./work-attribution/billing-source.js" import { createWorkCommitTrackingOperations } from "./work-attribution/commits.js" import * as continuation from "./work-attribution/continuation.js" @@ -43,6 +48,8 @@ import { tryWorkAttribution, tryWorkAttributionAsync, WORK_CHANGED_EVENT, + WORK_DETAILS_REQUEST_EVENT, + type WorkDetailsRequest, } from "./work-attribution.js" let dir: string @@ -201,11 +208,11 @@ describe("local work attribution", () => { expect(getWorkId(ctx)).toBe(target.workId) await flushWorkSummaries() const path = join(dir, "work", target.workId, "work.json") - expect(JSON.parse(readFileSync(path, "utf8")).workLinks).toHaveLength(1) + expect(savedWorkSummary(dir, target.workId).workLinks).toHaveLength(1) writeFileSync(path, "damaged") recoverWorkSummaries() await flushWorkSummaries() - expect(JSON.parse(readFileSync(path, "utf8")).workLinks).toHaveLength(1) + expect(savedWorkSummary(dir, target.workId).workLinks).toHaveLength(1) await command.handler(`unlink ${links[0].linkId}`, ctx) expect(records().filter((row) => row.type === "work_link")).toEqual([ links[0], @@ -453,11 +460,11 @@ describe("local work attribution", () => { requestId: calls[1].requestId, }) await flushWorkSummaries() - const requests = JSON.parse(readFileSync(join(dir, "work", getWorkId(ctx), "work.json"), "utf8")).requests + const { requests } = savedWorkSummary(dir, getWorkId(ctx)) expect(requests).toHaveLength(2) expect(records().filter((row) => row.type === "request")).toHaveLength(2) expect(records().filter((row) => row.type === "request_dispatch")).toHaveLength(2) - expect(requests[0].billingSelector.tag).toBe(calls[0].tag) + expect(requests[0].billingSelector).toMatchObject({ tag: calls[0].tag }) expect(requests[1]).toMatchObject({ requestId: calls[1].requestId, parentRequestId: logicalId, @@ -596,7 +603,7 @@ describe("local work attribution", () => { }) recordProviderResponse(attempt.requestId, { status: 500, headers: new Headers() }) await flushWorkSummaries() - const summary = JSON.parse(readFileSync(join(dir, "work", attempt.workId, "work.json"), "utf8")) + const summary = savedWorkSummary(dir, attempt.workId) expect(summary.requests).toHaveLength(1) expect(summary.requests[0]).toMatchObject({ requestId: attempt.requestId, @@ -619,7 +626,7 @@ describe("local work attribution", () => { }) recordProviderResponse("unknown-request", { status: 200, headers: new Headers() }) await flushWorkSummaries() - const summary = JSON.parse(readFileSync(join(dir, "work", attempt.workId, "work.json"), "utf8")) + const summary = savedWorkSummary(dir, attempt.workId) expect(summary.requests).toHaveLength(1) expect(summary.requests[0].response).toEqual({ status: 503, receivedAt: expect.any(String) }) }) @@ -636,7 +643,7 @@ describe("local work attribution", () => { vi.spyOn(continuation, "findWorkContinuation").mockImplementation(async () => { // Count only what the adoption announces, after the input bound the session. api.api.events.on(WORK_CHANGED_EVENT, announced) - return selected + return { match: selected, owned: true } }) createWorkAttributionExtension()(api.api) await api.getHandler("input")({ type: "input", text: "Implement ADR.md", source: "rpc" }, ctx) @@ -644,23 +651,32 @@ describe("local work attribution", () => { const event: BeforeProviderHeadersEvent = { type: "before_provider_headers", headers: {} } await api.getHandler("before_provider_headers")(event, ctx) expect(records().find((row) => row.requestId === event.headers["X-Request-Id"]).workId).toBe(workId) + const accepted = { + source: selected.source, + evidence: { + ...selected.evidence, + ...createWorkScopeSnapshot(join(dir, ".git")).scope, + segmentId: getWorkSegment(ctx)?.id, + }, + } expect(records()).toContainEqual( expect.objectContaining({ type: "work", workId, - continuation: { source: selected.source, evidence: selected.evidence }, + continuation: accepted, }), ) expect(api.getAppendedEntries("work_identity")).toContainEqual({ workId, - continuation: { source: selected.source, evidence: selected.evidence }, + continuation: accepted, }) }) it("keeps an accepted plan continuation for later inputs until a reference to other work ends it", async () => { const planned = getWorkId(createContext({ cwd: dir, sessionManager: { getSessionId: () => "planning" } })) - const find = vi - .spyOn(continuation, "findWorkContinuation") - .mockResolvedValueOnce({ workId: planned, source: "saved-plan", evidence: { path: "/plan.md" } }) + const find = vi.spyOn(continuation, "findWorkContinuation").mockResolvedValueOnce({ + match: { workId: planned, source: "saved-plan", evidence: { path: "/plan.md" } }, + owned: true, + }) const manager = SessionManager.inMemory(dir) const ctx = { ...createContext({ cwd: dir }), sessionManager: manager } const api = createExtensionApi() @@ -691,19 +707,23 @@ describe("local work attribution", () => { expect(getWorkSegment(ctx)).toMatchObject({ attribution: "explicit", reason: "saved-plan" }) expect(find).toHaveBeenCalledOnce() - // Naming a file this work wrote keeps the continuation, even after a command changed it and it no longer - // matches the recorded edit. - mkdirSync(join(dir, "work-attribution"), { recursive: true }) - appendFileSync( - join(dir, "work-attribution", "edits.jsonl"), - `${JSON.stringify({ version: 1, type: "file_transition", workId: planned, sessionId: "implementer", transitionId: "write-notes", toolCallId: "write-notes", repository: join(dir, ".git"), worktree: dir, path: "notes.md" })}\n`, - ) - find.mockResolvedValueOnce(undefined) + // Naming a file only this work recorded keeps the continuation, even after a command changed it and it no + // longer matches the recorded edit. + find.mockResolvedValueOnce({ owned: false }) await input({ type: "input", text: "What does notes.md contain now?", source: "interactive" }, ctx) + expect(find).toHaveBeenLastCalledWith( + expect.anything(), + "What does notes.md contain now?", + expect.anything(), + planned, + ) expect(getWorkSegment(ctx)).toMatchObject({ attribution: "explicit", reason: "saved-plan" }) const other = "22222222-2222-4222-8222-222222222222" - find.mockResolvedValueOnce({ workId: other, source: "saved-plan", evidence: { path: "/other.md" } }) + find.mockResolvedValueOnce({ + match: { workId: other, source: "saved-plan", evidence: { path: "/other.md" } }, + owned: true, + }) await input({ type: "input", text: "Implement /other.md", source: "interactive" }, ctx) expect(getWorkId(ctx)).toBe(planned) expect(getWorkSegment(ctx)).toMatchObject({ attribution: "unknown", reason: "unresolved-reference" }) @@ -712,7 +732,7 @@ describe("local work attribution", () => { await api.getHandler("session_shutdown")({ type: "session_shutdown", reason: "quit" }, ctx) }) it("resolves external input before work output and ignores extension input", async () => { - const find = vi.spyOn(continuation, "findWorkContinuation").mockResolvedValue(undefined) + const find = vi.spyOn(continuation, "findWorkContinuation").mockResolvedValue({ owned: false }) const api = createExtensionApi() createWorkAttributionExtension()(api.api) const input = api.getHandler("input") @@ -720,25 +740,45 @@ describe("local work attribution", () => { await input({ type: "input", text: "Implement", source: "extension" }, ctx) expect(find).not.toHaveBeenCalled() await input({ type: "input", text: "Implement", source: "interactive" }, ctx) - expect(find).toHaveBeenLastCalledWith(expect.objectContaining({ cwd: dir }), "Implement", expect.any(Object)) + expect(find).toHaveBeenLastCalledWith( + expect.objectContaining({ cwd: dir }), + "Implement", + expect.any(Object), + expect.any(String), + ) await input({ type: "input", text: "Continue", source: "interactive" }, ctx) - expect(find).toHaveBeenLastCalledWith(expect.objectContaining({ cwd: dir }), "Continue", expect.any(Object)) + expect(find).toHaveBeenLastCalledWith( + expect.objectContaining({ cwd: dir }), + "Continue", + expect.any(Object), + expect.any(String), + ) const started = createContext({ cwd: dir, sessionManager: { getSessionId: () => "requested" } }) await api.getHandler("before_provider_headers")( { type: "before_provider_headers", headers: {} }, started, ) await input({ type: "input", text: "Later", source: "interactive" }, started) - expect(find).toHaveBeenLastCalledWith(expect.objectContaining({ cwd: dir }), "Later", expect.any(Object)) + expect(find).toHaveBeenLastCalledWith( + expect.objectContaining({ cwd: dir }), + "Later", + expect.any(Object), + expect.any(String), + ) }) it("allows fresh continuation after an earlier startup hook allocated the ledger", async () => { const ctx = createContext({ cwd: dir }) getWorkId(ctx) - const find = vi.spyOn(continuation, "findWorkContinuation").mockResolvedValue(undefined) + const find = vi.spyOn(continuation, "findWorkContinuation").mockResolvedValue({ owned: false }) const api = createExtensionApi() createWorkAttributionExtension()(api.api) await api.getHandler("input")({ type: "input", text: "Implement", source: "rpc" }, ctx) - expect(find).toHaveBeenCalledWith(expect.objectContaining({ cwd: dir }), "Implement", expect.any(Object)) + expect(find).toHaveBeenCalledWith( + expect.objectContaining({ cwd: dir }), + "Implement", + expect.any(Object), + expect.any(String), + ) }) it("preserves restored identity even when a crash left no native tool result", async () => { const ctx = createContext({ cwd: dir }) @@ -749,9 +789,10 @@ describe("local work attribution", () => { // An earlier startup hook may restore the ledger before attribution binds. expect(getWorkId(ctx)).toBe(original) const selected = "11111111-1111-4111-8111-111111111111" - const find = vi - .spyOn(continuation, "findWorkContinuation") - .mockResolvedValue({ workId: selected, source: "named-artifact", evidence: { path: "/project/ADR.md" } }) + const find = vi.spyOn(continuation, "findWorkContinuation").mockResolvedValue({ + match: { workId: selected, source: "named-artifact", evidence: { path: "/project/ADR.md" } }, + owned: true, + }) const resumed = createExtensionApi() createWorkAttributionExtension()(resumed.api) await resumed.getHandler("input")({ type: "input", text: "Implement ADR.md", source: "rpc" }, ctx) @@ -764,9 +805,10 @@ describe("local work attribution", () => { }) it("preserves explicit work selection and historical native output without a summary", async () => { const selected = "11111111-1111-4111-8111-111111111111" - const find = vi - .spyOn(continuation, "findWorkContinuation") - .mockResolvedValue({ workId: selected, source: "saved-plan", evidence: { path: "/plan.md" } }) + const find = vi.spyOn(continuation, "findWorkContinuation").mockResolvedValue({ + match: { workId: selected, source: "saved-plan", evidence: { path: "/plan.md" } }, + owned: true, + }) const manager = SessionManager.inMemory(dir) const ctx = { ...createContext({ cwd: dir }), sessionManager: manager } const api = createExtensionApi() @@ -802,7 +844,7 @@ describe("local work attribution", () => { }) it("does not switch work after an asynchronous continuation lookup became stale", async () => { const ctx = createContext({ cwd: dir }) - let release!: (value: continuation.WorkContinuation) => void + let release!: (value: Awaited>) => void vi.spyOn(continuation, "findWorkContinuation").mockImplementation( () => new Promise((resolve) => { @@ -817,9 +859,12 @@ describe("local work attribution", () => { await api.getRegisteredCommand("work").handler("new", { ...createCommandContext(), ...ctx }) const chosen = getWorkId(ctx) release({ - workId: "11111111-1111-4111-8111-111111111111", - source: "named-artifact", - evidence: { path: "/plan.md" }, + match: { + workId: "11111111-1111-4111-8111-111111111111", + source: "named-artifact", + evidence: { path: "/plan.md" }, + }, + owned: true, }) await input expect(getWorkId(ctx)).toBe(chosen) @@ -964,9 +1009,8 @@ describe("local work attribution", () => { it("never adopts a plan automatically in a child whose inherited identity was unavailable", async () => { const selected = "11111111-1111-4111-8111-111111111111" const find = vi.spyOn(continuation, "findWorkContinuation").mockResolvedValue({ - workId: selected, - source: "saved-plan", - evidence: { path: "/plan.md" }, + match: { workId: selected, source: "saved-plan", evidence: { path: "/plan.md" } }, + owned: true, }) const api = createExtensionApi() createWorkAttributionExtension(null)(api.api) @@ -1110,6 +1154,55 @@ describe("local work attribution", () => { expect(getWorkSegment(resumedCtx)).toEqual(originalSegment) }) + it.each([ + false, + true, + ])("does not invent accounting history from a teleported conversation (explicit: %s)", async (explicit) => { + vi.mocked(scope.readWorkScope).mockRestore() + const captured = createWorkScopeSnapshot(join(dir, "sandbox", ".git")) + vi.mocked(scope.captureWorkScope).mockResolvedValue(captured) + const original = randomUUID() + const oldRequest = randomUUID() + const local = SessionManager.inMemory("/local/project") + local.appendMessage({ role: "user", content: "Plan a small change", timestamp: 1 }) + local.appendCustomEntry("work_identity", { workId: original, explicit }) + local.appendCustomEntry("request_diagnostics", { requestId: oldRequest }) + const source = join(dir, "local-session.jsonl") + const text = `${[local.getHeader(), ...local.getEntries()].map((entry) => JSON.stringify(entry)).join("\n")}\n` + writeFileSync(source, text) + const copy = copySessionFileAndAddHandoffNote(source, "[Teleport] Moved to a fresh sandbox.") + if (!copy) throw new Error("Expected annotated session copy") + try { + const sessionManager = SessionManager.open(copy) + const ctx = { ...createContext({ cwd: join(dir, "sandbox") }), sessionManager } + const api = createExtensionApi() + api.appendEntry.mockImplementation((type, data) => sessionManager.appendCustomEntry(type, data)) + createWorkAttributionExtension()(api.api) + await api.getHandler("session_start")({ type: "session_start", reason: "resume" }, ctx) + expect(getWorkId(ctx)).toBe(original) + expect(scope.readWorkScope(original)).toBeUndefined() + await api.getHandler("input")( + { type: "input", source: "interactive", text: "Continue implementing the change" }, + ctx, + ) + const current = recordProviderRequest(ctx, ctx.model) + const request = records().find((row) => row.type === "request" && row.requestId === current.requestId) + // Conversation transfer does not carry the original scope or request ledger. + if (explicit) { + expect(current.workId).toBe(original) + expect(request.scope).toBeNull() + } else { + expect(current.workId).not.toBe(original) + expect(request.scope).toEqual(captured.scope) + } + expect(records().some((row) => row.type === "request" && row.requestId === oldRequest)).toBe(false) + expect(readFileSync(source, "utf8")).toBe(text) + await api.getHandler("session_shutdown")({ type: "session_shutdown", reason: "quit" }, ctx) + } finally { + removeTempDir(dirname(copy)) + } + }) + it("ignores malformed copied work metadata and lets the session's ledger win", async () => { const sessionManager = SessionManager.inMemory(dir) sessionManager.appendCustomEntry("work_identity", { workId: "../../not-a-uuid" }) @@ -1127,6 +1220,43 @@ describe("local work attribution", () => { expect(getWorkId(ctx)).toBe(workId) }) + it("opens a work panel in the terminal and prints the chosen work as /work prints it elsewhere", async () => { + const panels: Component[] = [] + const custom = vi.fn().mockImplementation( + (factory: Parameters[0]) => + new Promise((done) => { + void Promise.resolve(factory({} as TUI, testTheme, {} as never, done)).then((panel) => panels.push(panel)) + }), + ) + const ctx = { ...createCommandContext(), ...createContext({ cwd: dir, ui: { custom } }) } + const mock = createExtensionApi() + createWorkAttributionExtension()(mock.api) + mock.api.events.on(WORK_DETAILS_REQUEST_EVENT, (request) => { + ;(request as WorkDetailsRequest).lines.push("PR #7 open: https://github.com/example/repo/pull/7") + }) + const command = mock.getRegisteredCommand("work") + const workId = getWorkId(ctx) + await flushWorkSummaries() + + const browsing = command.handler("", ctx) + await vi.waitFor(() => expect(panels).toHaveLength(1)) + const view = panels[0].render(100).map(stripTerminalSequences) + expect(view[1]).toBe(" Kimchi work · 1 work · no requests") + expect(view).toContain(` Work ID: ${workId}`) + expect(view).toContain(" PR #7 open: https://github.com/example/repo/pull/7") + expect(ctx.ui.notify).not.toHaveBeenCalled() + panels[0].handleInput?.("\r") + await browsing + const printed = vi.mocked(ctx.ui.notify).mock.lastCall + expect(printed?.[0]).toMatch(new RegExp(`^Work ID: ${workId}\nPR #7 open: \\S+\nCost: unknown`)) + + await command.handler("", { ...ctx, mode: "rpc" }) + expect(vi.mocked(ctx.ui.notify).mock.lastCall).toEqual(printed) + await command.handler("new", ctx) + expect(custom).toHaveBeenCalledOnce() + expect(vi.mocked(ctx.ui.notify).mock.lastCall?.[0]).toMatch(/^Work ID: /) + expect(getWorkId(ctx)).not.toBe(workId) + }) it("persists plan identity and explicitly continues it in another session", async () => { const parent = createContext({ cwd: dir }) const workId = getWorkId(parent) diff --git a/src/extensions/work-attribution.ts b/src/extensions/work-attribution.ts index 4462fa614..0f5a9fd9c 100644 --- a/src/extensions/work-attribution.ts +++ b/src/extensions/work-attribution.ts @@ -29,18 +29,21 @@ import { isResourceEnabled } from "../resources/store.js" import { readPlanWorkId, savePlanMarkdown, UNRETAINED_PLAN_NOTICE } from "../shared/planning/plan-markdown.js" import { isWorkId } from "../shared/work-id.js" import { isHarnessSteer } from "./steer-marker.js" +import { trackPRCostMetric } from "./telemetry/pr-cost.js" import { type BillingSource, captureBillingSource, requestTagSelector } from "./work-attribution/billing-source.js" +import { readWorkBrowser } from "./work-attribution/browser.js" +import { WorkBrowserPanel } from "./work-attribution/browser-panel.js" import { createCommitTrackingBashTool } from "./work-attribution/commits.js" -import { - findWorkContinuation, - hasOwnedWorkReference, - hasWorkReference, - type WorkContinuation, -} from "./work-attribution/continuation.js" +import { findWorkContinuation, hasWorkReference, type WorkContinuation } from "./work-attribution/continuation.js" import { workCostDetails } from "./work-attribution/cost-details.js" import { debugWorkAttribution } from "./work-attribution/diagnostics.js" import { createTrackedEditTool, createTrackedWriteTool } from "./work-attribution/file-transitions.js" -import { confirmWorkContinuation, correctWorkLink } from "./work-attribution/links.js" +import { + confirmWorkContinuation, + correctWorkLink, + pinWorkContinuation, + readContinuationHistory, +} from "./work-attribution/links.js" import { subscribeCostReconciliation, subscribeFileReconciliation } from "./work-attribution/reconcile-supervisor.js" import { prepareBillingTag } from "./work-attribution/request-tags.js" import { COST_PER_PR_RESOURCE_ID } from "./work-attribution/resource.js" @@ -65,7 +68,6 @@ import { flushWorkSummaries, markNewWork, object, - readWorkRecords, recoverWorkSummaries, updateWorkSummary, } from "./work-attribution/summary.js" @@ -187,8 +189,10 @@ export function appendWorkRecord( const size = fstatSync(fd).size const last = Buffer.alloc(1) if (size) readSync(fd, last, 0, 1, size - 1) - const prefix = size && last[0] !== 10 ? "\n" : "" - writeFileSync(fd, `${prefix}${JSON.stringify(record)}\n`) + // An unterminated tail is an interrupted append. Start a new line instead of truncating: + // another process may be appending, and readers skip only that cut-off record. + const repair = size && last[0] !== 10 ? "\n" : "" + writeFileSync(fd, `${repair}${JSON.stringify(record)}\n`) fsyncSync(fd) } finally { closeSync(fd) @@ -638,6 +642,7 @@ export function createWorkAttributionExtension( if (event.source === "extension") return // Optional model matching never interrupts the user; its failures leave the input unresolved. let matching = false + let matchingOutcome: "failed" | "limited" | undefined try { bind(ctx) const key = workLedgerPath(ctx) @@ -712,27 +717,48 @@ export function createWorkAttributionExtension( workLedgerPath(ctx) === key && getWorkId(ctx) === current && explicitSelection.get(key) === explicitReason - const records = referencesWork ? readWorkRecords(getAgentDir()) : undefined - const found = - captured && (eligible() || referencesWork) - ? await findWorkContinuation(pinWorkContext(ctx), event.text, captured) + // Ownership needs no scope: an input naming recorded work stays unresolved even without one. + const references = + referencesWork || (captured && eligible()) + ? await findWorkContinuation(pinWorkContext(ctx), event.text, captured, current) : undefined if (!unchanged()) return - if (found && (found.workId === current || eligible())) { + const found = references?.match + if (found && captured && (found.workId === current || eligible())) { + const accepted = { ...found, evidence: { ...found.evidence, segmentId, ...captured.scope } } + // One history read serves the pin and the confirmation. If it fails, the receipt stays unpinned and + // the confirmation reads again to report why. + const history = tryWorkAttribution(() => readContinuationHistory(getAgentDir())) + const pinned = (history && tryWorkAttribution(() => pinWorkContinuation(accepted, history))) ?? accepted + const continuation = { + source: pinned.source, + evidence: pinned.evidence, + } if (found.workId !== current) { - setWorkId(ctx, found.workId, pi, { source: found.source, evidence: found.evidence }) + setWorkId(ctx, found.workId, pi, continuation) notify( ctx, `Continuing the saved ${found.source === "named-artifact" ? "artifact" : "plan"}'s work: ${found.workId}`, ) - } + } else appendWorkRecord(ctx, { type: "work", continuation }, current) explicitSelection.set(key, found.source) useSegment("explicit", found.source) - if (captured) confirmWorkContinuation(ctx, found, captured.scope, records) + // Appends since the read are this input's receipt and segment, which confirmation does not use. + // Unreadable history blocks only the confirmation; the adopted work still remembers this input. + try { + confirmWorkContinuation( + ctx, + { ...found, ...continuation }, + captured.scope, + history ?? readContinuationHistory(getAgentDir()), + ) + } catch (error) { + warnWorkAttribution(ctx, error) + } if (model) await rememberWorkIntent(ctx.cwd, found.workId, event.text) return } - if (found || (records && hasOwnedWorkReference(ctx, event.text, records, current))) { + if (references?.owned) { // A reference this session's continuation cannot account for may start another task. explicitSelection.delete(key) useSegment("unknown", "unresolved-reference") @@ -802,9 +828,21 @@ export function createWorkAttributionExtension( ) await rememberWorkIntent(ctx.cwd, workId, event.text, intents.repository, intents.account) } catch (error) { - if (error instanceof WorkMatchingLimit) matchingLimit = error.message - if (matching || error instanceof WorkMatchingLimit) debugWorkAttribution("Work matching skipped:", error) - else warnWorkAttribution(ctx, error) + if (error instanceof WorkMatchingLimit) { + matchingLimit = error.message + matchingOutcome = "limited" + debugWorkAttribution("Work matching skipped:", error) + } else { + matchingOutcome = "failed" + if (matching) debugWorkAttribution("Work matching skipped:", error) + else warnWorkAttribution(ctx, error) + } + } finally { + if (generation === inputGeneration) + trackPRCostMetric({ + kind: "matching", + outcome: matchingOutcome ?? getWorkSegment(ctx)?.attribution ?? "unknown", + }) } } pi.on("before_provider_headers", (event, ctx) => { @@ -931,10 +969,23 @@ export function createWorkAttributionExtension( } const details: WorkDetailsRequest = { workId: getWorkId(ctx), lines: [] } pi.events.emit(WORK_DETAILS_REQUEST_EVENT, details) - if (matchingLimit && workMatchingEnabled()) - details.lines.push( - `Work matching stopped: ${matchingLimit}. New inputs stay unresolved in the current work.`, + const stopped = + matchingLimit && workMatchingEnabled() + ? `Work matching stopped: ${matchingLimit}. New inputs stay unresolved in the current work.` + : undefined + if (stopped) details.lines.push(stopped) + // The terminal browses recent works; other modes and every subcommand print text. + if (!value && ctx.mode === "tui") { + // The browser lists works, so say here that matching stopped. + if (stopped) notify(ctx, stopped) + const browser = await readWorkBrowser(getAgentDir(), details) + const selected = await ctx.ui.custom( + (_tui, theme, _keybindings, done) => new WorkBrowserPanel(browser, theme, done), ) + const row = browser.rows.find((candidate) => candidate.workId === selected) + if (row) notify(ctx, row.details) + return + } details.lines.push(...workCostDetails(getAgentDir(), details.workId)) notify(ctx, [`Work ID: ${details.workId}`, ...details.lines].join("\n")) } catch (error) { diff --git a/src/extensions/work-attribution/browser-panel.test.ts b/src/extensions/work-attribution/browser-panel.test.ts new file mode 100644 index 000000000..4f913dbdc --- /dev/null +++ b/src/extensions/work-attribution/browser-panel.test.ts @@ -0,0 +1,103 @@ +import { stripTerminalSequences, visibleWidth } from "@earendil-works/pi-tui" +import { describe, expect, it, vi } from "vitest" +import { lightTestTheme, testTheme } from "../__mocks__/theme.js" +import type { WorkBrowser } from "./browser.js" +import { WorkBrowserPanel } from "./browser-panel.js" + +const DOWN = "\x1b[B" +const ENTER = "\r" +const ESCAPE = "\x1b" +const browser: WorkBrowser = { + spend: "$0.0746 known so far", + rows: [ + { + workId: "a7c533fc-b51c-4af3-8446-0f38573c8b50", + label: "● a7c533fc Add CSV export with quoted fields", + value: "$0.0623 · PR #731 open", + description: + "Work ID: a7c533fc-b51c-4af3-8446-0f38573c8b50\nPR #731 open: https://github.com/example/kimchi/pull/731", + details: "Work ID: a7c533fc-b51c-4af3-8446-0f38573c8b50", + }, + { + workId: "9f8e7d6c-1111-4111-8111-111111111111", + label: " 9f8e7d6c kimchi · feat/browse-work-costs", + value: "$0.0123 known so far · 2 PRs", + description: "Work ID: 9f8e7d6c-1111-4111-8111-111111111111", + details: "Work ID: 9f8e7d6c-1111-4111-8111-111111111111", + }, + { + workId: "1b2c3d4e-2222-4222-8222-222222222222", + label: " 1b2c3d4e app", + value: "cost unknown · no PR", + description: "Work ID: 1b2c3d4e-2222-4222-8222-222222222222", + details: "Work ID: 1b2c3d4e-2222-4222-8222-222222222222", + }, + ], +} +const plain = (lines: string[]) => lines.map(stripTerminalSequences) + +describe("WorkBrowserPanel", () => { + it("keeps every value whole and aligned in wide and narrow terminals", () => { + for (const width of [150, 100, 60]) { + const lines = new WorkBrowserPanel(browser, testTheme, vi.fn()).render(width) + const text = plain(lines) + + for (const line of lines) expect(visibleWidth(line), `${width}: ${line}`).toBeLessThanOrEqual(width) + expect(text[1]).toBe(" Kimchi work · 3 works · $0.0746 known so far") + expect(text[2]).toBe(" Current: ● a7c533fc Add CSV export with quoted fields") + const columns = browser.rows.map(({ value }) => text.find((line) => line.endsWith(value))?.indexOf(value)) + expect( + columns.every((column) => column !== undefined && column === columns[0]), + `${width}`, + ).toBe(true) + expect(text.at(-2)).toBe(" ↑↓ select · Enter print details · Esc close") + expect(text.join("\n")).not.toContain("Enter/Space to change") + } + expect(plain(new WorkBrowserPanel(browser, testTheme, vi.fn()).render(150))).toContain( + "→ ● a7c533fc Add CSV export with quoted fields $0.0623 · PR #731 open", + ) + expect(plain(new WorkBrowserPanel(browser, testTheme, vi.fn()).render(60))).toContain( + "→ ● a7c533fc Add CSV export… $0.0623 · PR #731 open", + ) + }) + + it("shows the selected work's details and returns its ID on Enter", () => { + const done = vi.fn() + const panel = new WorkBrowserPanel(browser, testTheme, done) + + expect(plain(panel.render(100))).toContain(" PR #731 open: https://github.com/example/kimchi/pull/731") + panel.handleInput(DOWN) + const text = plain(panel.render(100)) + expect(text).toContain("→ 9f8e7d6c kimchi · feat/browse-work-costs $0.0123 known so far · 2 PRs") + expect(text).toContain(" Work ID: 9f8e7d6c-1111-4111-8111-111111111111") + expect(text.join("\n")).not.toContain("pull/731") + panel.handleInput(ENTER) + + expect(done).toHaveBeenCalledExactlyOnceWith(browser.rows[1].workId) + }) + + it("closes without a work on Escape", () => { + const done = vi.fn() + new WorkBrowserPanel(browser, testTheme, done).handleInput(ESCAPE) + + expect(done).toHaveBeenCalledExactlyOnceWith() + }) + + it("stays within tiny terminals", () => { + const panel = new WorkBrowserPanel(browser, testTheme, vi.fn()) + for (const width of [0, 1, 8, 20]) + for (const line of panel.render(width)) expect(visibleWidth(line)).toBeLessThanOrEqual(Math.max(1, width)) + }) + + it("takes its colors from the light or dark theme it is given", () => { + for (const theme of [testTheme, lightTestTheme]) { + const lines = new WorkBrowserPanel(browser, theme, vi.fn()).render(100) + + expect(lines[0]).toContain(theme.getFgAnsi("accent")) + expect(lines.find((line) => stripTerminalSequences(line).startsWith(" Work ID:"))).toContain( + theme.getFgAnsi("text"), + ) + } + expect(lightTestTheme.getFgAnsi("text")).not.toBe(testTheme.getFgAnsi("text")) + }) +}) diff --git a/src/extensions/work-attribution/browser-panel.ts b/src/extensions/work-attribution/browser-panel.ts new file mode 100644 index 000000000..529eccc3f --- /dev/null +++ b/src/extensions/work-attribution/browser-panel.ts @@ -0,0 +1,99 @@ +import type { Theme } from "@earendil-works/pi-coding-agent" +import { + type Component, + type SettingItem, + SettingsList, + type SettingsListTheme, + truncateToWidth, + visibleWidth, +} from "@earendil-works/pi-tui" +import type { WorkBrowser } from "./browser.js" + +const MAX_VISIBLE = 10 +const MIN_LABEL_WIDTH = 12 +const MAX_LABEL_WIDTH = 64 + +/** Settings-style list of works: Enter resolves with the selected work ID, Esc with none. */ +export class WorkBrowserPanel implements Component { + private readonly items: SettingItem[] + private readonly list: SettingsList + private labelsWidth = 0 + + constructor( + private readonly browser: WorkBrowser, + private readonly theme: Theme, + done: (workId?: string) => void, + ) { + // A single value makes Enter report the row instead of changing a setting. + this.items = browser.rows.map((row) => ({ + id: row.workId, + label: row.label, + description: row.description, + currentValue: row.value, + values: [row.value], + })) + this.list = new SettingsList( + this.items, + MAX_VISIBLE, + listTheme(theme), + (workId) => done(workId), + () => done(), + ) + } + + render(width: number): string[] { + const theme = this.theme + const { rows, spend } = this.browser + const lineWidth = Math.max(1, width) + this.fitLabels(lineWidth) + const separator = theme.fg("dim", " · ") + const rule = theme.fg("accent", "─".repeat(lineWidth)) + const works = `${rows.length} work${rows.length === 1 ? "" : "s"}` + return [ + rule, + ` ${theme.bold(theme.fg("text", "Kimchi work"))}${separator}${theme.fg("text", works)}${separator}${theme.fg("text", spend)}`, + ` ${theme.fg("dim", "Current:")} ${theme.fg("text", rows[0].label)}`, + "", + // SettingsList ends with its own "Enter/Space to change" hint; this panel prints details instead. + ...this.list.render(lineWidth).slice(0, -2), + "", + theme.fg("dim", " ↑↓ select · Enter print details · Esc close"), + rule, + ].map((line) => truncateToWidth(line, lineWidth, "…")) + } + + handleInput(data: string): void { + this.list.handleInput(data) + } + + invalidate(): void { + this.list.invalidate() + } + + /** + * SettingsList aligns labels in a column of at most 36 cells and cuts values to fit. Equal-width labels + * keep values aligned in a wider column too, and labels give way first so a PR number is never cut. + */ + private fitLabels(width: number): void { + if (width === this.labelsWidth) return + this.labelsWidth = width + const { rows } = this.browser + const valueWidth = Math.max(...rows.map((row) => visibleWidth(row.value))) + const longest = Math.max(...rows.map((row) => visibleWidth(row.label))) + // Around the label column the list draws a 2-cell cursor, a 2-cell gap and a 2-cell margin. + const labelWidth = Math.max(MIN_LABEL_WIDTH, Math.min(MAX_LABEL_WIDTH, longest, width - valueWidth - 6)) + for (const [index, item] of this.items.entries()) + item.label = truncateToWidth(rows[index].label, labelWidth, "…", true) + } +} + +/** Uses the theme passed to the panel, with readable text for the values and details. */ +function listTheme(theme: Theme): SettingsListTheme { + return { + label: (text, selected) => (selected ? theme.fg("accent", text) : text), + value: (text, selected) => theme.fg(selected ? "accent" : "text", text), + description: (text) => theme.fg("text", text), + cursor: theme.fg("accent", "→ "), + hint: (text) => theme.fg("dim", text), + } +} diff --git a/src/extensions/work-attribution/browser.test.ts b/src/extensions/work-attribution/browser.test.ts new file mode 100644 index 000000000..68295c6ac --- /dev/null +++ b/src/extensions/work-attribution/browser.test.ts @@ -0,0 +1,572 @@ +import * as fs from "node:fs" +import { mkdirSync, mkdtempSync, rmSync, utimesSync, writeFileSync } from "node:fs" +import * as asyncFs from "node:fs/promises" +import { tmpdir } from "node:os" +import { join } from "node:path" +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest" +import { saveWorkSummary } from "../__mocks__/work-summary.js" +import { buildWorkBrowser, readWorkBrowser, type SavedWork } from "./browser.js" +import { type WorkCostReport, workCostDetails, workCostTotals } from "./cost-details.js" +import type { PullRequestCost } from "./costs.js" +import { type Collection, type Row, readWorkHead, type WorkHead } from "./row-log.js" + +vi.mock("node:fs/promises", async (importOriginal) => ({ ...(await importOriginal()) })) + +const NOW = Date.parse("2026-10-08T12:00:00Z") +const HOUR = 60 * 60_000 +const DAY = 24 * HOUR +let agentDir: string + +beforeEach(() => { + agentDir = mkdtempSync(join(tmpdir(), "kimchi-work-browser-")) + vi.stubEnv("TZ", "UTC") +}) +afterEach(() => { + vi.restoreAllMocks() + vi.unstubAllEnvs() + rmSync(agentDir, { recursive: true, force: true }) +}) + +function workId(index: number): string { + return `${String(index).padStart(8, "0")}-0000-4000-8000-000000000000` +} +type Summary = Parameters[1] +function summary(id: string, rows: Partial> = {}): Summary { + return { workId: id, sessions: ["session"], ...rows } +} +function request(requestId: string, startedAt = "2026-10-08T09:00:00Z") { + return { requestId, sessionId: "session", startedAt, cwd: "/src/kimchi", scope: { repository: "/src/kimchi/.git" } } +} +function costs( + id: string, + requests: { requestId: string; usd?: string; priced?: boolean; workIds?: string[]; linkedWorkIds?: string[] }[], + pullRequests: PullRequestCost[] = [], +): WorkCostReport { + return { + workId: id, + pullRequests, + requests: requests.map(({ requestId, usd = "0", priced = true, workIds = [id], linkedWorkIds }) => ({ + requestId, + account: null, + workIds, + ...(linkedWorkIds ? { linkedWorkIds } : {}), + sessionIds: ["session"], + startedAt: null, + pullRequestIds: [], + allocation: "unlinked", + billingRecordIds: [], + priceStatus: priced ? "priced" : "missing", + knownCostUsd: usd, + totalCostUsd: priced ? usd : null, + })), + } +} +/** A saved PR total; request lists and portions not given are empty. */ +function pullRequestCost( + row: Pick & + Partial, +): PullRequestCost { + const empty = { requestIds: [], knownCostUsd: "0.000000000", totalCostUsd: "0.000000000" } + return { + account: null, + requestIds: [], + explicit: empty, + inferred: empty, + sharedRequestIds: [], + inferredRequestIds: [], + unknownRequestIds: [], + ...row, + } +} +function pullRequest(number: number, state: "open" | "closed" | "merged", provider: "github" | "gitlab" = "github") { + const finished = state === "open" ? null : "2026-10-07T10:00:00Z" + const path = + provider === "github" ? `example/kimchi/pull/${number}` : `example/team/kimchi/-/merge_requests/${number}` + return { + provider, + host: `${provider}.com`, + repository: provider === "github" ? "example/kimchi" : "example/team/kimchi", + number, + url: `https://${provider}.com/${path}`, + state, + headSha: "a".repeat(40), + mergeCommitSha: state === "merged" ? "b".repeat(40) : null, + mergedAt: state === "merged" ? finished : null, + closedAt: finished, + checkedAt: "2026-10-07T11:00:00Z", + } +} +function commit(sha: string, pullRequests: unknown[]) { + return { + sha: sha.repeat(40), + repository: "/src/kimchi/.git", + worktree: "/src/kimchi", + sessionId: "session", + recordedAt: "2026-10-07T09:00:00Z", + pullRequests, + } +} +/** Saves a work's files; a summary given as text is written to work.json as is. */ +async function save( + id: string, + files: { summary?: Summary | string; costs?: WorkCostReport; totals?: string; plans?: Record }, + at = NOW, +) { + const folder = join(agentDir, "work", id) + mkdirSync(join(folder, "plans"), { recursive: true }) + if (typeof files.summary === "string") writeFileSync(join(folder, "work.json"), files.summary) + else if (files.summary) await saveWorkSummary(agentDir, files.summary) + const totals = files.totals ?? (files.costs === undefined ? undefined : JSON.stringify(workCostTotals(files.costs))) + for (const [name, value] of [ + ["costs.json", files.costs], + ["cost-totals.json", totals], + ] as const) + if (value !== undefined) + writeFileSync(join(folder, name), typeof value === "string" ? value : JSON.stringify(value)) + for (const [name, text] of Object.entries(files.plans ?? {})) writeFileSync(join(folder, "plans", name), text) + if (files.summary !== undefined) utimesSync(join(folder, "work.json"), new Date(at), new Date(at)) +} +async function browse(works: { workId: string; summary?: Summary; costs?: WorkCostReport }[], lines: string[] = []) { + const saved: SavedWork[] = [] + for (const work of works) { + if (work.summary) await saveWorkSummary(agentDir, work.summary) + saved.push({ + workId: work.workId, + head: await readWorkHead(agentDir, work.workId), + totals: work.costs && workCostTotals(work.costs), + }) + } + return buildWorkBrowser(agentDir, { workId: works[0].workId, lines }, saved) +} + +describe("work browser rows", () => { + it("lists the current work first, then recent works by last activity, and leaves out old works", async () => { + const [current, older, newer, stale, idle] = [1, 2, 3, 4, 5].map(workId) + await save(current, { summary: summary(current, { requests: [request("r1", "2026-09-20T09:00:00Z")] }) }, NOW - DAY) + // A later background update does not make older work more recent than newer activity. + await save(older, { summary: summary(older, { requests: [request("r2", "2026-10-07T10:00:00Z")] }) }, NOW - HOUR) + await save( + newer, + { summary: summary(newer, { requests: [request("r3", "2026-10-08T09:00:00Z")] }) }, + NOW - 2 * HOUR, + ) + await save( + stale, + { summary: summary(stale, { requests: [request("r4", "2026-08-20T09:00:00Z")] }) }, + NOW - 40 * DAY, + ) + // A request's final cost record rewrites the summary of work idle for over a month. + await save(idle, { summary: summary(idle, { requests: [request("r5", "2026-08-19T09:00:00Z")] }) }, NOW - HOUR) + mkdirSync(join(agentDir, "work", "not-a-work")) + + const { rows } = await readWorkBrowser(agentDir, { workId: current, lines: [] }, NOW) + + expect(rows.map((row) => row.workId)).toEqual([current, newer, older]) + expect(rows[0].label).toBe("● 00000001 kimchi") + expect(rows[1].label).toBe(" 00000003 kimchi") + }) + + it("reads at most 30 works", async () => { + for (let index = 1; index <= 35; index++) + await save(workId(index), { summary: summary(workId(index)) }, NOW - index * HOUR) + + const { rows } = await readWorkBrowser(agentDir, { workId: workId(35), lines: [] }, NOW) + + expect(rows).toHaveLength(30) + expect(rows[0].workId).toBe(workId(35)) + expect(rows.map((row) => row.workId)).toContain(workId(29)) + expect(rows.map((row) => row.workId)).not.toContain(workId(30)) + }) + + it("keeps the most recently active works when more works changed recently", async () => { + const current = workId(1) + await save(current, { summary: summary(current) }) + // Background cost and PR updates rewrote these summaries after their last activity. + for (let index = 2; index <= 30; index++) + await save( + workId(index), + { summary: summary(workId(index), { requests: [request(`r${index}`, "2026-09-28T09:00:00Z")] }) }, + NOW - HOUR, + ) + const active = workId(31) + await save( + active, + { summary: summary(active, { requests: [request("active", "2026-10-07T09:00:00Z")] }) }, + NOW - DAY, + ) + + const { rows } = await readWorkBrowser(agentDir, { workId: current, lines: [] }, NOW) + + expect(rows).toHaveLength(30) + expect(rows[1].workId).toBe(active) + expect(rows.map((row) => row.workId)).not.toContain(workId(30)) + }) + + it("shows priced spend and never shows unpriced spend as $0", async () => { + const ids = [1, 2, 3, 4, 5, 6].map(workId) + const { rows, spend } = await browse([ + { + workId: ids[0], + summary: summary(ids[0], { requests: [request("a1"), request("a2")] }), + costs: costs(ids[0], [ + { requestId: "a1", usd: "0.031133200" }, + { requestId: "a2", usd: "0.031133200" }, + ]), + }, + { + workId: ids[1], + summary: summary(ids[1], { requests: [request("b1"), request("b2")] }), + costs: costs(ids[1], [ + { requestId: "b1", usd: "0.012300000" }, + { requestId: "b2", priced: false }, + ]), + }, + { + workId: ids[2], + summary: summary(ids[2], { requests: [request("c1")] }), + costs: costs(ids[2], [{ requestId: "c1", priced: false }]), + }, + // Requests recorded after the last cost pass are not priced yet. + { + workId: ids[3], + summary: summary(ids[3], { requests: [request("d1"), request("d2")] }), + costs: costs(ids[3], [{ requestId: "d1", usd: "1.234567890" }]), + }, + { workId: ids[4], summary: summary(ids[4]) }, + { + workId: ids[5], + summary: summary(ids[5], { requests: [request("f1")] }), + costs: costs(ids[5], [{ requestId: "f1" }]), + }, + ]) + + expect(Object.fromEntries(rows.map((row) => [row.workId, row.value]))).toEqual({ + [ids[0]]: "$0.0623 · no PR", + [ids[1]]: "$0.0123 known so far · no PR", + [ids[2]]: "cost unknown · no PR", + [ids[3]]: "$1.235 known so far · no PR", + [ids[4]]: "no requests · no PR", + // A billed zero is a known price. + [ids[5]]: "$0.00 · no PR", + }) + expect(spend).toBe("$1.309 known so far") + expect(rows.find((row) => row.workId === ids[1])?.description).toContain( + "Prices: 1/2 requests priced, $0.0123 known so far.", + ) + }) + + it("shows each connected work's own spend and counts their shared report once", async () => { + const [first, second] = [1, 2].map(workId) + // The cost pass saves the whole connected group in each member's costs.json. + const group = (id: string) => + costs( + id, + [ + { requestId: "own", usd: "0.500000000", workIds: [first] }, + // A correction linked this planning request into the second work. + { requestId: "linked", usd: "0.250000000", workIds: [first], linkedWorkIds: [second] }, + { requestId: "other", usd: "0.125000000", workIds: [second] }, + ], + [ + pullRequestCost({ + key: '["github","github.com","7"]', + pullRequest: pullRequest(7, "merged"), + workIds: [first, second], + requestIds: ["own", "linked", "other"], + knownCostUsd: "0.875000000", + totalCostUsd: "0.875000000", + explicit: { + requestIds: ["own", "linked", "other"], + knownCostUsd: "0.875000000", + totalCostUsd: "0.875000000", + }, + }), + ], + ) + const { rows, spend } = await browse([ + { + workId: first, + summary: summary(first, { requests: [request("own"), request("linked")] }), + costs: group(first), + }, + { workId: second, summary: summary(second, { requests: [request("other")] }), costs: group(second) }, + ]) + const row = (id: string) => rows.find((candidate) => candidate.workId === id) + + expect(row(first)?.value).toBe("$0.75 · no PR") + expect(row(second)?.value).toBe("$0.375 · no PR") + expect(row(first)?.description).toContain("Prices: 2/2 requests priced, $0.75.") + expect(row(second)?.description).toContain("Prices: 2/2 requests priced, $0.375.") + // The PR's total covers every contributing work. + expect(row(second)?.description).toContain("Cost: $0.875 — https://github.com/example/kimchi/pull/7") + // The header counts requests the group shares once, as deduplicating by request ID did. + expect(spend).toBe("$0.875") + }) + + it("shows each work's own spend while the saved reports repeat their connected works", async () => { + const [first, second] = [1, 2].map(workId) + // The cost pass saves the whole connected group in each member's costs.json. + const group = [ + { requestId: "planning", usd: "0.500000000", workIds: [first] }, + { requestId: "linked", usd: "0.250000000", workIds: [first], linkedWorkIds: [second] }, + { requestId: "implementation", usd: "0.125000000", workIds: [second] }, + ] + const { rows, spend } = await browse([ + { + workId: first, + summary: summary(first, { requests: [request("planning"), request("linked")] }), + costs: costs(first, group), + }, + { + workId: second, + summary: summary(second, { requests: [request("implementation")] }), + costs: costs(second, group), + }, + ]) + + expect(rows.map((row) => row.value.split(" · ")[0])).toEqual(["$0.75", "$0.375"]) + expect(spend).toBe("$0.875") + }) + + it("counts a listed work's whole connected report in the header, unlisted works included", async () => { + const [listed, older] = [1, 2].map(workId) + const { rows, spend } = await browse([ + { + workId: listed, + summary: summary(listed, { requests: [request("listed")] }), + costs: costs(listed, [ + { requestId: "listed", usd: "0.500000000", workIds: [listed] }, + // A connected work too old to be listed. + { requestId: "older", usd: "0.125000000", workIds: [older] }, + ]), + }, + ]) + + expect(rows[0].value).toBe("$0.50 · no PR") + expect(spend).toBe("$0.625") + }) + + it("names each work's PR state from its saved links", async () => { + const ids = [1, 2, 3, 4, 5].map(workId) + const { rows } = await browse( + [ + { workId: ids[0], summary: summary(ids[0], { commits: [commit("1", [pullRequest(731, "open")])] }) }, + { workId: ids[1], summary: summary(ids[1], { commits: [commit("2", [])] }) }, + // A later observation of the same PR replaces the earlier one. + { + workId: ids[2], + summary: summary(ids[2], { + commits: [ + commit("3", [{ ...pullRequest(7, "open"), checkedAt: "2026-10-06T11:00:00Z" }]), + commit("4", [pullRequest(7, "merged")]), + ], + }), + }, + { + workId: ids[3], + summary: summary(ids[3], { + requests: [request("d1"), request("d2")], + commits: [commit("5", [pullRequest(8, "merged"), pullRequest(9, "closed")])], + }), + costs: costs(ids[3], [{ requestId: "d1", usd: "0.012300000" }]), + }, + { workId: ids[4], summary: summary(ids[4], { commits: [commit("6", [pullRequest(12, "open", "gitlab")])] }) }, + ], + ["PR/MR lookup: 1 commit waiting"], + ) + const value = (id: string) => rows.find((row) => row.workId === id)?.value + + expect(value(ids[0])).toBe("no requests · PR #731 open") + expect(value(ids[1])).toBe("no requests · no PR") + expect(value(ids[2])).toBe("no requests · PR #7 merged") + expect(value(ids[3])).toBe("$0.0123 known so far · 2 PRs") + expect(value(ids[4])).toBe("no requests · MR !12 open") + // The current work shows the PR extension's live lines; others show their saved links. + expect(rows[0].details).toContain("PR/MR lookup: 1 commit waiting") + expect(rows[0].details).not.toContain("pull/731") + expect(rows.find((row) => row.workId === ids[3])?.details).toContain( + "PR #8 merged: https://github.com/example/kimchi/pull/8\nPR #9 closed: https://github.com/example/kimchi/pull/9", + ) + }) + + it("labels work by its saved plan title, else by repository and branch", async () => { + const [planned, edited, asked] = [1, 2, 3].map(workId) + const planRecord = { + sessionId: "session", + path: "/src/kimchi/.kimchi/plans/csv.md", + snapshotPath: `/old/agent/work/${planned}/plans/csv-abc.md`, + recordedAt: "2026-10-08T08:00:00Z", + } + await save(planned, { + summary: summary(planned, { plans: [planRecord] }), + plans: { "csv-abc.md": `\n# Add CSV\u001b[31m export\n\nSteps\n` }, + }) + await save(edited, { + summary: summary(edited, { + requests: [request("e1")], + fileTransitions: [ + { + sessionId: "session", + transitionId: "t1", + repository: "/src/kimchi/.git", + worktree: "/src/kimchi-csv", + branch: "main", + recordedAt: "2026-10-07T08:00:00Z", + }, + { + sessionId: "session", + transitionId: "t2", + repository: "/src/kimchi/.git", + worktree: "/src/kimchi-csv", + branch: "feat/csv", + recordedAt: "2026-10-08T08:00:00Z", + }, + ], + }), + }) + await save(asked, { + summary: summary(asked, { requests: [{ ...request("a1"), scope: { repository: "/srv/app.git" } }] }), + }) + + const { rows } = await readWorkBrowser(agentDir, { workId: planned, lines: [] }, NOW) + const label = (id: string) => rows.find((row) => row.workId === id)?.label + + expect(label(planned)).toBe("● 00000001 Add CSV export") + expect(label(edited)).toBe(" 00000002 kimchi · feat/csv") + expect(label(asked)).toBe(" 00000003 app") + }) + + it("shows a missing summary and damaged totals as unknown, and deletes a damaged work.json", async () => { + const [missing, damaged, wrongCosts] = [1, 2, 3].map(workId) + await save(damaged, { summary: "{", costs: costs(damaged, [{ requestId: "x1", usd: "0.100000000" }]) }) + await save(wrongCosts, { + summary: summary(wrongCosts, { + requests: [request("y1")], + plans: [{ sessionId: "session", path: "/gone.md", snapshotPath: "/gone/plans/missing.md" }], + }), + totals: '{"requests":"not a list"}', + }) + + const { rows, spend } = await readWorkBrowser(agentDir, { workId: missing, lines: [] }, NOW) + const row = (id: string) => rows.find((candidate) => candidate.workId === id) + + // The damaged work has no summary now and is listed again once an update rebuilds it; its costs stay. + expect(rows.map((candidate) => candidate.workId)).toEqual([missing, wrongCosts]) + expect(fs.readdirSync(join(agentDir, "work", damaged)).sort()).toEqual(["cost-totals.json", "costs.json", "plans"]) + expect(row(missing)).toMatchObject({ label: "● 00000001 summary unavailable", value: "cost unknown · no PR" }) + expect(row(missing)?.description).toBe( + [ + `Work ID: ${missing}`, + `Summary unavailable: ${join(agentDir, "work", missing, "work.json")}`, + "Cost: unknown; waiting for billing", + ].join("\n"), + ) + expect(row(wrongCosts)).toMatchObject({ label: " 00000003 kimchi", value: "cost unknown · no PR" }) + expect(row(wrongCosts)?.description).toContain("Cost: unknown; waiting for billing") + expect(spend).toBe("cost unknown") + }) + + it("opens 30 long works from their manifests and cost totals alone", async () => { + const ids = Array.from({ length: 30 }, (_, index) => workId(index + 1)) + for (const [index, id] of ids.entries()) { + const folder = join(agentDir, "work", id) + mkdirSync(join(folder, "rows"), { recursive: true }) + const manifest: WorkHead = { + version: 2, + workId: id, + updatedAt: new Date(NOW).toISOString(), + logs: { requests: { generation: 3, bytes: 33_441_207, rows: 15_000 } }, + latest: { + activityAt: new Date(NOW - index * HOUR).toISOString(), + request: { startedAt: new Date(NOW - index * HOUR).toISOString(), repository: "/src/kimchi/.git" }, + branch: { recordedAt: "2026-10-08T08:00:00Z", branch: "feat/search" }, + }, + pullRequests: [pullRequest(700 + index, "open")], + } + writeFileSync(join(folder, "work.json"), JSON.stringify(manifest)) + // The logs and the full report are large; the panel must not open them. + writeFileSync(join(folder, "rows", "requests.3.jsonl"), "") + writeFileSync(join(folder, "costs.json"), "") + const priced = Array.from({ length: 14_990 }, (_, at) => ({ requestId: `${id}-${at}`, usd: "0.001000000" })) + writeFileSync(join(folder, "cost-totals.json"), JSON.stringify(workCostTotals(costs(id, priced)))) + } + const reads = vi.spyOn(asyncFs, "readFile") + const opened = vi.spyOn(asyncFs, "open") + + const { rows, spend } = await readWorkBrowser(agentDir, { workId: ids[0], lines: [] }, NOW) + + expect(rows).toHaveLength(30) + expect(rows[0].value).toBe("$14.99 known so far · PR #700 open") + expect(rows[0].description).toContain("Repository: kimchi · branch feat/search") + expect(rows[0].description).toContain("· 15000 requests") + expect(spend).toBe("$449.70 known so far") + const files = reads.mock.calls.map(([path]) => String(path)) + expect(files.every((path) => path.endsWith("work.json") || path.endsWith("cost-totals.json"))).toBe(true) + expect(opened).not.toHaveBeenCalled() + }) + + it("describes a work with the details /work prints plus where and when it ran", async () => { + const id = workId(1) + await save(id, { + summary: summary(id, { + requests: [request("r1", "2026-10-08T08:00:00Z"), request("r2", "2026-10-08T09:30:00Z")], + fileTransitions: [ + { + sessionId: "session", + transitionId: "t1", + repository: "/src/kimchi/.git", + branch: "feat/csv", + recordedAt: "2026-10-08T09:00:00Z", + }, + ], + }), + costs: costs(id, [{ requestId: "r1", usd: "0.031133200" }]), + }) + const lines = ["PR #7 open: https://github.com/example/kimchi/pull/7"] + + const { rows } = await readWorkBrowser(agentDir, { workId: id, lines }, NOW) + const printed = [`Work ID: ${id}`, ...lines, ...workCostDetails(agentDir, id)] + + expect(rows[0].details).toBe(printed.join("\n")) + expect(rows[0].description).toBe( + [ + printed[0], + "Repository: kimchi · branch feat/csv", + "Last activity: 2026-10-08 09:30 · 2 requests", + ...printed.slice(1), + ].join("\n"), + ) + }) + + it("explains unknown prices with the work's own untagged requests and failed refreshes only", async () => { + const [first, second] = [1, 2].map(workId) + // The cost pass saves the whole connected group in each member's costs.json. + const group = (id: string) => { + const report = costs(id, [ + { requestId: "planning", usd: "0.500000000", workIds: [first] }, + { requestId: "untagged", priced: false, workIds: [second] }, + ]) + Object.assign(report.requests[1], { + billingTagSkipped: "tag-limit", + billingLookup: { + status: "unavailable", + reason: "Billing lookup unavailable", + checkedAt: "2026-10-08T10:00:00Z", + }, + }) + return report + } + const { rows } = await browse([ + { workId: first, summary: summary(first, { requests: [request("planning")] }), costs: group(first) }, + { workId: second, summary: summary(second, { requests: [request("untagged")] }), costs: group(second) }, + ]) + const details = (id: string) => rows.find((row) => row.workId === id)?.details + + expect(details(first)).toContain("Prices: 1/1 requests priced, $0.50.") + expect(details(first)).not.toContain("untagged") + expect(details(first)).not.toContain("Last billing refresh failed") + expect(details(second)).toContain("Prices: 0/1 requests priced, $0.00 known so far.") + expect(details(second)).toContain("1 request untagged: tag limit") + expect(details(second)).toContain("Last billing refresh failed for 1 request: Billing lookup unavailable.") + }) +}) diff --git a/src/extensions/work-attribution/browser.ts b/src/extensions/work-attribution/browser.ts new file mode 100644 index 000000000..3fc599a4d --- /dev/null +++ b/src/extensions/work-attribution/browser.ts @@ -0,0 +1,264 @@ +import { readdir, readFile, stat } from "node:fs/promises" +import { basename, dirname, join } from "node:path" +import { stripTerminalSequences } from "@earendil-works/pi-tui" +import { DEFAULT_SLUG, derivePlanTitle } from "../../shared/planning/plan-markdown.js" +import { isWorkId } from "../../shared/work-id.js" +import { mergePullRequestLinks, pullRequestLabel } from "../pull-request-status/links.js" +import { storedPullRequests } from "../pull-request-status/provider-records.js" +import type { WorkPullRequest } from "../pull-request-status/pull-requests.js" +import type { WorkDetailsRequest } from "../work-attribution.js" +import { costDetailLines, type knownSpend } from "./cost-details.js" +import { decimalNanos, roundedUsd, time } from "./costs.js" +import { readWorkHead, type WorkHead } from "./row-log.js" +import { object } from "./summary.js" + +/** Besides the current work, the panel lists works with activity this recent. */ +const RECENT_MS = 35 * 24 * 60 * 60_000 +const MAX_WORKS = 30 +/** Larger files are skipped rather than read; a plan is read only for its title. */ +const MAX_TOTALS_BYTES = 8 * 1024 * 1024 +const MAX_PLAN_BYTES = 256 * 1024 + +type Spend = ReturnType +type Link = Pick + +/** One work's saved files; any of them can be missing or damaged. Each stays small however long the work runs. */ +export interface SavedWork { + workId: string + /** The `work.json` manifest. */ + head?: WorkHead + /** Parsed `cost-totals.json`. */ + totals?: unknown + planTitle?: string + /** `work.json` modification time, used when the summary has no timestamps. */ + modifiedAt?: number +} + +export interface WorkRow { + workId: string + /** Plain text; the panel styles and shortens it. */ + label: string + value: string + description: string + /** The text `/work` prints for this work. */ + details: string +} + +export interface WorkBrowser { + /** The current work first, then the others by last activity. */ + rows: WorkRow[] + /** Spend across the listed works; connected works share requests, which count once. */ + spend: string +} + +/** Reads only local files, so opening the panel never waits for the network. */ +export async function readWorkBrowser( + agentDir: string, + current: WorkDetailsRequest, + now = Date.now(), +): Promise { + const directory = join(agentDir, "work") + // Recording any activity commits a new work.json, so its modification time bounds what is read. + const modified = new Map() + await Promise.all( + (await readdir(directory).catch(() => [])).filter(isWorkId).map(async (workId) => { + const info = await stat(join(directory, workId, "work.json")).catch(() => undefined) + if (info) modified.set(workId, info.mtimeMs) + }), + ) + + const candidates = [...modified] + .filter(([workId, at]) => workId !== current.workId && at >= now - RECENT_MS) + .map(([workId]) => workId) + const [own, ...others] = await Promise.all( + [current.workId, ...candidates].map( + async (workId): Promise => ({ + workId, + head: await readWorkHead(agentDir, workId), + modifiedAt: modified.get(workId), + }), + ), + ) + // Cost passes rewrite work.json weeks after the last request, so saved activity times pick the works listed. + // Reading deletes a work.json that is not a manifest; that work is listed again once an update rebuilds it. + const recent = others + .filter((work) => work.head) + .map((work) => ({ work, activity: lastActivity(work) ?? 0 })) + .filter(({ activity }) => activity >= now - RECENT_MS) + .sort((left, right) => right.activity - left.activity || left.work.workId.localeCompare(right.work.workId)) + .slice(0, MAX_WORKS - 1) + .map(({ work }) => work) + const works = await Promise.all( + [own, ...recent].map(async (work): Promise => { + const folder = join(directory, work.workId) + const [totals, planTitle] = await Promise.all([ + readJson(join(folder, "cost-totals.json")), + work.head && readPlanTitle(folder, work.head), + ]) + return { ...work, totals, planTitle } + }), + ) + return buildWorkBrowser(agentDir, current, works) +} + +/** Newest request start, native edit or retained plan; the file time when the manifest has none. */ +function lastActivity(work: SavedWork): number | undefined { + return time(work.head?.latest.activityAt) ?? work.modifiedAt +} + +/** Rows from saved summaries. Only the current work has live PR lookup lines; the rest use their saved links. */ +export function buildWorkBrowser(agentDir: string, current: WorkDetailsRequest, works: SavedWork[]): WorkBrowser { + const built = works + .map((work) => workRow(agentDir, work, work.workId === current.workId ? current : undefined)) + .sort( + (left, right) => + Number(right.current) - Number(left.current) || + (right.activity ?? 0) - (left.activity ?? 0) || + left.row.workId.localeCompare(right.row.workId), + ) + // Connected works' totals cover the same requests: count each group once, plus requests not priced yet. + const groups = new Map() + const spend: Spend = { priced: 0, total: 0, nanos: 0n } + for (const { group, unpriced } of built) { + spend.total += unpriced + if (group && !groups.has(group.key)) groups.set(group.key, group.spend) + } + for (const group of groups.values()) { + spend.priced += group.priced + spend.total += group.total + spend.nanos += group.nanos + } + return { + rows: built.map(({ row }) => row), + spend: spendText( + spend, + built.some(({ partial }) => partial), + ), + } +} + +function spendOf(value: unknown): Spend | undefined { + return object(value) && typeof value.priced === "number" && typeof value.total === "number" + ? { priced: value.priced, total: value.total, nanos: decimalNanos(value.knownCostUsd) ?? 0n } + : undefined +} + +/** A work's saved `cost-totals.json`, when it is readable. */ +function costTotals(value: unknown, workId: string) { + if (!object(value) || value.workId !== workId || !Array.isArray(value.group) || !Array.isArray(value.pullRequests)) + return undefined + const own = spendOf(value.requests) + const group = spendOf(value.groupRequests) + if (!own || !group || !object(value.requests) || typeof value.requests.recorded !== "number") return undefined + return { + own, + recorded: value.requests.recorded, + group: { key: JSON.stringify(value.group), spend: group }, + } +} + +function workRow(agentDir: string, work: SavedWork, current?: WorkDetailsRequest) { + const { workId } = work + const folder = join(agentDir, "work", workId) + const { head } = work + const totals = costTotals(work.totals, workId) + const requests = head?.logs.requests?.rows ?? 0 + // Requests recorded after the last cost pass are not in the totals yet; they count as unpriced. + const unpriced = head ? Math.max(0, requests - (totals?.recorded ?? 0)) : 0 + const spent: Spend | undefined = totals + ? { ...totals.own, total: totals.own.total + unpriced } + : head && { priced: 0, total: unpriced, nanos: 0n } + const links: Link[] = head ? mergePullRequestLinks(storedPullRequests(head.pullRequests)) : [] + const latest = head?.latest + const activity = lastActivity(work) + const path = + latest?.edit?.repository ?? latest?.commit?.repository ?? latest?.request?.repository ?? latest?.request?.cwd + const name = path === undefined ? undefined : repositoryName(path) + const branch = latest?.branch?.branch + const title = plain(work.planTitle ?? [name, branch].filter(Boolean).join(" · ")) + const pullRequests = current ? current.lines : links.map((pr) => `${pullRequestLabel(pr)}: ${pr.url}`) + const costLines = costDetailLines(work.totals, join(folder, "costs.json")) + const context = head + ? [ + `Repository: ${name ? plain(name) : "unknown"}${branch ? ` · branch ${plain(branch)}` : ""}`, + `Last activity: ${activity === undefined ? "unknown" : localTime(activity)} · ${requests} request${requests === 1 ? "" : "s"}`, + ] + : [`Summary unavailable: ${join(folder, "work.json")}`] + const row: WorkRow = { + workId, + label: `${current ? "●" : " "} ${workId.slice(0, 8)} ${title || (head ? "untitled work" : "summary unavailable")}`, + value: `${spendText(spent)} · ${pullRequestState(links)}`, + description: [`Work ID: ${workId}`, ...context, ...pullRequests, ...costLines].join("\n"), + details: [`Work ID: ${workId}`, ...pullRequests, ...costLines].join("\n"), + } + return { + row, + current: current !== undefined, + activity, + partial: !spent, + unpriced, + group: totals?.group, + } +} + +/** `/src/app/.git` and `/srv/app.git` name `app`; a working directory names itself. */ +function repositoryName(path: string): string { + const name = basename(path) + return name === ".git" ? basename(dirname(path)) : name.replace(/\.git$/, "") +} + +/** Compact for the list, with exact amounts left to the description. Unpriced spend never shows as $0. */ +function spendText(spend: Spend | undefined, partial = false): string { + if (!spend) return "cost unknown" + const complete = !partial && spend.priced === spend.total + if (complete && !spend.total) return "no requests" + if (!spend.nanos && !complete) return "cost unknown" + return `${roundedUsd(spend.nanos)}${complete ? "" : " known so far"}` +} + +function pullRequestState(links: Link[]): string { + if (links.length === 1) return pullRequestLabel(links[0]) + if (!links.length) return "no PR" + return `${links.length} ${links.every((pr) => pr.provider === "gitlab") ? "MRs" : "PRs"}` +} + +function localTime(at: number): string { + const date = new Date(at) + const pad = (value: number) => String(value).padStart(2, "0") + return `${date.getFullYear()}-${pad(date.getMonth() + 1)}-${pad(date.getDate())} ${pad(date.getHours())}:${pad(date.getMinutes())}` +} + +/** Saved names come from model and Git output; keep them to one printable line. */ +function plain(text: string): string { + return stripTerminalSequences(text) + .replace(/\p{Cc}+/gu, " ") + .trim() +} + +/** Missing, special and oversized files read as undefined. */ +async function readText(path: string, limit: number): Promise { + try { + const info = await stat(path) + return info.isFile() && info.size <= limit ? await readFile(path, "utf8") : undefined + } catch { + return undefined + } +} + +async function readJson(path: string): Promise { + const text = await readText(path, MAX_TOTALS_BYTES) + try { + return text === undefined ? undefined : JSON.parse(text) + } catch { + return undefined + } +} + +/** The latest plan's title, read from its retained copy inside this work's folder. */ +async function readPlanTitle(folder: string, head: WorkHead): Promise { + const snapshot = head.latest.plan?.snapshotPath + if (typeof snapshot !== "string") return undefined + const text = await readText(join(folder, "plans", basename(snapshot)), MAX_PLAN_BYTES) + const title = text === undefined ? undefined : derivePlanTitle(text) + return title === DEFAULT_SLUG ? undefined : title +} diff --git a/src/extensions/work-attribution/commits.test.ts b/src/extensions/work-attribution/commits.test.ts index caa4794ca..baf2a9977 100644 --- a/src/extensions/work-attribution/commits.test.ts +++ b/src/extensions/work-attribution/commits.test.ts @@ -2,7 +2,7 @@ import { execFileSync } from "node:child_process" import * as fs from "node:fs" import { mkdtempSync, realpathSync, rmSync, writeFileSync } from "node:fs" import { tmpdir } from "node:os" -import { join } from "node:path" +import { basename, join, sep } from "node:path" import { afterEach, beforeEach, describe, expect, it, vi } from "vitest" import { createContext } from "../__mocks__/context.js" import { createExtensionApi } from "../__mocks__/extension-api.js" @@ -464,6 +464,50 @@ await flushWorkSummaries();` ]) }, 15000) + it("follows another session's commit of the same work through a rebase, reading only commit rows", async () => { + vi.stubEnv("PI_CODING_AGENT_DIR", join(directory, "agent")) + const first = createContext({ cwd: repository, sessionManager: { getSessionId: () => "first" } }) + const workId = attribution.getWorkId(first) + git(repository, "commit", "--allow-empty", "-qm", "base") + await createCommitTrackingBashTool(first).execute( + "first-tool", + { command: "git checkout -qb feat && echo ours > ours.txt && git add ours.txt && git commit -qm ours" }, + undefined, + undefined, + first, + ) + const ours = git(repository, "rev-parse", "HEAD") + for (let index = 0; index < 50; index++) + attribution.appendWorkRecord(first, { type: "request", requestId: `r${index}` }) + await flushWorkSummaries() + git(repository, "checkout", "-q", "main") + git(repository, "commit", "--allow-empty", "-qm", "upstream") + git(repository, "checkout", "-q", "feat") + // A later session of the same work only knows the commit from the work's summary. + const second = createContext({ cwd: repository, sessionManager: { getSessionId: () => "second" } }) + attribution.setWorkId(second, workId) + const opened = vi.spyOn(fs, "openSync") + await createCommitTrackingBashTool(second).execute( + "second-tool", + { command: "git rebase -q main" }, + undefined, + undefined, + second, + ) + const logs = opened.mock.calls.map(([path]) => String(path)).filter((path) => path.includes(`${sep}rows${sep}`)) + expect(logs.length).toBeGreaterThan(0) + expect(logs.every((path) => basename(path).startsWith("commits."))).toBe(true) + const recorded = fs + .readFileSync(attribution.workLedgerPath(second), "utf8") + .trim() + .split("\n") + .map((line) => JSON.parse(line)) + .filter((row) => row.type === "commit") + expect(recorded).toEqual([ + expect.objectContaining({ sha: git(repository, "rev-parse", "HEAD"), rewrittenFrom: ours, workId }), + ]) + }) + it("records same-work cherry-picks, reverts and new merge commits", async () => { vi.stubEnv("PI_CODING_AGENT_DIR", join(directory, "agent")) const ctx = createContext({ cwd: repository, sessionManager: { getSessionId: () => "sequencer" } }) diff --git a/src/extensions/work-attribution/commits.ts b/src/extensions/work-attribution/commits.ts index 9144ce025..dd86f4e86 100644 --- a/src/extensions/work-attribution/commits.ts +++ b/src/extensions/work-attribution/commits.ts @@ -22,6 +22,7 @@ import { import { debugWorkAttribution } from "./diagnostics.js" import { observeToolFiles } from "./file-observations.js" import { COST_PER_PR_RESOURCE_ID } from "./resource.js" +import { readWorkRows } from "./row-log.js" const GIT_LOOKUP_TIMEOUT_MS = 2000 @@ -310,7 +311,9 @@ function recordedCommits(ctx: WorkContext, workId: string): Set { } catch {} } try { - for (const row of JSON.parse(readText(join(getAgentDir(), "work", workId, "work.json"))).commits) shas.add(row.sha) + // Only the commit rows: a long work's summary holds many more request rows. + for (const row of readWorkRows(getAgentDir(), workId, "commits") ?? []) + if (typeof row.sha === "string") shas.add(row.sha) } catch {} return shas } diff --git a/src/extensions/work-attribution/continuation.integration.test.ts b/src/extensions/work-attribution/continuation.integration.test.ts index 57a767a0c..dad5635db 100644 --- a/src/extensions/work-attribution/continuation.integration.test.ts +++ b/src/extensions/work-attribution/continuation.integration.test.ts @@ -1,5 +1,7 @@ import { execFileSync } from "node:child_process" +import { createHash } from "node:crypto" import { + appendFileSync, existsSync, mkdirSync, mkdtempSync, @@ -17,13 +19,15 @@ import { savePlanMarkdown } from "../../shared/planning/plan-markdown.js" import { createContext } from "../__mocks__/context.js" import { createExtensionApi } from "../__mocks__/extension-api.js" import { createWorkScopeSnapshot } from "../__mocks__/work-scope.js" -import { appendWorkRecord, createWorkAttributionExtension, getWorkId } from "../work-attribution.js" +import { savedWorkSummary } from "../__mocks__/work-summary.js" +import { appendWorkRecord, createWorkAttributionExtension, getWorkId, getWorkSegment } from "../work-attribution.js" import { createCommitTrackingBashTool } from "./commits.js" import { findWorkContinuation } from "./continuation.js" import { calculatePullRequestCosts } from "./costs.js" import { createTrackedWriteTool } from "./file-transitions.js" -import { correctWorkLink } from "./links.js" +import { correctWorkLink, reconcileWorkContinuations } from "./links.js" import * as scope from "./scope.js" +import * as summary from "./summary.js" import { flushWorkSummaries, readWorkRecords, recoverWorkSummaries, type WorkRecord } from "./summary.js" let root: string | undefined @@ -55,11 +59,11 @@ async function planningRepository() { const path = "docs/adr/decision.md" await createTrackedWriteTool(ctx, "write-adr").execute("write-adr", { path, content: "# Build the feature\n" }) const workId = getWorkId(ctx) - return { cwd, git, path, workId, agentDir: join(root, "agent") } + return { cwd, git, path, workId, captured, agentDir: join(root, "agent") } } it.each([0, 7])("records a candidate observation for an ADR written by Bash before exit %s", async (exitCode) => { - const { cwd, agentDir } = await planningRepository() + const { cwd, captured, agentDir } = await planningRepository() const ctx = createContext({ cwd, sessionManager: { getSessionId: () => `bash-planner-${exitCode}` } }) const workId = getWorkId(ctx) const path = "docs/adr/from-bash.md" @@ -89,24 +93,23 @@ it.each([0, 7])("records a candidate observation for an ADR written by Bash befo }), ) await flushWorkSummaries() - const summaryPath = join(agentDir, "work", workId, "work.json") - const saved = JSON.parse(readFileSync(summaryPath, "utf8")) + const saved = savedWorkSummary(agentDir, workId) expect(saved.fileObservations).toContainEqual(expect.objectContaining({ toolCallId, source: "bash" })) - rmSync(summaryPath) + rmSync(join(agentDir, "work", workId, "work.json")) recoverWorkSummaries() await flushWorkSummaries() - expect(JSON.parse(readFileSync(summaryPath, "utf8")).fileObservations).toEqual(saved.fileObservations) - expect(await findWorkContinuation({ cwd }, `Implement ${path}`)).toBeUndefined() + expect(savedWorkSummary(agentDir, workId).fileObservations).toEqual(saved.fileObservations) + expect(await findWorkContinuation({ cwd }, `Implement ${path}`, captured)).toEqual({ owned: false }) }) it("does not claim a human-written ADR after a read-only Bash call", async () => { - const { cwd, agentDir } = await planningRepository() + const { cwd, captured, agentDir } = await planningRepository() const path = "docs/adr/from-human.md" writeFileSync(join(cwd, path), "# Human decision\n") const ctx = createContext({ cwd, sessionManager: { getSessionId: () => "bash-reader" } }) await createCommitTrackingBashTool(ctx).execute("read-adr", { command: `cat ${path}` }, undefined, undefined, ctx) - expect(await findWorkContinuation({ cwd }, `Implement ${path}`)).toBeUndefined() + expect(await findWorkContinuation({ cwd }, `Implement ${path}`, captured)).toEqual({ owned: false }) expect(readWorkRecords(agentDir)).not.toContainEqual( expect.objectContaining({ type: "file_observation", @@ -156,25 +159,43 @@ it.each( }) it("continues a named skill-written ADR across worktrees without native plan mode", async () => { - const { cwd, git, path, workId } = await planningRepository() - expect(await findWorkContinuation({ cwd }, `/skill:implement ${path}`)).toMatchObject({ + const { cwd, git, path, workId, captured } = await planningRepository() + expect((await findWorkContinuation({ cwd }, `/skill:implement ${path}`, captured)).match).toMatchObject({ workId, source: "named-artifact", evidence: { path: join(cwd, path), branch: "feature" }, }) - expect(await findWorkContinuation({ cwd }, "/skill:implement")).toBeUndefined() + expect(await findWorkContinuation({ cwd }, "/skill:implement", captured)).toEqual({ owned: false }) git("add", path) git("commit", "-q", "-m", "Save the ADR") const otherTree = join(cwd, "..", "implementation") git("worktree", "add", "-q", "-b", "implementation", otherTree) - expect(await findWorkContinuation({ cwd: otherTree }, `Implement ${path}`)).toMatchObject({ + expect((await findWorkContinuation({ cwd: otherTree }, `Implement ${path}`, captured)).match).toMatchObject({ workId, source: "named-artifact", evidence: { path: join(otherTree, path), worktree: cwd }, }) - expect(await findWorkContinuation({ cwd: otherTree }, "/skill:implement")).toBeUndefined() + expect(await findWorkContinuation({ cwd: otherTree }, "/skill:implement", captured)).toEqual({ owned: false }) writeFileSync(join(otherTree, path), "# Different work\n") - expect(await findWorkContinuation({ cwd: otherTree }, `Implement ${path}`)).toBeUndefined() + // The changed copy cannot continue the ADR's work, but it still names that work. + expect(await findWorkContinuation({ cwd: otherTree }, `Implement ${path}`, captured)).toEqual({ owned: true }) +}) + +it("leaves an input naming an ADR owned from another worktree unresolved without reading every journal", async () => { + const { cwd, git, path, workId } = await planningRepository() + git("add", path) + git("commit", "-q", "-m", "Save the ADR") + const otherTree = join(cwd, "..", "implementation") + git("worktree", "add", "-q", "-b", "implementation", otherTree) + writeFileSync(join(otherTree, path), "# Revised by hand\n") + const ctx = createContext({ cwd: otherTree, sessionManager: { getSessionId: () => "other-tree" } }) + const api = createExtensionApi() + createWorkAttributionExtension()(api.api) + const journals = vi.spyOn(summary, "readWorkRecords") + await api.getHandler("input")({ type: "input", source: "interactive", text: `Revise ${path}` }, ctx) + expect(getWorkId(ctx)).not.toBe(workId) + expect(getWorkSegment(ctx)).toMatchObject({ attribution: "unknown", reason: "unresolved-reference" }) + expect(journals).not.toHaveBeenCalled() }) it.each(["interactive", "rpc"] as const)("adopts a pasted plan before the first %s request", async (source) => { @@ -192,12 +213,31 @@ it.each(["interactive", "rpc"] as const)("adopts a pasted plan before the first expect.objectContaining({ type: "request", requestId: request.headers["X-Request-Id"], workId }), ) await flushWorkSummaries() - const summary = JSON.parse(readFileSync(join(agentDir, "work", workId, "work.json"), "utf8")) + const summary = savedWorkSummary(agentDir, workId) expect(summary.continuations).toContainEqual( expect.objectContaining({ sessionId: "pasted-session", source: "pasted-plan", - evidence: { path: saved.snapshotPath }, + evidence: expect.objectContaining({ path: saved.snapshotPath, contentHash: saved.contentHash }), + }), + ) +}) + +it("reads work history once to pin and confirm a plan continuation", async () => { + const flow = await recordedPlan("path") + const reads = vi.spyOn(summary, "readWorkRecords") + await flow.input({ type: "input", source: "interactive", text: flow.text }, flow.implementer) + expect(reads).toHaveBeenCalledOnce() + expect(getWorkId(flow.implementer)).toBe(flow.workId) + const rows = readWorkRecords(flow.agentDir) + expect(rows.filter((row) => row.type === "work_link")).toMatchObject([ + { requestIds: [flow.research, flow.producer].sort() }, + ]) + expect(rows).toContainEqual( + expect.objectContaining({ + type: "work", + sessionId: "consumer", + continuation: expect.objectContaining({ evidence: expect.objectContaining({ requestId: flow.producer }) }), }), ) }) @@ -523,3 +563,74 @@ it.each([ // Keeping a work marker is not proof that the old request produced this new content. expect(readWorkRecords(flow.agentDir).filter((row) => row.type === "work_link")).toEqual([]) }) + +it.each( + (["path", "snapshot", "paste", "artifact"] as const).flatMap((kind) => + [false, true].map((sameWork) => ({ kind, sameWork })), + ), +)("retains accepted $kind evidence and original scope (same work: $sameWork)", async ({ kind, sameWork }) => { + const flow = await recordedPlan(kind) + const ctx = sameWork ? flow.planner : flow.implementer + await flow.input({ type: "input", source: "rpc", text: flow.text }, ctx) + const rows = readWorkRecords(flow.agentDir) + const receipt = rows.find( + (row) => row.type === "work" && row.sessionId === ctx.sessionManager.getSessionId() && row.continuation, + ) + expect(receipt).toMatchObject({ + workId: flow.workId, + continuation: { + source: kind === "artifact" ? "named-artifact" : kind === "paste" ? "pasted-plan" : "saved-plan", + evidence: { + segmentId: expect.any(String), + requestId: flow.producer, + ...createWorkScopeSnapshot(join(flow.cwd, ".git")).scope, + ...(kind === "artifact" ? {} : { contentHash: rows.find((row) => row.type === "plan")?.contentHash }), + }, + }, + }) +}) + +it("retains the edited snapshot's accepted bytes after the original snapshot is restored", async () => { + const flow = await recordedPlan("snapshot") + const plan = readWorkRecords(flow.agentDir).find((row) => row.type === "plan") + if (typeof plan?.snapshotPath !== "string") throw new Error("Missing retained plan") + const original = readFileSync(plan.snapshotPath) + const edited = `\n# Different accepted task\n` + writeFileSync(plan.snapshotPath, edited) + await flow.input({ type: "input", source: "interactive", text: flow.text }, flow.implementer) + writeFileSync(plan.snapshotPath, original) + const rows = readWorkRecords(flow.agentDir) + expect(rows.filter((row) => row.type === "work_link")).toEqual([]) + expect(rows.find((row) => row.type === "work" && row.continuation)).toMatchObject({ + continuation: { evidence: { contentHash: createHash("sha256").update(edited).digest("hex") } }, + }) + await reconcileWorkContinuations(flow.agentDir, new AbortController().signal, () => {}) + expect(readWorkRecords(flow.agentDir).filter((row) => row.type === "work_link")).toEqual([]) +}) + +it.each([ + "interactive", + "rpc", +] as const)("repairs same-work acceptance after missing producer requests return (%s)", async (source) => { + const flow = await recordedPlan("path") + const ledger = join(flow.agentDir, "work-attribution", "producer.jsonl") + const original = readWorkRecords(flow.agentDir).filter((row) => row.sessionId === "producer") + const request = original.find((row) => row.type === "request" && row.requestId === flow.producer) + writeFileSync( + ledger, + `${original + .filter((row) => row !== request) + .map((row) => JSON.stringify(row)) + .join("\n")}\n`, + ) + await flow.input({ type: "input", source, text: flow.text }, flow.planner) + expect(readWorkRecords(flow.agentDir).filter((row) => row.type === "work_link")).toEqual([]) + appendFileSync(ledger, `${JSON.stringify(request)}\n`) + await reconcileWorkContinuations(flow.agentDir, new AbortController().signal, () => {}) + expect(readWorkRecords(flow.agentDir).filter((row) => row.type === "work_link")).toMatchObject([ + { + workId: flow.workId, + requestIds: [flow.research, flow.producer].sort(), + }, + ]) +}) diff --git a/src/extensions/work-attribution/continuation.owned-reference.test.ts b/src/extensions/work-attribution/continuation.owned-reference.test.ts index 850ef28bc..4e8c6db94 100644 --- a/src/extensions/work-attribution/continuation.owned-reference.test.ts +++ b/src/extensions/work-attribution/continuation.owned-reference.test.ts @@ -4,67 +4,109 @@ import { mkdirSync, mkdtempSync, realpathSync, rmSync, symlinkSync } from "node: import { tmpdir } from "node:os" import { join } from "node:path" import { afterEach, beforeEach, expect, it, vi } from "vitest" -import { hasOwnedWorkReference } from "./continuation.js" -import type { WorkRecord } from "./summary.js" +import { savePlanMarkdown } from "../../shared/planning/plan-markdown.js" +import { createContext } from "../__mocks__/context.js" +import { appendWorkRecord } from "../work-attribution.js" +import { findWorkContinuation } from "./continuation.js" +import { type FileTransition, readAttributedFileState, readRepositoryTransitions } from "./file-transitions.js" +import { flushWorkSummaries } from "./summary.js" vi.mock("node:fs", async (original) => { const actual = await original() return { ...actual, realpathSync: Object.assign(vi.fn(actual.realpathSync), { native: actual.realpathSync.native }) } }) +vi.mock("./file-transitions.js", () => ({ readRepositoryTransitions: vi.fn(), readAttributedFileState: vi.fn() })) const workId = "11111111-1111-4111-8111-111111111111" let cwd: string +let transitions: FileTransition[] beforeEach(() => { cwd = realpathSync(mkdtempSync(join(tmpdir(), "kimchi-owned-reference-"))) + vi.stubEnv("PI_CODING_AGENT_DIR", join(cwd, "agent")) mkdirSync(join(cwd, "docs")) mkdirSync(join(cwd, "other")) + transitions = [] + vi.mocked(readRepositoryTransitions).mockImplementation(async () => ({ + repository: join(cwd, ".git"), + worktree: cwd, + transitions, + })) + // Each recorded edit was changed since, so a mention can only name its work. + vi.mocked(readAttributedFileState).mockResolvedValue(null) }) -afterEach(() => { +afterEach(async () => { + await flushWorkSummaries() + vi.clearAllMocks() + vi.unstubAllEnvs() rmSync(cwd, { recursive: true, force: true }) }) -function row(type: "plan" | "file_transition", fields: Record): WorkRecord { - return { version: 1, type, workId, sessionId: "planner", ...fields } +function edit(path: string, worktree = cwd): FileTransition { + return { + type: "file_transition", + transitionId: randomUUID(), + toolCallId: "write", + sessionId: "planner", + workId, + cwd: worktree, + repository: join(cwd, ".git"), + worktree, + path, + baseline: null, + baselineFile: null, + before: null, + after: { blob: "b".repeat(40), mode: "100644" }, + cursor: { bytes: 0, digest: "empty" }, + } +} +async function owned(text: string): Promise { + return (await findWorkContinuation({ cwd }, text, undefined)).owned } it.each([ ["an unowned file", () => [], false], - [ - "a file with the same name in another directory", - () => [row("file_transition", { worktree: cwd, path: "other/adr.md" })], - false, - ], - ["an edited file", () => [row("file_transition", { worktree: cwd, path: "docs/adr.md" })], true], - ["a plan's editable path", () => [row("plan", { cwd, path: "docs/adr.md" })], true], - ["a plan's retained path", () => [row("plan", { cwd, path: "gone.md", snapshotPath: "docs/adr.md" })], true], -] as const)("finds whether a mention of %s has an owner", (_case, records, owned) => { - expect(hasOwnedWorkReference({ cwd }, "Implement docs/adr.md", records())).toBe(owned) + ["a file with the same name in another directory", () => [edit("other/adr.md")], false], + ["an edited file", () => [edit("docs/adr.md")], true], + ["a file edited in another worktree of the repository", () => [edit("docs/adr.md", "/elsewhere/worktree")], true], +] as const)("finds whether a mention of %s has an owner", async (_case, rows, expected) => { + transitions.push(...rows()) + expect(await owned("Implement docs/adr.md")).toBe(expected) }) -it("does not count plans and files that only the current work recorded", () => { - const own = [ - row("file_transition", { worktree: cwd, path: "docs/adr.md" }), - row("plan", { cwd, path: "docs/plan.md" }), - ] - expect(hasOwnedWorkReference({ cwd }, "What do docs/adr.md and docs/plan.md say?", own, workId)).toBe(false) +it("does not count plans and files that only the current work recorded", async () => { + transitions.push(edit("docs/adr.md")) + const saved = savePlanMarkdown({ cwd, name: "own", planText: "# Plan\n", workId }) + appendWorkRecord(createContext({ cwd }), { type: "plan", ...saved }, workId) + await flushWorkSummaries() + const mention = `What do docs/adr.md and ${saved.path} say?` + expect((await findWorkContinuation({ cwd }, mention, undefined, workId)).owned).toBe(false) // A file that other work also recorded may still belong to that work. - const other = { ...row("file_transition", { worktree: cwd, path: "docs/adr.md" }), workId: randomUUID() } - expect(hasOwnedWorkReference({ cwd }, "What does docs/adr.md say?", [...own, other], workId)).toBe(true) + transitions.push({ ...edit("docs/adr.md"), workId: randomUUID() }) + expect((await findWorkContinuation({ cwd }, mention, undefined, workId)).owned).toBe(true) }) -it("matches an owner through a directory alias", () => { +it("matches an owner through a directory alias", async () => { symlinkSync(join(cwd, "docs"), join(cwd, "alias"), "dir") - const records = [row("file_transition", { worktree: cwd, path: "docs/adr.md" })] - expect(hasOwnedWorkReference({ cwd }, "Implement alias/adr.md", records)).toBe(true) + transitions.push(edit("docs/adr.md")) + expect(await owned("Implement alias/adr.md")).toBe(true) +}) + +it.each(["path", "snapshotPath"] as const)("keeps the owner of a deleted plan's %s", async (key) => { + const saved = savePlanMarkdown({ cwd, name: "deleted", planText: "# Plan\n", workId }) + appendWorkRecord(createContext({ cwd }), { type: "plan", ...saved }, workId) + const path = saved[key] + if (!path) throw new Error("Missing retained plan") + rmSync(path) + expect(await owned(`Implement ${path}`)).toBe(true) }) -it("resolves only the journal rows that name a mentioned file", () => { - const unrelated = Array.from({ length: 200 }, (_, index) => [ - row("file_transition", { worktree: cwd, path: `docs/other-${index}.md` }), - row("plan", { cwd, path: `.kimchi/plans/plan-${index}.md`, snapshotPath: `/retained/plan-${index}.md` }), - ]).flat() - const owner = row("file_transition", { worktree: cwd, path: "docs/adr.md" }) +it("resolves only the plan records that name a deleted plan", async () => { + mkdirSync(join(cwd, ".kimchi/plans"), { recursive: true }) + const planner = createContext({ cwd }) + for (let index = 0; index < 200; index++) + appendWorkRecord(planner, { type: "plan", path: `.kimchi/plans/plan-${index}.md` }, workId) + appendWorkRecord(planner, { type: "plan", path: ".kimchi/plans/deleted.md" }, workId) + await flushWorkSummaries() vi.mocked(realpathSync).mockClear() - expect(hasOwnedWorkReference({ cwd }, "Implement docs/adr.md", [...unrelated, owner])).toBe(true) - // One call for the mention itself and one for its owner's row. - expect(realpathSync).toHaveBeenCalledTimes(2) + expect(await owned(`Implement ${join(cwd, ".kimchi/plans/deleted.md")}`)).toBe(true) + expect(realpathSync).toHaveBeenCalledOnce() }) diff --git a/src/extensions/work-attribution/continuation.pasted-plan.test.ts b/src/extensions/work-attribution/continuation.pasted-plan.test.ts index b70506fd3..d41901f3e 100644 --- a/src/extensions/work-attribution/continuation.pasted-plan.test.ts +++ b/src/extensions/work-attribution/continuation.pasted-plan.test.ts @@ -15,13 +15,14 @@ vi.mock("./file-transitions.js", () => ({ })) let cwd: string +let captured: scope.WorkScopeSnapshot const planningWork = "11111111-1111-4111-8111-111111111111" const otherWork = "22222222-2222-4222-8222-222222222222" const after = { blob: "b".repeat(40), mode: "100644" } beforeEach(() => { cwd = realpathSync(mkdtempSync(join(tmpdir(), "kimchi-pasted-plan-"))) vi.stubEnv("PI_CODING_AGENT_DIR", join(cwd, "agent")) - const captured = createWorkScopeSnapshot(join(cwd, ".git")) + captured = createWorkScopeSnapshot(join(cwd, ".git")) vi.spyOn(scope, "captureWorkScope").mockResolvedValue(captured) vi.spyOn(scope, "readWorkScope").mockReturnValue(captured.scope) mkdirSync(join(cwd, "docs/adr"), { recursive: true }) @@ -75,5 +76,5 @@ it.each([ expect(hasWorkReference(prompt)).toBe(true) // Docs: "A changed plan, unknown ID, or conflicting plan stays unresolved. File paths inside the // verified plan are its instructions, not additional work selections." - expect((await findWorkContinuation({ cwd }, prompt))?.workId).toBeUndefined() + expect(await findWorkContinuation({ cwd }, prompt, captured)).toEqual({ owned: true }) }) diff --git a/src/extensions/work-attribution/continuation.test.ts b/src/extensions/work-attribution/continuation.test.ts index 10813395a..92d8c6914 100644 --- a/src/extensions/work-attribution/continuation.test.ts +++ b/src/extensions/work-attribution/continuation.test.ts @@ -15,6 +15,7 @@ vi.mock("./file-transitions.js", () => ({ })) let cwd: string +let captured: scope.WorkScopeSnapshot const planningWork = "11111111-1111-4111-8111-111111111111" const otherWork = "22222222-2222-4222-8222-222222222222" const after = { blob: "b".repeat(40), mode: "100644" } @@ -43,7 +44,7 @@ function artifact(overrides: Partial = {}): FileTransition { beforeEach(() => { cwd = realpathSync(mkdtempSync(join(tmpdir(), "kimchi-continuation-"))) vi.stubEnv("PI_CODING_AGENT_DIR", join(cwd, "agent")) - const captured = createWorkScopeSnapshot(join(cwd, ".git")) + captured = createWorkScopeSnapshot(join(cwd, ".git")) vi.spyOn(scope, "captureWorkScope").mockResolvedValue(captured) vi.spyOn(scope, "readWorkScope").mockReturnValue(captured.scope) mkdirSync(join(cwd, "docs/adr"), { recursive: true }) @@ -57,6 +58,10 @@ beforeEach(() => { })) vi.mocked(readAttributedFileState).mockResolvedValue(after) }) +/** The work an input continues; ownership of what it names has its own tests. */ +async function find(text: string) { + return (await findWorkContinuation({ cwd }, text, captured)).match +} afterEach(() => { vi.clearAllMocks() vi.restoreAllMocks() @@ -65,6 +70,32 @@ afterEach(() => { }) describe("continuing saved work", () => { + it.each([ + "path", + "paste", + ])("keeps the accepted %s bytes when the file changes during the final account check", async (kind) => { + const saved = savePlanMarkdown({ + cwd, + name: "accepted", + planText: "# Accepted\r\nOriginal bytes.\r\n", + workId: planningWork, + }) + const content = readFileSync(saved.path, "utf8") + const captured = createWorkScopeSnapshot(join(cwd, ".git")) + vi.mocked(scope.captureWorkScope).mockImplementation(async () => { + const path = kind === "path" ? saved.path : saved.snapshotPath + if (!path) throw new Error("Missing retained plan") + writeFileSync(path, `\n# Replacement\n`) + return captured + }) + expect( + (await findWorkContinuation({ cwd }, kind === "path" ? `Implement ${saved.path}` : content, captured)).match, + ).toMatchObject({ + workId: planningWork, + evidence: { contentHash: saved.contentHash }, + }) + }) + it.each(["plain", "fenced", "skill"])("continues a %s pasted retained plan without its path", async (wrapper) => { const saved = savePlanMarkdown({ cwd, @@ -80,7 +111,7 @@ describe("continuing saved work", () => { : wrapper === "skill" ? `\nSee examples/template.md.\n\n\n${content}` : `Implement this plan:\n${content}` - expect(await findWorkContinuation({ cwd }, prompt.replaceAll("\n", "\r\n"))).toMatchObject({ + expect(await find(prompt.replaceAll("\n", "\r\n"))).toMatchObject({ workId: planningWork, source: "pasted-plan", evidence: { path: saved.snapshotPath }, @@ -89,7 +120,7 @@ describe("continuing saved work", () => { it("does not use a saved plan pasted into the skill template", async () => { const saved = savePlanMarkdown({ cwd, name: "template", planText: "# Template\n", workId: planningWork }) const prompt = `\n${readFileSync(saved.path, "utf8")}\n` - expect(await findWorkContinuation({ cwd }, prompt)).toBeUndefined() + expect(await find(prompt)).toBeUndefined() }) it.each(["unknown", "changed", "marker-only"])("does not adopt a %s pasted plan", async (kind) => { const saved = savePlanMarkdown({ @@ -105,22 +136,22 @@ describe("continuing saved work", () => { : kind === "changed" ? content.replace("Add export.", "Remove authentication.") : content.split("\n")[0] - expect(await findWorkContinuation({ cwd }, prompt)).toBeUndefined() + expect(await find(prompt)).toBeUndefined() }) it("ignores an unowned output but does not choose between conflicting plan owners", async () => { const first = savePlanMarkdown({ cwd, name: "first", planText: "# First\n", workId: planningWork }) const second = savePlanMarkdown({ cwd, name: "second", planText: "# Second\n", workId: otherWork }) const content = readFileSync(first.path, "utf8") - expect(await findWorkContinuation({ cwd }, `${content}\n${readFileSync(second.path, "utf8")}`)).toBeUndefined() - expect(await findWorkContinuation({ cwd }, `${content}\nAlso write missing.md`)).toMatchObject({ + expect(await find(`${content}\n${readFileSync(second.path, "utf8")}`)).toBeUndefined() + expect(await find(`${content}\nAlso write missing.md`)).toMatchObject({ workId: planningWork, }) - expect(await findWorkContinuation({ cwd }, `${content}\nAlso follow ${second.path}`)).toBeUndefined() + expect(await find(`${content}\nAlso follow ${second.path}`)).toBeUndefined() }) it("resolves a scoped native plan without requiring file-transition evidence", async () => { const saved = savePlanMarkdown({ cwd, name: "feature", planText: "# Plan", workId: planningWork }) vi.mocked(readRepositoryTransitions).mockResolvedValue(undefined) - expect(await findWorkContinuation({ cwd }, `Implement ${saved.path}`)).toMatchObject({ + expect(await find(`Implement ${saved.path}`)).toMatchObject({ workId: planningWork, source: "saved-plan", evidence: { path: saved.path }, @@ -129,7 +160,7 @@ describe("continuing saved work", () => { it("does not select a modified native plan from its work marker alone", async () => { const saved = savePlanMarkdown({ cwd, name: "feature", planText: "# Plan", workId: planningWork }) writeFileSync(saved.path, `${readFileSync(saved.path, "utf8")}\nUnrecorded changes.`) - expect(await findWorkContinuation({ cwd }, `Implement ${saved.path} and write missing.md`)).toBeUndefined() + expect(await find(`Implement ${saved.path} and write missing.md`)).toBeUndefined() }) it.each([ "src/new.ts", @@ -141,16 +172,14 @@ describe("continuing saved work", () => { ])("does not ignore a competing owner of the named output %s", async (output) => { const saved = savePlanMarkdown({ cwd, name: "feature", planText: "# Plan", workId: planningWork }) const prompt = `Implement ${saved.path} and write ${output}` - expect(await findWorkContinuation({ cwd }, prompt)).toMatchObject({ workId: planningWork }) + expect(await find(prompt)).toMatchObject({ workId: planningWork }) transitions.push(artifact({ path: output.replaceAll("`", ""), workId: otherWork })) - expect(await findWorkContinuation({ cwd }, prompt)).toBeUndefined() + expect(await find(prompt)).toBeUndefined() }) it("keeps a named output on its existing work even if its contents changed", async () => { const saved = savePlanMarkdown({ cwd, name: "feature", planText: "# Plan", workId: planningWork }) vi.mocked(readAttributedFileState).mockResolvedValue({ ...after, blob: "changed" }) - expect( - await findWorkContinuation({ cwd }, `Implement ${saved.path} and update docs/adr/decision.md`), - ).toMatchObject({ + expect(await find(`Implement ${saved.path} and update docs/adr/decision.md`)).toMatchObject({ workId: planningWork, }) }) @@ -160,7 +189,7 @@ describe("continuing saved work", () => { "Follow [the ADR](docs/adr/decision.md).", "Follow [the ADR section](docs/adr/decision.md#steps).", ])("continues the exact saved artifact named by the user: %s", async (text) => { - expect(await findWorkContinuation({ cwd }, text)).toMatchObject({ + expect(await find(text)).toMatchObject({ workId: planningWork, source: "named-artifact", evidence: { path: join(cwd, "docs/adr/decision.md"), transitionId: transitions[0].transitionId }, @@ -169,37 +198,35 @@ describe("continuing saved work", () => { it("uses user arguments after skill expansion and ignores paths in the skill body", async () => { const prompt = '\nImplement examples/example.md.\n' - expect(await findWorkContinuation({ cwd }, prompt)).toBeUndefined() - expect(await findWorkContinuation({ cwd }, `${prompt}\n\ndocs/adr/decision.md`)).toMatchObject({ + expect(await find(prompt)).toBeUndefined() + expect(await find(`${prompt}\n\ndocs/adr/decision.md`)).toMatchObject({ workId: planningWork, source: "named-artifact", }) }) it("requires the saved artifact to remain unchanged", async () => { vi.mocked(readAttributedFileState).mockResolvedValue({ ...after, blob: "changed" }) - expect(await findWorkContinuation({ cwd }, "Implement docs/adr/decision.md")).toBeUndefined() + expect(await find("Implement docs/adr/decision.md")).toBeUndefined() }) it("does not choose an artifact that multiple works edited", async () => { transitions.push(artifact({ workId: otherWork })) - expect(await findWorkContinuation({ cwd }, "Implement docs/adr/decision.md")).toBeUndefined() + expect(await find("Implement docs/adr/decision.md")).toBeUndefined() }) it("does not resolve conflicting saved-plan and artifact owners", async () => { const plan = savePlanMarkdown({ cwd, name: "other", planText: "# Other", workId: otherWork }) - expect(await findWorkContinuation({ cwd }, `Implement docs/adr/decision.md and ${plan.path}`)).toBeUndefined() + expect(await find(`Implement docs/adr/decision.md and ${plan.path}`)).toBeUndefined() }) it("leaves an unknown artifact unresolved", async () => { - expect(await findWorkContinuation({ cwd }, "Implement docs/adr/missing.md")).toBeUndefined() + expect(await find("Implement docs/adr/missing.md")).toBeUndefined() }) it("does not ignore an unknown Markdown link with a section fragment", async () => { - expect( - await findWorkContinuation({ cwd }, "Implement [the other design](docs/adr/missing.md#details)."), - ).toBeUndefined() + expect(await find("Implement [the other design](docs/adr/missing.md#details).")).toBeUndefined() }) it.each([ "Explain what a closure is.", "/skill:implement", ])("leaves a message without a saved file unresolved: %s", async (text) => { - expect(await findWorkContinuation({ cwd }, text)).toBeUndefined() + expect(await find(text)).toBeUndefined() }) it("does not forget an older competing artifact owner", async () => { transitions.unshift( @@ -208,7 +235,7 @@ describe("continuing saved work", () => { recordedAt: new Date(Date.now() - 25 * 60 * 60 * 1000).toISOString(), }), ) - expect(await findWorkContinuation({ cwd }, "Implement docs/adr/decision.md")).toBeUndefined() + expect(await find("Implement docs/adr/decision.md")).toBeUndefined() }) it("resolves paths, URLs and dotted words in this worktree from one read of its evidence", async () => { await findWorkContinuation( diff --git a/src/extensions/work-attribution/continuation.ts b/src/extensions/work-attribution/continuation.ts index dfcc89a89..dafc78de5 100644 --- a/src/extensions/work-attribution/continuation.ts +++ b/src/extensions/work-attribution/continuation.ts @@ -1,4 +1,5 @@ -import { readFileSync, realpathSync } from "node:fs" +import { createHash } from "node:crypto" +import { realpathSync } from "node:fs" import { readdir, readFile, realpath } from "node:fs/promises" import { basename, dirname, isAbsolute, join, relative, resolve } from "node:path" import { getAgentDir, parseSkillBlock } from "@earendil-works/pi-coding-agent" @@ -6,7 +7,7 @@ import { readPlanWorkId } from "../../shared/planning/plan-markdown.js" import type { WorkContext } from "../work-attribution.js" import { type FileTransition, readAttributedFileState, readRepositoryTransitions } from "./file-transitions.js" import { captureWorkScope, readWorkScope, sameWorkScope, type WorkAccount, type WorkScopeSnapshot } from "./scope.js" -import type { WorkRecord } from "./summary.js" +import { readWorkRecords } from "./summary.js" const MARKDOWN_REFERENCE = /(?:^|[\s@'"`([])([^\s@'"`()[\]<>#]+\.md)(?:#[^\s'"`()[\]<>]*)?(?=$|[\s'"`()[\],;:.!?])/gi const FILE_REFERENCE = @@ -26,57 +27,13 @@ export function hasWorkReference(text: string): boolean { return message.includes("\n# Add a greeting\n`) - const costsPath = join(agentDir, "work", workId, "costs.json") - const costs = () => { + // `/work` reads these totals; the full `costs.json` is refreshed at most every five minutes. + const totalsPath = join(agentDir, "work", workId, "cost-totals.json") + const totals = () => { try { - return JSON.parse(readFileSync(costsPath, "utf8")) + return JSON.parse(readFileSync(totalsPath, "utf8")) } catch { return undefined } @@ -131,7 +132,7 @@ it("shows an open MR's pending cost, then its confirmed and inferred total once const sessionId = await newSession(fixture, cwd) expect((await prompt(fixture, sessionId, "/work .kimchi/plans/costs.md")).stopReason).toBe("end_turn") - await waitFor(costs, (value) => value?.requests?.length === 1, 45_000) + await waitFor(totals, (value) => value?.requests?.total === 1, 45_000) const pending = await workDetails(fixture, sessionId) expect(pending).toContain("Cost so far: unknown; $0.00 priced (open)") expect(pending).toContain("Prices: 0/1 requests priced, $0.00 known so far.") @@ -139,7 +140,7 @@ it("shows an open MR's pending cost, then its confirmed and inferred total once billed = true setGitLab("merged") await waitFor( - costs, + totals, (value) => value?.pullRequests?.some((row: { totalCostUsd?: unknown }) => row.totalCostUsd === "0.000166000"), 75_000, ) diff --git a/tests/e2e/acp/work-pr-discovery.test.ts b/tests/e2e/acp/work-pr-discovery.test.ts index 5afe7f80f..bbb868172 100644 --- a/tests/e2e/acp/work-pr-discovery.test.ts +++ b/tests/e2e/acp/work-pr-discovery.test.ts @@ -5,6 +5,7 @@ import { tmpdir } from "node:os" import { join } from "node:path" import { fileURLToPath } from "node:url" import { afterEach, expect, it, vi } from "vitest" +import { readWorkSummary } from "../tui/support/work-summary.js" import { type AcpFixture, startAcpFixture } from "./support/acp-fixture.js" import { newSession, prompt } from "./support/scenarios.js" @@ -149,7 +150,7 @@ it("shows pending and discovered GitLab MR details over ACP without a model requ beforeLinked, ) await expect - .poll(() => JSON.parse(readFileSync(join(agentDir, "work", workId, "work.json"), "utf8")).commits) + .poll(() => readWorkSummary(agentDir, workId)?.commits) .toEqual([ expect.objectContaining({ sha, diff --git a/tests/e2e/tui/pr-cost-reporting.test.ts b/tests/e2e/tui/pr-cost-reporting.test.ts new file mode 100644 index 000000000..37fd6d6a5 --- /dev/null +++ b/tests/e2e/tui/pr-cost-reporting.test.ts @@ -0,0 +1,70 @@ +import { existsSync, readFileSync, writeFileSync } from "node:fs" +import { join } from "node:path" +import { expect, test } from "@microsoft/tui-test" +import { fullText, waitForText } from "./support/assertions.js" +import { runKimchiSession, TUI_TEST_CONFIG } from "./support/kimchi-fixture.js" + +test.use(TUI_TEST_CONFIG) + +// The variables `automation.ts` treats as CI; the runner's own would hide the interactive default notice. +const CI_VARIABLES = [ + "CI", + "GITHUB_ACTIONS", + "GITLAB_CI", + "BUILDKITE", + "JENKINS_URL", + "TEAMCITY_VERSION", + "CIRCLECI", + "TF_BUILD", +] + +for (const telemetry of [false, true]) { + test(`PR reporting follows SaaS uploads (${telemetry ? "on" : "off"}) until explicitly changed`, async ({ + terminal, + }) => { + await runKimchiSession( + terminal, + { + artifactName: `pr-cost-reporting-${telemetry}`, + responses: [], + gitInit: true, + env: { + KIMCHI_TELEMETRY_ENABLED: String(telemetry), + ...Object.fromEntries(CI_VARIABLES.map((name) => [name, ""])), + }, + seedHome(homeDir) { + const path = join(homeDir, ".config/kimchi/config.json") + const config = JSON.parse(readFileSync(path, "utf8")) + config.telemetry = { + endpoint: `${config.llmEndpoint}/logs`, + metricsEndpoint: `${config.llmEndpoint}/metrics`, + } + writeFileSync(path, JSON.stringify(config)) + }, + }, + async (fixture, trace) => { + const path = join(fixture.agentDir, "pr-cost-reporting", "state.json") + if (telemetry) { + await waitForText(terminal, "PR costs are reported to your account") + expect(fullText(terminal)).toMatch(/\/pr-reporting\s+off/) + expect(JSON.parse(readFileSync(path, "utf8")).defaultNoticeShown).toBe(true) + } else expect(fullText(terminal)).not.toContain("PR costs are reported to your account") + terminal.submit("/pr-reporting status") + await waitForText(terminal, `PR reporting ${telemetry ? "Enabled" : "Disabled"} (SaaS default)`) + if (!telemetry) expect(existsSync(path)).toBe(false) + trace.step("reporting follows the existing SaaS upload setting") + terminal.submit("/pr-reporting on") + await waitForText(terminal, "PR reporting on.") + expect(JSON.parse(readFileSync(path, "utf8")).enabled).toBe(true) + trace.step("explicit consent is saved and the upload fields are explained") + terminal.submit("/pr-reporting off") + await waitForText(terminal, "Pending uploads were deleted") + const state = JSON.parse(readFileSync(path, "utf8")) + expect(state.enabled).toBe(false) + expect(state.entries).toEqual({}) + expect(fixture.fake.requests.filter((request) => request.url.includes("/chat/completions"))).toHaveLength(0) + trace.step("reporting is off again with no model calls") + }, + ) + }) +} diff --git a/tests/e2e/tui/support/work-summary.ts b/tests/e2e/tui/support/work-summary.ts new file mode 100644 index 000000000..3e6fff2b4 --- /dev/null +++ b/tests/e2e/tui/support/work-summary.ts @@ -0,0 +1,58 @@ +/** + * Reads a work's local summary the way docs/work-attribution.md describes it, independently of the product reader. + * `work.json` is a manifest whose `logs` name `rows/..jsonl`, sessions included; only the + * first `bytes` of each log are committed, and a later row with the same key replaces an earlier one in place. + */ +import { readFileSync } from "node:fs" +import { join } from "node:path" + +type Row = Record + +const KEYS: Record string> = { + workLinks: (row) => + JSON.stringify([ + row.linkId, + row.revision, + row.sourceWorkId, + row.targetWorkId, + row.requestIds, + row.scope, + row.status, + row.evidence, + ]), + requests: (row) => JSON.stringify(row.requestId), + plans: (row) => JSON.stringify([row.sessionId, row.path, row.snapshotPath]), + commits: (row) => JSON.stringify([row.sessionId, row.sha, row.repository, row.worktree]), + fileTransitions: (row) => JSON.stringify(row.transitionId), + fileObservations: (row) => JSON.stringify(row.observationId), + continuations: (row) => JSON.stringify([row.sessionId, row.source, row.evidence]), + sessions: (row) => JSON.stringify(row.sessionId), +} + +/** Every row of the summary, or undefined while it is missing or a writer is replacing a log. */ +export function readWorkSummary(agentDir: string, workId: string): Record | undefined { + const folder = join(agentDir, "work", workId) + try { + const manifest = JSON.parse(readFileSync(join(folder, "work.json"), "utf8")) + if (manifest.version !== 2) return undefined + const view: Record = { workId: manifest.workId } + for (const [collection, key] of Object.entries(KEYS)) { + const log = manifest.logs[collection] + const rows = new Map() + if (log) { + const text = readFileSync(join(folder, "rows", `${collection}.${log.generation}.jsonl`)) + .subarray(0, log.bytes) + .toString("utf8") + for (const line of text.split("\n")) { + if (!line) continue + const row = JSON.parse(line) + rows.set(key(row), row) + } + } + view[collection] = collection === "sessions" ? [...rows.values()].map((row) => row.sessionId) : [...rows.values()] + } + return view + } catch { + return undefined + } +} diff --git a/tests/e2e/tui/work-attribution.test.ts b/tests/e2e/tui/work-attribution.test.ts index a500068ce..21b805126 100644 --- a/tests/e2e/tui/work-attribution.test.ts +++ b/tests/e2e/tui/work-attribution.test.ts @@ -5,9 +5,10 @@ import { join } from "node:path" import { setTimeout as sleep } from "node:timers/promises" import { expect, Key, test } from "@microsoft/tui-test" import { check } from "proper-lockfile" -import { fullText, waitForText } from "./support/assertions.js" +import { fullText, viewText, waitForText } from "./support/assertions.js" import { type FakeResponseRequest, isWorkMatchingRequest } from "./support/fake-openai-server.js" import { launchKimchi, PROMPT_READY, runKimchiSession, TUI_TEST_CONFIG } from "./support/kimchi-fixture.js" +import { readWorkSummary } from "./support/work-summary.js" test.use(TUI_TEST_CONFIG) const models = [{ slug: "basic", displayName: "Fake Basic", contextWindow: 200_000, maxTokens: 8192 }] @@ -234,16 +235,13 @@ async function waitForSummary( workId: string, minimum: Partial>, ) { - const path = join(agentDir, "work", workId, "work.json") const deadline = Date.now() + 15_000 while (Date.now() < deadline) { - try { - const summary = JSON.parse(readFileSync(path, "utf8")) - if (Object.entries(minimum).every(([key, count]) => summary[key]?.length >= count)) return summary - } catch {} + const summary = readWorkSummary(agentDir, workId) + if (summary && Object.entries(minimum).every(([key, count]) => summary[key]?.length >= count)) return summary await sleep(50) } - throw new Error(`Work summary did not become ready: ${path}`) + throw new Error(`Work summary did not become ready: ${join(agentDir, "work", workId, "work.json")}`) } test("the next message recovers account tracking after its saved scope is lost", async ({ terminal }) => { @@ -280,6 +278,11 @@ test("the next message recovers account tracking after its saved scope is lost", expect(existsSync(missing)).toBe(false) terminal.submit("/work") await waitForText(terminal, `Work ID: ${next.workId}`, { full: false }) + // The work panel selects the new current work and still lists the original work. + expect(viewText(terminal)).toContain(`→ ● ${next.workId.slice(0, 8)}`) + expect(viewText(terminal)).toContain(` ${original.workId.slice(0, 8)}`) + terminal.keyEscape() + await waitForText(terminal, PROMPT_READY, { full: false }) trace.step("the next message uses newly scoped work without assigning today's identity to old history") }, ) @@ -452,9 +455,12 @@ test("work and request IDs are durable before the first reply, and commits belon await expect(terminal.getByText("Created the attribution test commit.", { full: true })).toBeVisible() const sha = execFileSync("git", ["rev-parse", "HEAD"], { cwd: fixture.workDir, encoding: "utf8" }).trim() const commits = records().filter((record) => record.type === "commit") - expect(commits.length).toBe(1) - expect(commits[0].sha).toBe(sha) - expect(commits[0].workId).toBe(started[0].workId) + // Discovery can append updates; the summary below must still contain only one commit. + expect(commits.length).toBeGreaterThan(0) + for (const commit of commits) { + expect(commit.sha).toBe(sha) + expect(commit.workId).toBe(started[0].workId) + } const namingDeadline = Date.now() + 15_000 const completionRequests = () => fixture.fake.requests.filter((item) => item.url.endsWith("/chat/completions")) const isNamingRequest = (url: string) => url === "/chat/completions" @@ -476,7 +482,6 @@ test("work and request IDs are durable before the first reply, and commits belon } const summary = await waitForSummary(fixture.agentDir, started[0].workId, { requests: sent.length, commits: 1 }) expect(summary).toMatchObject({ - version: 1, workId: started[0].workId, sessions: [started[0].sessionId], plans: [], @@ -606,6 +611,18 @@ for (const reference of ["path", "paste"]) { }), }), ) + expect(summary.continuations).toContainEqual( + expect.objectContaining({ + source: reference === "paste" ? "pasted-plan" : "saved-plan", + evidence: expect.objectContaining({ + path: reference === "path" ? realpathSync(snapshotPath) : snapshotPath, + contentHash: planned.plans[0].contentHash, + requestId: producer.requestId, + segmentId: expect.any(String), + ...producer.scope, + }), + }), + ) expect(summary.workId).toBe(workId) expect(new Set(summary.sessions)).toEqual(new Set(requests.map((record) => record.sessionId))) expect(summary.plans).toContainEqual(expect.objectContaining({ path: planPath, snapshotPath })) diff --git a/tests/e2e/tui/work-costs.test.ts b/tests/e2e/tui/work-costs.test.ts index feefa58b3..b49acbc20 100644 --- a/tests/e2e/tui/work-costs.test.ts +++ b/tests/e2e/tui/work-costs.test.ts @@ -6,13 +6,19 @@ import { setTimeout as sleep } from "node:timers/promises" import { SessionManager } from "@earendil-works/pi-coding-agent" import { test } from "@microsoft/tui-test" import { waitForText } from "./support/assertions.js" -import { runKimchiSession, TUI_TEST_CONFIG } from "./support/kimchi-fixture.js" +import { PROMPT_READY, runKimchiSession, TUI_TEST_CONFIG } from "./support/kimchi-fixture.js" import { BILLING_ACCOUNT, seedPendingWorkCost } from "./support/work-costs.js" test.use(TUI_TEST_CONFIG) -/** Background passes run every 30 seconds; wait for the one that publishes the expected costs. */ -async function waitForCosts(path: string, ready: (costs: Record) => boolean): Promise { +/** The cost totals `/work` reads; the full `costs.json` is refreshed at most every five minutes. */ +interface CostTotals { + requests?: { total?: unknown } + pullRequests?: { totalCostUsd?: unknown }[] +} + +/** Background passes run every 30 seconds; wait for the one that publishes the expected totals. */ +async function waitForTotals(path: string, ready: (totals: CostTotals) => boolean): Promise { const deadline = Date.now() + 45_000 while (Date.now() < deadline) { try { @@ -33,7 +39,7 @@ test("an open PR's pending cost becomes a confirmed and inferred total once it i const extraArgs: string[] = [] let billed = false let statePath = "" - let costsPath = "" + let totalsPath = "" let headSha = "" const setGitHub = (mode: string) => { writeFileSync(`${statePath}.next`, JSON.stringify({ mode, headSha })) @@ -91,7 +97,7 @@ test("an open PR's pending cost becomes a confirmed and inferred total once it i checkedAt: new Date().toISOString(), }, }) - costsPath = join(agentDir, "work", workId, "costs.json") + totalsPath = join(agentDir, "work", workId, "cost-totals.json") const fixtureDir = join(home, "pr-api-fixture") mkdirSync(fixtureDir) statePath = join(fixtureDir, "state.json") @@ -129,15 +135,18 @@ test("an open PR's pending cost becomes a confirmed and inferred total once it i }, async (_fixture, trace) => { await waitForText(terminal, "PR #731 open", { full: false, timeoutMs: 10_000 }) - await waitForCosts(costsPath, (costs) => costs.requests?.length === 1) + await waitForTotals(totalsPath, (totals) => totals.requests?.total === 1) terminal.submit("/work") await waitForText(terminal, "Cost so far: unknown; $0.00 priced (open)") await waitForText(terminal, "Prices: 0/1 requests priced, $0.00 known so far.") + // The work panel shows the current work's cost lines; close it before opening it again. + terminal.keyEscape() + await waitForText(terminal, PROMPT_READY, { full: false }) trace.step("an open PR's spend stays unknown while its request has no bill") billed = true setGitHub("merged") - await waitForCosts(costsPath, (costs) => - costs.pullRequests?.some((row) => (row as { totalCostUsd?: unknown }).totalCostUsd === "0.000166000"), + await waitForTotals(totalsPath, (totals) => + Boolean(totals.pullRequests?.some((row) => row.totalCostUsd === "0.000166000")), ) terminal.submit("/work") await waitForText(terminal, `Cost: $0.000166 — ${url}`) diff --git a/tests/e2e/tui/work-pr-discovery.test.ts b/tests/e2e/tui/work-pr-discovery.test.ts index a25876974..6b4ec1ff7 100644 --- a/tests/e2e/tui/work-pr-discovery.test.ts +++ b/tests/e2e/tui/work-pr-discovery.test.ts @@ -3,9 +3,10 @@ import { randomUUID } from "node:crypto" import { mkdirSync, readFileSync, realpathSync, renameSync, writeFileSync } from "node:fs" import { join } from "node:path" import { SessionManager } from "@earendil-works/pi-coding-agent" -import { expect, test } from "@microsoft/tui-test" -import { waitForText } from "./support/assertions.js" -import { runKimchiSession, TUI_TEST_CONFIG } from "./support/kimchi-fixture.js" +import { expect, Key, test } from "@microsoft/tui-test" +import { viewText, waitForText } from "./support/assertions.js" +import { PROMPT_READY, runKimchiSession, TUI_TEST_CONFIG } from "./support/kimchi-fixture.js" +import { readWorkSummary } from "./support/work-summary.js" test.use(TUI_TEST_CONFIG) @@ -111,18 +112,31 @@ test("pending work survives a GitHub login error and finds a PR while coding con setGitHub("auth") await waitForText(terminal, "PR/MR check /work", { full: false, timeoutMs: 10_000 }) terminal.submit("/work") - await waitForText(terminal, "GitHub authentication failed. Check the token for github.com.") - trace.step("GitHub token rejection is visible while coding stays available") + await waitForText(terminal, "Kimchi work", { full: false }) + await waitForText(terminal, "PR/MR lookup: GitHub authentication failed. Check the token for github.com.", { + full: false, + }) + terminal.keyEscape() + await waitForText(terminal, PROMPT_READY, { full: false }) + trace.step("the work panel shows the GitHub token rejection; Esc returns to coding") terminal.submit("Can I keep coding while GitHub is unavailable?") await waitForText(terminal, "You can keep coding while GitHub is unavailable.") setGitHub("open") await waitForText(terminal, "PR #731 open", { full: false, timeoutMs: 35_000 }) terminal.submit("/work") - await waitForText(terminal, "https://github.com/example/kimchi-lab/pull/731") - trace.step("the next background check finds the externally created PR") - const summary = JSON.parse(readFileSync(join(fixture.agentDir, "work", workId, "work.json"), "utf8")) - expect(summary.commits).toHaveLength(1) - expect(summary.commits[0]).toMatchObject({ + // The footer ends with `· PR #731 open` too; only the panel row has the work ID before it. + await waitForText(terminal, new RegExp(`${workId.slice(0, 8)}.*· PR #731 open`), { full: false }) + trace.step("the work panel lists the work with its open PR") + terminal.keyPress(Key.Enter) + await waitForText(terminal, PROMPT_READY, { full: false }) + const printed = viewText(terminal) + expect(printed).not.toContain("Esc close") + expect(printed).toContain(`Work ID: ${workId}`) + expect(printed).toContain("PR #731 open: https://github.com/example/kimchi-lab/pull/731") + trace.step("the next background check finds the PR; Enter prints the work's details") + const summary = readWorkSummary(fixture.agentDir, workId) + expect(summary?.commits).toHaveLength(1) + expect(summary?.commits[0]).toMatchObject({ sha: headSha, prLookup: { status: "linked" }, pullRequests: [{ number: 731, state: "open", headSha, repository: "example/kimchi-lab", host: "github.com" }], @@ -131,7 +145,11 @@ test("pending work survives a GitHub login error and finds a PR while coding con .trim() .split("\n") .map((line) => JSON.parse(line)) - expect(calls.filter((call) => call.kind === "commit").map((call) => call.mode)).toEqual(["auth", "open"]) + // Turn completion can refresh discovery before the periodic retry. + expect([...new Set(calls.filter((call) => call.kind === "commit").map((call) => call.mode))]).toEqual([ + "auth", + "open", + ]) expect(calls.every((call) => call.tokenMatched)).toBe(true) }, )