Skip to content

[11] Store work summaries as row logs - #1389

Open
tautvydasLiekis wants to merge 18 commits into
feat/browse-work-costsfrom
feat/split-work-files
Open

tautvydasLiekis wants to merge 18 commits into
feat/browse-work-costsfrom
feat/split-work-files

Conversation

@tautvydasLiekis

@tautvydasLiekis tautvydasLiekis commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Makes work summary updates cost the same at any size. Every request, response, billing change, edit and commit used to read, merge and rewrite the whole work/<id>/work.json, so large works slowed the TUI and eventually broke /work. Now work.json is a small manifest and the rows go into append-only logs.

Stacked on #1386. Ship it in the same release as #1348 and #1386, or before them: #1348 makes each request row about five times larger, and #1386 adds the /work browser that reads these files.

Problem

  • Growing one work to 2,000 requests wrote 17.6 GB to disk.
  • Each update blocked the event loop longer as the work grew: 229 ms at 20,000 requests.
  • Above about 2,800 billed requests work.json passed the browser's 8 MiB read limit, so /work showed "Summary unavailable" and lost the work's PR state.
  • Opening /work with 30 medium works parsed about 190 MB of JSON.

How it works

work/<id>/work.json                      # version 2 manifest: what /work lists, one entry per row log
work/<id>/rows/<collection>.<g>.jsonl    # one row per line, sessions included; a later line with the same key wins
work/<id>/cost-totals.json               # new: the bounded figures /work prints
work/<id>/costs.json                     # unchanged shape, refreshed at most every five minutes
  • Writes: under the existing per-work lock, a writer appends only changed rows, syncs them, then replaces work.json, which commits them. Readers stop at each log's committed length, so an interrupted append is ignored and the next writer cuts it off. Merge rules are unchanged.
  • Reads during an update: each process remembers where rows are, up to 200,000 rows across works. It reads a whole log once, the first time it updates that work or after another process compacted it; later updates read only the lines they change.
  • Compaction: when superseded lines outweigh half a log's live rows and exceed 1 MiB, the latest line per key is copied into a new generation.
  • /work reads only manifests and cost-totals.json, never the row logs. The text /work in ACP and print mode uses the same totals.
  • costs.json keeps its shape but is rewritten at most once every five minutes per work (immediately on a process's first pass, or when it or cost-totals.json is missing).

Evidence

Measured during development on an Apple M4 Pro, base 74f8e0d0 against an earlier build of this branch, median of 15 updates in one process:

Requests Update time Bytes written per update Longest event-loop stall /work details
1,000 19.0 → 11.0 ms 3.06 MB → 2.4 KB 12.8 → 2.1 ms 1.6 → 0.0 ms
5,000 75.2 → 11.0 ms 15.3 MB → 4.2 KB 57.0 → 2.0 ms 9.2 → 0.0 ms
20,000 262.2 → 10.9 ms 61.0 MB → 10.8 KB 229.2 → 2.1 ms 37.6 → 0.1 ms
  • Growing one work from 0 to 2,000 requests: 17,591 MB written → 19.4 MB.
  • Opening /work with 30 works of 5,000 requests: 379 ms, +474 MB heap and 0 of 30 summaries readable → 3 ms, +1 MB heap, 30 of 30.
  • Lab on a binary built from this head: one work with 5,000 seeded requests had an 8.3 MiB requests log and a 1.1 KB manifest; live updates took 19, 40 and 40 ms and /work opened in at most 357 ms.

On 95614e7c6, after the final review fixes:

  • CI: build, pr-checks, master-checks, TUI e2e in 4 shards, ACP e2e and MCP e2e pass on this head. A duplicate tui-e2e entry from a superseded workflow run shows as cancelled.
  • pnpm check passes; the full unit suite passes except 5 tests that need tools this machine lacks: a DAP js-debug adapter, a built validate-skill.mjs and a binary rebuilt for the smoke test.
  • New test: a manifest that still lists its sessions reads unchanged, and the next update moves them into the sessions log.
  • Lab on a binary built from this head (scripted model; local GitHub, billing and report fixtures): 73/73 default scenarios pass.

Tests also cover:

  • three checked-in version 1 fixtures that read unchanged and fold byte-for-byte equal after migration;
  • seeded random sequences over all ten record types, through migration, compaction and a simulated v1.7.1 rewrite, compared after every batch with the same merge applied to an in-memory version 1 summary (the merge loop itself is unchanged);
  • interrupted appends, missing, short or damaged logs, deleted manifests, crashes during compaction, and concurrent readers and writers across processes;
  • a restarted process that reads an unchanged log once, not on every update;
  • /work cost lines from cost-totals.json equal to the previous output for every saved report in the cost-sync tests, connected works included, and totals that are written again after being deleted.

Compatibility and risks

  • Version 1 summaries stay readable everywhere and migrate lazily on their next write. Idle works are not rewritten.
  • Damage: a missing or invalid manifest, a short log or a damaged row rebuilds the work from its journals.
  • Downgrade to v1.7.1 or older is safe but slow once per switch: the old version rejects version: 2, rebuilds a version 1 file from the journals, and this build migrates it again on return.
  • Version 1 files over 8 MiB, written only by unreleased stack builds, show summary too large in /work with PR state and partial spend until their next update.
  • The /work header now adds each listed work's connected cost report once, so connected works that are not listed also count. [10] Browse works, costs and PRs in /work #1386 counted only the listed works' own requests.
  • First update after a restart reads the work's logs once; the cost is proportional to the work's size.
  • Not changed: a cost pass after a journal change still rereads every journal, and costs.json still repeats a connected group in each member's file.

docs/work-attribution.md lists each row log and its keys.

🤖 Generated with Claude Code

@readme-ai-writer

readme-ai-writer Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

📚 No documentation changes were needed for this PR.

@tautvydasLiekis
tautvydasLiekis force-pushed the feat/split-work-files branch 2 times, most recently from 5f0fa95 to f208f42 Compare October 9, 2026 08:17
@tautvydasLiekis
tautvydasLiekis force-pushed the feat/browse-work-costs branch 2 times, most recently from 41d9c6f to 1310db0 Compare October 9, 2026 08:39
@tautvydasLiekis
tautvydasLiekis force-pushed the feat/split-work-files branch 2 times, most recently from 466a5cc to f10ef52 Compare October 9, 2026 21:49

@mszostok mszostok left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I strongly doubt the compatibility is necessary.

I'd drop v1 entirely and not care about existing works: no v1 reader, no migration, no fixtures, no rebuild-on-upgrade. Just have the new version delete old v1 work.json files and let this PR focus solely on v2.

@mszostok

Copy link
Copy Markdown
Contributor

also quite potential alternative for that in the future can be SQLite 🤔

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants