Skip to content

feat: deep token reduction — 4 phases of output compression improvements#1039

Open
ToyStoryCoder wants to merge 1 commit intortk-ai:developfrom
ToyStoryCoder:feat/deep-token-reduction
Open

feat: deep token reduction — 4 phases of output compression improvements#1039
ToyStoryCoder wants to merge 1 commit intortk-ai:developfrom
ToyStoryCoder:feat/deep-token-reduction

Conversation

@ToyStoryCoder
Copy link
Copy Markdown

Summary

4 phases of improvements to reduce LLM token consumption across all RTK command handlers.

Phase 1 — Stability & Code Quality

  • Replace unwrap() calls with proper error handling
  • Eliminate code duplication across command handlers
  • Add supply chain security with deny.toml (cargo-deny)
  • CI workflow improvements

Phase 2 — Core Token Savings

  • Lockfile detection (is_generated_file) for snapshots, minified assets, codegen
  • SmartFilter level with import/test collapsing, long string truncation
  • JSON array truncation (>50 items) with summary
  • Grep output deduplication for repeated patterns
  • Similar-file grouping in find output

Phase 3 — Extended Filters

  • Tree output cap (MAX_TREE_LINES = 200)
  • Git show --stat filtering (trim long paths, cap file list)
  • Git log width cap (120 chars)
  • Find filename grouping by extension
  • JSON support for Smart filter level
  • 3 new TOML filters (docker-build, flutter-build, pip-install)
  • Grep --no-compact flag support

Phase 4 — Deep Token Reduction

  • NOISE_DIRS expansion (+11 entries: .gradle, .cargo, .npm, vendor, etc.)
  • Git branch remote cap reduced 10→5
  • Stash list cap (MAX_STASH_ENTRIES = 20)
  • SmartFilter blank line normalization (3+ → 2)
  • SmartFilter doc-comment collapse (>15 lines → first 3 + summary)
  • Lockfile summary in rtk read (14 lockfile names)
  • ls entry cap (MAX_LS_ENTRIES = 50)
  • Curl HTML tag stripping (script/style removal, entity decoding)

Test plan

  • 1283 tests passing, 0 failures
  • No new clippy warnings introduced
  • 24 files changed, +1542 -111
  • All filter levels preserve escape hatch (--filter none / -l none)

@CLAassistant
Copy link
Copy Markdown

CLAassistant commented Apr 5, 2026

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Phase 1 — Stability & code quality:
- Replace unwrap() calls with proper error handling
- Eliminate code duplication across command handlers
- Add supply chain security with deny.toml (cargo-deny)
- CI workflow improvements

Phase 2 — Core token savings:
- Lockfile detection (is_generated_file) for snapshots, minified assets, codegen
- SmartFilter level with import/test collapsing, long string truncation
- JSON array truncation (>50 items) with summary
- Grep output deduplication for repeated patterns
- Similar-file grouping in find output

Phase 3 — Extended filters:
- Tree output cap (MAX_TREE_LINES = 200)
- Git show --stat filtering (trim long paths, cap file list)
- Git log width cap (120 chars)
- Find filename grouping by extension
- JSON support for Smart filter level
- 3 new TOML filters (docker-build, flutter-build, pip-install)
- Grep --no-compact flag support

Phase 4 — Deep token reduction:
- NOISE_DIRS expansion (+11 entries: .gradle, .cargo, .npm, vendor, etc.)
- Git branch remote cap reduced 10→5
- Stash list cap (MAX_STASH_ENTRIES = 20)
- SmartFilter blank line normalization (3+ → 2)
- SmartFilter doc-comment collapse (>15 lines → first 3 + summary)
- Lockfile summary in rtk read (14 lockfile names)
- ls entry cap (MAX_LS_ENTRIES = 50)
- Curl HTML tag stripping (script/style removal, entity decoding)

All 1283 tests passing. No new clippy warnings introduced.
@ToyStoryCoder ToyStoryCoder force-pushed the feat/deep-token-reduction branch from e9a56ec to de2f134 Compare April 5, 2026 15:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants