feat: deep token reduction — 4 phases of output compression improvements#1039
Open
ToyStoryCoder wants to merge 1 commit intortk-ai:developfrom
Open
feat: deep token reduction — 4 phases of output compression improvements#1039ToyStoryCoder wants to merge 1 commit intortk-ai:developfrom
ToyStoryCoder wants to merge 1 commit intortk-ai:developfrom
Conversation
|
|
Phase 1 — Stability & code quality: - Replace unwrap() calls with proper error handling - Eliminate code duplication across command handlers - Add supply chain security with deny.toml (cargo-deny) - CI workflow improvements Phase 2 — Core token savings: - Lockfile detection (is_generated_file) for snapshots, minified assets, codegen - SmartFilter level with import/test collapsing, long string truncation - JSON array truncation (>50 items) with summary - Grep output deduplication for repeated patterns - Similar-file grouping in find output Phase 3 — Extended filters: - Tree output cap (MAX_TREE_LINES = 200) - Git show --stat filtering (trim long paths, cap file list) - Git log width cap (120 chars) - Find filename grouping by extension - JSON support for Smart filter level - 3 new TOML filters (docker-build, flutter-build, pip-install) - Grep --no-compact flag support Phase 4 — Deep token reduction: - NOISE_DIRS expansion (+11 entries: .gradle, .cargo, .npm, vendor, etc.) - Git branch remote cap reduced 10→5 - Stash list cap (MAX_STASH_ENTRIES = 20) - SmartFilter blank line normalization (3+ → 2) - SmartFilter doc-comment collapse (>15 lines → first 3 + summary) - Lockfile summary in rtk read (14 lockfile names) - ls entry cap (MAX_LS_ENTRIES = 50) - Curl HTML tag stripping (script/style removal, entity decoding) All 1283 tests passing. No new clippy warnings introduced.
e9a56ec to
de2f134
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
4 phases of improvements to reduce LLM token consumption across all RTK command handlers.
Phase 1 — Stability & Code Quality
Phase 2 — Core Token Savings
Phase 3 — Extended Filters
Phase 4 — Deep Token Reduction
Test plan