feat(gain): add --quality flag for filter quality analysis#900
Open
tmchow wants to merge 1 commit intortk-ai:developfrom
Open
feat(gain): add --quality flag for filter quality analysis#900tmchow wants to merge 1 commit intortk-ai:developfrom
tmchow wants to merge 1 commit intortk-ai:developfrom
Conversation
91c5f20 to
668661f
Compare
Add `rtk gain --quality` / `rtk gain -Q` that analyzes tracking data for filter quality signals: - Retry detection: commands re-run within 60 seconds (possible retries from insufficient filtered output) - Low savings detection: filters consistently delivering <30% savings (excludes proxy/passthrough commands) - Parse failures summary: filters that fell back to raw output - Net savings estimate: gross savings minus retry overhead Uses LAG window function for O(n log n) retry detection with base-command grouping (e.g., "git diff" groups all git diff variants). Addresses rtk-ai#831 (AI retry loops from over-filtering) and rtk-ai#839 (empirical benchmarks for savings claims). Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
668661f to
928a4a2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
rtk gain --quality/rtk gain -Qthat analyzes tracking data for filter quality signals--failuresflag pattern, no new dependenciesWhy this matters
RTK has 12 open issues tagged
filter-qualitywhere filtering removes too much information, causing AI agents to retry commands and waste more tokens than RTK saves. The tracking database records every command execution but provides no quality analysis.What it does
rtk gain --qualityreports four sections:git diffvariants). Uses a LAG window function for O(n log n) performance.parse_failurestable to show filters that fell back to raw output.The 60-second retry window is a heuristic. Normal workflow repetition (e.g.,
git statusduring staging) will be counted. The report surfaces patterns for investigation, not definitive quality problems.Demo
Changes
src/main.rs: Addquality: boolfield toGainvariant, wire early return togain::show_quality()src/tracking.rs: AddRetryRecord/LowSavingsRecordstructs,get_retry_commands(),get_low_savings_commands(),get_gross_savings()methodssrc/gain.rs: Addshow_quality()function following theshow_failures()patternTest plan
cargo fmt --all && cargo clippy --all-targets && cargo test(1123 passed, 0 failed)rtk gain --qualityagainst real tracking DB (36M tokens saved, 95% efficiency)Relates to #831, #839
This contribution was developed with AI assistance (Claude Code).