Skip to content

feat(gain): add --quality flag for filter quality analysis#900

Open
tmchow wants to merge 1 commit intortk-ai:developfrom
tmchow:osc/feat-gain-quality-scorecard
Open

feat(gain): add --quality flag for filter quality analysis#900
tmchow wants to merge 1 commit intortk-ai:developfrom
tmchow:osc/feat-gain-quality-scorecard

Conversation

@tmchow
Copy link
Copy Markdown

@tmchow tmchow commented Mar 28, 2026

Summary

  • Add rtk gain --quality / rtk gain -Q that analyzes tracking data for filter quality signals
  • Surfaces retry detection, low-savings commands, parse failures, and net savings estimate
  • Follows the existing --failures flag pattern, no new dependencies

Why this matters

RTK has 12 open issues tagged filter-quality where filtering removes too much information, causing AI agents to retry commands and waste more tokens than RTK saves. The tracking database records every command execution but provides no quality analysis.

  • #831 - AI agents burn extra tokens on retry loops from over-filtering
  • #839 - Request for empirical benchmarks (2 thumbs up)
  • #846, #844, #838, #803, #759 - additional filter-quality issues

What it does

rtk gain --quality reports four sections:

  1. Retry Detection - Commands re-run within 60 seconds, grouped by base command (e.g., all git diff variants). Uses a LAG window function for O(n log n) performance.
  2. Low Savings - Filters consistently delivering <30% savings, excluding proxy/passthrough commands that intentionally have 0%.
  3. Parse Failures - Reuses existing parse_failures table to show filters that fell back to raw output.
  4. Net Savings - Gross savings minus estimated retry overhead.

The 60-second retry window is a heuristic. Normal workflow repetition (e.g., git status during staging) will be counted. The report surfaces patterns for investigation, not definitive quality problems.

Demo

Demo

Changes

  • src/main.rs: Add quality: bool field to Gain variant, wire early return to gain::show_quality()
  • src/tracking.rs: Add RetryRecord/LowSavingsRecord structs, get_retry_commands(), get_low_savings_commands(), get_gross_savings() methods
  • src/gain.rs: Add show_quality() function following the show_failures() pattern

Test plan

  • cargo fmt --all && cargo clippy --all-targets && cargo test (1123 passed, 0 failed)
  • Manual testing: rtk gain --quality against real tracking DB (36M tokens saved, 95% efficiency)

Relates to #831, #839

This contribution was developed with AI assistance (Claude Code).

@CLAassistant
Copy link
Copy Markdown

CLAassistant commented Mar 28, 2026

CLA assistant check
All committers have signed the CLA.

@tmchow tmchow force-pushed the osc/feat-gain-quality-scorecard branch from 91c5f20 to 668661f Compare March 28, 2026 17:06
Add `rtk gain --quality` / `rtk gain -Q` that analyzes tracking data
for filter quality signals:

- Retry detection: commands re-run within 60 seconds (possible retries
  from insufficient filtered output)
- Low savings detection: filters consistently delivering <30% savings
  (excludes proxy/passthrough commands)
- Parse failures summary: filters that fell back to raw output
- Net savings estimate: gross savings minus retry overhead

Uses LAG window function for O(n log n) retry detection with
base-command grouping (e.g., "git diff" groups all git diff variants).

Addresses rtk-ai#831 (AI retry loops from over-filtering) and rtk-ai#839
(empirical benchmarks for savings claims).

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@tmchow tmchow force-pushed the osc/feat-gain-quality-scorecard branch from 668661f to 928a4a2 Compare March 28, 2026 17:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants