feat(telemetry): enrich daily ping with gap detection and quality metrics#1001
Open
pszymkowiak wants to merge 3 commits intodevelopfrom
Open
feat(telemetry): enrich daily ping with gap detection and quality metrics#1001pszymkowiak wants to merge 3 commits intodevelopfrom
pszymkowiak wants to merge 3 commits intodevelopfrom
Conversation
…rics Add 6 new fields to the anonymous daily telemetry ping to help identify which commands need filters and which filters need improvement: - passthrough_top: top 5 commands with 0% savings (missing filters) - parse_failures_24h: count of parse failures (filter fragility) - low_savings_commands: commands averaging <30% savings (weak filters) - avg_savings_per_command: unweighted average savings - hook_type: which AI agent hook is installed (claude/gemini/codex/etc) - custom_toml_filters: count of user-defined TOML filter files New tracking.rs queries: top_passthrough(), parse_failures_since(), low_savings_commands(), avg_savings_per_command(). Signed-off-by: Patrick szymkowiak <[email protected]>
Extend the daily anonymous ping with product-piloting metrics: Retention: first_seen_days, active_days_30d, commands_total Ecosystem: ecosystem_mix (category distribution percentages) Economics: tokens_saved_30d, estimated_savings_usd_30d Config: has_config_toml, exclude_commands_count, projects_count Features: meta_usage (gain, discover, proxy, verify, learn counts) Update README.md privacy section with full field-by-field table explaining what is collected and why it helps improve RTK. Signed-off-by: Patrick szymkowiak <[email protected]>
Comprehensive telemetry documentation covering: - Why we collect (roadmap prioritization, filter quality, value measurement) - How it works (daily ping, background thread, fire-and-forget) - Every field with example values and purpose - What is NOT collected (explicit exclusion list) - Opt-out instructions - Data handling and privacy guarantees - Contributor guide for adding new fields Link added from README.md privacy section. Signed-off-by: Patrick szymkowiak <[email protected]>
Contributor
|
Could you move new TELEMETRY.md documentation into docs/usage/ folder please ? This will match the docs folder re-organization of PR #978 |
Contributor
Possible data accuracy issues1. top_passthrough SQL condition is wrongWHERE input_tokens = 0 AND output_tokens = 0 2. parse_failures_since assumes a parse_failures table existsSELECT COUNT(*) FROM parse_failures WHERE timestamp >= ?1 3. projects_count assumes project_path column existsSELECT COUNT(DISTINCT project_path) FROM commands WHERE project_path != '' |
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Enrich the anonymous daily telemetry ping with product-piloting metrics. All data is aggregate counts or anonymized — no file paths, no arguments, no personal data.
New fields (21 total, up from 10)
passthrough_top,parse_failures_24h,low_savings_commands,avg_savings_per_commandfirst_seen_days,active_days_30d,commands_totalecosystem_mixtokens_saved_30d,estimated_savings_usd_30dhook_type,custom_toml_filtershas_config_toml,exclude_commands_count,projects_countmeta_usageFiles changed
src/core/tracking.rs— 10 new query methods (first_seen_days, active_days_30d, commands_total, ecosystem_mix, tokens_saved_30d, projects_count + previous 4)src/core/telemetry.rs— EnrichedStats struct with all fields, detect_hook_type, detect_has_config, count_exclude_commands, count_custom_toml_filters, build_meta_usage, categorize_commandREADME.md— Updated privacy section with full field-by-field table explaining what is collected and whyPrivacy
README.md now has a detailed table showing every collected field and its purpose. All new fields follow the same principles: aggregate counts, anonymized command names (first 3 words), no file paths, no arguments, no user data.
Test plan
cargo fmt --all— cleancargo clippy --all-targets— no new warningscargo test telemetry— 13 tests passed (3 new)cargo test --all— 1256 passed