Repository navigation
perf(benchmarks): P1 — establish a reproducible ADR Guard performance baseline #196
Description
Child of P1 quality roadmap #193. Start implementation after PR #192 is merged or explicitly abandoned. Recheck current
mainfirst.
Problem / existing baseline
ADR Guard has bounded parsing, Git inventory, MCP search/listing and architecture-analysis contracts, but the existence of hard bounds is not evidence of latency, throughput or memory characteristics across realistic ADR sets. Optimize only after measuring.
Goal
Create a small, repeatable informational performance baseline for core ADR workflows that future changes can compare against, without production data, telemetry or fragile CI gates.
Scope / methodology
- Inventory any existing benchmark scripts, relevant fixture generators, limits and release/build paths. Reuse them where appropriate.
- Generate deterministic, version-controlled synthetic scenarios/seed for approximately 100, 500 and 900 ADRs, below the documented MCP 1,000-ADR ceiling. Include plausible statuses, relationships, sizes, and some representative errors; avoid accidental quadratic fixture generation.
- Measure at minimum
checkandindexusing a packaged/release-built CLI in a disposable repository, without mutating source fixtures; optionally add an MCPlist_adrs/search_adrshandshake suite only if straightforward and meaningfully comparable. - Record cold/warm behavior separately where practical, repeated-run timings (median and spread, optionally p95), peak memory where portable, scenario sizes, OS/CPU/RAM, .NET runtime, exact commit/tool version and commands. Separate setup/generation time from measured invocation time.
- Keep the run manual/local or informational at first. Document runner noise and prevent automated fail/pass performance thresholds until a stable measured baseline exists.
- Identify any observed bottlenecks, but open follow-up optimization issues only when supported by profiling and user-relevant evidence.
Acceptance criteria
- A deterministic generator or checked-in small fixture recipe is available and documented; no confidential or identifiable repository content.
- Reproduction works from a clean checkout with explicit build/install, seed and command steps; outputs carry source revision/environment metadata.
- Baseline measurements for 100, 500 and 900 ADRs, for
checkandindex, include multiple samples, reported variability and (where feasible) peak process memory. - Benchmarks isolate setup cost, avoid external network/provider calls and leave repository files clean.
- Findings clearly separate observed numbers from hypotheses and proposed optimizations.
- Scripts/tests and EN/PT-BR usage docs (if publicly exposed) are reviewed; no brittle mandatory CI threshold or release commitment.
Non-goals
No speculative algorithm rewrites, mandatory benchmark gating on shared runners, arbitrary latency SLAs, repository telemetry, AI-provider benchmarks, SaaS load testing or promises based on fabricated results.
Completion: A focused benchmark harness + documented real baseline, or an evidence-based no-change report if an equivalent reproducible harness already exists.
Activity
perf(benchmarks): add reproducible performance harness (#196)
docs(perf): record reproducible Windows baseline (#196)
rodri-oliveira-dev commentedon Oct 18, 2026
Checkpoint da #196 concluído na mesma branch.
Implementação:
benchmarks/AdrGuard.Benchmarks;finally, sem dados reais, rede ou provider;Commits:
9c2a27069f4a5035f5b19c5082b1bcf6e63d89a04c34bfe480277f64147f9a98aad4645c4b30752dMedição real no Windows x64, .NET 10.0.13, SHA da CLI/harness
9c2a270...(medianas de 5 amostras subsequentes):O JSON completo está em
benchmarks/results/windows-x64-2026-10-18.json; comandos e interpretação estão emdocs/quality-baseline.mde.pt-BR.md. Duas gerações completas produziram hashes idênticos; nenhum diretório temporário permaneceu. A primeira execução de check/100 teve ruído alto (1160,496 ms), portanto não foi inferido gargalo nem proposta otimização.Validação local:
Limitações: somente Windows foi medido neste ambiente; o relatório marca worktree dirty devido à alteração preexistente e não relacionada em
.gitignore. Nenhum número de macOS/Linux foi inventado.fix(benchmarks): normalize synthetic datasets across platforms (#196)
docs(perf): refresh baseline after portability review (#196)
rodri-oliveira-dev commentedon Oct 18, 2026
Atualização após a revisão integrada:
Foi identificado que o gerador usava separadores de linha do host, o que permitiria hashes diferentes entre Windows e Unix. A correção normaliza todo dataset sintético para LF.
f32b81956037b4935fc3da2cd64bd763e8b7491c0ad75d106521575a5f363865e4c2691f906e6c7dO benchmark foi reconstruído e reexecutado integralmente com CLI/versionamento no SHA
f32b819.... Medianas atualizadas (5 amostras):Uma segunda geração independente (3 amostras, relatório temporário removido) confirmou os mesmos seis hashes de dataset. O JSON e as tabelas EN/PT-BR foram substituídos pelos resultados pós-correção; os números anteriores devem ser desconsiderados.