Skip to content

perf(benchmarks): P1 — establish a reproducible ADR Guard performance baseline #196

Closed

Description

@rodri-oliveira-dev

Child of P1 quality roadmap #193. Start implementation after PR #192 is merged or explicitly abandoned. Recheck current main first.

Problem / existing baseline

ADR Guard has bounded parsing, Git inventory, MCP search/listing and architecture-analysis contracts, but the existence of hard bounds is not evidence of latency, throughput or memory characteristics across realistic ADR sets. Optimize only after measuring.

Goal

Create a small, repeatable informational performance baseline for core ADR workflows that future changes can compare against, without production data, telemetry or fragile CI gates.

Scope / methodology

  1. Inventory any existing benchmark scripts, relevant fixture generators, limits and release/build paths. Reuse them where appropriate.
  2. Generate deterministic, version-controlled synthetic scenarios/seed for approximately 100, 500 and 900 ADRs, below the documented MCP 1,000-ADR ceiling. Include plausible statuses, relationships, sizes, and some representative errors; avoid accidental quadratic fixture generation.
  3. Measure at minimum check and index using a packaged/release-built CLI in a disposable repository, without mutating source fixtures; optionally add an MCP list_adrs / search_adrs handshake suite only if straightforward and meaningfully comparable.
  4. Record cold/warm behavior separately where practical, repeated-run timings (median and spread, optionally p95), peak memory where portable, scenario sizes, OS/CPU/RAM, .NET runtime, exact commit/tool version and commands. Separate setup/generation time from measured invocation time.
  5. Keep the run manual/local or informational at first. Document runner noise and prevent automated fail/pass performance thresholds until a stable measured baseline exists.
  6. Identify any observed bottlenecks, but open follow-up optimization issues only when supported by profiling and user-relevant evidence.

Acceptance criteria

  • A deterministic generator or checked-in small fixture recipe is available and documented; no confidential or identifiable repository content.
  • Reproduction works from a clean checkout with explicit build/install, seed and command steps; outputs carry source revision/environment metadata.
  • Baseline measurements for 100, 500 and 900 ADRs, for check and index, include multiple samples, reported variability and (where feasible) peak process memory.
  • Benchmarks isolate setup cost, avoid external network/provider calls and leave repository files clean.
  • Findings clearly separate observed numbers from hypotheses and proposed optimizations.
  • Scripts/tests and EN/PT-BR usage docs (if publicly exposed) are reviewed; no brittle mandatory CI threshold or release commitment.

Non-goals

No speculative algorithm rewrites, mandatory benchmark gating on shared runners, arbitrary latency SLAs, repository telemetry, AI-provider benchmarks, SaaS load testing or promises based on fabricated results.

Completion: A focused benchmark harness + documented real baseline, or an evidence-based no-change report if an equivalent reproducible harness already exists.

Activity

  1. rodri-oliveira-dev commented on Oct 18, 2026

    @rodri-oliveira-dev
    OwnerAuthor

    Checkpoint da #196 concluído na mesma branch.

    Implementação:

    • harness .NET sem dependências externas em benchmarks/AdrGuard.Benchmarks;
    • geração determinística de corpora válidos/inválidos com 100, 500 e 900 ADRs, status variados, referências/dependências resolvidas e erros representativos;
    • preparação medida separadamente, primeira execução + 5 amostras subsequentes, mediana/min/max/spread e pico de working set;
    • metadados de SO/arquitetura/runtime/CPU/memória, versão da CLI, SHA, dirty state e hashes dos datasets;
    • limpeza em finally, sem dados reais, rede ou provider;
    • sem threshold de CI/SLA.

    Commits:

    • harness: 9c2a27069f4a5035f5b19c5082b1bcf6e63d89a0
    • resultados/documentação: 4c34bfe480277f64147f9a98aad4645c4b30752d

    Medição real no Windows x64, .NET 10.0.13, SHA da CLI/harness 9c2a270... (medianas de 5 amostras subsequentes):

    • 100 ADRs: check inválido 249,938 ms / 30,7 MiB; index válido 288,936 ms / 30,8 MiB;
    • 500 ADRs: check 323,914 ms / 35,5 MiB; index 304,722 ms / 35,7 MiB;
    • 900 ADRs: check 360,756 ms / 36,4 MiB; index 357,520 ms / 36,9 MiB.

    O JSON completo está em benchmarks/results/windows-x64-2026-10-18.json; comandos e interpretação estão em docs/quality-baseline.md e .pt-BR.md. Duas gerações completas produziram hashes idênticos; nenhum diretório temporário permaneceu. A primeira execução de check/100 teve ruído alto (1160,496 ms), portanto não foi inferido gargalo nem proposta otimização.

    Validação local:

    • report/SHAs/shape/cleanup validados;
    • restore e build Release: 0 warnings, 0 errors;
    • suíte .NET: Passed 699, Failed 0, Skipped 0;
    • sincronização oficial do portal: 78 documentos.

    Limitações: somente Windows foi medido neste ambiente; o relatório marca worktree dirty devido à alteração preexistente e não relacionada em .gitignore. Nenhum número de macOS/Linux foi inventado.

  2. rodri-oliveira-dev commented on Oct 18, 2026

    @rodri-oliveira-dev
    OwnerAuthor

    Atualização após a revisão integrada:

    Foi identificado que o gerador usava separadores de linha do host, o que permitiria hashes diferentes entre Windows e Unix. A correção normaliza todo dataset sintético para LF.

    • correção de portabilidade: f32b81956037b4935fc3da2cd64bd763e8b7491c
    • relatório atualizado: 0ad75d106521575a5f363865e4c2691f906e6c7d

    O benchmark foi reconstruído e reexecutado integralmente com CLI/versionamento no SHA f32b819.... Medianas atualizadas (5 amostras):

    • 100: check 255,384 ms / 30,5 MiB; index 242,322 ms / 30,6 MiB;
    • 500: check 417,161 ms / 35,5 MiB; index 304,760 ms / 35,6 MiB;
    • 900: check 368,928 ms / 36,5 MiB; index 367,128 ms / 36,3 MiB.

    Uma segunda geração independente (3 amostras, relatório temporário removido) confirmou os mesmos seis hashes de dataset. O JSON e as tabelas EN/PT-BR foram substituídos pelos resultados pós-correção; os números anteriores devem ser desconsiderados.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions