Skip to content

QualityAssurance.md

Latest commit

 

History

History
206 lines (149 loc) · 7.53 KB

QualityAssurance.md

File metadata and controls

206 lines (149 loc) · 7.53 KB
Raw

Automated QA for AI Coding Agents (Compiler Projects)

This is a concise, fully automated QA framework for compiler codebases. It is designed for AI coding agents, not humans. Human review is optional and used only for policy overrides.

Scope

Applies to:

  • New changes (features, refactors, fixes).
  • Historical code (legacy modules, regressions, untested paths).

Core Rules (Non‑Negotiable)

  1. Evidence or reject: If required checks are missing, the change is blocked.
  2. Fail closed: Unknown risk defaults to highest risk.
  3. Self-correcting: The agent must repair failures and re-run.
  4. No silent behavior change: All behavior changes must be declared and tested.

Quick Start (Actionable)

  1. Classify risk: low | medium | high
  2. Select gates based on impacted compiler work and artefacts.
  3. Run gates via the unified entrypoint: scripts/qa_local.sh --risk <risk>
  4. Collect evidence under qa/evidence/ and finalize the decision.

Notes:

  • scripts/qa_local.sh is the primary local gate runner. It orchestrates work-appropriate checks and is the default path for both CI and local verification.
  • Evidence is mandatory: if an applicable gate is missing or evidence is missing, the change is blocked.

Gate Entrypoints (Authoritative)

Use these scripts as the canonical entrypoints for QA gates:

  • scripts/qa_local.sh --risk <low|medium|high> — Primary local/CI gate runner. Always use this for formal validation.
  • scripts/verify_semantics_matrix.sh — Semantic contract gate (IR snapshots + hash guard). Called by qa_local.sh at medium+.
  • scripts/verify_prod.sh — High-risk supplemental checks (risk inventory + rust-ui + jit e2e). This is not a full release gate by itself.

Release Gate (Non-Bypassable)

Production releases must satisfy all of the following:

  • scripts/qa_local.sh --risk high passes.
  • Evidence record exists under qa/evidence/ for the release change set.
  • Semantic matrix gate evidence (qa/ir_snapshots/ + qa/ir_snapshots/ir_hashes.json) is captured and referenced.

scripts/verify_prod.sh is a subset of high-risk checks; it does not replace qa_local.sh --risk high.

Risk Levels

Risk Required Gates
Low Build + lint + unit tests
Medium Low + integration + compiler-work gates
High Medium + runtime fixtures + bootstrap + perf

If risk is unknown, treat as High. Evidence must reflect the chosen risk level and its required gates.

Compiler Work Gates

Select only the gates that match the change:

  • Parsing/AST: parser tests, AST artefacts, round-trip tests.
  • Typing: inference tests, negative cases, ambiguity checks.
  • Requests/comptime: CompileTimeNeed, RequestId, generated AST, and parity vs runtime where constructs overlap.
  • Lowering (typed AST -> HIR -> MIR -> LIR): IR artefacts, structural invariants, dependency invalidation.
  • Diagnostics: UI/artefact tests for messages, spans, request ids, and codes.
  • Codegen/Backend: IR/ASM artefacts, executable fixtures.
  • Optimization: pass unit tests, semantic equivalence checks.
  • Bootstrap: self-host build if supported.
  • Performance: compile‑time and runtime benchmarks for hot paths.

If a gate is applicable but missing, the agent must add tests or block.

Semantic Contract Gate (Required for Medium+)

Changes that can affect observable semantics must run the semantic matrix gate:

  • scripts/verify_semantics_matrix.sh
    • Runs scripts/snapshot_ir.sh to generate IR snapshots from tests/fixtures/semantic.
    • Enforces qa/ir_snapshots/ir_hashes.json via scripts/ir_hash_guard.py.

UPDATE_IR_HASHES Policy

UPDATE_IR_HASHES=1 is only allowed when the semantic change is declared:

  • The semantic matrix entry must be updated with a rationale and expected delta.
  • The baseline suite mapping must be updated (or explicitly marked as new coverage).
  • The evidence record must include the change justification and updated artifacts.

If the change is not intentional or not declared, UPDATE_IR_HASHES must remain unset and the gate must fail.

Evidence (Required)

Each change produces a single evidence record. Use the template and generator:

  • Template: qa/evidence/template.md
  • Generator: scripts/qa_evidence.sh

The evidence file must align with the selected risk level and the gates actually executed by scripts/qa_local.sh.

Required Evidence Artifacts (Semantic Gates)

For any semantic or lowering change (including IR-affecting refactors), attach:

  • qa/ir_snapshots/ (generated by scripts/snapshot_ir.sh).
  • qa/ir_snapshots/ir_hashes.json (baseline hash file enforced by scripts/ir_hash_guard.py).
  • Any delta summary produced by scripts/verify_semantics_matrix.sh (stdout/stderr logs).

Evidence Format (Machine‑Readable)

Each change produces a single file:

qa/evidence/<change-id>.yaml

change_id: <unique id>
intent: <short description>
risk: <low|medium|high>
work: [parser, typing, lowering, ...]
invariants: [ ... ]
expected_behavior: [ ... ]
checks:
  build: pass
  lint: pass
  unit: pass
  integration: pass
  compiler_work_gates: pass
  runtime_fixtures: pass
  bootstrap: pass
  perf: pass
artifacts:
  logs: [ ... ]
  reports: [ ... ]

Missing fields or failed checks are hard failures.

Automated Review Rules (No Human Required)

  • Reject broad refactors unless explicitly requested.
  • Reject any new unwrap/panic on user-controlled inputs.
  • Require stable IR/diagnostic snapshots unless the change declares deltas.
  • Require regression tests for all fixed bugs.

File-by-File Review (AI Agent, One-by-One)

When the agent is asked to review code quality or historical code, it performs a deterministic one-by-one read of each relevant source file (in repo order), not just pattern matching. Each file gets a short, explicit reasoning note from the source file, not individual lines.

Procedure

  1. Enumerate files in a stable order (lexicographic), excluding target/, .git/, and any tests/ paths unless explicitly requested.
  2. Read the entire file top to bottom; do not stop at the first match.
  3. Summarize file intent from module docs, public APIs, and main types.
  4. Identify concerns with explicit reasoning:
    • Correctness gaps (incomplete lowering, missing invariants, undefined behavior).
    • Confusing semantics (implicit fallbacks, silent behavior changes).
    • Reliability risks (panic/unwrap in user paths, unsafe without invariants).
    • Each finding must include why it is a problem in this file, not just a label.
  5. Record findings with:
    • File path and key lines.
    • Why it is risky/confusing (explicit reasoning tied to actual code flow).
    • Suggested remediation class (diagnostic, refactor, test, or guard).

Output Format (Recommended)

File: <path>
Intent: <short statement>
Findings:
- <issue> — <reasoning> — <remediation>
Notes:
- <any follow-ups>

The agent must keep a cumulative report and never skip files unless told to.

Historical Code (Legacy Stabilization)

When touching legacy areas:

  1. Add a baseline gate first (snapshot or fixture) to lock current behavior.
  2. Then change code and re-run gates.
  3. Any change must be explicitly declared in the evidence file.

This prevents accidental regressions while enabling safe modernization.

Automation Hooks (Suggested)

  • CI pipeline runs all selected gates.
  • Agent loop:
    1. Run gates.
    2. Fix failures.
    3. Re-run until pass or block.
  • Evidence stored under qa/evidence/.

This framework is intentionally strict: the agent must prove correctness.