Skip to content

Task: Build fair Kida comparison fixtures from selected pain cases #320

Description

@lbliii

Parent: #318
Blocked by: #319
Related: #199

Goal

Create three runnable, pinned comparison fixtures that demonstrate outcomes Kida can honestly change. Equal output and behavior are proven before any timing or complexity comparison.

Fixtures

  1. Checked component call — a mistyped/missing component parameter. Compare Kida check --validate-calls with Jinja configured with StrictUndefined; disclose that dynamic data and non-component names remain runtime concerns.
  2. Custom CI report — a conventional inline/scripted report workflow versus a Kida report component/template. Compare the feedback stage, change surface, and custom-report workflow—not synthetic render speed.
  3. Focused report Action — Kida Report versus one mature focused Action. Compare input formats, output targets, customization, permissions, runner/platform constraints, and unsupported cases; no universal “winner” conclusion.

Acceptance and proof

  • Every fixture pins tool/package versions, runner/platform, Python build, input artifact, command, expected output, and fixture-specific limitations.
  • Output parity/sanity assertions run before any measurement.
  • Jinja uses an explicit undefined policy; the fixture identifies which failures remain runtime-only.
  • Any timing result names hardware, OS, Python build, GIL status, warm/cold state, repetitions, and variance rationale. Linux 3.14t baselines are authoritative; Darwin is development evidence only.
  • No benchmark compares unequal rendered output, different input parsing, or different work.
  • Negative findings remain published in the fixture record.

Likely perimeter

benchmarks/, tests/, templates/, action_support/, .github/, and fixture/example collateral only as justified by the selected corpus. No new syntax, public API, runtime dependency, Action permission, or template schema is authorized by this task.

Verification

Run the narrow fixture tests first, then applicable action/template/analysis/CLI/benchmark sanity suites. Record exact commands in the implementation PR.

Stewards and collateral

  • Stewards: benchmarks, tests, analysis, cli, parser/nodes/compiler/template, action, github, templates, markdown, schemas, utils, public, docs/site/examples.
  • Collateral: reproducible fixture instructions and results artifacts. A public claim needs a later evidence-card task.
  • No downstream pilot: fixture-only proof of existing contracts; a target repository is not changed here.

Stop and ask

Stop before adding a benchmark dependency, changing Actions permissions/runners, altering public CLI/API/syntax, or publishing a performance/security/free-threading claim.

Metadata

Metadata

Assignees

No one assigned

    Labels

    priority/P2Valuable quality, capability, or evidence workstatus/not-nowPreserved proposal intentionally outside the current execution queuetaskConcrete work itemtheme/analysisArea: static analysis and type checkingtheme/performanceArea: performance and benchmarkstheme/testing-ciArea: tests and continuous integrationtype/testTest coverage or test infrastructure

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions