Skip to content

CLAUDE.md

Latest commit

 

History

History
48 lines (39 loc) · 2.46 KB

CLAUDE.md

File metadata and controls

48 lines (39 loc) · 2.46 KB
Raw

CLAUDE.md — Repository evaluation harness

This repo (wrtnlabs/autobe-examples) hosts backend projects generated by the AutoBe function-calling harness across 11 LLMs × 4 benchmark projects (todo, reddit, erp, shopping) = 44 projects total.

When asked to evaluate quality, always use the rubric at estimate/RUBRIC.md. Key rules:

What we score

  • Requirements depth (docs/analysis/*.md): spec completeness, business rules, edge cases.
  • DB schema (prisma/schema/*.prisma): entity completeness, relationships, indexes, audit.
  • API design (src/api/**, src/providers/*.ts): coverage, authorization, DTO richness.
  • Test scenarios (test/features/**/*.ts): happy paths, error paths, auth boundaries, lifecycles, edges.

Each category is 25 points; total is 100.

What we do NOT score

  • Coding style, formatting, imports, NestJS/nestia/typia boilerplate — all harness-controlled.
  • File names, folder layout, decorator patterns — harness-controlled.
  • README.md, LICENSE, package.json, build config — scaffolded.

Process

  1. Read docs/user-message/user.message.1.md first — ground truth.
  2. Line-by-line read every doc/schema/provider/test file.
  3. Exclude any {vendor}/{model}/{project}/autobe/ subtree from estimate.
  4. For each sub-criterion (A1..D5) cite file:line evidence for every point awarded or withheld.
  5. After all 44 reports land, run relative calibration (see estimate/RUBRIC.md §5).
  6. Write ranked summary in estimate/SUMMARY.md.

Directory map

estimate/
  RUBRIC.md              — scoring criteria
  INDEX.md               — pointers to every per-project report
  SUMMARY.md             — rankings, findings (produced last)
  CALIBRATION.md         — relative-adjustment log
  prompts/               — canonical user prompts (one per project)
  reports/
    <vendor>/<model>/<project>.md

Anti-bias rules

  • No brand boost (Anthropic/OpenAI/etc. are peers).
  • README benchmark scores (all "100") are build/compile metrics, not quality — ignore them.
  • Size ≠ quality: a 10-line test with strong assertions beats a 100-line test with none.
  • Docs that read well but lack enumerated rules, invariants, and state machines get no credit.

Workflow shortcuts

  • All 11 models share the same 4 prompts byte-for-byte (verified). Compare results head-to-head without re-reading the prompt for every project.
  • Measured baseline counts (docs, schema files, providers, tests) are in estimate/RUBRIC.md §2.