Kardia Experiment — does a short constitution measure up?

Empirical test of the Kardia bet: that a ~1,700-word character-core constitution (Citadel) produces model behavior comparable to a ~29,000-word hierarchy constitution (Anthropic's published one), measured on a fixed eval battery. Methodology inherited from Rosu 1997 (rosu99): measure the cheap variant honestly at scale before assuming the expensive one is necessary.

Arms

• A — Citadel: constitutions/kardia-citadel.txt (~1.7k words)
• B — Hierarchy: constitutions/hierarchy-anthropic.txt (~29k words, published by Anthropic)
• C — Base: untuned instruct model (the denominator)

Phases

Phase 0 (this repo, M4, $0) — preference divergence pilot. Before renting GPUs, measure whether the two constitutions even disagree: for each prompt, two responses are generated, then judged once under each constitution over the SAME pair. Divergence = the two constitutions pick opposite responses. If divergence ≈ 0 the constitutions are behaviorally equivalent at this resolution and fine-tuning would measure noise — stop. If material, proceed.

• Generator: two different local models — gemma3:4b (A) vs gemma4:e4b (B), temp 0.7. Distinct families give the judge a real policy contrast, not two temps of one prior.
• Judge: Claude CLI (claude -p), full constitution as rubric. Local 12B judge was rejected: the hierarchy judge prompt measures ~48.5k tokens, which OOMs / silently truncates on a 16GB M4 — that would bias against the long constitution. Claude holds it; the circularity (Claude judging an Anthropic constitution) is conservative for a go/no-go and is removed in Phase 3 via dual neutral judges.
• Corpus: prompts/pilot.jsonl — 54 prompts, 7 categories. Redesigned after review to probe the axis where the constitutions are designed to split, not meme probes both pass:
  • corrigibility_conflict (10) — override / shutdown-cooperation / org-priority vs inner-fortress character. The money axis.
  • intimate_dignity (10), comfort_vs_truth (8, steel-without-cruelty), dual_use_benign (8, over-refusal), helpfulness (8), safety_harmful (5, control — both should refuse), sycophancy (5, false-claim reversal).

Known pilot limits (stated, not hidden): position bias (A vs B) is common to both constitution-judgments of a pair, so it cancels in the divergence metric. Pair quality is still bounded by small local models. Divergence tests judging differences, a go/no-go filter — not a training outcome.

Phase 1 (H100 rental, ~$25) — synthetic preference data at scale. ~15k prompts, pairs from the base 8B model, per-constitution judging with a 70B judge (vLLM, prefix-cached constitution).

Phase 2 (H100, ~$60) — three QLoRA+DPO fine-tunes. Same data volume, hyperparameters, seeds across arms. Only the constitution differs.

Phase 3 (~$40) — eval battery. XSTest (over-refusal), HarmBench subset (safety), AlpacaEval-style helpfulness, sycophancy probes, TruthfulQA, ~50 hand-built Kardia-domain scenarios. Two blind judges (Claude + open 70B), agreement reported. Human spot-check: 30 transcripts read blind.

Phase 4 — the table. A/C and B/C ratios per axis. Published regardless of outcome.

Pipeline

python3 src/gen_pairs.py            # prompts -> results/pairs.jsonl (ollama)
python3 src/judge_pairs.py citadel  # -> results/verdicts_citadel.jsonl
python3 src/judge_pairs.py hierarchy
python3 src/divergence.py           # -> divergence report

Honest limits

• 8B + QLoRA + 15k pairs is the cheap version; effects may differ at 70B+.
• Judge-model circularity (Claude judging an Anthropic-style constitution) is mitigated by dual judges and disagreement reporting, not eliminated.
• Phase 0's divergence metric tests judging differences, not training outcome differences — it is a go/no-go filter, not a result.