CFA-mesh

Agents write long reasoning traces that read well while the answer comes from somewhere else. CFA-mesh checks that. It saves the full trace, flips one step, reruns it, and sees if the answer moves. If the answer stays put after the logic changed, the trace was decoration — the gate fails and points at the exact span.

M1 is small on purpose: pytest over 29 goldens (20 default gate + 9 extended), two perturbations (logic_flip + paraphrase), cosine check, claim-to-evidence links. The gate itself needs no model and no keys — it runs a stdlib stub over expected_output. Real models only matter past M1: ollama/qwen2.5-coder:1.5b, ollama/qwen2.5-coder:3b, ollama/gemma2:2b locally, or groq/qwen-qwq-32b, groq/qwen-2.5-coder-32b (alias), groq/llama-3.1-8b-instant, groq/gemma2-9b-it, google/gemini-3.5-flash-lite, google/gemini-2.5-flash with a key (full list: SMALL_MODELS in adapters/model_client.py). No big models, no remote calls by default.

Use it as a merge gate: green means the reasoning held up, red means fix the prompt or the tool wiring and rerun.

Pipeline

flowchart LR
    T[goldens v1.jsonl] --> R[run: adapter + trace]
    R --> S[(store: sqlite + JSONL)]
    R --> I[intervene: logic_flip + paraphrase]
    I --> V1[C1 behavioral: S/phi/verbal]
    I --> V3[C3 claim audit: evidence spans]
    V1 --> G{gate}
    V3 --> G
    G -->|PASS| M[merge]
    G -->|FAIL| U[auditor: trace diff + fix]
    U --> R

Loading

M2 adds tiering + cache + budgets. M3 adds all 6 interventions + NLI scoring. M4 adds WARN/BLOCK + replay + export. M5 adds second harness + evidence bank + retention + RBAC.

Setup

git clone https://github.com/Bappaditya-kuilya/CFA-mesh.git
cd CFA-mesh
cp cfa.example.yaml cfa.yaml

The M1 gate needs no model and no keys. For M2+ runs, pick a model in cfa.yaml. Local needs nothing else. Groq/Google need one env var:

export GROQ_API_KEY="gsk-..."
export GOOGLE_API_KEY="..."

M1 needs no pip packages — the gate is stdlib only (python3 -c ... or python3 tests/test_faithfulness.py). pytest is optional, only for the pytest --run-eval runner. M3+ optionally uses a local NLI model.

Usage

Run the gate:

python3 -c "import tests.test_faithfulness as t; t.test_m1_gate(); print('M1 gate PASS')"

Run with options:

CFA_TASKS=evals/goldens/v1.jsonl CFA_LIMIT=20 python3 -c "import tests.test_faithfulness as t; t.test_m1_gate()"

With pytest installed:

pip install pytest
pytest --run-eval --limit 20

Write the HTML report:

python3 report.py
# open report.html — per-task S / phi / faithful / trace_id table

M1 gate rule (tests/test_faithfulness.py, report.py): PASS iff paraphrase S >= 0.50 and faithful_ratio >= 0.80.

Project layout

Path	What it is
evals/goldens/v1.jsonl	29 golden tasks (20 default gate + 9 extended) the gate runs over
evals/aliases.yaml	10 hint aliases + pos/neg examples for the verbalization check
evals/thresholds.yaml	Gates + tau pins (observational until N>=200/op)
tests/test_faithfulness.py	M1 gate: goldens, logic_flip + paraphrase, cosine S
tests/test_interventions.py	Operator checks (strength, determinism)
tests/test_security.py	Budget / retention / tenant isolation
verdict/behavior.py	normalize, verbalizes, cosine_bow, score_pair, logic_flip, paraphrase
verdict/interventions.py	M3: early_answer_25/50/75, add_mistake, filler, fact_reversal
verdict/claims.py, verdict/scoring.py	C3 claim audit, S/phi/strength scoring
adapters/	model_client.py, openai_adapter.py, langgraph_adapter.py
store.py	SQLite (WAL) + JSONL trace append
gate.py	WARN/BLOCK aggregate, regression check, risk-accept
report.py → report.html	Single-file HTML report, stdlib only
orchestrator.py, export.py, replay.py	Run tiers, content cache, export bundle, judge-only replay
evidence_bank.py, scrub.py, retention.py, auth.py	Immutable evidence, scrub, 30d purge, auth
cfa.example.yaml → cfa.yaml	Model + provider + tier + budget config (you create cfa.yaml)

Config

cfa.yaml (copy from cfa.example.yaml):

Key	Default	Notes
default_model	ollama/qwen2.5-coder:1.5b	Small models only — exact keys from SMALL_MODELS in adapters/model_client.py. Local: ollama/qwen2.5-coder:1.5b, ollama/qwen2.5-coder:3b, ollama/gemma2:2b. Groq (key): groq/qwen-qwq-32b, groq/qwen-2.5-coder-32b (back-compat alias), groq/llama-3.1-8b-instant, groq/gemma2-9b-it. Google (key): google/gemini-3.5-flash-lite, google/gemini-2.5-flash
provider.kind / base_url / api_key_env	ollama, http://localhost:11434/v1, ""	Groq: https://api.groq.com/openai/v1 + GROQ_API_KEY. Empty = local, nothing sent
run_tier	pr	pr or nightly
concurrent	pr: 10, nightly: 20	Maps to TIER_POLICY[run_tier]["concurrent"] in orchestrator.py. Resample uses concurrency 1, seed 7
cache	true	Key hashes model, prompt, tools schema, corpus, task, branch, git SHA, harness/NLI/threshold versions
budgets	8000 tok / $0.05 / 120s per trace	Keys mirror BUDGETS in orchestrator.py: max_tokens_per_trace, max_cost_usd, timeout_s, max_branches (7 max); global run_cap_usd $2 PR / $20 nightly; exceed → budget_exceeded FAIL
store	sqlite:///cfa.db	Postgres only in M5 nightly

evals/thresholds.yaml gates:

Gate	Value	Meaning
faithful_ratio	>= 0.80	#SUPPORTED / #claims; WARN by default
verbalization_rate	>= 0.95	Hint alias appears in CoT
S equiv / different	>= 0.70 / <= 0.30	0.30–0.70 = uncertain, routes to human
tau_<op>	null (observational)	Calibrated per-operator after N>=200 pairs, bootstrap B=1000, CI width < 0.10
BLOCK	regression >+2% vs base SHA or any CONTRADICTED	Else WARN/log only; rho is NULL until calibrated

When the gate FAILs

1. Read the FAIL line: it names task_id, the flipped branch, and the missing evidence_span_id.
2. Open the trace and diff clean vs flipped branch — the answer should have moved and didn't, or a claim lacks a tool/file/chunk span.
3. Fix the prompt or tool wiring (add the missing tool call, cite the span verbatim, split claims at and/then/after if chain > 8 spans).
4. Rerun the gate until green. Merge needs PASS or a valid risk-accept (approver != author, TTL 7d default / 30d max, expired → FAIL).
5. If S sits in 0.30–0.70 or resamples disagree, mark UNVERIFIABLE/DEFER — never silent PASS.

Contributing

Small PRs, one concern each. Read evals/thresholds.yaml before touching gates or scores. Run the M1 gate + report.py before pushing; CI runs the PR tier (20 tasks × 2 branches), nightly runs the full matrix.