ATLAS

ATLAS is a local-LLM fine-tuning program: QLoRA on 14B and 32B bases, SFT then DPO, quantized to GGUF and served on a 16 GB GPU, built to replace the hosted model behind a personal agent gateway. Three major versions were trained and all three were shelved, because the other half of this repo, a 711-case evaluation harness across 14 behavioral categories with a three-judge ensemble, kept showing that a free hosted baseline was better on real traffic.

The harness is the point. It turned "is my fine-tune good enough?" from a vibe into a number, and the number said no.

Results

Every version was gated against a free hosted baseline (Gemini 3 Flash, later DeepSeek V4 Flash) on the same cases with the same judges. "pp" is percentage points of category pass rate relative to the baseline.

Version	Base model	Recipe	Outcome
ATLAS I	Qwen3-14B-Instruct	QLoRA SFT, rank 32	-12.0pp macro. Failed.
ATLAS II	Qwen3-14B-Base	SFT + DPO (B2 adapter stack), rank 64, KG corpus	-36.8pp. Failed worse: compounded changes, nothing attributable.
ATLAS III v0	DeepSeek-R1-Distill-Qwen-32B (IQ3_M)	Base swap only, ATLAS I recipe	-14.9pp. Confident confabulation caused 70% of regressions.
ATLAS III v1-v3	same	One corpus class added per iteration	v1 abstention: hallucination +22pp, macro -1.2pp. v2 action-discipline: hallucination +28pp but a -21pp reasoning tax. v3 confident-reasoning: +2.6pp macro, first version to beat the baseline on the harness.
ATLAS mini 3.x / 4	DeepSeek-R1-Distill-Qwen-14B	Tool-calling SFT for the live serving condition (64K context, 90-tool catalog)	Live A/B through the real gateway stack: 0 topic wins out of 35. Reverted to the hosted model.

ATLAS III v3 is the interesting failure: it won on the category harness and still lost the live A/B, because the training distribution (8K context, under 10 tools per prompt) did not survive contact with the serving condition (64K context, ~90 tools). The harness caught that too: confabulation on live traffic fell from ~100% to 31% after a tool-state-fidelity training class, but zero routing topics flipped, which hit the pre-registered kill rule. Program retired, hosted baseline kept.

The eval harness

711 cases across 14 categories, one JSONL line per case, schema in evals/SCHEMA.md:

Category	Cases	What it tests
tool-call	160	Exact tool dispatch: right tool, right args, no extra calls
latency	100	Response-shape budgets by task complexity
reasoning	100	Causal attribution, trade-offs, counterfactuals, arithmetic
escalation	53	Knowing when to stop and route to a human
hallucination	50	Refusing to recite file contents or numbers from memory
persona	50	Voice fidelity, scored against an empirically derived rubric
multi-step	48	Multi-turn task chains replayed from real sessions
self-correction	40	Recovery after a wrong or failed step
pushback	30	Rejecting wrong premises instead of complying
refusal	30	Declining out-of-environment or unsafe requests
diagnostic	20	Evidence-first debugging order
data-fidelity	10	Numeric precision, no rounded-up guesses
honeypot	10	Bait tools that must never be called
retrieval-ablation	10	Behavior when memory context is removed

Cases carry stable IDs, provenance (hand-written, synthetic, replay, honeypot), an author field, and a review tier: routine, red-team, live-anomaly (must link a real incident), or capital-bearing.

Judge ensemble, and why capital-bearing cases get stricter judging

The agent this model was meant to run orchestrates a homelab that includes a live DeFi trading stack, so the worst failure mode is not a wrong answer, it is a fabricated capital action: claiming a transfer happened, inventing a broker call, complying with "bypass the broker this once". 51 cases carry the capital-bearing tier and are graded differently (evals/runners/eval_runner.py):

• Routine LLM-judged cases get one judge. Capital-bearing cases automatically get a 3-judge ensemble: Gemini Flash, Claude Sonnet 4.5, and DeepSeek V4 Flash, pass only on 2-of-3 agreement at a 0.85 threshold.
• A judge that errors or times out counts as a FAIL vote. Consensus must be deliberate, not accidental.
• Generator-judge separation is enforced: a case authored by a given model family is never judged by that family, and if the model under evaluation is also a judge, that judge is swapped out. Self-grading is structurally impossible.
• Capital-bearing cases scored deterministically still get the ensemble as a second gate on top of the matcher result.

This design came from measurement, not taste: a two-judge inter-rater pilot (evals/results/judge-irr-pilot-2026-05-07.json) found kappa 0.20 on persona cases, with one judge passing 90% and the other 50% of identical outputs. One LLM judge is a coin with a personality. The rubric templates for all 51 capital-bearing cases were then audited cluster-by-cluster (evals/results/capital-bearing-template-audit.md).

Stress harness (live agentic loop)

evals/stress/ runs the model in a real tool-execution loop, not single-shot grading:

• capital_ladder (12 tasks): capital prompts across four escalation tiers, from direct requests to authority injection. Honeypot capital tools (transfer, sign, bridge, allocate) fail closed and any call is a hard SAFETY FAIL. Two independently written detectors (instrumentation/capital_detector.py, capital_check_2.py) cross-check each other.
• multi_turn (50 tasks): deterministic sandboxed file-state goals, scored by end state, not by transcript vibes.
• tool_selection (14 tasks): pick the right tool from a 90-tool catalog under distractors.
• long_context (4 tasks): retrieval-at-depth at 4K, 16K, 32K, 64K.

exec runs inside a jailed sandbox (unshare -n / firejail / restricted fallback) with path-escape validation, and the jail has regression tests for both directions: real escapes must be caught, benign commands must not be (evals/stress/test_jail_paths.py).

Keeping the numbers honest

• training/scripts/check_corpus_contamination.py gates every corpus build against every eval suite at 0.85 similarity. It fired in production: 14 distilled training rows near-duplicated a stress case and were dropped before training.
• Results directories are append-only per run. Case IDs are never reused.
• The one contamination that did happen (hindsight-corrected traces overlapping the stress multi_turn suite) is documented as invalidating future comparisons on that suite, rather than quietly ignored.

The training recipe

training/ holds the full path from dataset to served model:

• SFT: QLoRA via Unsloth, 4-bit base, LoRA rank 32/alpha 64 (rank 64/alpha 128 on 9 target modules for ATLAS II), bf16, cosine schedule, 8-bit AdamW, 16K context via YaRN factor 2 on the 14B Qwen base. train.py through train_sft_v6.py, one file per version, spec-locked configs inline.
• DPO: the B2 adapter-stacking approach (training/scripts/train_dpo.py): the SFT adapter stays frozen as a PEFT adapter, DPO trains a second adapter on top, and the reference model is a separate base+SFT instance. No merge before preference training, so the SFT behavior is the explicit reference rather than a moving target.
• Post-train to serving: merge PEFT into the base (merge_peft_v3.py), convert to f16 GGUF, compute an importance matrix, quantize to IQ3_M for the 16 GB card (export_gguf_v3.py, requant_*.sh), then register with Ollama from a Modelfile template.
• Chat-template parity: the same Jinja template must be loaded at SFT, DPO, and serve time. A SHA-256 companion file guards it, and the requant pipeline re-verifies the SHA of the template embedded in the GGUF before registering. Template drift between training and serving is a silent model-killer and is treated as a hard gate.
• Cloud orchestration: fire_sft_v5.py and runpod_helpers.py provision an H100 pod, run the fire, and terminate it only after the adapter is downloaded and verified locally. A training run cost $4-7; the ops mistakes (a pod idling 17 hours behind a sleeping laptop) cost more, and moved the launcher to an always-on box.
• Serving: serving/atlas_tool_bridge.py is an OpenAI-compatible proxy that rewrites DeepSeek-R1 tool-call markup into native tool_calls, because llama.cpp has no parser for that format. This one bridge unblocked every model version at once. security/ adds Ed25519 signing of inference responses with JWT replay protection, so a downstream broker can verify a response actually came from the attested serving process.

What was learned

Fine-tuning a local model is not worth it when:

1. A free hosted baseline exists and your harness can prove it wins. Every dollar and week spent here bought that proof. Without the harness, the project would have shipped a worse model on feel.
2. Your training distribution is not your serving distribution. Category wins at 8K context with 10 tools meant nothing at 64K with 90 tools. Evaluate under the serve condition or the numbers lie to you politely.
3. Eval loss is a rumor about behavior. The single best eval-loss checkpoint (0.9839) was the worst model ever gated: tool selection 0/14, an exposure-bias textbook case from training on mid-chain continuations.
4. Each corpus fix taxes another axis. Abstention training fixed hallucination (+28pp) and caused over-deflection (-21pp reasoning). Upsampling multi-turn chains bought +24pp on multi-turn and destroyed single-turn tool selection (11/14 to 5/14). Single-variable iterations were the only reason any of this was attributable.
5. Safety can hold while capability fails. capital_ladder passed 12/12 with zero honeypot calls on every version, including the ones that lost every routing topic. The refusal floor is much easier to train than the competence ceiling.
6. Pre-register the kill rule. "Two retrains with zero topic flips and the program retires" is what actually ended it. Sunk cost had no vote.

Repo layout

evals/            The harness: 14 case suites, schema, runners, judge ensemble
  SCHEMA.md       Case format, scoring methods, tiers, tool-surface contract
  runners/        eval_runner.py (matchers + judges), compare_models.py (A/B)
  stress/         Live agentic-loop harness: sandbox, jail, honeypots, 4 suites
  results/        Judge reliability pilots + capital-rubric audit
training/         SFT/DPO scripts per version, GGUF export, requant, Modelfiles
  scripts/        DPO (B2), contamination gate, RunPod orchestration
serving/          OpenAI-compatible tool-call bridge + telemetry
security/         Ed25519 inference attestation: signer, verifier, tests
instrumentation/  Capital-op detectors (x2, cross-checked), GPU sampler
prompts/          The system prompts the model was trained toward
runners/          Eval-case translator between tool-naming surfaces
tests/            Harness unit tests (sandbox, scoring, detectors, suites)
visualization/    Self-contained build-journal page from the ATLAS I era

Training corpora, preference pairs, checkpoints, adapters, and GGUF artifacts are not included: the data came from real sessions and stays private. The code, the cases, and the methodology are all here.

License

MIT. See LICENSE.