| tags |
|
||||||
|---|---|---|---|---|---|---|---|
| created | 2026-04-28 | ||||||
| updated | 2026-06-06 |
A reusable playbook of agent-architecture decision patterns — memory tiers, retrieval pipelines, multi-agent delegation, agent/RAG evaluation, and the LLM-ops to run them. Every pattern is grounded in a built + measured lab (W1…W3.5.9, on-disk RESULTS); each cites its source. Built to be reused in later labs (W4+ ReAct, tool harness, system design, capstone) without re-deriving.
Scope (refocused 2026-06-02). This doc was pruned to agent-development domain patterns only. Generic engineering-process discipline (scope-estimate, options-table, verification, etc.) and curriculum-authoring discipline (forward-link panels, spec/disk fidelity, cross-ref invariants) were removed — they were correct but not agent-development knowledge. Pattern NUMBERS are preserved from the original (gaps are intentional) so existing chapter cross-references like
#Pattern 14still resolve. Build frontier = W3.5.9; nothing below cites an unbuilt chapter as evidence.
Each pattern: Rule (when X → do Y) · Evidence (the measured lab behind it) · Anti-pattern · Conditionality (when it does NOT apply). Skim the names; reach for them by name when designing an agent's memory/retrieval/eval/coordination.
Distilled 2026-05-07 from [[Week 2.7 - Structure-Aware RAG#Production Considerations — Storage, Concurrency, Observability]]. Generalizes beyond tree-index — applies to any persistent-state decision (vector indexes, graph data, eval harnesses, agent memory).
The pattern. When a system has multiple storage candidates (filesystem / SQLite / Postgres / object store / cache layer), match the choice to the current scale, not the eventual scale. Build the decision boundary around concrete numbers (doc count, QPS, concurrency, dataset size), not vibes. Three or four scale tiers with explicit thresholds beat one "production-ready" backend that's wrong for both prototype and prod.
Three-tier template:
| Scale | Backend | Why |
|---|---|---|
| Dev / lab / single-user | Local filesystem (JSON / SQLite / disk file) | Operational simplicity. git diff shows changes; full rebuild is one command; zero infrastructure. |
| Single-tenant prod (10–1,000 units) | SQLite or Postgres jsonb / on-disk index |
Concurrent reads. Per-record version history. ACID. No multi-tenancy load yet. |
| Multi-tenant prod (1,000+ units, multi-user) | Postgres + S3/blob + Redis cache | Tier the storage by access pattern. Hot in cache, warm in DB, cold in object store. |
Decision matrix specifics from W2.7 tree-index:
| Artifact | Property | Storage choice |
|---|---|---|
tree.json (50–100 KB, hierarchical, read-mostly) |
Document-shape, indexable JSON | Filesystem → Postgres jsonb (with index on tree -> 'title' for cross-doc title search) |
| Source PDF (10s–100s MB, immutable, page-range access) | Append-only blob | Filesystem → S3 with byte-range reads + Redis LRU cache for hot ranges |
| Cross-document index | Doc registry | None → SQLite → Postgres |
Why it works. A premature production backend forces every developer running the lab to bring up infrastructure that has no value at single-user scale (Postgres for one document is theatrical). A premature filesystem backend at 1,000-doc scale destroys concurrency. The match-to-scale decision lets each user pay only for the complexity their scale actually demands. The abstraction seam (in W2.7's case, the page_provider Protocol) is what makes scale transitions cheap — same AgenticTreeRetriever works against any of the three storage tiers; only the closure around the page fetch changes shape.
Anti-pattern A — premature production scaffolding. "We might scale to 100K docs someday, so let's start with Postgres + S3 + Redis." Now every developer setting up the lab needs three services running. The next 6 months of dev time is spent fighting infrastructure for the value of "we're production-ready" — which is fictional value if you have one user.
Anti-pattern B — locked-in dev simplicity. Filesystem-only design that has no abstraction seam between storage and the application logic. When you DO need to scale, the lift is a multi-week refactor, not a single closure swap. The fix at design time: introduce the seam (Protocol, callable, ABC) even if the filesystem implementation is the only one shipped. The seam is free; the absence of the seam costs a refactor.
Anti-pattern C — wrong backend for the data shape. Common mistake: putting structurally non-vector data (a tree, a graph, an event log) into a vector database because that's the database the project already runs. Vector DBs are for embeddings; trees go in jsonb / document stores; graphs go in graph DBs; event logs go in append-only logs. The blast radius is silent — the system "works" but every read pays JSON ↔ vector conversion overhead and cross-record queries break.
5-second sanity test before adopting a backend:
- What's the unit of work? If hierarchical → document store / jsonb; if relational multi-hop → graph DB; if embedding similarity → vector DB; if append-only event → log store; if immutable blob → object store.
- What's the access pattern? Read-mostly per-record vs read-many cross-record; concurrent vs single-writer; hot-tail vs uniform.
- What's the concurrency requirement? 1 reader = file is fine. Multiple concurrent writers = need ACID. Multiple processes reading + occasional writes = SQLite is fine until QPS > ~100.
- What's the data size at the upper end of the current tier? Filesystem fine to ~100 GB; SQLite to ~10 GB; Postgres jsonb to ~1 TB. Above those, scale tier up.
See also:
- Pattern 26 — Write-Time Primitive Preserves Signal (the data-shape sanity test feeds the write-primitive choice)
- Pattern 28 — Memory-Tier Graduation (the memory-tier version of match-to-current-scale)
- [[Week 2.7 - Structure-Aware RAG#Production Considerations — Storage, Concurrency, Observability]] — the tree-index storage decision this distills
When it applies: any time a parent agent spawns a child / worker / subagent / delegate. Codex Use parallel subagents, Claude Code description-routed subagent, Hermes delegate_task, OpenClaw /subagents spawn, in-house crew.kickoff-style orchestrators — same pattern.
Failure mode this prevents: the single most common multi-agent failure mode in the wild — "subagents know nothing" (Hermes docs phrase, 2026). A parent agent has the full project context in its head; the child gets only what the parent decides to pass in. Saying "fix the auth bug" to a fresh child is the same as DM'ing "fix the auth bug" to a new hire on day one. They will guess, and the cost of guessing wrong is paid in token spend, time, AND main-context contamination when the wrong patch comes back.
The 8-field contract — fill every field for every delegation, even if the answer is "none":
Role: you are a <read-only auth explorer | scoped implementation worker | security reviewer | …>
Goal: what to answer or complete; what the boundary is
Context: project paths, relevant files, error reproduction, user goal, judgements already made
Allowed actions: which files readable; can run shell; can write files; can hit network
Ownership: if writes are allowed, which directories / files are within bounds
Forbidden actions: do-not-modify list; do-not-refactor list; do-not-ask-user; do-not-spawn-child
Output format: findings / patch summary / test result / confidence / open questions
Stop condition: what counts as done; when to stop and report blocked
Why each field is load-bearing:
- Role sets the persona priors.
read-only auth explorerandscoped implementation workerproduce different code even with identical Goal + Context. - Goal is the contract. If the parent can't write it in one sentence, the parent doesn't know what it wants — spawning a child won't fix that.
- Context is what the child knows. Everything not in Context, the child guesses. Production rule: the child should never have to ask the user a clarifying question; if the parent omitted critical context, that's a parent bug, not a child bug.
- Allowed actions + Ownership + Forbidden actions are the sandbox. They are NOT redundant with each other: Allowed = capability gate (can-it), Ownership = scope gate (where-can-it), Forbidden = explicit anti-goal (don't-do-this-even-though-you-can). Codex's
agents.sandboxconfig maps to Allowed; Hermes'sleafworker restrictions map to Forbidden. - Output format is the merge contract. If the parent has to free-text-parse the child's response, the parent's reduce phase becomes the new bottleneck. Pre-declare the shape; child returns structured fields.
- Stop condition prevents two failure modes: child runs forever because "done" isn't defined; child stops at the first plausible answer because "good enough" isn't defined. Explicit stop conditions also enable retry semantics in durable systems like Hermes Kanban.
★ Insight ─────────────────────────────────────
- The 8 fields ARE the audit log schema. Compare to the AuditEntry primitive (W3.5.8 §3.4):
actor_agent_id= Role,payload_summary= Goal+Context distilled,metadata= Allowed/Ownership/Forbidden/Output,target_id+new_id= Stop-condition outcome. Write-time contracts and read-time replay are two views of the same data. - Filling all 8 fields takes ~30 seconds and saves ~5 minutes per botched delegation. If a child returns nonsense, look at the contract first: which of the 8 fields was empty or vague? That field is the bug. ~80% of cases will be Context (parent assumed shared state that doesn't exist) or Forbidden (parent didn't constrain blast radius).
- Topology choice is downstream of the contract, not upstream. Star fan-out / pipeline / mesh / durable-board don't change whether the contract is needed; they only change how many copies of the contract get written per task. A team-mesh with 4 teammates needs 4 contracts. A solo subagent needs 1. None of them work with 0.
─────────────────────────────────────────────────
See also:
- Pattern 32 — Metered-Proxy Role-Split (delegating to a metered model needs the same context-passing + a no-persona local fallback)
- [[Week 3.5.8 - Two-Tier Memory Architecture]] §3.4 — AuditEntry primitive (the read-side mirror of this write-side contract)
(SPEC)W4.6 Durable Agent Runtime, W6.5 Hermes — where the topologies that use this contract + the "subagents know nothing" maxim live (chapters drafted, labs not yet built)
Source: synthesized from Russell (2026), 多智能体协作调查：Agent 到底该怎么分工 — engineering survey of Codex / Claude Code / OpenClaw / Hermes delegation patterns. The 8-field template combines field names used across all 4 systems into one normative shape.
When it applies: any data-processing or AI pipeline where the SAME primitive (summarisation, atomisation, embedding, classification, compression, masking) could plausibly run at multiple stages — write/ingest time, read/query time, or both. Specifically when the primitive is lossy and downstream consumers have heterogeneous needs.
The invariant. The lifecycle position of a primitive is load-bearing — same code, different stage, opposite outcome. Before placing a primitive, ask:
| Question | Early-binding (write-time) wins when… | Late-binding (read-time) wins when… |
|---|---|---|
| Is the consumer's query known? | No (logs, archival) — write-time still works because future queries are uniform | Yes (per-request agent memory) — atomise under the lens of the actual query |
| Is the primitive lossy? | Acceptable: the schema is known and stable | Hazardous: losing detail at write means it cannot be re-derived |
| Queries per memory? | ≪ 1 (log ingestion) — amortise the cost at write | ≫ 1 (agent memory) — pay per query, since writes are rare per memory |
| Error compounding? | Acceptable if downstream stages don't depend on this primitive's output for retrieval | Hazardous: error at write poisons embed → retrieve → compose downstream |
| Schema stability? | Stable, known in advance | Schema-of-interest = the query's own structure (idiosyncratic, late-bound) |
| Can the primitive be upgraded in production? | No — re-ingest required to apply new logic | Yes — change the read-time prompt/model and ALL old memories benefit immediately |
The math. Write-time = fixed projection π_write. Read-time = query-indexed family π_query(q). The family always dominates the fixed choice when q is observable, which it is at read time. Same logic as parametric vs hand-tuned models, JIT vs AOT compilation, dynamic vs static dispatch.
Curriculum instance: W3.5.8 atomisation at write-time destroys signal; at read-time lifts +5pts across the capability range. §3.2.1's extract_atomic_facts invoked as part of consolidate() at WRITE time on LongMemEval conversational haystacks: 0/20 correct, conversational facts skipped or paraphrased into tech-flavored summaries. The SAME primitive invoked at READ time (after Qdrant retrieval, before LLM compose) lifted BOTH a 4-bit dense Qwen3.6-27B model (60% → 65%) AND a Claude-Opus-distilled Qwen 27B model (70% → 75%) by +5pts each on the same slice. The architectural primitive is correct; the lifecycle position was wrong for conversational data. See [[Week 3.5.8 - Two-Tier Memory Architecture#5.3.3 Atomisation lifecycle — write-time vs read-time (the deeper §3.2.1 lesson)|W3.5.8 §5.3.3]] for the five-reason decomposition and the data-shape-vs-lifecycle architectural table.
The discipline.
- Name the lifecycle stage explicitly when describing a primitive. "Atomisation" is ambiguous; "write-time atomisation" vs "read-time atomisation" are different decisions with different failure modes.
- Default to late-binding when query distribution is unknown or heterogeneous. Most agent-memory and retrieval-augmented systems fit this shape. The intuition that "compress at write to save query cost" is imported from log-processing pipelines and is the wrong default for agent memory.
- Bind early only when the schema is genuinely known AND queries are uniform. Structured durable facts (user preferences, ACID-eligible records) fit this. Conversational episodic data does not.
- Test the same primitive at both positions when in doubt. A/B by environment flag, not by re-architecture. W3.5.8's
ATOMISE_AT_READ=1flag is the minimal viable ablation harness. - Treat lifecycle position as a tunable parameter, not a fixed pipeline shape. Different data shapes inside the same system can have different lifecycle policies for the same primitive.
Anti-pattern: log-processing intuition imported wholesale into agent-memory. Many "two-tier memory" articles split by storage engine (SQL + vector) and assume write-time compression because that's what log pipelines do. The actual split that matters is early-bound structured facts vs late-bound retrievable raw. Same engine could serve both with different lifecycle policies; different engines could serve the same lifecycle. The discipline is the lifecycle choice, not the SQL-vs-vector choice.
Sub-rule: Authority-Weight Calibration (refinement, added 2026-05-20). Lifecycle position is necessary but not sufficient. The volume and prominence of derived facts in the consumer's prompt also matter — independently of lifecycle stage. Measured 2026-05-20: constrained read-time atomise (top-K=5 triples, prominently positioned at top of composer prompt, raw context preserved below) collapsed Qwen3.6-27B by −35pts AND Qwen-Opus by −30pts (uniform regression across a 40-pt capability gap). Same lifecycle stage as the successful unconstrained variant — only the volume changed. Compressed derived representations carry per-item authority weight that exceeds the consumer's threshold for overriding via raw context, regardless of model size. Production guardrails:
- Minimum-volume floor (K_min ≥ 8). If extractor returns fewer than K_min triples for a non-trivial input, drop derived entirely; fall back to raw-only.
- Deployment calibration gate. A/B test the deployed extractor: if
composer(raw + facts) ≤ composer(raw alone), do not ship — the extractor is poisoning, not helping. - Position discipline. Raw context FIRST in composer prompts; derived facts LAST with neutral framing. Composers anchor on early-prompt structured content.
- Never ship "K most relevant facts" without these guardrails. The seductive design pattern of "let the extractor pick the 3 most relevant facts" appears frequently in production memory write-ups and is unsafe by construction at any model size when extractor accuracy < consumer trust threshold.
The Bayesian framing: many triples = Bayesian model averaging (errors cancel); few triples = MAP selection (errors fatal). Same math that makes random forests dominate single decision trees. See [[Week 3.5.8 - Two-Tier Memory Architecture#5.3.4 Volume buffers extraction error — the Bayesian framing of why unconstrained atomise works|W3.5.8 §5.3.4]] for the empirical phase transition.
★ Insight ─────────────────────────────────────
- Read-time primitives are iterable in production; write-time primitives are not. Ship a better atomiser at read-time and ALL old memories benefit immediately. Ship a better atomiser at write-time and only newly-ingested data benefits — old data requires re-ingest. This is the production-ops corollary of late-binding and a strong argument against eagerly compressing data at write time when the read path can absorb the cost.
- "Where does this primitive belong in the pipeline?" is usually the wrong question. The right question is "what data shape is being processed, and is its query distribution known at write time?" Lifecycle position is downstream of data-shape commitment, not an independent design choice.
- The pattern generalises beyond memory. RAG re-rankers, embedding-time chunking, write-time summarisation, schema-on-write databases, and ahead-of-time compilation all sit in the early-binding family and all break the same way when the consumer's needs are not known in advance. The early-vs-late binding tradeoff is a chapter-level invariant worth keeping at the front of your mind whenever you're placing a primitive in a pipeline.
- The empirical signature of a lifecycle mismatch: signal destruction at write time (zero recall on questions whose answers are in the raw input) AND recovery when the same primitive is moved to read time. If you see this signature, suspect lifecycle position before suspecting the primitive itself.
─────────────────────────────────────────────────
See also:
- Pattern 26 — Write-Time Primitive Preserves Signal (Pattern 22 = where the primitive runs; Pattern 26 = what it extracts — paired decisions)
- Pattern 30 — New Complexity A/B-Earns Keep (the volume-buffer / K_min sub-rule that refines this pattern)
- [[Week 3.5.8 - Two-Tier Memory Architecture]] §3.2.1 (write-time) ↔ §5.3.3 (read-time + lifecycle decomposition)
- BCJ 2026-05-19 (write-time failure) + BCJ 2026-05-20 (read-time recovery) — the matched-pair empirical record
Category note. Patterns 1-12 are process/collaboration discipline; 17-21 are curriculum-authoring discipline; this block is technical-domain discipline for retrieval-augmented + memory systems — joining the existing domain patterns 13 (Storage-Scale), 15 (Read/Write Mirror), 22 (Lifecycle Position). Every pattern below is grounded in a built + measured lab (W1…W3.5.9, all with on-disk RESULTS); each cites its source. Extracted 2026-06-02 to be reusable in W4+ (ReAct, tool harness, system design, capstone) without re-deriving.
Rule. Retrieval quality has a cost ladder: dense → +BM25 hybrid → +cross-encoder rerank → +LLM-rerank. Each rung costs ~10× the last. Climb only when the measured gap justifies it. W2: hybrid (BM25+dense) captured ~70-80% of reranking's lift at ~5% of the latency; fp16 was 63% of the reranker speedup. Start at the cheapest rung that clears your quality bar. Anti-pattern. Reaching for an LLM-reranker first because it's "best," paying 100× latency for a gap a $0 BM25 channel would have closed. Conditionality (W2, measured). Hybrid only beats dense on ceiling-free benchmarks (BEIR-FiQA: +0.5pp); it ties on saturated ones (MS MARCO recall ≥0.99). And SPLADE++ flipped the hybrid story — the sparse encoder choice matters. Measure on YOUR corpus; benchmark ceiling effects mask real differences. State the latency cost of each rung explicitly.
Rule. Two paired guards against RAG fabrication: (a) retrieve with a structural/tokenized index, not substring CONTAINS — and (b) prompt the reader to answer ONLY from retrieved facts and refuse when they're absent. W2.5: substring match made "meta" hit metal/metalloid/metabolism → confident chemistry fabrication; a full-text Lucene index + "answer using ONLY the graph facts, else say so" made the LLM correctly refuse the out-of-corpus "Mark Zuckerberg" query (9 token-overlap hits, 0 Zuckerberg facts → honest refusal).
Anti-pattern. Substring/LIKE %x% entity matching feeding a reader with no refusal contract → the model dresses noise as an answer.
Conditionality. The refusal contract trades recall for precision — on a workload where partial/inferred answers are wanted, soften it. (Pattern 24b, the refusal half, is also the reader-side guard behind W3.5.9's cloak-framing — frame the reader as an extraction function.)
Rule. Spend your strongest model on the answer step, a cheap/fast model on extraction. W3.5.9 probe: the multi-session counting failures were the weak reader (gemma flaked 1↔2 at temp=0), not retrieval — Haiku as reader was stable and lifted every backend; swapping the extraction model among competent locals barely moved accuracy. The needles were already in the store; the reasoning over them was the bottleneck. Anti-pattern. Burning the capable (metered) model on high-volume extraction while a weak model does the final reasoning — exactly inverted. Conditionality. Holds when retrieval surfaces the answer items at all (a strong reader on empty retrieval honestly returns "0" — that's the tell it's a retrieval problem, not a reader one). (See Pattern 22 — and W3.5.9 §4.10.)
Rule. Pattern 22 governs where a primitive runs; this governs what it extracts. The dimension you erase at write cannot be recovered at read. W3.5.8: whole-scroll summarize SKIPped conversational detail → 0/20; per-message atomic extraction preserved the needles. W3.5.9: a memory that stores user-action facts answers count questions; one that summarizes narrative (qdrant, 0%) loses them. Corollary — aggregation can live at read-time: count questions need a deeper retrieval window + an enumerate-then-count reader, NOT a special write-time counting tier.
Anti-pattern. Choosing the write-time primitive for storage economy (summarize to save space) when the workload needs the detail summarize discards.
Conditionality (the load-bearing one). User-turn-only extraction is workload-dependent: it gave 9× S/N on user-centric count questions (W3.5.9) but would LOSE single-session-assistant questions where the assistant's recommendation is the answer (W3.5.9 §2.2 refinement). Route the extraction policy by question shape; don't hard-code one.
Pattern 27 — Router Selects, Ensemble Unions; both can LOSE to the best single backend (RRF is non-monotonic for read-then-reason)
Rule. Two ways to combine backends, both with a ceiling — and on a read-then-reason task the simplest single backend can beat both.
- A question-type router dispatches each query to one backend → upper-bounded by best-single-backend-per-axis, and it only wins if (a) its table CONTAINS each axis's winner and (b) it routes each axis to that winner. W3.5.9: the
hybridrouter scored 75% — 10 pts BELOW the best single backend (atomic_fact85%) — because its table was built from the design-time PREDICTION ("knowledge-update → 2-tier dedup") that measurement falsified (KU's real winner is atomic_fact's read-time latest-wins reader, 100%, not the 2-tier path, 80%), AND the multi-session winner (mem0, 80%) wasn't even in its table. - An ensemble queries multiple backends and RRF-merges their retrieved facts (rank-based → fuses heterogeneous score spaces). It does NOT have "no ceiling": RRF maximizes recall@k of the union, but the downstream reader reasons over a fixed top-k window — so fusion is NON-MONOTONIC for read-then-reason. W3.5.9: the ensemble scored 80% — also BELOW
atomic_fact85%. It tied at the knowledge-update ceiling (100%, ≥ both members ✓) but dropped to 60% on multi-session, below BOTH members (af 70%, mem0 80%): net +1 gain (surfaced a needle af alone missed) −2 loss. Three measured loss mechanisms: window truncation (a needle both members kept individually falls out of the fused top-k), recall dilution (a low-recall member's facts displace a high-recall member's in the window), distractor injection (the union pulls one member's distractors into the other's clean set → over-count). - The actual best is a DATA-DRIVEN router that routes each axis to its measured winner (KU→atomic_fact 100%, multi-session→mem0 80%) → 90%, beating every single backend, the blind ensemble (80%), and the prediction-built router (75%). Also a ceiling, but the correct-and-reachable one.
Anti-pattern. Calling an ensemble "combines the best of each, so it can't lose" — it unions, and union ≠ improvement when a reader reasons over a fixed window (three_tier, a tier-union, fails the same way: 75% < atomic_fact 85%). And: building a router's table from design-time predictions instead of the measured matrix.
Conditionality. Fuse for pure retrieval (recall@k of the union ≥ either member — genuinely no ceiling there); route by selection for read-then-reason (fusion's non-monotonicity bites a reasoning reader). A router beats a single backend only when axes have different winners AND its table contains them AND it routes from measurement, not prediction. (See W3.5.9 §4.16 for the full per-question accounting.)
Rule. Add a memory tier only at its trigger condition, and publish the null result when the trigger is absent. W3.5.9: the L3 graph tier (HyperMem) serves multi-entity intersection queries; the slice had none, so three_tier (75%) scored as ≈ its atomic-fact L2 — L3 never fired (and the L1+L2 tier-union actually diluted it below standalone atomic_fact's 85%). That null result ("the third tier earns nothing on this workload") is more honest + more useful than a synthetic win. Graduate: 1-tier → 2-tier when you need dedup/supersede; → graph-tier when you need multi-entity relational joins.
Anti-pattern. Adding a graph tier speculatively, then reporting it "matches" the cheaper tier as if that validates it — it indicts it (wasted operational cost on this workload). (Mirror of Pattern 13's "no premature scaling," at the memory-tier level.)
Rule. A benchmark number is only as trustworthy as the behavior it rewards. Disciplines, all measured:
- Hold the judge constant across compared systems (W3.5.8: judge-confound moved every score ≤1pt once fixed).
- Calibrate dev-set difficulty — a too-easy set hides differences (W3 Entry 1).
- Exclude broken golds — a question whose gold contradicts its own text (W3.5.9
0a995998: gold=3 counts a non-store item the question excludes) rewards crude reasoning over correct; quarantine it, don't tune to it. - Commitment bias — golds that score a confident wrong guess and an honest abstention identically reward committing over calibrating (W3.5.8): the benchmark winner may be the worse production choice.
- Aggregate over single questions (N=20 → ±10pt; treat the shape, not the rank). Anti-pattern. Tuning a prompt to hit one noisy gold; trusting a surprising number without suspecting the instrument first.
Rule. Every added pipeline stage (HyDE, query expansion, a longer prompt, a reranker, an extra tier) is a hypothesis — A/B it against the simpler baseline before shipping. W3 Entry 2: a more detailed prompt made answers WORSE; Entry 6: HyDE added cost without improving the default pipeline. W3.5.8: constrained "top-K=5 best facts" extraction collapsed accuracy 30-35pts. More machinery ≠ better. Anti-pattern. Adding HyDE / multi-query / a tier because the literature uses it, without measuring that it beats your baseline on your data. Conditionality. Volume buffers extraction error — more (unconstrained) extractions can help where fewer "high-confidence" ones poison (W3.5.8 K_min≥8). Counterintuitive: sometimes more-but-noisier beats fewer-but-authoritative (Bayesian model averaging vs MAP selection).
Pattern 31 — Hand-Roll vs Production Library: A/B on YOUR Data; the Lib's "Failures" Are Often Contracts
Rule. Before adopting a memory/RAG library, A/B it against a minimal hand-roll on your own test set. W3.5: hand-roll scored 15/15 vs mem0 v2 10/14 — but the 4 "failures" were different semantic contracts (contradiction archival, episodic/semantic separation), not bugs. W3.5.9: the homebrew atomic-fact (85%) BEAT the mem0 SDK (75%) outright once the read-time levers (user-turn extraction + count-aware + latest-wins readers) were added — the write-time primitive plus a capable reader carried it past the production library on this slice. Always test the cheaper hypothesis first (W3.5: a 1-env-var model-swap, 72s, reframed "mem0 is flaky" → "mem0 has different contracts" — saved hours of wrong-direction patching). Anti-pattern. Adopting the library on its published number, or dismissing it on a single local run, without isolating WHY the gap exists (model? contract? reader?).
Rule. When a capable model is reached through a metered/cloaking gateway (a Claude-subscription proxy, a shared key), treat it as a scarce, quirky resource: (a) role-split — high-volume roles (per-message extraction, ~1000s of calls) on a local model, capability-critical low-volume roles (the reader, ~120 calls) on the gateway; (b) retry/backoff on cooldown (503); (c) make the judge non-fatal — save predictions, rejudge later; (d) cloak defense — frame structured tasks as data-extraction (an injected persona will do "extract from these records" but refuse "answer my personal question"), detect residual persona refusals, fall back to a local model. All measured in W3.5.9 (BCJ 5-7): all-Haiku-via-VibeProxy crashed the eval until the role-split + retry + cloak-fallback made it complete. Anti-pattern. Routing every LLM call through the metered gateway "for quality" → cooldown 503s mid-run, persona refusals scored as wrong answers, lost imprints. Conditionality. Only relevant when the capable model is gateway-mediated; with a clean API key, just rate-limit politely. The local fallback is the read-side mirror of "the gateway might refuse the write."
Rule. A memory system answers many question shapes (current-value, count, as-of/ordering, preference, lookup); each needs a DIFFERENT read-time assembly operator, not one reader. Build a read-side operator router keyed on question type: latest-[sN]-wins (knowledge-update), enumerate-then-count (count), earliest-[sN]-wins (temporal ordering), generate-aligned-with-preferences (preference), deeper-k terse lookup (single-session). W3.5.9 6-axis: read-side operators lifted base 45% → 62% → 79% → 83% (temporal-reasoning 0/4 → 3/4 → 4/4 from the ordering operator + per-operator seq recency), all without changing storage. Selection beats union: a per-axis router that picks the right operator/backend per question beats both a blind ensemble (RRF union) and tier-stacking (three_tier), which dilute (see Pattern 27).
Anti-pattern. One universal reader prompt for all question types; or routing STORAGE backends (the hybrid router, 75%) while leaving the READ assembly generic.
Conditionality. Requires knowing/classifying the question type; a real system classifies it, the eval has it labeled.
Recency-operator sub-lesson (measured W3.5.9 §4.17). Recency-signal granularity is per-operator, not global. Three tiers: (1) session-index [sN] — per-session, assumes index=chrono (violated ~17%); correct for current-value / superlative latest-wins (KU); (2) insert-sequence seq — per-fact monotonic counter at imprint, needs sessions date-sorted; use for temporal-ORDERING operators (fixes intra-session order; validated: gpt4_2487a7cb now correct); do NOT apply globally to KU latest-wins — measured regression: global seq on a superlative KU question re-exposed a re-mentioned old value that session-level [sN] suppressed (4/4→3/4 on 6a1eabeb); per-operator assignment restores both axes to 4/4 and lifts base 79%→83%; (3) extracted event-time — per-fact LLM temporal-expression extraction, highest fidelity, LLM-dependent ceiling. The operator-routing thesis extends to the recency signal: not just "which prompt?" but "which recency resolution?" — both are operator-specific choices.
Extraction-volume sub-lesson (measured W3.5.9 §4.17 14B run). Extraction VOLUME must be controlled at READ time, not write time. A more verbose extraction model (14B vs 7B) over-extracted: ~5 near-duplicate restatements of the same item per fact (measured: F-15 model kit restated ~5 ways, 445 total facts). Under a fixed retrieval top-k (40), duplicates crowd the window and rare items fall below the cutoff — the Spitfire kit existed in the store but ranked below #40, so the reader enumerated 4 of 5. Bigger extraction model ≠ better recall when the reader's context window is fixed. The non-destructive remedy is read-time diversity rerank (MMR or per-entity cap), scoped per-operator — it diversifies what the reader sees without altering the store. Do not attempt to control volume by deduplicating at write (see Pattern 34).
See also:
- Pattern 27 — Router Selects / Ensemble Unions (the backend-selection version; this pattern is its read-assembly mirror — selection beats union in both)
- Pattern 25 — The Reader/Composer Is the Quality Lever (the operators this pattern routes ARE read-side reasoning steps)
- Pattern 34 — Evidence-Before-Belief Extraction (write-time dedup is the same category of mistake as write-time discard; diversify at read, preserve at write)
- [[Week 3.5.9 - Requirement-Driven Memory Architecture]] — the 6-axis read-operator runs (45→62→79%→83%) + §4.17 recency-signal tiers + per-operator seq result + 14B over-extraction finding
Rule. Write-time extraction filters that DISCARD data are irreversible and workload-brittle. W3.5.9: a user-turn-only extraction filter (built to de-flood multi-session counts) ZEROED the single-session-assistant axis (0/4) — the answer lived in dropped assistant turns. Fix: extract from all roles, TAG provenance (role=user/assistant), and filter at READ time per question type — the de-flooding moves to a read-side role filter, losing no data (evidence-before-belief). Also: parse TURNS not lines (multi-line turns mis-tag if split on raw newlines — measured 283 user / 5 assistant), and CHUNK long turns (~700 chars) so a long narrative turn isn't one extraction call that returns 0 facts (a 2473-char turn: 0 → 47 facts after chunking). Refs: Eywa evidence-before-belief (arXiv 2605.30771), MemIR provenance (arXiv 2605.25869).
Anti-pattern. Dropping turns/roles at write to reduce noise — it's unrecoverable and breaks the axis whose answer lived there. Splitting role-tagged scrolls on raw newlines (mis-tags multi-line turns). Write-time fact dedup is the same category of mistake: "started X" and "finished X" are distinct facts; old and new value of the same attribute are distinct facts; merging at write destroys the signal that knowledge-update and temporal-reasoning operators depend on. Preserve at write; diversify/deduplicate at read.
Conditionality. Read-side role filtering needs the question's provenance need (assistant-stated vs user-stated), routed by type. Read-time diversity rerank (MMR or per-entity cap) is the non-destructive remedy for extraction-volume dilution — it keeps the store intact and scopes the policy per-operator and per-query.
See also:
- Pattern 22 — Lifecycle Position Matters (this is late-binding the role filter: keep all roles at write, filter at read so the policy stays iterable)
- Pattern 26 — Write-Time Primitive Preserves Signal (Pattern 26 = don't erase a dimension; Pattern 34 = don't erase a role — same write-time-discard hazard, and 26's user-turn-only conditionality is exactly the failure 34 fixes)
- [[Week 3.5.9 - Requirement-Driven Memory Architecture]] — the role-aware extraction run (37→45%) + the chunking fix
Rule. To make a reader abstain on unanswerable questions WITHOUT over-refusing answerable ones, ask "is the question's SUBJECT present in the records at all?" (topic-presence) — NOT "is the answer grounded?" W3.5.9: a binary GROUNDED/UNGROUNDED gate over-refused catastrophically (answerable 19/24 → 10/24, −9) because it conflated "answer not verbatim" with "unanswerable"; a topic-presence gate (biased to PRESENT + few-shot) cut over-refusal to −1 (answerable 18/24) while lifting abstention 3/8 → 5/8. Same wiring, same model — only the epistemic FRAMING of the prompt changed. Decouple the abstention judgment from answer generation (a separate lightweight classifier pass). Refs: AbstentionBench (arXiv 2506.09038 — a good prompt boosts abstention without precision loss, but prompt-only has a ceiling), Decision-aware Answer/Ask/Abstain (arXiv 2604.04565), Know Your Limits survey (TACL). Anti-pattern. Tuning a grounding gate's STRICTNESS to fix over-refusal (it's a framing problem, not a threshold). Shipping prompt-only abstention as if it solves the problem (it has a hard ceiling — keep it opt-in, measure precision). Conditionality. Prompt-only abstention is fundamentally limited; uncertainty/training-based methods (GRACE arXiv 2601.04525, Abstain-R1 arXiv 2604.17073) preserve answerable accuracy by design where prompts can't.
See also:
- Pattern 24 — Structural Index + Faithful-Refusal Contract (24's refusal contract is answer-groundedness; Pattern 35 reframes it to topic-presence to kill over-refusal — the framing 24 was missing)
- Pattern 30 — New Complexity Must A/B-Earn Its Keep (the topic-presence gate shipped OPT-IN at marginal +1 — it had to A/B-earn its default-on, and didn't)
- [[Week 3.5.9 - Requirement-Driven Memory Architecture]] — the abstention gate A/B (grounding −9 vs topic-presence −1)
Rule. When a store has both a READ ranker and a WRITE/retention policy, wire the read to record what it used so retention acts on real signal — not a synthetic proxy. W3.5.95: metacog_recall.recall_block reads a self-pattern AND records the visit (track=True) → heat (visits + recency + importance) accrues → enforce(budget) dedups + evicts by heat → bounded store. Before the wiring, heat was demo-only: eviction ran on simulated visits, so retention had no connection to what the agent actually reached for. Closing the loop is what makes eviction earned (survivors are the facts recall kept using) rather than arbitrary. The general shape: a bounded store needs its keep/drop decision fed by its own access pattern, or it's guessing.
Sub-rule A — Default by SEAM, not by caller (so wiring a side-effect changes no caller). The READ primitive (recall) defaults track=False — pure-read keeps tests and ablation harnesses side-effect-free; the live-loop convenience wrapper (recall_block) defaults track=True because that's the path that should record usage. Behavior follows the layer, so every existing caller got the right default for free and nothing had to change. When adding a side-effect to a widely-called primitive, put the opt-in on the primitive and the opt-out on the wrapper that fans out to it (or vice-versa) — never force every call site to update.
Sub-rule B — Lazy import is the escape hatch for a legitimate two-way dependency. heat_eviction imports metacog_recall (shared tokenizer + decay constant); wiring recall → touch creates a genuine cycle. A top-level reverse import would deadlock at module load. Defer it: from heat_eviction import touch inside recall(), resolved at call-time when both modules are fully loaded. Don't refactor a real bidirectional dependency into a fake one-way one just to satisfy the import graph — defer resolution instead.
Anti-pattern. Shipping a retention/eviction policy whose signal is seeded by a demo or a heuristic, never by live reads — the store looks bounded but evicts by guesswork. Also: breaking a legitimate two-way module dependency by hoisting a constant into a third "util" module purely to dodge a cycle the language already lets you defer.
See also:
- Pattern 28 — Memory-Tier Graduation Triggers + Null-Result Discipline (graduation/eviction are the two ends of the same bounded-store lifecycle)
- [[Week 3.5.95 - Self-Observability Memory#Phase 7 — Bounding the store: heat-scored eviction|W3.5.95 Phase 7]] — the heat/eviction mechanism + the
recall(track=)wiring (BAI-LAB/MemoryOS leverage)
Pattern 37 — Measured Search-Policy Architecture (selector / gate / calibrator; score the prompt, not the retrieval)
Rule. A self-tuning retrieval policy needs exactly three layers, and almost nothing else: a cheap deterministic SELECTOR that runs every ingest, a rare expensive GATE that runs only on a policy change, and an offline CALIBRATOR that proves the selector's cheap metric still tracks the real objective. Build the simplest version that the evidence supports; every richer thing must measure its way in. W3.5.96 (mixed 67-page brain, golden eval): SELECTOR = discounted grounding@C over keyword/vector/hybrid → one global arm per corpus, re-fired on ingest; GATE = answer-judge (pinned strong model) on a flip; CALIBRATOR = corr(metric, answer-pass) on a snapshot.
Sub-rule A — Score the prompt, not the retrieval. The selector metric must use the cutoff C = the chunks the generator actually reads, and discount by rank — not an arbitrary top-K. W3.5.96: rank-blind grounding@5 tied vector and hybrid (0.972=0.972, decided by sort order); budget-aware discounted grounding@3 separated them (0.910 > 0.832) because vector hid ~10% of its answer mass at ranks 4–5, outside a 3-chunk prompt the generator never sees. C is measured off the agent's context-assembly (here limit=3), never hand-tuned; sweep C∈{1,2,3,5} to confirm the verdict isn't on a cliff.
Sub-rule B — Cheap in the hot loop, expensive at the edges. The deterministic metric runs every ingest (zero-LLM, free); the answer-judge GATE fires only when the policy flips. Cost: an answer-judge selector = 6G LLM calls every ingest forever; the gate = 4G·P(flip) → ~15× cheaper early, →0 as the policy converges. Same metric, opposite placement, completely different cost curve — and the gate's cost falls exactly as the system matures and risk drops.
Sub-rule C — Build the ceiling before the classifier. Before engineering any router/reranker/sub-agent, compute its unattainable upper bound against the baseline. W3.5.96: the per-query routing oracle (peeks at labels) equalled global hybrid in one free number (0.910 = 0.910) — killing per-query routing before any classifier was tuned. If the perfect-play ceiling doesn't clear the baseline, no engineering will. (See Pattern 27 — both router and ensemble can lose to the best single backend.)
Sub-rule D — Pin the generator before attributing a delta to retrieval. Scoring a retrieval choice by answer quality mixes in generation variance. W3.5.96: per-query routing showed −0.250 answer-pass on a local 14B, but Δ0 on Claude Opus 4.5 — identical even on the differing-context questions. The "regression" was the weak generator's context-composition sensitivity, not retrieval. Run a two-tier judge (weak + strong); the gap between them localizes whether an effect is retrieval or generation. (This is Pattern 29 — the instrument shapes the result — applied to the generator.)
Anti-pattern. Putting an LLM answer-judge in the every-ingest hot loop (confounds retrieval choice with generation variance and meters the loop); per-query routing built without a ceiling check (complexity for Δ0); a rank-blind grounding metric (structurally can't see RRF demotion); a hand-picked C (scores a context the model never reads); trusting a single weak generator (manufactures phantom retrieval effects). Most of the win here was deletion — the final system is simpler than the naive one and better, because each cut piece failed a measurement.
Conditionality. The architecture is corpus-agnostic; the verdict ("global hybrid wins, routing rejected") is not. Re-fire the selector on drift (built-in), re-run the calibrator on domain shift, and re-check the routing oracle if a future corpus is genuinely bimodal (e.g. a code corpus + a prose corpus where one arm is useless on half the queries). The loop signals when its own assumptions break.
Rule. When a parent spawns an async/background subagent, the subagent's terminal state must be a closed, enumerated status contract — not a free-form string parsed differently on each side — and the parent's poll loop needs its own stuck-detector that is independent of the task's own timeout, because a hung child can report RUNNING forever. deer-flow (contracts/subagent_status_contract.json + subagents/status_contract.py + task_tool.py): statuses are completed / failed / cancelled / timed_out / polling_timed_out, and polling_timed_out ("RUNNING for 15 min ⇒ assume stuck") is a separate timer from the task's timed_out (900 s). This is the production extension of W4.6's heartbeat watchdog from in-process workers to async subagents.
Sub-rule A — Stuck-detection ≠ task timeout (two independent timers). The task timeout bounds expected work; the polling timeout catches a child that lies about being alive. One can't substitute for the other: a 900 s task that wedges at second 5 still says RUNNING for 895 s unless a separate poll-stuck timer fires.
Sub-rule B — Enumerate pre-execution failures too. unknown subagent type, host bash disabled by config, background task disappeared — validate and return a clear failed status before spending tokens, not after. Fail fast with a contract-shaped error.
Sub-rule C — Make the contract a cross-boundary test fixture. One JSON (status values + cases + expected_status) that both the producer (backend) and consumer (frontend/parent) load and must agree on — so the status field can't silently drift apart across the boundary.
Anti-pattern. Trusting a child's RUNNING indefinitely (the canonical "stuck subagent burns the budget" failure); ad-hoc status strings each side parses with its own regex; reporting failure only after the spend that a pre-check would have caught.
See also:
- [[Week 4.6 - Durable Agent Runtime and Process Topologies]] — heartbeat watchdog (L0/L1/L2) + stall; this pattern is the lab phase extending it to async subagents (lab
src/subagent.py; stuck child →polling_timed_outin 0.32 s atpoll_timeout=0.3; source: bytedance/deer-flow). Productionised mirror in the agentkit shared lib atagentkit/runtime/subagent.py. - Pattern 39 — fan-out cost aggregation (the other half of "don't let a fan-out run away").
Rule. A fan-out (star / tree / mesh) must aggregate every child's token/cost up to the parent and enforce a budget ceiling that aborts the whole fan-out — per-child max_tokens does not bound the total, so N children at the per-child limit is an N× blow-up. deer-flow (subagents/token_collector.py): a collector sums subagent token usage to the parent run. The ceiling is checked against the running sum, not per child.
Sub-rule — Meter before AND during. A pre-flight estimate gates the spawn (refuse a fan-out that can't fit the budget); a running sum aborts mid-flight when it crosses the ceiling. Both, because estimates are wrong and a single runaway child can blow a budget the estimate approved.
Anti-pattern. Setting per-agent max_tokens and calling it budgeted — the classic-AutoGPT cost failure reappears at fan-out scale (one task, N parallel children, no parent-level sum, surprise bill).
See also:
- [[Week 11.6 - Production Tracing and Cost Telemetry]] — per-node cost; this adds the cross-subagent rollup + ceiling.
- [[Week 4.6 - Durable Agent Runtime and Process Topologies]] — Phase 8 lab (
src/cost_ceiling.py:FanoutBudget+run_fanout; uncapped 10×100 = 1000, ceiling 350 → aborts on child 4, spent 400, saved 600; source: deer-flow). The fan-out shapes whose cost this bounds.
Rule. A tool or subagent exception must be caught and reframed as a structured observation the model can act on — never propagated as a crash, never silently swallowed. deer-flow (ToolErrorHandlingMiddleware): a tool raise becomes a message like "Error: Tool 'task' failed with TypeError: …. Continue with available context, or choose an alternative tool." — the agent reads it and recovers (picks another tool / replans) instead of dying.
Sub-rule — Include the error TYPE + a recovery hint; keep the raw error in the trace. The model needs the type (to decide whether to retry vs switch tools) and an explicit "what to do next"; the audit log keeps the untruncated original (the framed message is for the model, the trace is for you).
Anti-pattern. A tool exception aborting the whole run (one bad tool call kills an hour of work); OR catching it and returning "" (the agent loops blind, never learning the tool failed). agentkit already frames tool output via quarantine + records mark_failed; this pattern is the recovery-hint half on the model-facing side.
See also:
- [[Week 7 - Tool Harness]] — implemented as Concept 4 (error-as-prompt) + Diagram 1c (error-recovery flow) + the
run()loop (errors are tool results with a different payload shape); deer-flow'sToolErrorHandlingMiddlewareis the convergent third source (alongside Claude Codequery.tsand Codexexecpolicy). - Pattern 41 — implement this as one middleware in the stack, not a per-tool try/except.
Rule. Guardrails, sandboxing, cost-metering, and error-handling are cross-cutting — implement each as a middleware that wraps every tool/sandbox call, behind a Provider Protocol, composed as an ordered stack — not as checks copy-pasted into each call site. deer-flow (guardrails/{middleware,provider}.py, sandbox/middleware.py): a guardrail provider and a sandbox middleware intercept calls uniformly; swapping the policy is a provider swap, not a code sweep.
Sub-rule — Order is part of the design. The stack is guardrail (allow/deny) → cost-check (budget) → execute → error-wrap (Pattern 40) → trace. Each layer is independently testable and swappable; the order encodes the policy (deny before spend; wrap errors before they reach the trace).
Anti-pattern. An allowlist / timeout / cost check hand-pasted at each tool call site — guaranteed to drift, leave gaps (the one call site someone forgot), and resist testing. This is the same DI discipline as agentkit's Embedder/LLMClient/UrlChecker seams, applied to enforcement rather than capability.
See also:
- [[Week 11.5 - Agent Security]] — guardrail policy; this is the architecture that delivers it uniformly.
- [[Week 7 - Tool Harness]] — where tool middleware composes (source: deer-flow).
See also:
- Pattern 27 — Router Selects, Ensemble Unions; both can LOSE to the best single backend (the ceiling-first check operationalizes this)
- Pattern 29 — Eval Integrity (the instrument shapes the result) — Sub-rule D is this applied to the generator
- Pattern 30 — New Complexity Must A/B-Earn Its Keep — routing failed exactly this test
- Pattern 33 — Route the READ assembly by question type; select, don't union — W3.5.96 tested per-query routing and rejected it on this corpus (Pattern 33's conditionality made concrete)
- [[Week 3.5.96 - Self-Wiring Memory (GBrain)#Phase 9 — A corpus-adaptive search policy, tuned on a real golden eval set|W3.5.96 Phase 9]] — selector (
policy_eval.ts), routing test (route_eval.ts), answer A/B (answer_route_ab.py), verifier (verify_arch.py)
Rule. Two subagents writing the same artifact race two distinct ways that need distinct primitives: a lost update (two read-modify-write cycles interleave; the second clobbers the first with stale data) is fixed by serializing the whole RMW under a cross-process file lock; a torn read (a reader observes a half-written file) is fixed by os.replace atomic publication — which needs no lock at all. Use both; neither substitutes for the other. deer-flow (sandbox file operations): subagent deliverables are written through a locked, atomically-published path.
Sub-rule A — Atomic publish is filesystem-scoped. The temp must live on the target's own filesystem (tempfile.mkstemp(dir=target.parent, …)), or os.replace silently degrades from an atomic rename to a non-atomic cross-device copy.
Sub-rule B — Unique temp per call, not per process. A pid-scoped temp name (f"{name}.{pid}.tmp") passes every single-threaded test and crashes the instant two threads share a process — both create the same temp, and one writer's rename races the other's away. (Observed live: W4.6 BCJ Entry 3, FileNotFoundError.) mkstemp gives a guaranteed-unique name.
Anti-pattern. "Just add a lock" — fixes lost updates but still ships half-written files to readers. "Just write atomically" — fixes torn reads but still drops concurrent edits. Per-pid temp names — green in serial tests, crash under real concurrency.
See also:
- [[Week 4.6 - Durable Agent Runtime and Process Topologies]] — Phase 7 lab (
src/artifact_writer.py; locked 400/400 every run, unlocked ≈ 51/400 — ~349 lost; source: bytedance/deer-flow) + BCJ Entry 3 (the pid-temp-collision bug). Reuses Phase 3'sFileLockprimitive for a second job. - Pattern 38 — Subagent Status Contract (the other deer-flow durability pattern labbed on W4.6).
Rule. A long-horizon agent must stream partial output — tokens, tool-call boundaries, intermediate steps — as it produces them, never block until a terminal answer. The metric that matters is time-to-first-token (TTFT), not total latency: a 40 s run that streams its first token in 800 ms feels responsive; the same run that blocks for 40 s reads as hung. deer-flow (stream_bridge): partial results are bridged to the client over a streaming channel (SSE / chunked / websocket) as they are generated.
Sub-rule — One event spine for streaming AND tracing. Streaming forces the harness to emit an event at every gate (tool start/end, model delta); that is the same event stream your tracing/telemetry needs. Build one event bus, feed both — don't bolt on a separate ad-hoc emit path for the UI.
Sub-rule — Silence is a signal. A stream that stalls is exactly the "alive-but-stuck" case Pattern 38's polling-timeout watches for; the two are complementary, not redundant.
Anti-pattern. Block-to-completion (the user assumes it hung; you get zero visibility into where a slow run spends time); a UI-only emit path separate from the trace bus (they drift).
See also:
- [[Week 11 - System Design]] Concept 6 — serving-layer treatment (source: deer-flow).
- [[Week 11.6 - Production Tracing and Cost Telemetry]] — the event bus streaming reuses.
- Pattern 38 — a silent stream is the stuck-subagent signal.
Rule. In a multi-tenant agent, thread a per-request user_context (tenant id, permissions, quota) through the entire run — memory, sandbox paths, tool credentials — and validate it before execution begins. Retrieval ACL filtering is necessary but not sufficient: the agent's own state must be tenant-scoped structurally, or you leak across tenants through the harness, not the index. deer-flow (user_context + pre-execution validation): the request context is established and checked before any tool runs.
Sub-rule A — Scope state by context, don't filter after. Memory namespace, sandbox dir, and tool auth are keyed on user_context, so isolation is structural — not a filter a future call site can forget.
Sub-rule B — Validate pre-execution (fail fast, contract-shaped). Unknown tenant / missing permission / exhausted quota → return a clear failed before spending tokens (the same pre-execution enumeration as Pattern 38 Sub-rule B). Validating after execution has already leaked or already spent.
Anti-pattern. Relying on retrieval ACL alone → cross-tenant leakage via subagent scratch memory, sandbox files, or shared tool creds; validating the request only after the run has touched state.
See also:
- [[Week 11 - System Design]] Concept 6 + the Exercise-1 retrieval-ACL checklist (the necessary-but-insufficient half) (source: deer-flow).
- Pattern 38 — pre-execution failure enumeration (the fail-fast-before-spend move).
Pattern 45 — Group-Relative Experience Distillation (rank rollouts, keep the above-mean lesson; no weights)
Rule. To make an experience/reflection memory select better lessons without fine-tuning, sample a group of N rollouts per task, score each with a verifier, and keep the natural-language lesson only from rollouts whose reward beats the group mean — GRPO's group-relative advantage $\hat{A}i = (r_i - \mu{\text{group}})/\sigma_{\text{group}}$, consumed by the prompt layer instead of an optimizer. Youtu-agent (Training-Free GRPO, arXiv:2510.08191): the same variance-reduction statistic that stabilizes RL, applied to text — the model stays frozen, the experience buffer is the policy.
Sub-rule — A lesson needs a comparison class, not just a post-mortem. Plain Reflexion critiques one trajectory with no baseline, so a confidently-wrong lesson enters memory unchallenged. Group-relative ranking gives every lesson a baseline (the group mean); below-mean rollouts become "what not to do" counter-lessons.
Anti-pattern. Reaching for weight-GRPO (LoRA + RL trainer infra) when textual distillation hasn't plateaued — paying GPU-hours for selection a prompt layer does for free. Inverse anti-pattern: assuming textual distillation lifts capability — it only reweights behaviors the frozen model already has; new capability still needs weights.
See also:
- [[Week 5.5 - Metacognition]] Theory Primer — Training-Free GRPO (the textual cousin) vs [[Week 9.5 - Agentic RL Fine-Tuning]] (weight-GRPO; same statistic, optimizer-consumed).
- Pattern 29 — Eval Integrity (the verifier scoring the rollouts is the lever; a convenient verifier distills convenient lessons).
Pattern 46 — Auto-Tool-Gen Gated by an In-Loop Debugger (generation is cheap; the repair loop is the contract)
Rule. When an agent generates its own tools, the load-bearing stage is not generation but validation+repair: draft schema → draft code → run in a subprocess sandbox → feed the traceback to an in-loop debugger agent that patches and re-runs until it executes once → only then emit as an MCP manifest and register. Youtu-agent (auto-tool-generation): query → schema → code → manifest, subprocess-validated, with a debugger agent in the loop. Schema-first fixes the contract before implementation.
Sub-rule — "Passed validation" is only as strong as the validation inputs. Smoke inputs that miss the production distribution make a self-written tool a Pattern 29 failure in disguise. Side-effecting generated tools (write/pay/deploy) stay gated behind a human or a real suite; read-only tools are the safe auto-register entry point.
Anti-pattern. Registering an LLM-drafted tool because it looks right (imports a nonexistent lib, mishandles the one edge case the agent hits) — hallucinated capability with zero executions behind it. The repair loop is what converts "plausible code" into "ran successfully at least once."
See also:
- [[Week 7 - Tool Harness]] Concept 7 (the five-stage pipeline) + Concept 5 (MCP manifest out).
- [[Week 7.8 - Code-Agent Patterns AST Coverage Mocks]] — the same generate→measure→repair loop, run for test generation instead of tool generation.
- Pattern 29 — Eval Integrity (convenient validation inputs = convenient pass).
Rule. Compose a multi-agent topology — which agents, which edges, which models, which tools — from declarative config (YAML) rather than hand-wired code, so a new topology is a config file, not a code change. Youtu-agent (Hydra defaults + ${oc.env:}): the whole agent graph is assembled from composable YAML groups with env-interpolated secrets; swapping STAR→MESH or adding a worker is a config edit the framework reads, not a new Python module.
Sub-rule — Config-composed ≠ config-toggled. A boolean flag that picks between two hard-coded topologies is still code-defined. The pattern is true composition: the config names the nodes and edges, and the runtime builds the graph from that description (Hydra defaults lists, group overrides).
Anti-pattern. Declarative-everything when the topology set is small and stable — a YAML DSL that reimplements Python with worse errors and no type-checking (config-as-system carries a real debuggability + onboarding tax). Code-defined topologies win when there are four shapes that rarely change; declarative wins when topologies are many, user-authored, or hot-swapped without a deploy.
See also:
- [[Week 4.6 - Durable Agent Runtime and Process Topologies]] §2.5 (the 4-trigger × 6-topology design space, code-defined in
topologies.py) — declarative composition is the config-side alternative. - agentkit
select_topology(code/rule-based selection) — the same choose-a-topology decision, made in code.
| Pattern | When in the work cycle | Prevents |
|---|---|---|
| 13 — Storage-Scale Match | Picking a store for retrieval/memory data | Wrong backend for data shape (vector DB for a tree/graph); premature scaffolding |
| 14 — Delegation Contract | Every parent → subagent spawn | "Subagents know nothing" + token-burn from under-briefed workers |
| 22 — Lifecycle Position Matters | Placing a lossy primitive (atomise/summarise/embed) in a memory pipeline | Right primitive at the wrong stage — silent signal destruction |
| 23 — Cheap-First Retrieval Ladder | Choosing retrieval quality vs cost | Paying 100× for a rerank a $0 BM25 channel would match |
| 24 — Structural Index + Faithful Refusal | Entity/keyword retrieval feeding an LLM | Substring-match fabrication; reader dressing noise as an answer |
| 25 — Reader Is the Quality Lever | Allocating models across a RAG/memory pipeline | Capable model on extraction, weak model on reasoning (inverted) |
| 26 — Write-Time Primitive Preserves Signal | Choosing WHAT to extract at write | Erasing a dimension the workload needs; hard-coding one extraction policy |
| 27 — Router Chooses / Ensemble Combines | Building a "hybrid" memory/RAG | Calling a router an ensemble; expecting it to beat its best member |
| 28 — Memory-Tier Graduation Triggers | Adding a tier (2-tier, graph) | Speculative tiers; reporting a "match" as validation not indictment |
| 29 — Eval Integrity | Any benchmark / A-B measurement | Drifting judge, easy dev-set, broken golds, commitment bias, single-Q ranks |
| 30 — New Complexity A/B-Earns Keep | Adding HyDE / multi-query / a tier / longer prompt | Shipping machinery the literature uses without measuring it beats baseline |
| 31 — Hand-Roll vs Library A/B | Deciding to adopt a memory/RAG library | Adopting on published number; dismissing on one run; not isolating WHY |
| 32 — Metered-Proxy Role-Split | Capable model behind a rate-limited/cloaking gateway | All-calls-through-gateway → cooldown crash, persona refusals, lost work |
| 33 — Route the READ Assembly by Question Type | Building a multi-shape memory reader | One universal reader prompt; routing storage but not the read assembly |
| 34 — Evidence-Before-Belief Extraction | Choosing what to keep/drop at write | Discarding turns/roles at write — unrecoverable, breaks the axis whose answer lived there |
| 35 — Abstention Is Topic-Presence, Not Groundedness | Making a reader refuse unanswerable questions | Catastrophic over-refusal from a grounding gate; tuning strictness instead of reframing |
| 37 — Measured Search-Policy Architecture | Building a self-tuning retrieval policy / eval loop | LLM-judge in the hot loop; routing without a ceiling check; rank-blind metric; hand-picked C; phantom retrieval deltas from a weak generator |
These patterns compound. The teams (or solo engineers) who use them feel "calm and deliberate" to work with; the ones who don't feel "frantic and surprising." The difference isn't IQ or experience — it's the discipline of slowing down at the right 5-10% of moments.
- [[Bad-Case Journal]] — the empirical failure log these patterns are distilled from (W1…W3.5.9 incidents)
- [[Week 3.5.9 - Requirement-Driven Memory Architecture]] — source of Patterns 25-35 (reader-as-lever, router-vs-ensemble, memory-tier graduation, eval integrity, metered-proxy, read-assembly routing, evidence-before-belief extraction, topic-presence abstention)
- [[Week 3.5.8 - Two-Tier Memory Architecture]] — source of Patterns 22, 26, 29-31 (lifecycle, write-time primitive, commitment-bias, volume buffers, hand-roll-vs-library)
- [[Week 2 - Rerank and Context Compression]] — source of Pattern 23 (cheap-first retrieval ladder)
- [[Week 2.5 - GraphRAG]] — source of Pattern 24 (structural index + faithful refusal)
- Engineering-process + curriculum-authoring patterns (scope-estimate, options-table, forward-link panels, spec/disk fidelity, …) were removed from this doc on 2026-06-02 to keep it agent-development-focused. They were generic, not agent-specific.
— end —