Empirical & Architectural Study: LangGraph Agent vs. Interpretable Context Methodology (ICM) Agent for automated repository analysis and ADR generation.
In early 2026, Jake Van Clief and David McDermott published research introducing the Interpretable Context Methodology (ICM). They hypothesized that agentic AI workflows do not necessarily require heavyweight orchestration frameworks, state graphs, or vector databases. Instead, a transparent, filesystem-first paradigm with layered markdown contexts can achieve comparable or superior results with fewer tokens, lower code complexity, and built-in human-in-the-loop inspection.
This repository provides a complete, side-by-side empirical benchmark of two autonomous systems evaluated on real open-source repositories (e.g. pallets/flask, tiangolo/fastapi, encode/httpx, django/django):
- LangGraph Agent (
agent/): Graph-orchestrated pipeline with ChromaDB vector storage, embedding clustering, and centralizedTypedDictstate. - ICM Agent (
icm_agent/): Zero-framework, file-driven pipeline using a 5-layer context hierarchy and deterministic Git ingestion.
[Repository URL / Path]
│
▼
┌─────────────────────────┐
│ IngestAgent │ (ingest_agent.py)
│ ─ git clone / log │
│ ─ nomic-embed-text │
│ ─ Write ChromaDB L0 │
└────────────┬────────────┘
│
▼
┌─────────────────────────┐
│ AnalysisAgent │ (analysis_agent.py)
│ ─ Fetch L0 embeddings │
│ ─ Greedy Cosine > 0.75 │
│ ─ Write Centroids to L1 │
└────────────┬────────────┘
│
▼
┌─────────────────────────┐
│ RationaleAgent │ (rationale_agent.py)
│ ─ In-memory SHA lookup │
│ ─ git show diffs / TODOs│
│ ─ 1 LLM call per cluster│
│ ─ Write Rationale to L2 │
└────────────┬────────────┘
│
▼
┌─────────────────────────┐
│ SynthesisAgent │ (synthesis_agent.py)
│ ─ 1 LLM call per ADR │
│ ─ Writes docs/adr/*.md │
└────────────┬────────────┘
│
▼
┌─────────────────────────┐
│ QualityChecker / Judge │ (quality_checker.py)
│ ─ 1 Judge call per ADR │
│ ─ Rubric 0-10 (5 axes) │
└────────────┬────────────┘
│
┌───────────────┴───────────────┐
│ score < 3.5 & revision < 2 │ score >= 3.5 or retries exhausted
▼ ▼
┌─────────────────────┐ ┌─────────┐
│ Revise Synthesize │ │ [Output]│
└──────────┬──────────┘ └─────────┘
│ (loops back to Judge)
- Clustering: Pure Python greedy cosine threshold (
> 0.75) grouping commit embedding vectors. - State Management: Centralized
ADRState(TypedDict) containing commits, file trees, decision points, and ADR objects. - LLM Calls: Isolated prompt slices per node; 14–22 LLM calls per repo run (1 per decision cluster / ADR).
┌────────────────────────────────────────────────────────────────────────┐
│ Layer 0: Global Instructions & Philosophy │
│ └── adr-icm-workspace/CLAUDE.md │
├────────────────────────────────────────────────────────────────────────┤
│ Layer 1: Workspace Context & Injected Target Repository URL │
│ └── adr-icm-workspace/CONTEXT.md │
├────────────────────────────────────────────────────────────────────────┤
│ Layer 2: Stage-Specific Context │
│ └── adr-icm-workspace/stages/{stage_num}/CONTEXT.md │
├────────────────────────────────────────────────────────────────────────┤
│ Layer 3: Reference Documentation & Rubrics │
│ ├── adr-icm-workspace/stages/{stage_num}/references/*.md │
│ └── adr-icm-workspace/_config/shared/repo-conventions.md │
├────────────────────────────────────────────────────────────────────────┤
│ Layer 4: Previous Stage Outputs (Cumulative & Truncated) │
│ └── adr-icm-workspace/stages/{01..prev}/output/*.md │
└────────────────────────────────────────────────────────────────────────┘
- Deterministic Ingestion: Stage
01_ingestextracts filtered architectural commits, dependency histories, file trees, and README snippets via native Git CLI. Zero LLM tokens consumed. - Stage Pipeline: Stages
02_analysis→03_rationale→04_synthesis→05_judge→06_content. - LLM Calls: Exactly 5 LLM calls per repo run (1 cumulative call per stage).
- Review Gates: Built-in human-in-the-loop review gates (
review_gate()) allowing inspection/editing of intermediate markdown artifacts before downstream synthesis.
Verified results on pallets/flask (Ollama llama3.1:8b, temperature=0.1; judge temperature=0.0):
| Dimension | LangGraph Agent (agent/) |
ICM Agent (icm_agent/) |
|---|---|---|
| Framework Dependencies | LangGraph, LangChain, ChromaDB, Pydantic | Zero (Pure Python, httpx, Git CLI) |
| State Management | Centralized in-memory TypedDict |
Layered filesystem (stages/*/output/*.md) |
| Total LLM Calls / Repo | 14–22 calls (Per-item prompt slicing) | Exactly 5 calls (1 call per stage) |
Total Prompt Tokens (flask) |
34,457 tokens | 24,077 tokens (-30%) |
| Overall Quality Score | 8.11 / 10.0 | 7.44 / 10.0 |
| — Context Specificity | 4.2 / 5.0 | 5.0 / 5.0 |
| — Decision Clarity | 4.4 / 5.0 | 4.0 / 5.0 |
| — Rationale Depth | 4.0 / 5.0 | 2.6 / 5.0 |
| — Alternatives Quality | 3.8 / 5.0 | 2.4 / 5.0 |
| — Consequences Completeness | 3.9 / 5.0 | 4.6 / 5.0 |
| Structural Compliance | 0.68 / 1.0 | 0.73 / 1.0 |
| Hallucination Rate | 0.0% (Grounded in Git log) | 0.0% (Grounded in Git log) |
| ChromaDB Storage | 3 collections (layer0, layer1, layer2) |
0 collections (Markdown files only) |
| Debug Steps to Root Cause | 6 steps (Trace state dict, DB, logs) | 3 steps (Inspect stage markdown folder) |
Latency (flask, 8B local) |
135.7s (~2.2 mins) | 1,296.5s (~21 mins) |
📚 Detailed Analysis: For full architectural deep dives, state growth profiles, and implementation gap analyses, see ARCHITECTURE_DOCS.md.
.
├── ARCHITECTURE_DOCS.md # In-depth architectural & empirical report
├── paper.pdf / paper.md # Reference research paper
├── benchmark_repos.py # Shared test dataset (sparse, medium, large, known_decisions)
├── score.py # Structural compliance validation script
├── eval.py # Absolute LLM-as-a-judge grading rubric
├── comparison_report.md # Empirical head-to-head rerun report
├── .env.example # Global environment configuration template
│
├── agent/ # LangGraph Multi-Agent Implementation
│ ├── main.py # Typer CLI entrypoint & interactive TUI
│ ├── test_all_repos.py # Benchmark runner across repo categories
│ ├── requirements.txt # Dependencies (LangGraph, ChromaDB, etc.)
│ ├── src/
│ │ ├── config.py # Settings (Ollama models, URLs, thresholds)
│ │ ├── agents/ # Ingest, Analysis, Rationale, Synthesis, Judge
│ │ ├── models/ # ADR & State data schemas
│ │ ├── tools/ # Git, embedding, metrics, and LLM utilities
│ │ └── tui/ # Textual UI for live agent progress
│ └── tests/ # Pytest suite
│
└── icm_agent/ # Interpretable Context Methodology (ICM) Agent
├── run_icm.py # Core 6-stage filesystem-driven runner
├── test_icm_repos.py # ICM benchmark runner across repo suites
├── benchmark_results_icm.json # Raw benchmark outputs
├── adr-icm-workspace/ # Layered filesystem workspace
│ ├── CLAUDE.md # Layer 0: Global instructions & philosophy
│ ├── CONTEXT.md # Layer 1: Workspace target repo placeholder
│ ├── _config/shared/ # Layer 3: Shared repo conventions
│ └── stages/ # Stages 01 to 06 (CONTEXT.md, references, output)
└── result/ # Final synthesized ADR.md, evaluation.md, social-posts.md
- Python 3.11+
- Git
- Ollama running locally
# Pull recommended models
ollama pull llama3.1:8b
ollama pull nomic-embed-text# Clone the repository
git clone https://github.com/your-username/adr_agent.git
cd adr_agent
# Create virtual environment
python -m venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
# Install dependencies for both runners
pip install -r agent/requirements.txt httpx tiktoken typer rich textualCopy the example environment file:
cp .env.example .envBoth agents support local models via Ollama as well as custom endpoints.
Edit .env (or pass environment variables):
# Main synthesis and rationale model
OLLAMA_MODEL_MAIN=llama3.1:8b
# Evaluation & Judge model
OLLAMA_MODEL_JUDGE=llama3.1:8b
# Ollama endpoint & Embeddings
OLLAMA_BASE_URL=http://localhost:11434
EMBED_MODEL=nomic-embed-text
NUM_GPU=-1Alternative models to try: qwen2.5:7b, mistral:7b, deepseek-r1:8b.
You can pass the model dynamically via the --model CLI argument or set the OLLAMA_MODEL environment variable:
# 1. Via CLI argument (recommended)
python run_icm.py https://github.com/pallets/flask --model llama3.1:8b
# 2. Faster lightweight models for low latency
python run_icm.py https://github.com/pallets/flask --model qwen2.5-coder:0.5b
python run_icm.py https://github.com/pallets/flask --model llama3.2:1b
# 3. Via environment variable
export OLLAMA_MODEL=llama3.1:8b
python run_icm.py https://github.com/pallets/flaskcd agent
python main.py generate https://github.com/pallets/flaskcd agent
python main.py generate /path/to/local/git/repofrom agent.src.evaluation.benchmark import run_pipeline
results = run_pipeline("https://github.com/pallets/flask")
print(f"Generated {len(results.get('adrs', []))} ADRs")Output files are stored in agent/docs/adr/ADR-001.md, ADR-002.md, etc.
cd icm_agent
python run_icm.py https://github.com/pallets/flaskAt each stage boundary, the agent pauses, allowing you to review and adjust markdown files in adr-icm-workspace/stages/{stage_num}/output/ before pressing ENTER to proceed.
cd icm_agent
python run_icm.py https://github.com/pallets/flask --skip-review- Final ADRs:
icm_agent/result/ADR.md - Quality Evaluation:
icm_agent/result/evaluation.md - Execution Metrics & Token Counts:
icm_agent/result/benchmark.json - Social Summary Posts:
icm_agent/result/social-posts.md - Stage-by-Stage Intermediates:
icm_agent/adr-icm-workspace/stages/*/output/
The repository provides a standardized 20-repository benchmark suite partitioned into 4 categories in benchmark_repos.py:
sparse: Repositories with small commit counts (~30–100 commits)medium: Moderate-sized frameworks (fastapi,httpx,pydantic,click,starlette)large: Mature codebases (flask,django,sqlalchemy,aiohttp,tornado)known_decisions: Targeted repos with known major architectural pivots (uvicorn,celery, etc.)
cd agent
# Run a specific category (e.g. sparse) with 1 iteration
python test_all_repos.py sparse --runs 1
# Run medium category with 3 iterations for statistical stability
python test_all_repos.py medium --runs 3
# Run all 20 repositories
python test_all_repos.py --runs 1Results are saved to agent/benchmark_results_lg.json.
cd icm_agent
# Run a specific category
python test_icm_repos.py sparse
# Run all categories
python test_icm_repos.pyResults and markdown tables are saved to icm_agent/benchmark_results_icm.json and icm_agent/benchmark_summary.md.
The benchmark suite assesses three primary dimensions:
-
LLM-as-a-Judge Quality (0–10 scale): Scored across 5 criteria (1–5 scale each) via
agent/src/tools/metrics_tools.py:context_specificity: Explains the underlying problem and technical constraints.decision_clarity: Concrete, unambiguous architectural choice.rationale_depth: Explains why without simply paraphrasing commit messages.alternatives_quality: Outlines distinct alternatives considered and reasons for rejection.consequences_complete: Captures both positive benefits and realistic tradeoffs/risks.
-
Structural Compliance (
score.py/score_adr_structure): Deterministic regex checks for date formatting, rationale length, distinct alternatives, tradeoff vocabulary, and confidence tagging. -
Hallucination Verification (
check_hallucination): Cross-references factual claims against the raw Git commit history (repo-data.md) to ensure zero invented libraries or phantom features.
- Jake Van Clief & David McDermott (2026): Interpretable Context Methodology (ICM).
- LangGraph: Stateful, multi-actor applications with LLMs.