Agent Comparisons

Benchmark harness for evaluating how reliably different agent architectures follow a required sequential execution order as complexity (number of steps N) increases.

Current focus:

• Agent with tools: single orchestrator agent delegating to N sub-agent tools
• Agent graph: N-node pydantic-graph where an LLM routes between nodes
• Deep agents: planned

Prerequisites

• Python 3.12+
• An OpenAI API key

Local Setup

From the repo root:

python -m venv .venv
pip install -e .

Then activate the virtual environment in your shell.

Create .env from .env.example and set environment variables:

• OPENAI_API_KEY

Optional (for Logfire telemetry):

• LOGFIRE_KEY
• LOGFIRE_ENVIRONMENT