🏗️ AgentArch: A Comprehensive Benchmark to Evaluate Agent Architectures in Enterprise

A systematic evaluation framework for agentic AI systems across diverse architectural configurations and enterprise use cases.

---

🌟 Overview

AgentArch provides empirical insights into how different design dimensions interact within complex multi-agent systems. This benchmark evaluates 18 distinct agentic configurations across state-of-the-art large language models, examining four critical system dimensions:

🎯 Orchestration Strategy Single-agent vs. multi-agent systems	⚙️ Agent Implementation ReAct vs. function calling approaches
🧠 Memory Architecture Complete vs. summarized memory management	🔧 Thinking Tool Integration Mathematical reasoning and information synthesis tools

---

🔍 Key Findings

TL;DR: No one-size-fits-all solution exists for enterprise agentic systems

Finding	Impact	📊
No Universal Architecture	Models demonstrate significant architectural preferences that vary by use case complexity	🎯
Performance Gaps	Even top models achieve only 35.3% success on complex enterprise tasks and 70.8% on simpler workflows	📉
Multi-Agent ReAct Limitations	Consistent underperformance across all models in multi-agent ReAct configurations	⚠️
Reliability Challenges	Pass^K scores peak at only 6.34%, indicating fundamental gaps for production deployment	🚨

---

🚀 Quick Start

Installation

# Clone the repository
git clone https://github.com/ServiceNow/AgentArch.git
cd AgentArch

# Install dependencies
pip install -r requirements.txt

# Set up environment
cp .env.example .env
# 🔑 Replace placeholders with real API keys and endpoints

Run Your First Evaluation

python -m src.run \
  --mode single_agent \
  --usecase requesting_time_off \
  --model claude_sonnet_4 \
  --agent_type function_calling \
  --project test \
  --debug

---

📁 Repository Structure

AgentArch/
├── 📁 configs/
│   ├── 🔧 mocked_data/
│   │   ├── requesting_time_off_mocked_tool_calls.json
│   │   └── triage_cases_mocked_tool_calls.json
│   ├── ⚙️ use_case_configs/
│   │   ├── requesting_time_off.yaml
│   │   ├── triage_cases.yaml
│   ├── ⚙📜 prompts.yaml
├── 📁 src/
│   ├── 🛠️ tools/            
│   ├── 🔧 utils/
│   ├── 🤖 agent.py     
│   ├── 📊 metrics.py    
│   └── ▶️ run.py  # Main execution script
├── 📄 .env.example  
├── 📄 .gitignore
├── 📄 LICENSE
└── 📄 requirements.txt

---

🏢 Enterprise Use Cases

1. 📅 Requesting Time Off (TO) - Simple Workflow

Aspect	Details
🎯 Complexity	Basic multi-step reasoning with clear success criteria
🛠️ Tools	8 custom enterprise tools
🤖 Agents	3 specialized agents
💡 Challenges	Date calculations, leave balance verification, policy compliance

2. 🎫 Customer Request Routing (CR) - Complex Workflow

Aspect	Details
🎯 Complexity	Intelligent classification and escalation decisions
🛠️ Tools	31 custom enterprise tools
🤖 Agents	9 specialized agents
💡 Challenges	Ambiguous request handling, context preservation, routing logic

---

🤖 Evaluated Models

Provider	Models	Status
OpenAI	GPT-4.1, GPT-4o, GPT-4.1-mini, o3-mini	✅
Meta	LLaMA 3.3 70B	✅
Anthropic	Claude Sonnet 4	✅

*Framework includes support for evaluating Gemini family models as well as Qwen models

---

🏗️ Architectural Dimensions

🎭 Orchestration Strategies

1. 🎪 Orchestrator-led, Isolated Agents

Centralized task assignment with mediated communication

2. 🌐 Orchestrator-led, Open Network

Initial task assignment with direct agent-to-agent communication

3. 🤖 Single Agent

Unified agent with access to all tools

🎨 Agent Styles

📞 Function Calling Direct tool selection using native model capabilities

🧠 ReAct Structured reasoning-action framework with explicit thought processes

💾 Memory Management

📚 Complete Memory Full visibility into all previous tool calls and responses

📝 Summarized Memory Condensed information sharing to manage context length

🧮 Thinking Tools

➕ Math Tool Structured mathematical reasoning and calculations

🔍 Synthesis Tool Information organization and analysis capabilities

---

📊 Evaluation Metrics

🎯 Primary Metric: Acceptable Score

Success requires simultaneous achievement of:

• ✅ Correct tool selection
• ✅ Accurate tool arguments (100% accuracy required)
• ✅ Correct final decision

🔄 Reliability Metrics

• Pass@1: Success rate over k=8 trials
• Pass^K: Probability of all k trials succeeding

📈 Behavioral Metrics

• 🚫 Hallucination rates (non-existent tool/agent selection)
• 🔄 Tool repetition rates
• ❌ Missing required tools

---

💡 Key Recommendations

👨‍💼 For Practitioners

Recommendation	Rationale
❌ Avoid Multi-Agent ReAct	Poor performance across all tested models
✅ Use Multi-Agent for Final Decisions	Higher accuracy in decision-making despite tool selection challenges
🎯 Model-Specific Architectures	Test multiple configurations rather than assuming universal optima
🧮 Thinking Tools for Non-Reasoning Models	Significant performance improvements on calculation-heavy tasks for non-reasoning models

🔬 For Researchers

Focus Area	Insight
🔄 Architecture-Use Case Interaction	Models perform optimally under different architectures depending on task complexity
⚖️ Reliability vs Performance	Consider both accuracy and consistency for enterprise deployment
💾 Memory Management Impact	Minimal performance differences between complete and summarized memory

---

📚 Citation

@misc{bogavelli2025agentarchcomprehensivebenchmarkevaluate,
      title={AgentArch: A Comprehensive Benchmark to Evaluate Agent Architectures in Enterprise}, 
      author={Tara Bogavelli and Roshnee Sharma and Hari Subramani},
      year={2025},
      eprint={2509.10769},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2509.10769}, 
}

---

📄 License

AgentArch is licensed under the Apache 2.0 License.

📞 Contact

For questions or collaboration opportunities:

---

⭐ If this project helps your research, please consider giving it a star! ⭐