Chapter 4: Tool-Calling Toolkit - Structured LLM Interactions

[jrbrowning]

Adding a Toolkit to our workshop

Chapters 1-3 establish a foundational chat system that connects multiple local models through a unified API gateway, supporting both streaming (for interactive use) and completions (for programmatic processing). This chapter focuses on OpenAI-compatible tool calling, also known as function calling.

(missed a chapter? You can find them here. The main branch will always contain the latest code, synchronized with the latest released chapter. You can review earlier chapters by selecting the branch matching the chapter title.)

ToolKit

Most tutorials without a FastAPI gateway demonstrate passing tool definitions directly with each request. However, when building a system with a decoupled frontend, this approach raises an architectural question: where should tool definitions live?

In a decoupled architecture, tool definitions belong in the gateway layer, where they can be accessed by the dispatcher's handler. This design choice keeps tool function calling and validation within the same logical boundary. The alternative would require sending validation results over HTTP connections, adding unnecessary network overhead.

By consolidating tool definitions and validation logic into a centralized toolkit class, all related components remain encapsulated together. This toolkit can then be registered in a central registry, providing a single source of truth for tool management across the system.

While models can receive all available tools and orchestrate selection independently, this approach presents challenges for quantized models due to computational constraints. By explicitly structuring routes with specific tool requirements for each request type, the system supports multiple routes for different tool combinations.

This architecture creates a flexible developer-orchestration pattern, where developers define tool routing logic, as contrasted with a model-orchestration pattern where the model makes all tool selection decisions. The developer maintains control over tool availability per route while the model focuses on execution within those boundaries.

The test case includes an additional Pydantic dataclass strategy called "tool_call" that routes requests to endpoints containing tool definitions. Separating this into a strategy that accompanies the request enables tool selection based on user intention.

  ToolchainTestCase(
        id="tool_completion_1",
        stage_a=ToolchainModelStage(
            system_prompt=(
                "You are a weather assistant who answers location-based weather queries. "
                "When a user asks about the weather in a location, convert the city to coordinates."
                "Call the `get_weather` tool with latitude and longitude, and return the result. "
                "After the tool call, summarize the weather condition using simple, factual language. "
                "Avoid making predictions or interpretations not supported by the data. "
                "Only respond with information directly derived from tool output or coordinates."
            ),
            user_prompt="What's the weather in Antarctica today?",
            model_container="reasoning",
            stream=False,
            prompt_tool_spec=ToolchainPromptSpec(
                strategy="tool_call",  <-- HERE
                max_tokens=1024,
                temperature=0.0,
            ),
            synthesis=False,
        ),
        evaluation=False,
    ),

Test Your Tool Calling

Run your first multi-model comparison with: poe test. This executes the same configuration across all five models updated to run the tool_call strategy in your CLI.

The architecture diagram below illustrates a single-stage tool calling operation. Tool completion (returning the result as JSON) and streaming (each chunk arrives as it becomes available).

Tool Call

Streaming Tool Call

Course Prompts

For this chapter, try using these prompts:

Option 1: Copilot - Explain something in the codebase:
ex: @workspace where is tool calling implemented and how does it work?
ex: @workspace Explain how the tool_registry allows for capturing both the tool definition and validation
ex: @workspace Explain how validate_tool_call works in the toolkit and within the dispatcher
ex: @workspace explain how the temperature affects the results for a tool_call.

Option 2: Local Models - RAG Experiment (You can read more about the idea Course Goal)

(use # in ollama to include file in context)
ex: #(select the file) tool_registry.py explain how ToolRegistry works to organize tool runners?

---

Questions? → Questions board