Skip to content

[0.x] Add tool call assertions for agent responses#383

Open
yousefkadah wants to merge 1 commit intolaravel:0.xfrom
yousefkadah:feature/agent-tool-call-assertions
Open

[0.x] Add tool call assertions for agent responses#383
yousefkadah wants to merge 1 commit intolaravel:0.xfrom
yousefkadah:feature/agent-tool-call-assertions

Conversation

@yousefkadah
Copy link
Copy Markdown

Summary

Adds three assertions for verifying which tools an agent invoked while producing a TextResponse, plus matching static helpers on the Promptable trait so they can be called from any agent class.

$response = (new SalesCoach)->prompt('Find me wines from Napa Valley');

Ai::assertAgentCalledTool($response, 'search_wines');
Ai::assertAgentCalledTool($response, 'search_wines', ['region' => 'napa']);
Ai::assertAgentCalledTool($response, 'search_wines', fn (array $args) => str_contains($args['query'], 'napa'));
Ai::assertAgentDidNotCallTool($response, 'delete_database');
Ai::assertAgentCalledNoTools($greetingResponse);

// Or via the static helpers on the agent class
SalesCoach::assertCalledTool($response, 'search_wines');
SalesCoach::assertDidNotCallTool($response, 'delete_database');
SalesCoach::assertCalledNoTools($greetingResponse);

Methods added

On Concerns\InteractsWithFakeAgents (auto-exposed via the Ai facade since AiManager uses the trait):

  • assertAgentCalledTool(TextResponse $response, string $tool, Closure|array|null $arguments = null)
  • assertAgentDidNotCallTool(TextResponse $response, string $tool)
  • assertAgentCalledNoTools(TextResponse $response)

The third argument of assertAgentCalledTool may be:

  • null to match any invocation of the tool
  • an array of arguments that must be a subset of the recorded call's arguments (recursively for nested arrays — useful when the model adds optional fields you don't want to assert on)
  • a Closure that receives the recorded arguments and returns a bool (with the full ToolCall instance as a second argument for advanced inspection)

This mirrors the existing assertAgentWasPrompted API which already accepts both string and Closure callbacks.

Matching static helpers were added to the Promptable trait:

  • Promptable::assertCalledTool(...)
  • Promptable::assertDidNotCallTool(...)
  • Promptable::assertCalledNoTools(...)

Why no gateway / response changes

All assertion data already lives on TextResponse->toolCalls (a Collection<ToolCall>). The new methods are pure consumers of that data, so this PR touches zero gateway, provider, or response classes.

Tests

A new file tests/Feature/AgentToolAssertionsTest.php adds 17 Pest tests / 37 assertions covering:

  • Single and multi-tool responses
  • Failure messages for the named tool not being called
  • Exact-match argument subsets, nested argument subsets, and missing arguments
  • Closure-based argument matching (pass and fail)
  • The static Promptable helpers
  • Empty-response and multi-call cases for assertAgentCalledNoTools
Tests:    17 passed (37 assertions)
Duration: 0.25s

tests/Feature/AgentFakeTest.php (the existing related test file) still passes alongside the new file:

Tests:    37 passed (76 assertions)

Pint is clean on every changed file.

Why this is useful

Agent evaluation is a core pillar of LLM testing (DeepEval, RAGAS both ship tool-call assertions). The Laravel AI SDK already has rich primitives for asserting that an agent was prompted, but no first-class way to assert which tools the agent called or with what arguments — currently you'd have to reach into $response->toolCalls and write the loop yourself in every test.

Adds three assertions for verifying which tools an agent invoked while
producing a TextResponse, plus matching static helpers on the Promptable
trait so they can be called from any agent class.

Methods on InteractsWithFakeAgents (exposed via the Ai facade):

- assertAgentCalledTool(TextResponse $response, string $tool, Closure|array|null $arguments = null)
- assertAgentDidNotCallTool(TextResponse $response, string $tool)
- assertAgentCalledNoTools(TextResponse $response)

The third argument of assertAgentCalledTool may be:
- null to match any invocation of the tool
- an array of arguments that must be a subset of the recorded call's
  arguments (with recursive matching for nested arrays)
- a Closure that receives the recorded arguments and returns a bool

This mirrors the existing assertAgentWasPrompted API which already
accepts both string and Closure callbacks. All assertion data already
lives on TextResponse->toolCalls so no gateway, provider or response
class changes are required.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants