[0.x] Add tool call assertions for agent responses#383
Open
yousefkadah wants to merge 1 commit intolaravel:0.xfrom
Open
[0.x] Add tool call assertions for agent responses#383yousefkadah wants to merge 1 commit intolaravel:0.xfrom
yousefkadah wants to merge 1 commit intolaravel:0.xfrom
Conversation
Adds three assertions for verifying which tools an agent invoked while producing a TextResponse, plus matching static helpers on the Promptable trait so they can be called from any agent class. Methods on InteractsWithFakeAgents (exposed via the Ai facade): - assertAgentCalledTool(TextResponse $response, string $tool, Closure|array|null $arguments = null) - assertAgentDidNotCallTool(TextResponse $response, string $tool) - assertAgentCalledNoTools(TextResponse $response) The third argument of assertAgentCalledTool may be: - null to match any invocation of the tool - an array of arguments that must be a subset of the recorded call's arguments (with recursive matching for nested arrays) - a Closure that receives the recorded arguments and returns a bool This mirrors the existing assertAgentWasPrompted API which already accepts both string and Closure callbacks. All assertion data already lives on TextResponse->toolCalls so no gateway, provider or response class changes are required.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds three assertions for verifying which tools an agent invoked while producing a
TextResponse, plus matching static helpers on thePromptabletrait so they can be called from any agent class.Methods added
On
Concerns\InteractsWithFakeAgents(auto-exposed via theAifacade sinceAiManageruses the trait):assertAgentCalledTool(TextResponse $response, string $tool, Closure|array|null $arguments = null)assertAgentDidNotCallTool(TextResponse $response, string $tool)assertAgentCalledNoTools(TextResponse $response)The third argument of
assertAgentCalledToolmay be:nullto match any invocation of the toolarrayof arguments that must be a subset of the recorded call's arguments (recursively for nested arrays — useful when the model adds optional fields you don't want to assert on)Closurethat receives the recordedargumentsand returns abool(with the fullToolCallinstance as a second argument for advanced inspection)This mirrors the existing
assertAgentWasPromptedAPI which already accepts bothstringandClosurecallbacks.Matching static helpers were added to the
Promptabletrait:Promptable::assertCalledTool(...)Promptable::assertDidNotCallTool(...)Promptable::assertCalledNoTools(...)Why no gateway / response changes
All assertion data already lives on
TextResponse->toolCalls(aCollection<ToolCall>). The new methods are pure consumers of that data, so this PR touches zero gateway, provider, or response classes.Tests
A new file
tests/Feature/AgentToolAssertionsTest.phpadds 17 Pest tests / 37 assertions covering:PromptablehelpersassertAgentCalledNoToolstests/Feature/AgentFakeTest.php(the existing related test file) still passes alongside the new file:Pint is clean on every changed file.
Why this is useful
Agent evaluation is a core pillar of LLM testing (DeepEval, RAGAS both ship tool-call assertions). The Laravel AI SDK already has rich primitives for asserting that an agent was prompted, but no first-class way to assert which tools the agent called or with what arguments — currently you'd have to reach into
$response->toolCallsand write the loop yourself in every test.