Skip to content

Commit e648997

Browse files
dd-octo-sts[bot]ncybulchristophe-papazian
authored
fix(pydantic_ai): [MLOB-3476] fix tool call tracing in pydantic-ai >= 0.4.4 [backport 3.11] (#14236)
Backport efcbf0d from #14171 to 3.11. We received this issue where upgrading to `pydantic-ai-slim >= 0.4.4` breaks tool call tracing for Pydantic AI since the traced tool call method (`Tool.run`) was removed in this version due to the new [Toolsets](https://ai.pydantic.dev/toolsets/) interface being introduced. This PR fixes that issue by conditionally tracing either the `run` method of the Tool class for versions < 0.4.4 or tracing the `ToolManager.handle_call` in newer versions of Pydantic AI. Also, some minor changes were made to support annotating tools defined via toolsets for the agent manifest. ## Manual Testing I first ensured that conditionally patching `Tool.run` resolved the original error message where we were crashing during startup while trying to enable the Pydantic AI integration. I did this by making sure I had version `pydantic-ai>=0.4.4` and then running the following script with `ddtrace-run python startup.py`. ``` import pydantic_ai from ddtrace.llmobs import LLMObs LLMObs.enable() ``` This resulted in no error message. I then wanted to check that we indeed trace tool calls made either via regular tools or via toolsets. I first used the normal `tools` argument to register my tool and invoke it. ``` from pydantic_ai import Agent def calculate_square(x: int) -> int: return x * x agent = Agent( 'openai:gpt-4o', system_prompt='Use one of the tools provided to calculate the mathematical operation.', tools=[calculate_square] ) result = agent.run_sync('What is the square of 5?') print(result.output) ``` This resulted in this [trace](https://app.datadoghq.com/llm/traces?query=%40ml_app%3Anicole-test%20%40event_type%3Aspan%20%40parent_id%3Aundefined&agg_m=count&agg_m_source=base&agg_t=count&fromUser=true&llmPanels=%5B%7B%22t%22%3A%22sampleDetailPanel%22%2C%22rEID%22%3A%22AwAAAZhcffsFoQctBAAAABhBWmhjZmZzRkFBRDhQWGUzdTMwbEFBQUEAAAAkMTE5ODVjN2UtMGRiMC00MGQ1LWE3ZjAtZDk2ZmIwZjE5NTQxAAAAWQ%22%7D%5D&spanId=14197137118701788221&start=1753897528617&end=1753898428617&paused=false) where you can see the `calculate_square_tool` being invoked and also see the tool definition in the agent manifest. I then tried to use the toolsets argument by modifying the script slightly. ``` from pydantic_ai import Agent from pydantic_ai.toolsets import FunctionToolset def calculate_square(x: int) -> int: return x * x toolset = FunctionToolset(tools=[calculate_square]) agent = Agent( 'openai:gpt-4o', system_prompt='Use one of the tools provided to calculate the mathematical operation.', toolsets=[toolset] ) result = agent.run_sync('What is the square of 5?') print(result.output) ``` This resulted in this [trace](https://app.datadoghq.com/llm/traces?query=%40ml_app%3Anicole-test%20%40event_type%3Aspan%20%40parent_id%3Aundefined&agg_m=count&agg_m_source=base&agg_t=count&fromUser=true&llmPanels=%5B%7B%22t%22%3A%22sampleDetailPanel%22%2C%22rEID%22%3A%22AwAAAZhcga3TRDnHrwAAABhBWmhjZ2EzVEFBQ1dfOU1EX09rTkFBQUEAAAAkZjE5ODVjODEtY2M1Yi00ZjNiLTg5NzYtZDJiNTlmMDE5ODBmAAAAqw%22%7D%5D&spanId=4209762990183049031&start=1753897790055&end=1753898690055&paused=false) where again you see the tool call being traced along with the tool definition in the agent manifest. ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) --------- Signed-off-by: Nicole Cybul <[email protected]> Co-authored-by: ncybul <[email protected]> Co-authored-by: Christophe Papazian <[email protected]>
1 parent 97b001d commit e648997

File tree

16 files changed

+780
-34
lines changed

16 files changed

+780
-34
lines changed

‎.riot/requirements/1205fcd.txt‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
#
2+
# This file is autogenerated by pip-compile with Python 3.11
3+
# by the following command:
4+
#
5+
# pip-compile --allow-unsafe --no-annotate .riot/requirements/1205fcd.in
6+
#
7+
ag-ui-protocol==0.1.8
8+
aiohappyeyeballs==2.6.1
9+
aiohttp==3.12.15
10+
aiosignal==1.4.0
11+
annotated-types==0.7.0
12+
anthropic==0.60.0
13+
anyio==4.9.0
14+
argcomplete==3.6.2
15+
attrs==25.3.0
16+
boto3==1.39.16
17+
botocore==1.39.16
18+
cachetools==5.5.2
19+
certifi==2025.7.14
20+
charset-normalizer==3.4.2
21+
click==8.2.1
22+
cohere==5.16.1
23+
colorama==0.4.6
24+
coverage[toml]==7.10.1
25+
distro==1.9.0
26+
eval-type-backport==0.2.2
27+
fastavro==1.11.1
28+
filelock==3.18.0
29+
frozenlist==1.7.0
30+
fsspec==2025.7.0
31+
google-auth==2.40.3
32+
google-genai==1.27.0
33+
griffe==1.9.0
34+
groq==0.30.0
35+
h11==0.16.0
36+
hf-xet==1.1.5
37+
httpcore==1.0.9
38+
httpx==0.28.1
39+
httpx-sse==0.4.0
40+
huggingface-hub[inference]==0.34.3
41+
hypothesis==6.45.0
42+
idna==3.10
43+
importlib-metadata==8.7.0
44+
iniconfig==2.1.0
45+
jiter==0.10.0
46+
jmespath==1.0.1
47+
jsonschema==4.25.0
48+
jsonschema-specifications==2025.4.1
49+
logfire-api==4.0.0
50+
markdown-it-py==3.0.0
51+
mcp==1.12.2
52+
mdurl==0.1.2
53+
mistralai==1.9.3
54+
mock==5.2.0
55+
multidict==6.6.3
56+
openai==1.98.0
57+
opentelemetry-api==1.36.0
58+
opentracing==2.4.0
59+
packaging==25.0
60+
pluggy==1.6.0
61+
prompt-toolkit==3.0.51
62+
propcache==0.3.2
63+
pyasn1==0.6.1
64+
pyasn1-modules==0.4.2
65+
pydantic==2.11.7
66+
pydantic-ai==0.4.4
67+
pydantic-ai-slim[ag-ui,anthropic,bedrock,cli,cohere,evals,google,groq,huggingface,mcp,mistral,openai,vertexai]==0.4.4
68+
pydantic-core==2.33.2
69+
pydantic-evals==0.4.4
70+
pydantic-graph==0.4.4
71+
pydantic-settings==2.10.1
72+
pygments==2.19.2
73+
pytest==8.4.1
74+
pytest-asyncio==1.1.0
75+
pytest-cov==6.2.1
76+
pytest-mock==3.14.1
77+
python-dateutil==2.9.0.post0
78+
python-dotenv==1.1.1
79+
python-multipart==0.0.20
80+
pyyaml==6.0.2
81+
referencing==0.36.2
82+
requests==2.32.4
83+
rich==14.1.0
84+
rpds-py==0.26.0
85+
rsa==4.9.1
86+
s3transfer==0.13.1
87+
six==1.17.0
88+
sniffio==1.3.1
89+
sortedcontainers==2.4.0
90+
sse-starlette==3.0.2
91+
starlette==0.47.2
92+
tenacity==8.5.0
93+
tokenizers==0.21.4
94+
tqdm==4.67.1
95+
types-requests==2.32.4.20250611
96+
typing-extensions==4.14.1
97+
typing-inspection==0.4.1
98+
urllib3==2.5.0
99+
uvicorn==0.35.0
100+
vcrpy==7.0.0
101+
wcwidth==0.2.13
102+
websockets==15.0.1
103+
wrapt==1.17.2
104+
yarl==1.20.1
105+
zipp==3.23.0

‎.riot/requirements/17ed285.txt‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
#
2+
# This file is autogenerated by pip-compile with Python 3.12
3+
# by the following command:
4+
#
5+
# pip-compile --allow-unsafe --no-annotate .riot/requirements/17ed285.in
6+
#
7+
ag-ui-protocol==0.1.8
8+
aiohappyeyeballs==2.6.1
9+
aiohttp==3.12.15
10+
aiosignal==1.4.0
11+
annotated-types==0.7.0
12+
anthropic==0.60.0
13+
anyio==4.9.0
14+
argcomplete==3.6.2
15+
attrs==25.3.0
16+
boto3==1.39.16
17+
botocore==1.39.16
18+
cachetools==5.5.2
19+
certifi==2025.7.14
20+
charset-normalizer==3.4.2
21+
click==8.2.1
22+
cohere==5.16.1
23+
colorama==0.4.6
24+
coverage[toml]==7.10.1
25+
distro==1.9.0
26+
eval-type-backport==0.2.2
27+
fastavro==1.11.1
28+
filelock==3.18.0
29+
frozenlist==1.7.0
30+
fsspec==2025.7.0
31+
google-auth==2.40.3
32+
google-genai==1.27.0
33+
griffe==1.9.0
34+
groq==0.30.0
35+
h11==0.16.0
36+
hf-xet==1.1.5
37+
httpcore==1.0.9
38+
httpx==0.28.1
39+
httpx-sse==0.4.0
40+
huggingface-hub[inference]==0.34.3
41+
hypothesis==6.45.0
42+
idna==3.10
43+
importlib-metadata==8.7.0
44+
iniconfig==2.1.0
45+
jiter==0.10.0
46+
jmespath==1.0.1
47+
jsonschema==4.25.0
48+
jsonschema-specifications==2025.4.1
49+
logfire-api==4.0.0
50+
markdown-it-py==3.0.0
51+
mcp==1.12.2
52+
mdurl==0.1.2
53+
mistralai==1.9.3
54+
mock==5.2.0
55+
multidict==6.6.3
56+
openai==1.98.0
57+
opentelemetry-api==1.36.0
58+
opentracing==2.4.0
59+
packaging==25.0
60+
pluggy==1.6.0
61+
prompt-toolkit==3.0.51
62+
propcache==0.3.2
63+
pyasn1==0.6.1
64+
pyasn1-modules==0.4.2
65+
pydantic==2.11.7
66+
pydantic-ai==0.4.4
67+
pydantic-ai-slim[ag-ui,anthropic,bedrock,cli,cohere,evals,google,groq,huggingface,mcp,mistral,openai,vertexai]==0.4.4
68+
pydantic-core==2.33.2
69+
pydantic-evals==0.4.4
70+
pydantic-graph==0.4.4
71+
pydantic-settings==2.10.1
72+
pygments==2.19.2
73+
pytest==8.4.1
74+
pytest-asyncio==1.1.0
75+
pytest-cov==6.2.1
76+
pytest-mock==3.14.1
77+
python-dateutil==2.9.0.post0
78+
python-dotenv==1.1.1
79+
python-multipart==0.0.20
80+
pyyaml==6.0.2
81+
referencing==0.36.2
82+
requests==2.32.4
83+
rich==14.1.0
84+
rpds-py==0.26.0
85+
rsa==4.9.1
86+
s3transfer==0.13.1
87+
six==1.17.0
88+
sniffio==1.3.1
89+
sortedcontainers==2.4.0
90+
sse-starlette==3.0.2
91+
starlette==0.47.2
92+
tenacity==8.5.0
93+
tokenizers==0.21.4
94+
tqdm==4.67.1
95+
types-requests==2.32.4.20250611
96+
typing-extensions==4.14.1
97+
typing-inspection==0.4.1
98+
urllib3==2.5.0
99+
uvicorn==0.35.0
100+
vcrpy==7.0.0
101+
wcwidth==0.2.13
102+
websockets==15.0.1
103+
wrapt==1.17.2
104+
yarl==1.20.1
105+
zipp==3.23.0

‎.riot/requirements/1894006.txt‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
#
2+
# This file is autogenerated by pip-compile with Python 3.10
3+
# by the following command:
4+
#
5+
# pip-compile --allow-unsafe --no-annotate .riot/requirements/1894006.in
6+
#
7+
ag-ui-protocol==0.1.8
8+
aiohappyeyeballs==2.6.1
9+
aiohttp==3.12.15
10+
aiosignal==1.4.0
11+
annotated-types==0.7.0
12+
anthropic==0.60.0
13+
anyio==4.9.0
14+
argcomplete==3.6.2
15+
async-timeout==5.0.1
16+
attrs==25.3.0
17+
backports-asyncio-runner==1.2.0
18+
boto3==1.39.16
19+
botocore==1.39.16
20+
cachetools==5.5.2
21+
certifi==2025.7.14
22+
charset-normalizer==3.4.2
23+
click==8.2.1
24+
cohere==5.16.1
25+
colorama==0.4.6
26+
coverage[toml]==7.10.1
27+
distro==1.9.0
28+
eval-type-backport==0.2.2
29+
exceptiongroup==1.3.0
30+
fastavro==1.11.1
31+
filelock==3.18.0
32+
frozenlist==1.7.0
33+
fsspec==2025.7.0
34+
google-auth==2.40.3
35+
google-genai==1.27.0
36+
griffe==1.9.0
37+
groq==0.30.0
38+
h11==0.16.0
39+
hf-xet==1.1.5
40+
httpcore==1.0.9
41+
httpx==0.28.1
42+
httpx-sse==0.4.0
43+
huggingface-hub[inference]==0.34.3
44+
hypothesis==6.45.0
45+
idna==3.10
46+
importlib-metadata==8.7.0
47+
iniconfig==2.1.0
48+
jiter==0.10.0
49+
jmespath==1.0.1
50+
jsonschema==4.25.0
51+
jsonschema-specifications==2025.4.1
52+
logfire-api==4.0.0
53+
markdown-it-py==3.0.0
54+
mcp==1.12.2
55+
mdurl==0.1.2
56+
mistralai==1.9.3
57+
mock==5.2.0
58+
multidict==6.6.3
59+
openai==1.98.0
60+
opentelemetry-api==1.36.0
61+
opentracing==2.4.0
62+
packaging==25.0
63+
pluggy==1.6.0
64+
prompt-toolkit==3.0.51
65+
propcache==0.3.2
66+
pyasn1==0.6.1
67+
pyasn1-modules==0.4.2
68+
pydantic==2.11.7
69+
pydantic-ai==0.4.4
70+
pydantic-ai-slim[ag-ui,anthropic,bedrock,cli,cohere,evals,google,groq,huggingface,mcp,mistral,openai,vertexai]==0.4.4
71+
pydantic-core==2.33.2
72+
pydantic-evals==0.4.4
73+
pydantic-graph==0.4.4
74+
pydantic-settings==2.10.1
75+
pygments==2.19.2
76+
pytest==8.4.1
77+
pytest-asyncio==1.1.0
78+
pytest-cov==6.2.1
79+
pytest-mock==3.14.1
80+
python-dateutil==2.9.0.post0
81+
python-dotenv==1.1.1
82+
python-multipart==0.0.20
83+
pyyaml==6.0.2
84+
referencing==0.36.2
85+
requests==2.32.4
86+
rich==14.1.0
87+
rpds-py==0.26.0
88+
rsa==4.9.1
89+
s3transfer==0.13.1
90+
six==1.17.0
91+
sniffio==1.3.1
92+
sortedcontainers==2.4.0
93+
sse-starlette==3.0.2
94+
starlette==0.47.2
95+
tenacity==8.5.0
96+
tokenizers==0.21.4
97+
tomli==2.2.1
98+
tqdm==4.67.1
99+
types-requests==2.32.4.20250611
100+
typing-extensions==4.14.1
101+
typing-inspection==0.4.1
102+
urllib3==2.5.0
103+
uvicorn==0.35.0
104+
vcrpy==7.0.0
105+
wcwidth==0.2.13
106+
websockets==15.0.1
107+
wrapt==1.17.2
108+
yarl==1.20.1
109+
zipp==3.23.0

0 commit comments

Comments
 (0)