Skip to content

Commit 6f94ede

Browse files
maxzhangddYun-Kim
maxzhangdd
and
Yun-Kim
authored
feat(openai): added support for tool results, definitions (#14204)
This PR adds improved tracking of tool results and definitions to the LLMObs OpenAI integration ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) --------- Co-authored-by: Yun Kim <[email protected]>
1 parent 2a4773c commit 6f94ede

File tree

7 files changed

+689
-143
lines changed

7 files changed

+689
-143
lines changed

‎ddtrace/llmobs/_integrations/utils.py‎

Lines changed: 192 additions & 86 deletions
Large diffs are not rendered by default.

‎releasenotes/notes/openai_tool_usage-ef5eaf531317ac3b.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
features:
2+
- |
3+
LLM Observability: Adds support for collecting tool definitions, tool calls and tool results in the OpenAI integration.

‎tests/contrib/litellm/test_litellm_llmobs.py‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,13 @@ def test_completion_with_tools(self, litellm, request_vcr, llmobs_events, mock_t
121121
"stream_options": {"include_usage": True},
122122
"tool_choice": "auto",
123123
},
124+
tool_definitions=[
125+
{
126+
"name": tools[0]["function"]["name"],
127+
"description": tools[0]["function"]["description"],
128+
"schema": tools[0]["function"]["parameters"],
129+
}
130+
],
124131
token_metrics=token_metrics,
125132
tags={"ml_app": "<ml-app-name>", "service": "tests.contrib.litellm"},
126133
)

‎tests/contrib/openai/cassettes/v1/chat_completion_custom_tool_call.yaml‎

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
interactions:
2+
- request:
3+
body: '{"messages":[{"role":"user","content":"Use the math_exp tool to add four
4+
plus four."}],"model":"gpt-5","tools":[{"type":"custom","custom":{"name":"math_exp","description":"Creates
5+
valid mathematical expressions","format":{"type":"grammar","grammar":{"syntax":"lark","definition":"\nstart:
6+
expr\nexpr: term (SP ADD SP term)* -> add\n| term\nterm: factor (SP MUL SP factor)*
7+
-> mul\n| factor\nfactor: INT\nSP: \" \"\nADD: \"+\"\nMUL: \"*\"\n%import common.INT\n"}}}}]}'
8+
headers:
9+
accept:
10+
- application/json
11+
accept-encoding:
12+
- gzip, deflate
13+
connection:
14+
- keep-alive
15+
content-length:
16+
- '466'
17+
content-type:
18+
- application/json
19+
host:
20+
- api.openai.com
21+
user-agent:
22+
- OpenAI/Python 1.91.0
23+
x-stainless-arch:
24+
- arm64
25+
x-stainless-async:
26+
- 'false'
27+
x-stainless-lang:
28+
- python
29+
x-stainless-os:
30+
- MacOS
31+
x-stainless-package-version:
32+
- 1.91.0
33+
x-stainless-read-timeout:
34+
- '600'
35+
x-stainless-retry-count:
36+
- '0'
37+
x-stainless-runtime:
38+
- CPython
39+
x-stainless-runtime-version:
40+
- 3.12.11
41+
method: POST
42+
uri: https://api.openai.com/v1/chat/completions
43+
response:
44+
body:
45+
string: !!binary |
46+
H4sIAAAAAAAAA3RT0W6bMBR95yssvzZMQCEpvFXtJi1Ss27SXjZVyDMXcGtszzZTuij/XtkkQNKU
47+
B2T5+Jxzfe71LkAIswoXCNOWWNopHt4tX27N5rHJ4+u/9cO3+7z9sllG67b6teH/8cIx5J9noPbI
48+
+kRlpzhYJsUAUw3EglONV1m2Sq7T/MYDnayAO1qjbJiFSZRkYXQTRqsDr5WMgsEF+h0ghNDO/12F
49+
ooItLlC0OO50YAxpABfjIYSwltztYGIMM5YIixcTSKWwIFzRoud8BlgpeUkJ55Px8O1m6ykmwnn5
50+
XTXrH1+zu/yePn/ePmx+LrleP9J+5jdIvypfEO2Nld05etgtzowQwoJ0ntcR25awVWdMn4jqfQNS
51+
dIVSfALvg0vrp9mVNdS9Ifx9FkQIaYnrpA/j6YDsx9xrJphpSw3ESOH8Z+kFR0NvhfuTBmGlZads
52+
aeULePEkjQdVPI3PDI3TA2qlJXwC0ixbXBAsK7CE+RaOU0MJbaGaqNP0kL5icgYEs0u+L+eS9hAA
53+
E82kEufJhwYTQCkoC1WpNFSMnl56OqbBPbCPjo1B+5KxAf2PUSgtA+1aUkFNej4MPzavxkJX1kw0
54+
oJVm4wsI9sEbAAAA//8DAGftrEX+AwAA
55+
headers:
56+
CF-RAY:
57+
- 9724cebf8b2dd6f4-IAD
58+
Connection:
59+
- keep-alive
60+
Content-Encoding:
61+
- gzip
62+
Content-Type:
63+
- application/json
64+
Date:
65+
- Wed, 20 Aug 2025 20:58:22 GMT
66+
Server:
67+
- cloudflare
68+
Set-Cookie:
69+
- __cf_bm=K.tOaHC5Lf45ysCtP6USNAPbZgHy78FNC72qcH2X.Fo-1755723502-1.0.1.1-6gJa9NKR2lxcxp5WDhmS4X6xrB2zYsLnov89sMAupSGbYt8Hdbc_awPaQqbnQOdNmztzKstiMU8L_wFbmzt9f0Q7Qg_sho4.ecYi5i7_EiA;
70+
path=/; expires=Wed, 20-Aug-25 21:28:22 GMT; domain=.api.openai.com; HttpOnly;
71+
Secure; SameSite=None
72+
- _cfuvid=CkJlTNRJNqcMv3f9_YvhdnJLTb4jyPNrZypv2Xi49h4-1755723502477-0.0.1.1-604800000;
73+
path=/; domain=.api.openai.com; HttpOnly; Secure; SameSite=None
74+
Strict-Transport-Security:
75+
- max-age=31536000; includeSubDomains; preload
76+
Transfer-Encoding:
77+
- chunked
78+
X-Content-Type-Options:
79+
- nosniff
80+
access-control-expose-headers:
81+
- X-Request-ID
82+
alt-svc:
83+
- h3=":443"; ma=86400
84+
cf-cache-status:
85+
- DYNAMIC
86+
openai-organization:
87+
- datadog-staging
88+
openai-processing-ms:
89+
- '8067'
90+
openai-project:
91+
- proj_gt6TQZPRbZfoY2J9AQlEJMpd
92+
openai-version:
93+
- '2020-10-01'
94+
x-envoy-upstream-service-time:
95+
- '8091'
96+
x-ratelimit-limit-requests:
97+
- '15000'
98+
x-ratelimit-limit-tokens:
99+
- '40000000'
100+
x-ratelimit-remaining-requests:
101+
- '14999'
102+
x-ratelimit-remaining-tokens:
103+
- '39999987'
104+
x-ratelimit-reset-requests:
105+
- 4ms
106+
x-ratelimit-reset-tokens:
107+
- 0s
108+
x-request-id:
109+
- req_611954e1d30494f191e4680d1c99b6e2
110+
status:
111+
code: 200
112+
message: OK
113+
version: 1

‎tests/contrib/openai/cassettes/v1/response_custom_tool_call.yaml‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
interactions:
2+
- request:
3+
body: '{"input":"Use the math_exp tool to add four plus four.","model":"gpt-5","tools":[{"type":"custom","name":"math_exp","description":"Creates
4+
valid mathematical expressions","format":{"type":"grammar","syntax":"lark","definition":"\n start:
5+
expr\n expr: term (SP ADD SP term)* -> add\n | term\n term:
6+
factor (SP MUL SP factor)* -> mul\n | factor\n factor: INT\n SP:
7+
\" \"\n ADD: \"+\"\n MUL: \"*\"\n %import common.INT\n "}}]}'
8+
headers:
9+
accept:
10+
- application/json
11+
accept-encoding:
12+
- gzip, deflate
13+
connection:
14+
- keep-alive
15+
content-length:
16+
- '500'
17+
content-type:
18+
- application/json
19+
host:
20+
- api.openai.com
21+
user-agent:
22+
- OpenAI/Python 1.91.0
23+
x-stainless-arch:
24+
- arm64
25+
x-stainless-async:
26+
- 'false'
27+
x-stainless-lang:
28+
- python
29+
x-stainless-os:
30+
- MacOS
31+
x-stainless-package-version:
32+
- 1.91.0
33+
x-stainless-read-timeout:
34+
- '600'
35+
x-stainless-retry-count:
36+
- '0'
37+
x-stainless-runtime:
38+
- CPython
39+
x-stainless-runtime-version:
40+
- 3.12.11
41+
method: POST
42+
uri: https://api.openai.com/v1/responses
43+
response:
44+
body:
45+
string: !!binary |
46+
H4sIAAAAAAAAA4xUXW/bNhR9968gBAzY2riQFcmS/TCgmx86oO0CpMFQLIVwRV05bEhRI68ye1n+
47+
e0HKMqUlw/Zi+J7D+6FzPx4XjEWijrYsMmi7cl1AnuBlUlcJL1abdQUJFOnmMisSDlWNcZ7xTcNh
48+
vVnnwIsqunABdPUVOY1BdGtxwLlBIKxLcNwqz7K8WBdx7jlLQL11PlyrTiJhPThVwO/3Rvetq6oB
49+
adHDaIw20Za1vZQeEO3oWNZIIKSds5ZMz0nodoYrOJS6p66nkvQ9PidJa1lykPNwStcoXbH7jpbZ
50+
MomTbBkXyzg/CeAjRlv2+4Ixxh79b1DWjrrmdRanTtciTxOs+Aqw2iRpyl/U1YegY4eDsmB1K9p9
51+
oGyvFJijS/vFY08XL6XnxMf8fF0khcsPvObpJl1VVbyCCrL/zs97S1oFeSZl/EsnPemelqc63N93
52+
xWf7083u6vPXj++u/9g0v/36159IBxM8RDtIGaXsNUsD3oLyhSiguxIPXTR88oKxL74HHRiQEuW8
53+
g2T6YX46gw9C97YcR3So6tzhzmjVUcmB32F5j8cpF7TfnsSNsGm08WUqrEWvTnVOeuK8F6eeRBYa
54+
pGMpamxJNAJnk2zRPAiOJQ14BD3p6LQk2uBsDQhVhwao9/jqTXxCDxRqa7RREOxJE/276ahED2gq
55+
bQUdJ59yrnpQ8k4LjvO6HGGfj/t8VkLrarTciM4to6N/9mfBsgeQomaun6iABAfJ8NAZtNZv7dn9
56+
2fdMUu0NOMHPj322RrRiTHbbjoQlMLT1KQLorC0jNIp9f33F3u527PrK2z+8YssfGdR1ePy3J4Lt
57+
LNccTtp49w837537gAwBVC+nAQYqIIO9Zb98/BTA66stu43Y7aT4t7udw15PsQ837x32aop9J1Sn
58+
DTGuldLtm1nYqUj22BIcnEASzH10Ip7+97aR7kqp953RlRuE+Ax207E0fcth7EQtLFRyPPO9hT2G
59+
mfVbH07yKttcPCcmh/4xHBh+h3XwjGfj/c9Tn+TZS8xLgc9LP/HO1rPopAlkoNPL9Lw6vZ3vuEKC
60+
GghchqfF0zcAAAD//wMAtxAgsngHAAA=
61+
headers:
62+
CF-RAY:
63+
- 972ad85b7fefc942-IAD
64+
Connection:
65+
- keep-alive
66+
Content-Encoding:
67+
- gzip
68+
Content-Type:
69+
- application/json
70+
Date:
71+
- Thu, 21 Aug 2025 14:33:32 GMT
72+
Server:
73+
- cloudflare
74+
Set-Cookie:
75+
- __cf_bm=G7bjch5x1Todcb0Gu9nayz0bBGGnE2IZl7nhjCTjDuE-1755786812-1.0.1.1-YO8UMu1dHwkNmJ9yA3wWlJ_.h6qvgJb.5Y.Uzz.MjHf1WEXtbg_93NuyaiEeFUje_5Z0sKYk4jvbWMJkx2nwmnKlhgY7N1v8UL4Qzr.p7TI;
76+
path=/; expires=Thu, 21-Aug-25 15:03:32 GMT; domain=.api.openai.com; HttpOnly;
77+
Secure; SameSite=None
78+
- _cfuvid=xW34AolqPK2QSJ_rXrCmy7hO5U8DIRUefqk8mtN48YA-1755786812729-0.0.1.1-604800000;
79+
path=/; domain=.api.openai.com; HttpOnly; Secure; SameSite=None
80+
Transfer-Encoding:
81+
- chunked
82+
X-Content-Type-Options:
83+
- nosniff
84+
alt-svc:
85+
- h3=":443"; ma=86400
86+
cf-cache-status:
87+
- DYNAMIC
88+
openai-organization:
89+
- datadog-staging
90+
openai-processing-ms:
91+
- '9839'
92+
openai-project:
93+
- proj_gt6TQZPRbZfoY2J9AQlEJMpd
94+
openai-version:
95+
- '2020-10-01'
96+
strict-transport-security:
97+
- max-age=31536000; includeSubDomains; preload
98+
x-envoy-upstream-service-time:
99+
- '9853'
100+
x-request-id:
101+
- req_5b10ddfab4e1ea8e01f65920dc68c145
102+
status:
103+
code: 200
104+
message: OK
105+
version: 1

0 commit comments

Comments
 (0)