You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
TGI exposes an OpenAI-compatible API, which means you can use OpenAI's client libraries to interact with TGI's Messages API and Tool functions.
317
317
318
-
However there are some minor differences in the API, for example `tool_choice="auto"` will ALWAYS choose the tool for you. This is different from OpenAI's API where `tool_choice="auto"` will choose a tool if the model thinks it's necessary.
319
-
320
318
```python
321
319
from openai import OpenAI
322
320
@@ -362,3 +360,61 @@ print(called)
362
360
# },
363
361
# }
364
362
```
363
+
364
+
### Tool Choice Configuration
365
+
366
+
When configuring how the model interacts with tools during a chat completion, there are several options for determining if or how a tool should be called. These options are controlled by the `tool_choice` parameter, which specifies the behavior of the model in relation to tool usage. The following modes are supported:
367
+
368
+
1.**`auto`**:
369
+
370
+
- The model decides whether to call a tool or generate a response message based on the user's input.
371
+
- If tools are provided, this is the default mode.
372
+
- Example usage:
373
+
```python
374
+
tool_choice="auto"
375
+
```
376
+
377
+
2. **`none`**:
378
+
379
+
- The model will never call any tools and will only generate a response message.
380
+
- If no tools are provided, this is the default mode.
381
+
- Example usage:
382
+
```python
383
+
tool_choice="none"
384
+
```
385
+
386
+
3. **`required`**:
387
+
388
+
- The model must call one or more tools and will not generate a response message on its own.
389
+
- Example usage:
390
+
```python
391
+
tool_choice="required"
392
+
```
393
+
394
+
4. **Specific Tool Call by Function Name**:
395
+
- You can force the model to call a specific tool either by specifying the tool function directly or by using an object definition.
396
+
- Two ways to do this:
397
+
1. Provide the function name as a string:
398
+
```python
399
+
tool_choice="get_current_weather"
400
+
```
401
+
2. Use the function objectformat:
402
+
```python
403
+
tool_choice={
404
+
"type": "function",
405
+
"function": {
406
+
"name": "get_current_weather"
407
+
}
408
+
}
409
+
```
410
+
411
+
These options allow flexibility when integrating tools with the chat completions endpoint. You can configure the model to either rely on tools automatically or force it to follow a predefined behavior, based on the needs of the task at hand.
412
+
413
+
---
414
+
415
+
|**Tool Choice Option**|**Description**|**When to Use**|
|`auto`| The model decides whether to call a tool or generate a message. This is the default if tools are provided. | Use when you want the model to decide when a tool is necessary. |
418
+
|`none`| The model generates a message without calling any tools. This is the default if no tools are provided. | Use when you do not want the model to call any tools. |
419
+
|`required`| The model must call one or more tools and will not generate a message on its own. | Use when a tool call is mandatory, and you do not want a regular message generated. |
420
+
| Specific Tool Call (`name`orobject) | Force the model to call a specific tool either by specifying its name (`tool_choice="get_current_weather"`) or using an object. | Use when you want to restrict the model to calling a particular tool for the response. |
0 commit comments