Stream output does not support function call

[eyesspace]

ChatMessageRequest in need in ChatMessageResponseStream, use the add tool

[pepperoni21]

I don't understand

[eyesspace]

I don't understand

The add_tool method can only be used within the Coordinator. However, the Coordinator does not have a streaming output. Therefore, I hope to use the add_tool method in the ChatMessageRequest.

[eyesspace]

Or you can add streaming output to Coordinator

[leonidas1712]

They're referring to this issue with Ollama itself where including tool choices in the request automatically disables streaming: ollama/ollama#10415

The issue is currently unresolved but they are working to fix it. Once it's fixed, it would be good if this crate is updated to reflect the change!

[pepperoni21]

Oh I see, well then I'll keep this issue opened until they fix the issue on their side

[domWinter]

It got merged upstream ollama. Would be nice if it could be implemented in here :)

[pepperoni21]

i tried implementing it but the issue i'm facing is that with no streaming we check if the first response has tool calls, if it does not we return the first response, but since we need to return the stream we cannot know before the stream is done if there were tool calls

i'll try to explain it clearly:

first response <- call to llm

if first response has tool calls:
make tool calls and add results to history
second response <- call to llm
return second response
else:
return first response

so if the first response is a stream we cannot know if it has tool calls before it's done. So maybe we need to make the first response without streaming and make a "fake stream" to return it if needed?

[RobJellinghaus]

I'm wanting to add tool support to my Ollama project that uses streaming, so I have the same requirement.

Speaking personally, I don't want ollama-rs to "pipeline" the responses. I want to make the first call to the LLM, and get a streaming response back. If the response includes tool calls (e.g. it's a "thinking" response), I'll indicate that to the user the way the search.py demo does: ollama/ollama#10415

In other words, I would like to stream the first response myself, then get back the full response with tool calls, then make the tool calls and extend the context, then make the extended request and stream the second response myself. I don't want ollama-rs to generate the second request for me. For clarity:

• call ollama-rs with first request, get back stream
• stream response, show to user as "thinking"
• once response is complete, collect all tool calls, make all tool calls (optionally showing results to user), add to context
• repeat with second request + extended context (and maybe more repetitions, if more tool calls are needed)

Does this make sense? Would this be a simpler API? I may fork ollama-rs and try adding this just as an experiment....

[theguywhoburns]

Any updates related to this?

[RobJellinghaus]

I went ahead and experimented with using Claude Code to integrate streaming with tool usage.

I don't think this is really ready for a PR back into the main ollama-rs project, but it's at least an interesting reference point: https://github.com/RobJellinghaus/ollama-rs/tree/RobJellinghaus/streaming-tools

For an example of how this is used in my web app LLM prototype, see the chat_with_suppliers_stream_with_tools function in this file: https://github.com/RobJellinghaus/create-rust-app/blob/experiments/11.0.3/procuretoy/backend/services/chat.rs
The basic idea is that there is a loop in which you make a chat request and get a stream of responses, including tool invocation responses. Any tool invocations and completions get streamed to the UI, and if the response shows that any such tools were invoked, then the code adds the tool completion information to the history and loops to make another LLM call. Once there are no tool invocations in a given LLM call, the loop ends.

This seems to work quite well for supporting streaming information about tool invocations. It winds up rendering like this:

🔍 Looking up location coordinates with args: {"location":"Las Vegas, NV"}
✅ geocode_location completed: Coordinates for 'Las Vegas, NV': 36.1674263,-115.1484131 (Las Vegas, Clark County, Nevada, United States)

🔍 Looking up location coordinates with args: {"location":"Madrid"}
✅ geocode_location completed: Coordinates for 'Madrid': 40.4167047,-3.7035825 (Madrid, Comunidad de Madrid, España)
The distance between Madrid and Las Vegas is approximately 9,019 kilometers.

[eyesspace]

I went ahead and experimented with using Claude Code to integrate streaming with tool usage.

I don't think this is really ready for a PR back into the main ollama-rs project, but it's at least an interesting reference point: https://github.com/RobJellinghaus/ollama-rs/tree/RobJellinghaus/streaming-tools

For an example of how this is used in my web app LLM prototype, see the chat_with_suppliers_stream_with_tools function in this file: https://github.com/RobJellinghaus/create-rust-app/blob/experiments/11.0.3/procuretoy/backend/services/chat.rs The basic idea is that there is a loop in which you make a chat request and get a stream of responses, including tool invocation responses. Any tool invocations and completions get streamed to the UI, and if the response shows that any such tools were invoked, then the code adds the tool completion information to the history and loops to make another LLM call. Once there are no tool invocations in a given LLM call, the loop ends.

This seems to work quite well for supporting streaming information about tool invocations. It winds up rendering like this:

🔍 Looking up location coordinates with args: {"location":"Las Vegas, NV"} ✅ geocode_location completed: Coordinates for 'Las Vegas, NV': 36.1674263,-115.1484131 (Las Vegas, Clark County, Nevada, United States)

🔍 Looking up location coordinates with args: {"location":"Madrid"} ✅ geocode_location completed: Coordinates for 'Madrid': 40.4167047,-3.7035825 (Madrid, Comunidad de Madrid, España) The distance between Madrid and Las Vegas is approximately 9,019 kilometers.

Hello, when will the new version be released? thank you

[MariaGolubev]

Hello, it’s unfortunate that the function call doesn’t work with streams. Is there any ongoing work in this direction?