-
-
Notifications
You must be signed in to change notification settings - Fork 277
Description
Basic checks
- I searched existing issues - this hasn't been reported
- I can reproduce this consistently
- This is a RubyLLM bug, not my application code
What's broken?
Content extraction fails for Mistral because the chunk of data returned has the following structure.
choices/delta/content is an array, which isn't expected
{
"id": "836a02cf63f7429793b2a62e63dfce29",
"object": "chat.completion.chunk",
"created": 1758093959,
"model": "magistral-small-latest",
"choices": [{
"index": 0,
"delta": {
"content": [{
"type": "thinking",
"thinking": [{"type": "text", "text": "Okay"}]
}]
},
"finish_reason": null
}],
"p": "abcdefghijklmnopqr"
}
How to reproduce
- Configure RubyLLM with mistral and use mistral-small-latest
- Ask
calculate 1+1
- exception
TypeError (no implicit conversion of Array into String)
is raised
Note curiously enough, once in while the problematic message isn't included and the question succeeds.
Expected behavior
Process the streaming response without an exception.
What actually happened
The exception:
TypeError (no implicit conversion of Array into String):
/gems/ruby_llm-1.8.0/lib/ruby_llm/stream_accumulator.rb:23:in 'RubyLLM::StreamAccumulator#add'
/gems/ruby_llm-1.8.0/lib/ruby_llm/streaming.rb:20:in 'block (2 levels) in RubyLLM::Streaming#stream_response'
/gems/ruby_llm-1.8.0/lib/ruby_llm/streaming.rb:41:in 'block in RubyLLM::Streaming#handle_stream'
/gems/ruby_llm-1.8.0/lib/ruby_llm/streaming.rb:129:in 'block in RubyLLM::Streaming#handle_sse'
/gems/event_stream_parser-1.0.0/lib/event_stream_parser.rb:217:in 'EventStreamParser::Parser#dispatch_event'
/gems/event_stream_parser-1.0.0/lib/event_stream_parser.rb:73:in 'EventStreamParser::Parser#process_line'
/gems/event_stream_parser-1.0.0/lib/event_stream_parser.rb:46:in 'EventStreamParser::Parser#feed'
/gems/ruby_llm-1.8.0/lib/ruby_llm/streaming.rb:124:in 'RubyLLM::Streaming#handle_sse'
/gems/ruby_llm-1.8.0/lib/ruby_llm/streaming.rb:74:in 'RubyLLM::Streaming#process_stream_chunk'
/gems/ruby_llm-1.8.0/lib/ruby_llm/streaming.rb:87:in 'block in RubyLLM::Streaming#stream_processor'
/gems/faraday-2.13.4/lib/faraday/options/env.rb:176:in 'block in Faraday::Env#stream_response'
/gems/net-protocol-0.2.2/lib/net/protocol.rb:535:in 'Net::ReadAdapter#call_block'
/gems/net-protocol-0.2.2/lib/net/protocol.rb:526:in 'Net::ReadAdapter#<<'
/gems/net-protocol-0.2.2/lib/net/protocol.rb:168:in 'Net::BufferedIO#read'
/gems/net-http-0.6.0/lib/net/http/response.rb:631:in 'Net::HTTPResponse#read_chunked'
/gems/net-http-0.6.0/lib/net/http/response.rb:595:in 'block in Net::HTTPResponse#read_body_0'
/gems/net-http-0.6.0/lib/net/http/response.rb:588:in 'Net::HTTPResponse#inflater'
/gems/net-http-0.6.0/lib/net/http/response.rb:593:in 'Net::HTTPResponse#read_body_0'
/gems/net-http-0.6.0/lib/net/http/response.rb:363:in 'Net::HTTPResponse#read_body'
/gems/faraday-net_http-3.4.1/lib/faraday/adapter/net_http.rb:116:in 'block (2 levels) in Faraday::Adapter::NetHttp#request_with_wrapped_block'
/gems/net-http-0.6.0/lib/net/http.rb:2433:in 'block in Net::HTTP#transport_request'
/gems/net-http-0.6.0/lib/net/http/response.rb:320:in 'Net::HTTPResponse#reading_body'
/gems/net-http-0.6.0/lib/net/http.rb:2430:in 'Net::HTTP#transport_request'
/gems/net-http-0.6.0/lib/net/http.rb:2384:in 'Net::HTTP#request'
/gems/rack-mini-profiler-4.0.1/lib/patches/net_patches.rb:19:in 'block in Net::HTTP#request_with_mini_profiler'
/gems/rack-mini-profiler-4.0.1/lib/mini_profiler/profiling_methods.rb:51:in 'Rack::MiniProfiler::ProfilingMethods#step'
/gems/rack-mini-profiler-4.0.1/lib/patches/net_patches.rb:18:in 'Net::HTTP#request_with_mini_profiler'
/gems/airbrake-13.0.5/lib/airbrake/rails/net_http.rb:11:in 'block in Airbrake::Rails::NetHttp#request'
/gems/airbrake-13.0.5/lib/airbrake/rack.rb:21:in 'Airbrake::Rack.capture_timing'
/gems/airbrake-13.0.5/lib/airbrake/rails/net_http.rb:10:in 'Airbrake::Rails::NetHttp#request'
/gems/faraday-net_http-3.4.1/lib/faraday/adapter/net_http.rb:113:in 'block in Faraday::Adapter::NetHttp#request_with_wrapped_block'
/gems/net-http-0.6.0/lib/net/http.rb:1632:in 'Net::HTTP#start'
/gems/faraday-net_http-3.4.1/lib/faraday/adapter/net_http.rb:112:in 'Faraday::Adapter::NetHttp#request_with_wrapped_block'
/gems/faraday-net_http-3.4.1/lib/faraday/adapter/net_http.rb:98:in 'block in Faraday::Adapter::NetHttp#perform_request'
/gems/faraday-2.13.4/lib/faraday/options/env.rb:172:in 'Faraday::Env#stream_response'
/gems/faraday-net_http-3.4.1/lib/faraday/adapter/net_http.rb:97:in 'Faraday::Adapter::NetHttp#perform_request'
/gems/faraday-net_http-3.4.1/lib/faraday/adapter/net_http.rb:66:in 'block in Faraday::Adapter::NetHttp#call'
/gems/faraday-2.13.4/lib/faraday/adapter.rb:45:in 'Faraday::Adapter#connection'
/gems/faraday-net_http-3.4.1/lib/faraday/adapter/net_http.rb:65:in 'Faraday::Adapter::NetHttp#call'
/gems/ruby_llm-1.8.0/lib/ruby_llm/error.rb:39:in 'RubyLLM::ErrorMiddleware#call'
/gems/faraday-2.13.4/lib/faraday/middleware.rb:56:in 'Faraday::Middleware#call'
/gems/faraday-2.13.4/lib/faraday/middleware.rb:56:in 'Faraday::Middleware#call'
/gems/faraday-retry-2.3.2/lib/faraday/retry/middleware.rb:171:in 'block in Faraday::Retry::Middleware#call'
/gems/faraday-retry-2.3.2/lib/faraday/retry/retryable.rb:7:in 'Faraday::Retryable#with_retries'
/gems/faraday-retry-2.3.2/lib/faraday/retry/middleware.rb:167:in 'Faraday::Retry::Middleware#call'
/gems/faraday-2.13.4/lib/faraday/middleware.rb:56:in 'Faraday::Middleware#call'
/gems/faraday-2.13.4/lib/faraday/response/logger.rb:25:in 'Faraday::Response::Logger#call'
/gems/faraday-2.13.4/lib/faraday/rack_builder.rb:153:in 'Faraday::RackBuilder#build_response'
/gems/faraday-2.13.4/lib/faraday/connection.rb:452:in 'Faraday::Connection#run_request'
/gems/faraday-2.13.4/lib/faraday/connection.rb:280:in 'Faraday::Connection#post'
/gems/ruby_llm-1.8.0/lib/ruby_llm/connection.rb:38:in 'RubyLLM::Connection#post'
/gems/ruby_llm-1.8.0/lib/ruby_llm/streaming.rb:11:in 'RubyLLM::Streaming#stream_response'
/gems/ruby_llm-1.8.0/lib/ruby_llm/provider.rb:56:in 'RubyLLM::Provider#complete'
/gems/ruby_llm-1.8.0/lib/ruby_llm/chat.rb:125:in 'RubyLLM::Chat#complete'
/gems/ruby_llm-1.8.0/lib/ruby_llm/active_record/chat_methods.rb:190:in 'RubyLLM::ActiveRecord::ChatMethods#complete'
/gems/ruby_llm-1.8.0/lib/ruby_llm/active_record/chat_methods.rb:184:in 'RubyLLM::ActiveRecord::ChatMethods#ask'
/app/jobs/llm_chat_job.rb:72:in 'LlmChatJob#execute'
The cause
The accumulator adds content with the following code (stream_accumulator.rb:23).
This fails because chunk.content is an array
@content << (chunk.content || '')
Workaround / Possible Solution
The problem that needs to be resolved is that content from the choices chunk is an array. This seems to be unexpected. This can be resolved by not accepting arrays as content. (Or at least the thinking message from Mistral)
I'm not sure what the best solution is for this problem.
In the solution below, I've added a custom streaming handler to Mistral.
Perhaps the global implementation should ignore array content items.
Because if another provider responds with array, it also breaks.
The workaround I've implemented is described below.
(Perhaps this can be implemented better by only fixing the content extraction for Mistral, but I don't know the internals of RubyLLM good enough)
Include a custom streaming provider to the mistral provider providers/mistral.rb
include Mistral::Streaming
Create a provider chunk builder providers/mistral/streaming.rb
(Copied the used OpenAI implementation of build_chunk
and changed the content to invoke the extract_content
method)
# frozen_string_literal: true
module RubyLLM
module Providers
class Mistral
# Streaming methods of the OpenAI API integration
module Streaming
module_function
def build_chunk(data)
Chunk.new(
role: :assistant,
model_id: data['model'],
content: extract_content(data),
tool_calls: parse_tool_calls(data.dig('choices', 0, 'delta', 'tool_calls'), parse_arguments: false),
input_tokens: data.dig('usage', 'prompt_tokens'),
output_tokens: data.dig('usage', 'completion_tokens')
)
end
def extract_content(data)
data = data.dig('choices', 0, 'delta', 'content')
return '' if data.is_a?(Array) || data.is_a?(Hash) # the workaround
data.to_s
end
end
end
end
end
Environment
- Ruby 3.4.1
- RubyLLM 1.8.0
- Provider Mistral
- OS Mac OS X