Skip to content

[BUG] Content Streaming from Mistral results in TypeError (no implicit conversion of Array into String) #420

@gamecreature

Description

@gamecreature

Basic checks

  • I searched existing issues - this hasn't been reported
  • I can reproduce this consistently
  • This is a RubyLLM bug, not my application code

What's broken?

Content extraction fails for Mistral because the chunk of data returned has the following structure.
choices/delta/content is an array, which isn't expected

{
"id": "836a02cf63f7429793b2a62e63dfce29", 
"object": "chat.completion.chunk", 
"created": 1758093959, 
"model": "magistral-small-latest", 
"choices": [{
   "index": 0, 
   "delta": {
     "content": [{
       "type": "thinking", 
       "thinking": [{"type": "text", "text": "Okay"}] 
     }]
   }, 
   "finish_reason": null
}], 
"p": "abcdefghijklmnopqr"
}

How to reproduce

  1. Configure RubyLLM with mistral and use mistral-small-latest
  2. Ask calculate 1+1
  3. exception TypeError (no implicit conversion of Array into String) is raised

Note curiously enough, once in while the problematic message isn't included and the question succeeds.

Expected behavior

Process the streaming response without an exception.

What actually happened

The exception:

TypeError (no implicit conversion of Array into String):
/gems/ruby_llm-1.8.0/lib/ruby_llm/stream_accumulator.rb:23:in 'RubyLLM::StreamAccumulator#add'
/gems/ruby_llm-1.8.0/lib/ruby_llm/streaming.rb:20:in 'block (2 levels) in RubyLLM::Streaming#stream_response'
/gems/ruby_llm-1.8.0/lib/ruby_llm/streaming.rb:41:in 'block in RubyLLM::Streaming#handle_stream'
/gems/ruby_llm-1.8.0/lib/ruby_llm/streaming.rb:129:in 'block in RubyLLM::Streaming#handle_sse'
/gems/event_stream_parser-1.0.0/lib/event_stream_parser.rb:217:in 'EventStreamParser::Parser#dispatch_event'
/gems/event_stream_parser-1.0.0/lib/event_stream_parser.rb:73:in 'EventStreamParser::Parser#process_line'
/gems/event_stream_parser-1.0.0/lib/event_stream_parser.rb:46:in 'EventStreamParser::Parser#feed'
/gems/ruby_llm-1.8.0/lib/ruby_llm/streaming.rb:124:in 'RubyLLM::Streaming#handle_sse'
/gems/ruby_llm-1.8.0/lib/ruby_llm/streaming.rb:74:in 'RubyLLM::Streaming#process_stream_chunk'
/gems/ruby_llm-1.8.0/lib/ruby_llm/streaming.rb:87:in 'block in RubyLLM::Streaming#stream_processor'
/gems/faraday-2.13.4/lib/faraday/options/env.rb:176:in 'block in Faraday::Env#stream_response'
/gems/net-protocol-0.2.2/lib/net/protocol.rb:535:in 'Net::ReadAdapter#call_block'
/gems/net-protocol-0.2.2/lib/net/protocol.rb:526:in 'Net::ReadAdapter#<<'
/gems/net-protocol-0.2.2/lib/net/protocol.rb:168:in 'Net::BufferedIO#read'
/gems/net-http-0.6.0/lib/net/http/response.rb:631:in 'Net::HTTPResponse#read_chunked'
/gems/net-http-0.6.0/lib/net/http/response.rb:595:in 'block in Net::HTTPResponse#read_body_0'
/gems/net-http-0.6.0/lib/net/http/response.rb:588:in 'Net::HTTPResponse#inflater'
/gems/net-http-0.6.0/lib/net/http/response.rb:593:in 'Net::HTTPResponse#read_body_0'
/gems/net-http-0.6.0/lib/net/http/response.rb:363:in 'Net::HTTPResponse#read_body'
/gems/faraday-net_http-3.4.1/lib/faraday/adapter/net_http.rb:116:in 'block (2 levels) in Faraday::Adapter::NetHttp#request_with_wrapped_block'
/gems/net-http-0.6.0/lib/net/http.rb:2433:in 'block in Net::HTTP#transport_request'
/gems/net-http-0.6.0/lib/net/http/response.rb:320:in 'Net::HTTPResponse#reading_body'
/gems/net-http-0.6.0/lib/net/http.rb:2430:in 'Net::HTTP#transport_request'
/gems/net-http-0.6.0/lib/net/http.rb:2384:in 'Net::HTTP#request'
/gems/rack-mini-profiler-4.0.1/lib/patches/net_patches.rb:19:in 'block in Net::HTTP#request_with_mini_profiler'
/gems/rack-mini-profiler-4.0.1/lib/mini_profiler/profiling_methods.rb:51:in 'Rack::MiniProfiler::ProfilingMethods#step'
/gems/rack-mini-profiler-4.0.1/lib/patches/net_patches.rb:18:in 'Net::HTTP#request_with_mini_profiler'
/gems/airbrake-13.0.5/lib/airbrake/rails/net_http.rb:11:in 'block in Airbrake::Rails::NetHttp#request'
/gems/airbrake-13.0.5/lib/airbrake/rack.rb:21:in 'Airbrake::Rack.capture_timing'
/gems/airbrake-13.0.5/lib/airbrake/rails/net_http.rb:10:in 'Airbrake::Rails::NetHttp#request'
/gems/faraday-net_http-3.4.1/lib/faraday/adapter/net_http.rb:113:in 'block in Faraday::Adapter::NetHttp#request_with_wrapped_block'
/gems/net-http-0.6.0/lib/net/http.rb:1632:in 'Net::HTTP#start'
/gems/faraday-net_http-3.4.1/lib/faraday/adapter/net_http.rb:112:in 'Faraday::Adapter::NetHttp#request_with_wrapped_block'
/gems/faraday-net_http-3.4.1/lib/faraday/adapter/net_http.rb:98:in 'block in Faraday::Adapter::NetHttp#perform_request'
/gems/faraday-2.13.4/lib/faraday/options/env.rb:172:in 'Faraday::Env#stream_response'
/gems/faraday-net_http-3.4.1/lib/faraday/adapter/net_http.rb:97:in 'Faraday::Adapter::NetHttp#perform_request'
/gems/faraday-net_http-3.4.1/lib/faraday/adapter/net_http.rb:66:in 'block in Faraday::Adapter::NetHttp#call'
/gems/faraday-2.13.4/lib/faraday/adapter.rb:45:in 'Faraday::Adapter#connection'
/gems/faraday-net_http-3.4.1/lib/faraday/adapter/net_http.rb:65:in 'Faraday::Adapter::NetHttp#call'
/gems/ruby_llm-1.8.0/lib/ruby_llm/error.rb:39:in 'RubyLLM::ErrorMiddleware#call'
/gems/faraday-2.13.4/lib/faraday/middleware.rb:56:in 'Faraday::Middleware#call'
/gems/faraday-2.13.4/lib/faraday/middleware.rb:56:in 'Faraday::Middleware#call'
/gems/faraday-retry-2.3.2/lib/faraday/retry/middleware.rb:171:in 'block in Faraday::Retry::Middleware#call'
/gems/faraday-retry-2.3.2/lib/faraday/retry/retryable.rb:7:in 'Faraday::Retryable#with_retries'
/gems/faraday-retry-2.3.2/lib/faraday/retry/middleware.rb:167:in 'Faraday::Retry::Middleware#call'
/gems/faraday-2.13.4/lib/faraday/middleware.rb:56:in 'Faraday::Middleware#call'
/gems/faraday-2.13.4/lib/faraday/response/logger.rb:25:in 'Faraday::Response::Logger#call'
/gems/faraday-2.13.4/lib/faraday/rack_builder.rb:153:in 'Faraday::RackBuilder#build_response'
/gems/faraday-2.13.4/lib/faraday/connection.rb:452:in 'Faraday::Connection#run_request'
/gems/faraday-2.13.4/lib/faraday/connection.rb:280:in 'Faraday::Connection#post'
/gems/ruby_llm-1.8.0/lib/ruby_llm/connection.rb:38:in 'RubyLLM::Connection#post'
/gems/ruby_llm-1.8.0/lib/ruby_llm/streaming.rb:11:in 'RubyLLM::Streaming#stream_response'
/gems/ruby_llm-1.8.0/lib/ruby_llm/provider.rb:56:in 'RubyLLM::Provider#complete'
/gems/ruby_llm-1.8.0/lib/ruby_llm/chat.rb:125:in 'RubyLLM::Chat#complete'
/gems/ruby_llm-1.8.0/lib/ruby_llm/active_record/chat_methods.rb:190:in 'RubyLLM::ActiveRecord::ChatMethods#complete'
/gems/ruby_llm-1.8.0/lib/ruby_llm/active_record/chat_methods.rb:184:in 'RubyLLM::ActiveRecord::ChatMethods#ask'
/app/jobs/llm_chat_job.rb:72:in 'LlmChatJob#execute'

The cause

The accumulator adds content with the following code (stream_accumulator.rb:23).
This fails because chunk.content is an array

@content << (chunk.content || '')

Workaround / Possible Solution

The problem that needs to be resolved is that content from the choices chunk is an array. This seems to be unexpected. This can be resolved by not accepting arrays as content. (Or at least the thinking message from Mistral)

I'm not sure what the best solution is for this problem.
In the solution below, I've added a custom streaming handler to Mistral.
Perhaps the global implementation should ignore array content items.
Because if another provider responds with array, it also breaks.

The workaround I've implemented is described below.

(Perhaps this can be implemented better by only fixing the content extraction for Mistral, but I don't know the internals of RubyLLM good enough)

Include a custom streaming provider to the mistral provider providers/mistral.rb

include Mistral::Streaming

Create a provider chunk builder providers/mistral/streaming.rb
(Copied the used OpenAI implementation of build_chunk and changed the content to invoke the extract_content method)

# frozen_string_literal: true

module RubyLLM
  module Providers
    class Mistral
      # Streaming methods of the OpenAI API integration
      module Streaming
        module_function

        def build_chunk(data)
          Chunk.new(
            role: :assistant,
            model_id: data['model'],
            content: extract_content(data),
            tool_calls: parse_tool_calls(data.dig('choices', 0, 'delta', 'tool_calls'), parse_arguments: false),
            input_tokens: data.dig('usage', 'prompt_tokens'),
            output_tokens: data.dig('usage', 'completion_tokens')
          )
        end

        def extract_content(data)
          data = data.dig('choices', 0, 'delta', 'content')
          return '' if data.is_a?(Array) || data.is_a?(Hash)  # the workaround

          data.to_s
        end
      end
    end
  end
end

Environment

  • Ruby 3.4.1
  • RubyLLM 1.8.0
  • Provider Mistral
  • OS Mac OS X

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions