Skip to content

Commit 9aa7b7c

Browse files
compumikecrmne
andauthored
Add RubyLLM::Chat#with_params to add custom parameters to the underlying API payload (#265)
## What this does Implements `with_request_options` (renamed from `with_options` due to ActiveRecord conflict -- see conversation) with @crmne's suggestions from comment #130 (review) and tested against all providers. This allows users to set arbitrary options on the payload before it's sent to the provider's API endpoint. The render_payload takes precedence. Demo: ```ruby chat = RubyLLM .chat(model: "qwen3", provider: :ollama) .with_request_options(response_format: {type: "json_object"}) .with_instructions("Answer with a JSON object with the key `result` and a numerical value.") response = chat.ask("What is the square root of 64?") response.content => "{\n \"result\": 8\n}" ``` This is a power-user feature, and is specific to each provider (and model, to a lesser extent). I added a brief section to the docs. For tests: different providers supported different options, so tests are divided by provider. (Note that `deep_merge` is required for Gemini in particular because it relies on a top-level `generationConfig` object.) ## Type of change - [ ] Bug fix - [X] New feature - [ ] Breaking change - [ ] Documentation - [ ] Performance improvement ## Scope check - [X] I read the [Contributing Guide](https://github.com/crmne/ruby_llm/blob/main/CONTRIBUTING.md) - [X] This aligns with RubyLLM's focus on **LLM communication** - [X] This isn't application-specific logic that belongs in user code - [X] This benefits most users, not just my specific use case ## Quality check - [ ] I ran `overcommit --install` and all hooks pass - [X] I tested my changes thoroughly - [X] I updated documentation if needed - [X] I didn't modify auto-generated files manually (`models.json`, `aliases.json`) ## API changes - [ ] Breaking change - [X] New public methods/classes - [ ] Changed method signatures - [ ] No API changes ## Related issues - #130 - #131 - #221 --------- Co-authored-by: Carmine Paolino <[email protected]>
1 parent c87f6c3 commit 9aa7b7c

13 files changed

+646
-8
lines changed

docs/guides/chat.md

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -246,6 +246,19 @@ puts response2.content
246246

247247
You can set the temperature using `with_temperature`, which returns the `Chat` instance for chaining.
248248

249+
## Custom Request Parameters
250+
251+
You can configure additional provider-specific features by adding custom fields to each API request. Use the `with_params` method.
252+
253+
```ruby
254+
# response_format parameter is supported by :openai, :ollama, :deepseek
255+
chat = RubyLLM.chat.with_params(response_format: { type: 'json_object' })
256+
response = chat.ask "What is the square root of 64? Answer with a JSON object with the key `result`."
257+
puts JSON.parse(response.content)
258+
```
259+
260+
Allowed parameters vary widely by provider and model.
261+
249262
## Tracking Token Usage
250263

251264
Understanding token usage is important for managing costs and staying within context limits. Each `RubyLLM::Message` returned by `ask` includes token counts.
@@ -311,4 +324,4 @@ This guide covered the core `Chat` interface. Now you might want to explore:
311324
* [Using Tools]({% link guides/tools.md %}): Enable the AI to call your Ruby code.
312325
* [Streaming Responses]({% link guides/streaming.md %}): Get real-time feedback from the AI.
313326
* [Rails Integration]({% link guides/rails.md %}): Persist your chat conversations easily.
314-
* [Error Handling]({% link guides/error-handling.md %}): Build robust applications that handle API issues.
327+
* [Error Handling]({% link guides/error-handling.md %}): Build robust applications that handle API issues.

lib/ruby_llm/active_record/acts_as.rb

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -130,6 +130,11 @@ def with_context(...)
130130
self
131131
end
132132

133+
def with_params(...)
134+
to_llm.with_params(...)
135+
self
136+
end
137+
133138
def on_new_message(...)
134139
to_llm.on_new_message(...)
135140
self

lib/ruby_llm/chat.rb

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ module RubyLLM
1111
class Chat
1212
include Enumerable
1313

14-
attr_reader :model, :messages, :tools
14+
attr_reader :model, :messages, :tools, :params
1515

1616
def initialize(model: nil, provider: nil, assume_model_exists: false, context: nil)
1717
if assume_model_exists && !provider
@@ -25,6 +25,7 @@ def initialize(model: nil, provider: nil, assume_model_exists: false, context: n
2525
@temperature = 0.7
2626
@messages = []
2727
@tools = {}
28+
@params = {}
2829
@on = {
2930
new_message: nil,
3031
end_message: nil
@@ -78,6 +79,11 @@ def with_context(context)
7879
self
7980
end
8081

82+
def with_params(**params)
83+
@params = params
84+
self
85+
end
86+
8187
def on_new_message(&block)
8288
@on[:new_message] = block
8389
self
@@ -99,6 +105,7 @@ def complete(&)
99105
temperature: @temperature,
100106
model: @model.id,
101107
connection: @connection,
108+
params: @params,
102109
&wrap_streaming_block(&)
103110
)
104111

lib/ruby_llm/provider.rb

Lines changed: 21 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -10,14 +10,19 @@ module Provider
1010
module Methods
1111
extend Streaming
1212

13-
def complete(messages, tools:, temperature:, model:, connection:, &)
13+
def complete(messages, tools:, temperature:, model:, connection:, params: {}, &) # rubocop:disable Metrics/ParameterLists
1414
normalized_temperature = maybe_normalize_temperature(temperature, model)
1515

16-
payload = render_payload(messages,
17-
tools: tools,
18-
temperature: normalized_temperature,
19-
model: model,
20-
stream: block_given?)
16+
payload = deep_merge(
17+
params,
18+
render_payload(
19+
messages,
20+
tools: tools,
21+
temperature: normalized_temperature,
22+
model: model,
23+
stream: block_given?
24+
)
25+
)
2126

2227
if block_given?
2328
stream_response connection, payload, &
@@ -26,6 +31,16 @@ def complete(messages, tools:, temperature:, model:, connection:, &)
2631
end
2732
end
2833

34+
def deep_merge(params, payload)
35+
params.merge(payload) do |_key, params_value, payload_value|
36+
if params_value.is_a?(Hash) && payload_value.is_a?(Hash)
37+
deep_merge(params_value, payload_value)
38+
else
39+
payload_value
40+
end
41+
end
42+
end
43+
2944
def list_models(connection:)
3045
response = connection.get models_url
3146
parse_list_models_response response, slug, capabilities

spec/fixtures/vcr_cassettes/chat_with_params_anthropic_claude-3-5-haiku-20241022_supports_service_tier_param.yml

Lines changed: 81 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

spec/fixtures/vcr_cassettes/chat_with_params_bedrock_anthropic_claude-3-5-haiku-20241022-v1_0_supports_top_k_param.yml

Lines changed: 54 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

spec/fixtures/vcr_cassettes/chat_with_params_deepseek_deepseek-chat_supports_response_format_param.yml

Lines changed: 57 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)