Skip to content

Inference regions #338

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

ESegundoRolon
Copy link

What this does

Support region-aware inference profiles for AWS Bedrock

Problem

The current implementation hardcodes us. prefix for all Bedrock inference
profile models, causing failures when using cross-region inference in
non-US regions like EU, AP, or CA.

  • Error encountered:
    The provided model identifier is invalid. (RubyLLM::BadRequestError)

This occurs because:

  1. EU regions expect for example eu.anthropic.claude-sonnet-4-20250514-v1:0
  2. But the library sends us.anthropic.claude-sonnet-4-20250514-v1:0
  3. AWS Bedrock rejects the incorrect region prefix

Solution

Updated model_id_with_region method to dynamically extract the region
prefix from the configured bedrock_region:

  • eu-west-3 → eu. prefix
  • us-east-1 → us. prefix
  • ap-south-1 → ap. prefix
  • ca-central-1 → ca. prefix

Changes Made

Core Fix

File: lib/ruby_llm/providers/bedrock/models.rb

  • Modified model_id_with_region to use dynamic region prefix
  • Added inference_profile_region_prefix helper method
  • Uses first two characters of bedrock_region as prefix
  • Maintains backwards compatibility with us default

Tests Added

File: spec/ruby_llm/providers/bedrock/models_spec.rb

  • Comprehensive tests for different AWS regions (US, EU, AP, CA)
  • Tests for edge cases (empty/nil regions)
  • Tests for both inference profile and on-demand models
  • Ensures backwards compatibility

Testing

  • ✅ US regions continue to work (us. prefix)
  • ✅ EU regions now work (eu. prefix)
  • ✅ AP/CA regions supported (ap./ca. prefixes)
  • ✅ ON_DEMAND models unaffected
  • ✅ Backwards compatible with existing configurations

Breaking Changes

None. This change is fully backwards compatible.

Type of change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation
  • Performance improvement

Scope check

  • I read the Contributing Guide
  • This aligns with RubyLLM's focus on LLM communication
  • This isn't application-specific logic that belongs in user code
  • This benefits most users, not just my specific use case

Quality check

  • I ran overcommit --install and all hooks pass
  • I tested my changes thoroughly
  • I updated documentation if needed
  • I didn't modify auto-generated files manually (models.json, aliases.json)

API changes

  • Breaking change
  • New public methods/classes
  • Changed method signatures
  • No API changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant