Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat : Support AWS bedrock base models #25

Open
wants to merge 20 commits into
base: main
Choose a base branch
from

Conversation

detunjiSamuel
Copy link

@detunjiSamuel detunjiSamuel commented Jan 9, 2025

AWS Bedrock Provider Integration

Added support for AWS Bedrock as a new LLM provider:

  • Anthropic Claude models
  • Amazon Titan models (text & embeddings)
  • AI21 Jurassic models

Key Changes

Added Bedrock provider implementation with model-specific handlers:

  • Anthropic: Reused existing Anthropic provider code with Bedrock-specific modifications ( no actual changes)
  • Titan: Added chat completion and embedding support
  • AI21: Added completion and chat completion support
  • Request/response mapping between Hub formats and Bedrock formats

Testing Notes

All tests pass using AWS credentials in us-east-1/2 regions
Verified error handling for invalid credentials/models
Tested non-streaming responses ( models in Bedrock don't seem to have streaming types )

Review notes

The model ID from AWS link does not work consistently.
Instead, use the Inference profile ARN or Inference profile ID from the cross-region reference tab as your model_id.

Issue: #20

/claim #20

@CLAassistant
Copy link

CLAassistant commented Jan 9, 2025

CLA assistant check
All committers have signed the CLA.

@detunjiSamuel detunjiSamuel marked this pull request as draft January 9, 2025 03:18
@detunjiSamuel detunjiSamuel changed the title Feat : Support AWS bedrock base models [ DRAFT ] Feat : Support AWS bedrock base models Jan 9, 2025
@detunjiSamuel
Copy link
Author

Hey @nirga ,

Quick question about integrating Stability AI models in our Hub. I'm looking at AWS Bedrock's Stable Diffusion 3.5 integration (from their model catalog).

Not sure about the best format to implement this:

  1. Should we use the existing chat_completions since it takes a similar prompt format?
  2. Or would it make more sense to create a new Content type specifically for image generation?

Would appreciate your thoughts on this.

Thanks!

@nirga
Copy link
Member

nirga commented Jan 9, 2025

I think it should be in a new api @detunjiSamuel
But we can do it in a separate PR

@detunjiSamuel detunjiSamuel marked this pull request as ready for review January 10, 2025 23:11
@detunjiSamuel
Copy link
Author

Hey @nirga , I have completed this PR and would appreciate your review.
I've verified the changes through:

  • External Python file testing
  • Manual console verification
  • Updated test suite
  • Results exported to Traceloop from the Pipeline.

Please let me know if any additional information or changes are needed.

Thank you!

Copy link

algora-pbc bot commented Jan 10, 2025

💵 To receive payouts, sign up on Algora, link your Github account and connect with Stripe.

Copy link
Member

@nirga nirga left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @detunjiSamuel, can you take a look at the tests? Seems to be failing

@detunjiSamuel
Copy link
Author

@nirga , Thanks for pointing that out.
The error was caused by missing AWS credentials in the CI environment.
I've fixed this by adding recordings of my local AWS requests and using AWS Smithy to replay the responses.
The most recent commit resolves that issue.

Thanks again!

@detunjiSamuel detunjiSamuel requested a review from nirga January 14, 2025 18:27
@detunjiSamuel
Copy link
Author

Hi @nirga ,

I wanted to follow up on this PR as it's been a few weeks since the last update. As mentioned previously, I've addressed the test failures by implementing AWS Smithy to replay the test responses, which resolved the CI issues.

Everything is passing and ready for review.
Please let me know if you need any additional information or have any concerns I can address.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants