docs/integrations/llms/bedrock/

[giscus[bot]]

docs/integrations/llms/bedrock/

You are currently on a page documenting the use of Amazon Bedrock models as text completion models. Many popular models available on Bedrock are chat completion models.

https://python.langchain.com/docs/integrations/llms/bedrock/

[shrijayan]

Error while trying to use custom imported models in bedrock

Traceback (most recent call last):
  File "src/llm-bedrock.py", line 18, in <module>
    llm.invoke(messages)
  File ".venv/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py", line 286, in invoke
    self.generate_prompt(
  File ".venv/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py", line 786, in generate_prompt
    return self.generate(prompt_messages, stop=stop, callbacks=callbacks, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py", line 643, in generate
    raise e
  File ".venv/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py", line 633, in generate
    self._generate_with_cache(
  File ".venv/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py", line 851, in _generate_with_cache
    result = self._generate(
             ^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/langchain_aws/chat_models/bedrock.py", line 563, in _generate
    prompt = ChatPromptAdapter.convert_messages_to_prompt(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/langchain_aws/chat_models/bedrock.py", line 362, in convert_messages_to_prompt
    raise NotImplementedError(
NotImplementedError: Provider imported-model model does not support chat.

[luq7]

When I tried out the custom imported model(DeepSeek 70B) on Bedrock using LangChain I ran into this error:

ERROR:root:Error raised by bedrock service: An error occurred (ValidationException) when calling the 
InvokeModelWithResponseStream operation: User input fails validation.

To replicate this:

from langchain_aws import BedrockLLM

deepseek_70b_llm = BedrockLLM(
    aws_access_key_id="my-access-key-id",
    aws_secret_access_key="my-secret-access",
    aws_session_token="my-session-token",
    region="us-east-1",
    provider="deepseek",
    model_id="custom-imported-model-ARN",  # ARN like 'arn:aws:bedrock:...' obtained via provisioning the custom model
    model_kwargs={"temperature": 0.75, "max_tokens":512, "max_retries": 3},
    streaming=True,
)

deepseek_70b_llm.invoke("Who wrote the general relativity paper?")

[Bhanu-prakash33]

Fetching 5 files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:00<?, ?it/s]

Processing query: 'Expalin about the generation of carbon emissions in UK'
Query processed successfully in 2.22 seconds.
response I'm sorry, I can't respond to that.

event though the query is valid-output, nemo guardrails predicting correctly and sending to llm and there is no response from the LLM, every time getting the same error

23:52:39.397 | Invocation Params {'model_id': 'anthropic.claude-3-5-sonnet-20240620-v1:0', 'provider': 'anthropic', 'stream': True, 'trace': None, 'guardrailIdentifier': None, 'guardrailVersion': None, '_type':
'amazon_bedrock_chat', 'stop': None}

LLM Prompt (13e98..) - self_check_input

User
You are an ESG and sustainability expert. Your role is to determine if the query is related to Environmental, Social, and Governance (ESG) topics, with a focus on carbon calculations or sustainability metrics that
can be passed to an external API for processing.
Guidelines for acceptable queries (return "on-topic"): - Carbon footprint calculations (e.g., emissions for energy consumption) - Environmental metrics (e.g., water usage, waste generation) - Sustainability
reporting and metrics - Corporate governance related to ESG - Social responsibility measures - Climate risk assessments - Environmental compliance - Sustainable resource management - ESG investment criteria - Green
initiatives and renewable energy
Guidelines for queries to reject (return "off-topic"): - General business questions not related to ESG - Personal advice unrelated to sustainability - Non-environmental technical questions - Marketing or sales
queries - General financial questions not related to ESG - Entertainment or personal topics
Query to evaluate: Expalin about the generation of carbon emissions in UK
Respond with only "on-topic" or "off-topic" and proivide the output status.

LLM Completion (13e98..)
on-topic

23:52:41.869 | Output Stats None
23:52:41.871 | LLM call took 2.46 seconds