docs/tutorials/qa_chat_history/

[giscus[bot]]

docs/tutorials/qa_chat_history/

This guide assumes familiarity with the following concepts:

https://python.langchain.com/docs/tutorials/qa_chat_history/

[aditya11ad]

it can be a silly question, but please help me:
How can me apply "trimmer" , with create_react_agent()?

[ccurme]

We have an example of that in the guide here.

[SP3500]

The import code:
from langchain_core.vector_stores import InMemoryVectorStore

vector_store = InMemoryVectorStore(embeddings)
should be :
from langchain_core.vectorstores import InMemoryVectorStore

vector_store = InMemoryVectorStore(embeddings)

I think you wrongly added a "_" that will cause import error

[ccurme]

Thanks for the catch! Just merged a fix in #28388

[SP3500]

Hello!
Can I ask why this function can tell whether to let LLM directly answer the user question or enter the RAG part to retrieve?
Is it the cause of using llm_with_tools.invoke ?

def query_or_respond(state: MessagesState):
    """Generate tool call for retrieval or respond."""
    llm_with_tools = llm.bind_tools([retrieve])
    response = llm_with_tools.invoke(state["messages"])
    # MessagesState appends messages to state instead of overwriting
    return {"messages": [response]}

[SP3500]

Cause in my task , I also need to decide whether to enter the RAG to retrieve or just print the response directy.
Your chain framework is very helpful, so I want to have a deeper understanding of how to use it.
Your reply will help a lot!

[ccurme]

If response contains tool calls, we enter the retrieval step. If it does not, we return the response directly. This is controlled by tools_condition in the conditional edge downstream.

That is, we delegate the decision about entering the retrieval step to the model.

A simplified version of tools_condition looks like this:

def tools_condition(
    state: dict[str, Any],
    messages_key: str = "messages",
) -> Literal["tools", "__end__"]:
    ai_message = state[messages_key][-1]
    if hasattr(ai_message, "tool_calls") and len(ai_message.tool_calls) > 0:
        return "tools"
    return "__end__"

but you can swap this for a custom function.

[SP3500]

Thank you so much for this patient reply! It really helped a lot!

[YanzhongSu]

Good question and very helpful answer. Thank you both.

[HoganHPH]

[SP3500]

Did you add the condition edge when creating the chain?

[SP3500]

Hi there,
Is it possible to provide two function codes in LangGraph, where one function calls the other, outputs the result of its execution, and then passes it to the next module in LangGraph?

[Sanjeevkumar555]

Hi
I am running all commands copy and paste , However still I am not able to connect the link between retrieved docs and the llm output.
the LLM replies "I need more information to answer this question. I cannot find information about a named using the available tools."
But my retriever is working fine.
Any help pls how to debug?

[navono]

It seems like you might not have initialized the vector_store. Try initializing it with documents first, and then using it for retrieval.

[jasonzhang2022]

The memory saver resend all the previous messages (Human and AI message) to LLM so the LLM has context.

Wouldn't it be easy and consume less resource if we could the multi-turn chat session from LLM itself like this: https://ai.google.dev/gemini-api/docs/text-generation?lang=node#chat?

[Baanrow]

Great tutorials thanks team.

Do the MemorySaver references x 3 in this page need to be changed to InMemorySaver

https://langchain-ai.github.io/langgraph/reference/checkpoints/#langgraph.checkpoint.memory.InMemorySaver

Thanks again.

[khteh]

display(Image(graph.get_graph().draw_mermaid_png())) does not show anything at all in a python app.

[labilezhu]

I think below code may help as a standalone python app to display the image:

# Display the image in a popup window
def show_image():
    from IPython.display import Image, display
    import tkinter as tk
    from PIL import Image as PILImage, ImageTk
    import io    
    root = tk.Tk()
    root.title("Graph Image")
    
    # Get the image in memory buffer
    img_bytes = graph.get_graph().draw_mermaid_png()
    img_buffer = io.BytesIO()
    img_buffer.write(img_bytes)
    # graph.get_graph().draw_mermaid_png(img_buffer)

    img_buffer.seek(0)
    
    img = PILImage.open(img_buffer)
    img = ImageTk.PhotoImage(img)
    panel = tk.Label(root, image=img)
    panel.pack(side="bottom", fill="both", expand="yes")
    root.mainloop()

show_image()

[khteh]

This works but root.mainloop() blocks the execution. How to show the image without blocking the execution?

[labilezhu]

Great tutorials thanks team.

I am a nerd, so I draw a graph to help me understand how it works. Hope it helps you too:

https://blog.mygraphql.com/zh/posts/ai/langchain/rag-tool-chat-history-study/#my-notes-on-langchain-docs-build-a-retrieval-augmented-generation-rag-app---part-2

[sangwooklee0201]

Great tutorial. Thank you.

I am trying the "create_react_agent" method with Anthropic model (used ChatAnthropic from langchain library). When I ran the message from the tutorial following error is what I received:

== Error message starts ==
"UnprocessableEntityError: Error code: 422 - {'detail': [{'type': 'missing', 'loc': ['body', 'tools', 0, 'input_schema', 'properties', 'query', 'type'], 'msg': 'Field required', 'input': {}}]}
During task with name 'agent' and id '0d7e9770-7a14-04d3-28bc-3887dd88e7e5'"
== Error message ends ==

But when I tried with Open AI GPT 4o, it worked just fine. What changes should I make when I try it with Anthropic model? Following is a code:

== Code starts ==
gpt_4o_mini_model = AzureChatOpenAI(
api_key=API_KEY,
azure_endpoint=URL_BASE,
openai_api_version=OPENAI_API_VERSION,
azure_deployment=AZURE_DEPLOYMENT,
)

sonnet_3_5_model = ChatAnthropic(
anthropic_api_url=URL_BASE+"/anthropic",
api_key=API_KEY,
model_name=SONNET_3_5_DEPLOYMENT,
)

memory = MemorySaver()
agent_executor = create_react_agent(sonnet_3_5_model, [tool(response_format="content_and_artifact")(retrieve)], checkpointer=memory)

memory_config = {"configurable": {"thread_id": "def234"}}

inputs = {"messages": [("user", "What's your name?")]}

for s in agent_executor.stream(inputs, stream_mode="values", config=memory_config,):
message = s["messages"][-1]
if isinstance(message, tuple):
print(message)
else:
message.pretty_print()

== Code ends ==

[bihaosky2024]

Hi, may I ask how to fix the error below:

NotImplementedError:
During task with name 'query_or_respond' and id 'fe16a088-7be4-a59b-85ad-a46eeb4cd746'

It seems that there is a problem with the function 'query_or_respond'? I used the same code as this tutorial. Welcome to any comments, idea or possible solutions, thanks~

[bihaosky2024]

I have solved this problem myself. the local llm ("qwen2.5:32b") I used may be the reason, when I change to gpt-4o-mini, the error disappeared.

[khteh]

StateGraph generate step produces empty "Ai Message"
Code:

@tool
def ground_search(
    query: str, *, config: Annotated[RunnableConfig, InjectedToolArg]
)-> Optional[list[str]]:
    """Search for general web results.

    This function performs a search using the Google search engine, which is designed
    to provide comprehensive, accurate, and trusted results. It's particularly useful
    for answering questions about current events.
    """
    client = genai.Client(api_key=os.environ.get("VERTEX_API_KEY"))
    model_id = "gemini-2.0-flash"
    google_search_tool = Tool(
        google_search = GoogleSearch()
    )
    response = client.models.generate_content(
        model=model_id,
        contents=[{"role": "user", "parts": [{"text": query}]}], 
        config=GenerateContentConfig(
            tools=[google_search_tool],
            system_instruction="You are a helpful AI assistant named Bob.",
            response_modalities=["TEXT"],
        )
    )
    return "\n\n".join(content.text for content in response.candidates[0].content.parts)

@tool(response_format="content_and_artifact")
async def retrieve(query: str, *, config: RunnableConfig):
    """Retrieve information related to a query."""
    retrieved_docs = await vector_store.asimilarity_search(query, k=2)
    serialized = "\n\n".join(
        (f"Source: {doc.metadata}\n" f"Content: {doc.page_content}")
        for doc in retrieved_docs
    )
    return serialized, retrieved_docs

---

    # Step 3: Generate a response using the retrieved content.
    async def Generate(self, state: MessagesState, config: RunnableConfig):
        """Generate answer."""
        print(f"\n=== {self.Generate.__name__} ===")
        # Get generated ToolMessages
        recent_tool_messages = []
        for message in reversed(state["messages"]):
            if message.type == "tool":
                recent_tool_messages.append(message)
            else:
                break
        tool_messages = recent_tool_messages[::-1]
        # Format into prompt
        docs_content = "\n\n".join(doc.content for doc in tool_messages)
        system_message_content = f"""
        You are an assistant for question-answering tasks. 
        Use the following pieces of retrieved context to answer 
        the question. If you don't know the answer, say that you 
        don't know. Use three sentences maximum and keep the 
        answer concise.
        \n\n
        {docs_content}
        """
        conversation_messages = [
            message
            for message in state["messages"]
            if message.type in ("human", "system")
            or (message.type == "ai" and not message.tool_calls)
        ]
        prompt = [SystemMessage(system_message_content)] + conversation_messages
        # Run
        response = await self._llm.ainvoke(prompt, config)
        return {"messages": [response]}

---

        graph_builder = StateGraph(MessagesState)
        graph_builder.add_node("query_or_respond", self.query_or_respond)
        graph_builder.add_node("tools", ToolNode(TOOLS)) # Execute the retrieval.
        graph_builder.add_node("generate", self.generate)
        graph_builder.set_entry_point("query_or_respond")
        graph_builder.add_conditional_edges(
            "query_or_respond",
            tools_condition,
            {END: END, "tools": "tools"},
        )
        graph_builder.add_edge("tools", "generate")
        graph_builder.add_edge("generate", END)
        return graph_builder.compile(store=InMemoryStore(), checkpointer=MemorySaver(), name="Checkedpoint StateGraph RAG")

async def Chat(graph, config, messages: List[str]):
    print(f"\n=== {Chat.__name__} ===")
    messages = ["What is Task Decomposition?", "Can you look up some common ways of doing it?"]
    for message in messages:
        async for step in graph.astream(
            {"messages": [{"role": "user", "content": message}]},
            stream_mode="values",
            config = config
        ):
            step["messages"][-1].pretty_print()

================================ Human Message =================================

What is Task Decomposition?
================================== Ai Message ==================================
Tool Calls:
  retrieve (e1e31448-a4e7-4602-8d1c-ecf41cffa663)
 Call ID: e1e31448-a4e7-4602-8d1c-ecf41cffa663
  Args:
    query: Task Decomposition

=== asimilarity_search ===
================================= Tool Message =================================
Name: retrieve

Source: {'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/'}
Content: Fig. 1. Overview of a LLM-powered autonomous agent system.
Component One: Planning#
A complicated task usually involves many steps. An agent needs to know what they are and plan ahead.
Task Decomposition#
Chain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.

Source: {'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/'}
Content: Tree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.
Task decomposition can be done (1) by LLM with simple prompting like "Steps for XYZ.\n1.", "What are the subgoals for achieving XYZ?", (2) by using task-specific instructions; e.g. "Write a story outline." for writing a novel, or (3) with human inputs.

=== Generate ===
================================== Ai Message ==================================

Task decomposition is a method of breaking down a complex task into smaller, more manageable steps. This can be done by prompting an LLM, using task-specific instructions, or with human input. Chain of Thought (CoT) is a standard prompting technique for task decomposition, instructing the model to "think step by step". Tree of Thoughts extends CoT by exploring multiple reasoning possibilities at each step, creating a tree structure.
================================ Human Message =================================

Can you look up some common ways of doing it?
================================== Ai Message ==================================
Tool Calls:
  retrieve (4e69fd1b-ab9e-4a47-a00b-51ce34584004)
 Call ID: 4e69fd1b-ab9e-4a47-a00b-51ce34584004
  Args:
    query: common methods for task decomposition in AI

=== asimilarity_search ===
================================= Tool Message =================================
Name: retrieve

Source: {'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/'}
Content: Fig. 1. Overview of a LLM-powered autonomous agent system.
Component One: Planning#
A complicated task usually involves many steps. An agent needs to know what they are and plan ahead.
Task Decomposition#
Chain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.

Source: {'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/'}
Content: Tree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.
Task decomposition can be done (1) by LLM with simple prompting like "Steps for XYZ.\n1.", "What are the subgoals for achieving XYZ?", (2) by using task-specific instructions; e.g. "Write a story outline." for writing a novel, or (3) with human inputs.

=== Generate ===
================================== Ai Message ==================================
Tool Calls:
  retrieve (78d2ae63-8fab-4489-9422-5e5d95788f5b)
 Call ID: 78d2ae63-8fab-4489-9422-5e5d95788f5b
  Args:
    query: common ways to do task decomposition

^^^ Where is the response? ^^^

I check the log output of the Generate step:

conversation_messages: [HumanMessage(content='Hello, who are you?', additional_kwargs={}, response_metadata={}, id='838cc7bf-9361-4777-8359-e7a7a7b141c1'), AIMessage(content='I am Gemini, a large language model built by Google.\n', additional_kwargs={}, response_metadata={'is_blocked': False, 'safety_ratings': [], 'usage_metadata': {'prompt_token_count': 16, 'candidates_token_count': 13, 'total_token_count': 29, 'prompt_tokens_details': [{'modality': 1, 'token_count': 16}], 'candidates_tokens_details': [{'modality': 1, 'token_count': 13}], 'cached_content_token_count': 0, 'cache_tokens_details': []}, 'finish_reason': 'STOP', 'avg_logprobs': -0.002166313763994437}, id='run-0393aaea-5368-4183-86f3-2dd2d7ae3cbc-0', usage_metadata={'input_tokens': 16, 'output_tokens': 13, 'total_tokens': 29}), HumanMessage(content='What is Task Decomposition?', additional_kwargs={}, response_metadata={}, id='928ca68f-0353-49e1-9523-4fe6efb17963'), AIMessage(content='Task decomposition is breaking down a complex task into smaller, simpler steps. This can be done by prompting an LLM, using task-specific instructions, or with human input. Chain of Thought and Tree of Thoughts are examples of task decomposition techniques.\n', additional_kwargs={}, response_metadata={'is_blocked': False, 'safety_ratings': [], 'usage_metadata': {'prompt_token_count': 458, 'candidates_token_count': 50, 'total_token_count': 508, 'prompt_tokens_details': [{'modality': 1, 'token_count': 458}], 'candidates_tokens_details': [{'modality': 1, 'token_count': 50}], 'cached_content_token_count': 0, 'cache_tokens_details': []}, 'finish_reason': 'STOP', 'avg_logprobs': -0.16305770874023437}, id='run-01dd3c6c-7296-47bf-8e9a-40656c7650b8-0', usage_metadata={'input_tokens': 458, 'output_tokens': 50, 'total_tokens': 508}), HumanMessage(content='Can you look up some common ways of doing it?', additional_kwargs={}, response_metadata={}, id='d9fbbb13-febe-48ff-8543-988529c725b4')]

prompt: [SystemMessage(content='\n        You are an assistant for question-answering tasks. \n        Use the following pieces of retrieved context to answer \n        the question. If you don\'t know the answer, say that you \n        don\'t know. Use three sentences maximum and keep the \n        answer concise.\n        \n\n\n        Source: {\'source\': \'https://lilianweng.github.io/posts/2023-06-23-agent/\'}\nContent: Fig. 1. Overview of a LLM-powered autonomous agent system.\nComponent One: Planning#\nA complicated task usually involves many steps. An agent needs to know what they are and plan ahead.\nTask Decomposition#\nChain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.\n\nSource: {\'source\': \'https://lilianweng.github.io/posts/2023-06-23-agent/\'}\nContent: Tree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.\nTask decomposition can be done (1) by LLM with simple prompting like "Steps for XYZ.\\n1.", "What are the subgoals for achieving XYZ?", (2) by using task-specific instructions; e.g. "Write a story outline." for writing a novel, or (3) with human inputs.\n        ', additional_kwargs={}, response_metadata={}), HumanMessage(content='Hello, who are you?', additional_kwargs={}, response_metadata={}, id='838cc7bf-9361-4777-8359-e7a7a7b141c1'), AIMessage(content='I am Gemini, a large language model built by Google.\n', additional_kwargs={}, response_metadata={'is_blocked': False, 'safety_ratings': [], 'usage_metadata': {'prompt_token_count': 16, 'candidates_token_count': 13, 'total_token_count': 29, 'prompt_tokens_details': [{'modality': 1, 'token_count': 16}], 'candidates_tokens_details': [{'modality': 1, 'token_count': 13}], 'cached_content_token_count': 0, 'cache_tokens_details': []}, 'finish_reason': 'STOP', 'avg_logprobs': -0.002166313763994437}, id='run-0393aaea-5368-4183-86f3-2dd2d7ae3cbc-0', usage_metadata={'input_tokens': 16, 'output_tokens': 13, 'total_tokens': 29}), HumanMessage(content='What is Task Decomposition?', additional_kwargs={}, response_metadata={}, id='928ca68f-0353-49e1-9523-4fe6efb17963'), AIMessage(content='Task decomposition is breaking down a complex task into smaller, simpler steps. This can be done by prompting an LLM, using task-specific instructions, or with human input. Chain of Thought and Tree of Thoughts are examples of task decomposition techniques.\n', additional_kwargs={}, response_metadata={'is_blocked': False, 'safety_ratings': [], 'usage_metadata': {'prompt_token_count': 458, 'candidates_token_count': 50, 'total_token_count': 508, 'prompt_tokens_details': [{'modality': 1, 'token_count': 458}], 'candidates_tokens_details': [{'modality': 1, 'token_count': 50}], 'cached_content_token_count': 0, 'cache_tokens_details': []}, 'finish_reason': 'STOP', 'avg_logprobs': -0.16305770874023437}, id='run-01dd3c6c-7296-47bf-8e9a-40656c7650b8-0', usage_metadata={'input_tokens': 458, 'output_tokens': 50, 'total_tokens': 508}), HumanMessage(content='Can you look up some common ways of doing it?', additional_kwargs={}, response_metadata={}, id='d9fbbb13-febe-48ff-8543-988529c725b4')]

Generate() response: content='' additional_kwargs={'function_call': {'name': 'retrieve', 'arguments': '{"query": "common ways to do task decomposition"}'}} response_metadata={'is_blocked': False, 'safety_ratings': [], 'usage_metadata': {'prompt_token_count': 519, 'candidates_token_count': 8, 'total_token_count': 527, 'prompt_tokens_details': [{'modality': 1, 'token_count': 519}], 'candidates_tokens_details': [{'modality': 1, 'token_count': 8}], 'cached_content_token_count': 0, 'cache_tokens_details': []}, 'finish_reason': 'STOP', 'avg_logprobs': -0.0082077132537961} id='run-9d950ea3-bcdc-4234-be8f-1e14de4f4838-0' tool_calls=[{'name': 'retrieve', 'args': {'query': 'common ways to do task decomposition'}, 'id': 'cc6e4484-a863-48c4-8381-47f1e3d73f73', 'type': 'tool_call'}] usage_metadata={'input_tokens': 519, 'output_tokens': 8, 'total_tokens': 527}

The response content=''. Is it because it runs out of tokens?

[XariZaru]

How would one pass metadata filtering arguments at input time to this? Basically, I want the user to be able to select a "source" to further make the retrieval process more accurate.

[khteh]

https://api.python.langchain.com/en/latest/tools/langchain.tools.retriever.create_retriever_tool.html makes a reference to this tutorial. How does create_retriever_tool compare to using the retrieve method here?

[saketh395]

AttributeError: module 'mlflow' has no attribute 'trace'
File , line 10
8 from databricks.sdk import WorkspaceClient
9 from databricks.sdk.service.serving import ChatMessage, ChatMessageRole
---> 10 from databricks_langchain import ChatDatabricks
12 graph_builder = StateGraph(MessagesState)
14 llm = ChatDatabricks(
15 endpoint="databricks-meta-llama-3-3-70b-instruct"
16 )

[saketh395]

How can i solve this issue, as i am already using latest mlflow and databricks-langchain versions