knowledge-bases/4_customized-rag-retreive-api-titan-lite-evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building and evaluating Q&A application using Knowledge Bases for Amazon Bedrock - Retrieve API, Langchain, and LLaMa Index for Prompt Completion Evaluations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Context\n",
    "\n",
    "In this notebook, we will dive deep into building Q&A application using Retrieve API provide by Knowledge Bases for Amazon Bedrock, along with LangChain and LlamaIndex for evaluating the responses. Here, we will query the knowledge base to get the desired number of document chunks based on similarity search, prompt the query using Amazon Titan Text Lite, and then evaluate the responses effectively using LLaMaIndex evaluation metrics, such as faithfulness, correctness, relevancy or guideline based expectations.\n",
    "\n",
    "With knowledge bases, you can securely connect foundation models (FMs) in Amazon Bedrock to your company\n",
    "data for Retrieval Augmented Generation (RAG). Access to additional data helps the model generate more relevant,\n",
    "context-speciﬁc, and accurate responses without continuously retraining the FM. All information retrieved from\n",
    "knowledge bases comes with source attribution to improve transparency and minimize hallucinations. For more information on creating a knowledge base using console, please refer to this [post](!https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base.html).\n",
    "\n",
    "### Pattern\n",
    "\n",
    "We can implement the solution using Retreival Augmented Generation (RAG) pattern. RAG retrieves data from outside the language model (non-parametric) and augments the prompts by adding the relevant retrieved data in context. Here, we are performing RAG effectively on the knowledge base created in the previous notebook or using console. \n",
    "\n",
    "### Pre-requisite\n",
    "\n",
    "Before being able to answer the questions, the documents must be processed and stored in Amazon knowledge base.\n",
    "\n",
    "1. Load the documents into the knowledge base by connecting your s3 bucket (data source). \n",
    "2. Ingestion - Knowledge base will split them into smaller chunks (based on the strategy selected), generate embeddings and store it in the associated vectore store.\n",
    "\n",
    "![data_ingestion.png](./images/data_ingestion.png)\n",
    "\n",
    "\n",
    "#### Notebook Walkthrough\n",
    "\n",
    "\n",
    "\n",
    "For our notebook we will use the `Retreive API` provided by Knowledge Bases for Amazon Bedrock which converts user queries into\n",
    "embeddings, searches the knowledge base, and returns the relevant results, giving you more control to build custom\n",
    "workﬂows on top of the semantic search results. The output of the `Retrieve API` includes the the `retrieved text chunks`, the `location type` and `URI` of the source data, as well as the relevance `scores` of the retrievals. \n",
    "\n",
    "\n",
    "We will then use the text chunks being generated and augment it with the original prompt and pass it through the `titan-text-lite` model using prompt engineering patterns based for your use case.\n",
    "\n",
    "Finally we will evaluate the generated responses using LLaMaIndex on using metrics such as faithfulness, correctness, guideline and relevancy metrics. For evaluation, we will use `Anthropic Claude v2 model`.\n",
    "### Ask question\n",
    "\n",
    "\n",
    "![retrieveapi.png](./images/retrieveAPI.png)\n",
    "\n",
    "\n",
    "#### Evaluation\n",
    "1. Utilize LLaMa Index for answer evaluation on \n",
    "    1. Faithfulness\n",
    "    2. Correctness\n",
    "    3. Relevancy\n",
    "    4. Guidelines\n",
    "    \n",
    "\n",
    "### USE CASE:\n",
    "\n",
    "#### Dataset\n",
    "\n",
    "In this example, you will use several years of Amazon's Letter to Shareholders as a text corpus to perform Q&A on. This data is already ingested into the knowledge base. You will need the `knowledge base id` to run this example.\n",
    "In your specific use case, you can sync different files for different domain topics and query this notebook in the same manner to evaluate model responses using the retrieve API from knowledge bases.\n",
    "\n",
    "\n",
    "### Python 3.10\n",
    "\n",
    "⚠  For this lab we need to run the notebook based on a Python 3.10 runtime. ⚠\n",
    "\n",
    "### Setup\n",
    "\n",
    "To run this notebook you would need to install dependencies, langchain and LLaMa Index and the updated boto3, botocore whls.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install --upgrade pip\n",
    "%pip install boto3==1.33.2 --force-reinstall --quiet\n",
    "%pip install botocore==1.33.2 --force-reinstall --quiet\n",
    "%pip install langchain==0.0.342 --force-reinstall --quiet\n",
    "%pip install llama-index==0.9.3.post1 --force-reinstall --quiet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Restart the kernel with the updated packages that are installed through the dependencies above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# restart kernel\n",
    "from IPython.core.display import HTML\n",
    "HTML(\"<script>Jupyter.notebook.kernel.restart()</script>\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nest_asyncio\n",
    "nest_asyncio.apply()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Follow the steps below to initiate the bedrock client:\n",
    "\n",
    "1. Import the necessary libraries, along with langchain for bedrock model selection, llama index to store the service context containing the llm and embedding model instances. We will use this service context later in the notebook for evaluating the responses from our Q&A application. \n",
    "\n",
    "2. Initialize `amazon.titan-text-lite-v1` as our large language model to perform query completions using the RAG pattern with the given knowledge base, once we get all text chunk searches through the `retrieve` API.\n",
    "\n",
    "3. For evaluating the response with LlamaIndex we will use `anthropic.claude-v2` model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "import pprint\n",
    "from botocore.client import Config\n",
    "from langchain.llms.bedrock import Bedrock\n",
    "from llama_index import (\n",
    "    ServiceContext,\n",
    "    set_global_service_context\n",
    ")\n",
    "from langchain.embeddings.bedrock import BedrockEmbeddings\n",
    "from llama_index.embeddings import LangchainEmbedding\n",
    "\n",
    "pp = pprint.PrettyPrinter(indent=2)\n",
    "\n",
    "bedrock_config = Config(connect_timeout=120, read_timeout=120, retries={'max_attempts': 0})\n",
    "bedrock_client = boto3.client('bedrock-runtime')\n",
    "bedrock_agent_client = boto3.client(\"bedrock-agent-runtime\",\n",
    "                              config=bedrock_config)\n",
    "\n",
    "parameters = {\n",
    "    \"maxTokenCount\":2000,\n",
    "    \"stopSequences\":[],\n",
    "    \"temperature\":0,\n",
    "    \"topP\":0.9\n",
    "    }\n",
    "\n",
    "model_kwargs_claude = {\n",
    "    \"temperature\": 0,\n",
    "    \"top_k\": 10,\n",
    "    \"max_tokens_to_sample\": 3000\n",
    "}\n",
    "\n",
    "embed_model = LangchainEmbedding(\n",
    "    BedrockEmbeddings(model_id=\"amazon.titan-embed-text-v1\")\n",
    ")\n",
    "\n",
    "llm = Bedrock(model_id = \"amazon.titan-text-lite-v1\",\n",
    "              model_kwargs=parameters,\n",
    "              client = bedrock_client,)\n",
    "\n",
    "llm_claude = Bedrock(model_id = \"anthropic.claude-v2\",\n",
    "              model_kwargs=model_kwargs_claude,\n",
    "              client = bedrock_client,)\n",
    "\n",
    "service_context = ServiceContext.from_defaults(llm=llm_claude,\n",
    "                                               embed_model=embed_model)\n",
    "set_global_service_context(service_context)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrieve API: Process flow \n",
    "\n",
    "Define a retrieve function that calls the `Retreive API` provided by Knowledge Bases for Amazon Bedrock which converts user queries into\n",
    "embeddings, searches the knowledge base, and returns the relevant results, giving you more control to build custom\n",
    "workﬂows on top of the semantic search results. The output of the `Retrieve API` includes the the `retrieved text chunks`, the `location type` and `URI` of the source data, as well as the relevance `scores` of the retrievals. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def retrieve(query, kbId, numberOfResults=5):\n",
    "    return bedrock_agent_client.retrieve(\n",
    "        retrievalQuery= {\n",
    "            'text': query\n",
    "        },\n",
    "        knowledgeBaseId=kbId,\n",
    "        retrievalConfiguration= {\n",
    "            'vectorSearchConfiguration': {\n",
    "                'numberOfResults': numberOfResults\n",
    "            }\n",
    "        }\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Initialize your Knowledge base id before querying responses from the initialized LLM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "kb_id = \"<knowledge_base_id>\" # replace it with the Knowledge base id."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will call the `retreive API`, and pass `knowledge base id`, `number of results` and `query` as paramters. \n",
    "\n",
    "`score`: You can view the associated score of each of the text chunk that was returned which depicts its correlation to the query in terms of how closely it matches it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "query = \"What is Amazon's doing in the field of generative AI?\"\n",
    "response = retrieve(query, kb_id, 5)\n",
    "retrievalResults = response['retrievalResults']\n",
    "pp.pprint(retrievalResults)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prompt specific to the model to personalize responses \n",
    "\n",
    "Here, we will use the specific prompt below for the model to act as a financial advisor AI system that will provide answers to questions by using fact based and statistical information when possible. We will provide the `Retrieve API` responses from above as a part of the `{context_str}` in the prompt for the model to refer to, along with the user `query`.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.prompts import PromptTemplate\n",
    "\n",
    "PROMPT_TEMPLATE = \"\"\"\n",
    "You are a financial advisor AI system, and provides answers to questions by using fact based and statistical information when possible. \n",
    "Use the following pieces of information to provide a concise answer to the question enclosed in <question> tags. \n",
    "If you don't know the answer, just say that you don't know, don't try to make up an answer.\n",
    "<context>\n",
    "{context_str}\n",
    "</context>\n",
    "\n",
    "<question>\n",
    "{query_str}\n",
    "</question>\n",
    "\n",
    "The response should be specific and use statistics or numbers when possible.\n",
    "\"\"\"\n",
    "titan_prompt = PromptTemplate(template=PROMPT_TEMPLATE, \n",
    "                               input_variables=[\"context_str\",\"query_str\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fetch the text chunks from the RetrieveAPI response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# fetch context from the response\n",
    "def get_contexts(retrievalResults):\n",
    "    contexts = []\n",
    "    for retrievedResult in retrievalResults: \n",
    "        contexts.append(retrievedResult['content']['text'])\n",
    "    return contexts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "contexts = get_contexts(retrievalResults)\n",
    "pp.pprint(contexts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initiate the user prompt and response via the LLM\n",
    "\n",
    "Here, we are going to format our prompt using the context generated by the retrieve API associated to our KB as well as the user query to get the final response that we will use to evaluate generated answers using LLaMaIndex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "prompt = titan_prompt.format(context_str=contexts, \n",
    "                                 query_str=query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = llm(prompt)\n",
    "pp.pprint(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation Pipeline: Utilizing LLaMaIndex for end-end evaluations on Faithfulness, Correctness, Guidelines given, and Relevancy of answers generated by the model.\n",
    "\n",
    "- Faithfulness - to measure if the response from the model matches any source nodes. This is useful for measuring if the response was hallucinated.\n",
    "- Relevancy - to measure if the response + source nodes match the query.This is useful for measuring if the query was actually answered by the response.\n",
    "- Correctness - to evaluate the relevance and correctness of a generated answer against a reference answer.\n",
    "- Guidelines - to evaluate a question answer system given user specified guidelines for example, if the response generated is complete, not toxic, or biased or uses facts in the context.\n",
    "\n",
    "### 1. Faithfulness Evaluation of Prompt Completions: Using LLaMa Index\n",
    "\n",
    "This is useful for measuring if the response was hallucinated. Here we are going to essentially first focus on correlating the response of the model back to the context that was given to the model to answer the prompt/query, and check for how much the response and the context of the model correlates, or in other words, how faithful the prompt completion is based on the RAG performed using the query.\n",
    "\n",
    "![faithfullness.png](./images/rag-eval-flow-faithfulness.png)\n",
    "\n",
    "Here, we are matching whether the response generated correlates and matches with the corresponding context or not.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.evaluation import FaithfulnessEvaluator\n",
    "\n",
    "faithfulness_evaluator = FaithfulnessEvaluator(service_context=service_context)\n",
    "faith_eval = faithfulness_evaluator.evaluate(query=query,\n",
    "                                              response=response, \n",
    "                                              contexts=contexts)\n",
    "print(f\"Faithful response?: {str(faith_eval.passing)}\"  )\n",
    "pp.pprint(f\"Reason: {faith_eval.feedback} \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Relevancy Evaluation of Prompt Completions: Using LLaMa Index\n",
    "\n",
    " In this section will focus on using the `RelevancyEvaluator` module to measure if the response + source nodes match the query. This is useful for measuring if the query was actually answered by the response.\n",
    "\n",
    "![relevancy](./images/rag-eval-flow-relevancy.png)\n",
    "\n",
    "Here, we are using the query, along with the correlation with the response generated from that query, as well as the context utilized to evaluate whether the response generated by the model answers the question asked by user."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.evaluation import RelevancyEvaluator\n",
    "\n",
    "relevancy_evaluator = RelevancyEvaluator(service_context=service_context)\n",
    "relevant_eval = relevancy_evaluator.evaluate(query=query,\n",
    "                                              response=response, \n",
    "                                              contexts=contexts)\n",
    "print(f'Relevant response?: {str(relevant_eval.passing)}')\n",
    "pp.pprint(f\"Reason: {relevant_eval.feedback} \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Correctness Evaluation of Prompt Completions: Using LLaMa Index\n",
    "\n",
    "#### The CorrectnessEvaluator is used to evaluate the relevance and correctness of a generated answer against a reference answer.\n",
    "\n",
    "Here, we provide a reference answer along with the query in order to check for the correctness and exact accuracy of the response. This process can be viewed from a 'Ground Truth' perspective to check for specific use cases where correctness is the highest priority.\n",
    "![correctness](./images/rag-eval-flow-correctness.png)\n",
    "\n",
    "\n",
    "In this section we will use a batch of questions to evaluate on all three evaluators discussed above - Faithfullness, Relevancy and Correctness, to evaluate the performance of our RAG application.\n",
    "\n",
    "Now, we will check for all three in our case: Faithfulness of responses, relevancy as well as correctness and display that as a pandas report table:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_question_answer_pair = [(\"How many days has Amazon asked employees to come to work in office?\",\n",
    "                          \"Amazon has asked corporate employees to come back to office at least three days a week beginning May 2022.\"),\n",
    "                         (\"By what percentage did AWS revenue grow year-over-year in 2022?\",\n",
    "                          \"AWS had a 29% year-over-year ('YoY') revenue in 2022 on $62B revenue base.\"),\n",
    "                         (\"Compared to Graviton2 processors, what performance improvement did Graviton3 chips deliver according to the passage?\",\n",
    "                          \"In 2022, AWS delivered their Graviton3 chips, providing 25% better performance than the Graviton2 processors.\"),\n",
    "                         (\"Which was the first inference chip launched by AWS according to the passage?\",\n",
    "                          \"AWS launched their first inference chips (“Inferentia”) in 2019, and they have saved companies like Amazon over a hundred million dollars in capital expense.\"),\n",
    "                         (\"According to the context, in what year did Amazon's annual revenue increase from $245B to $434B?\",\n",
    "                          \"Amazon's annual revenue increased from $245B in 2019 to $434B in 2022.\"\n",
    "                          )\n",
    "                          ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Tuple, List\n",
    "import pandas as pd\n",
    "from llama_index.evaluation import (\n",
    "    FaithfulnessEvaluator,\n",
    "    RelevancyEvaluator,\n",
    "    CorrectnessEvaluator,\n",
    ")\n",
    "\n",
    "faithfulness_evaluator = FaithfulnessEvaluator(service_context=service_context)\n",
    "relevancy_evaluator = RelevancyEvaluator(service_context=service_context)\n",
    "correctness_evaluator = CorrectnessEvaluator(service_context=service_context)\n",
    "\n",
    "def run_evals(qa_pairs: List[Tuple[str, str]], topK):\n",
    "    results_list = []\n",
    "    for question, reference_answer in qa_pairs:\n",
    "        # retrieve matching documents\n",
    "        result = retrieve(question, kb_id, topK)\n",
    "        retrievalResults = result['retrievalResults']\n",
    "        contexts = get_contexts(retrievalResults=retrievalResults)\n",
    "        #call LLM with updated context and question.\n",
    "        prompt = titan_prompt.format(context_str=contexts, \n",
    "                                 query_str=question)\n",
    "        response = llm(prompt)\n",
    "        generated_answer = str(response)\n",
    "        # evaluate results.\n",
    "        correctness_results = correctness_evaluator.evaluate(\n",
    "            query=question,\n",
    "            response=generated_answer,\n",
    "            reference=reference_answer\n",
    "        )\n",
    "        faithfulness_results = faithfulness_evaluator.evaluate(\n",
    "            query=question,\n",
    "            response=generated_answer,\n",
    "            contexts=contexts\n",
    "            )\n",
    "        relevancy_results = relevancy_evaluator.evaluate(\n",
    "            query=question,\n",
    "            response=generated_answer,\n",
    "            contexts=contexts\n",
    "            )\n",
    "        cur_result_dict = {\n",
    "            \"query\": question,\n",
    "            \"generated_answer\": generated_answer,\n",
    "            \"correctness\": correctness_results.passing,\n",
    "            \"correctness_feedback\": correctness_results.feedback,\n",
    "            \"correctness_score\": correctness_results.score,\n",
    "            \"faithfulness\": faithfulness_results.passing,\n",
    "            \"faithfulness_feedback\": faithfulness_results.feedback,\n",
    "            \"faithfulness_score\": faithfulness_results.score,\n",
    "            \"relevancy\": relevancy_results.passing,\n",
    "            \"relevancy_feedback\": relevancy_results.feedback,\n",
    "            \"relevancy_score\": relevancy_results.score\n",
    "        }\n",
    "        results_list.append(cur_result_dict)\n",
    "    evals_df = pd.DataFrame(results_list)\n",
    "    return evals_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# please note the execution of this cell might take 3-5mins. \n",
    "evaluation_results = run_evals(eval_question_answer_pair, 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize evaluations - all the questions, generated responses, along with their correlating evaluation metrics\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluation_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overall score for all 3 metrics\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f'Correctness score: {evaluation_results.correctness.mean()} \\nFaithfulness score: {evaluation_results.faithfulness.mean()} \\nRelevancey score: {evaluation_results.relevancy.mean()}')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Note: Please note the scores above gives a relative idea on the performance of your RAG application and should be used with caution and not as standalone scores. Also note, that we have used only 5 question/answer pairs for evaluation, as best practice, you should use enough data to cover different aspects of your document for evaluating model.\n",
    "\n",
    "Based on the scores, you can review other components of your RAG workflow to further optimize the scores, few recommended options are to review your chunking strategy, prompt instructions, adding more numberOfResults for additional context and so on. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Guideline Evaluation of Prompt Completions: Using LLaMaIndex\n",
    "\n",
    "#### This section will focus on using the GuidelineEvaluator module to evaluate a question answer system given user specified guidelines.\n",
    "\n",
    "In this code below, we will define `certain` guidelines to look out for while evaluating a response to our query. Once we set these guidelines, we can check for certain pass and fail rates to make sure our query is following the guidelines.\n",
    "\n",
    "This may be useful for use cases where more than one component needs to be displayed within the response for a more holistic evaluation that is not only faithful, relevant or correct, but also follows expected guidelines.\n",
    "\n",
    "![guidelines.png](./images/rag-eval-flow-guidelines.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.evaluation import GuidelineEvaluator\n",
    "\n",
    "GUIDELINES = [\n",
    "    \"The response should fully answer the query.\",\n",
    "    \"The response should avoid being vague or ambiguous.\",\n",
    "    \"The response should not use toxic or profane language.\",\n",
    "    \"The response should not be bias or discriminatory.\",\n",
    "    (\n",
    "        \"The response should be specific and use statistics or numbers when\"\n",
    "        \" possible.\"\n",
    "    ),\n",
    "]\n",
    "\n",
    "evaluators = [\n",
    "    GuidelineEvaluator(service_context=service_context, guidelines=guideline)\n",
    "    for guideline in GUIDELINES\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"What is Amazon's generative AI strategy?\"\n",
    "result = retrieve(query, kb_id, 3)\n",
    "retrievalResults = result['retrievalResults']\n",
    "contexts = get_contexts(retrievalResults=retrievalResults)\n",
    "#call LLM with updated context and question.\n",
    "prompt = titan_prompt.format(context_str=contexts, \n",
    "                            query_str=query)\n",
    "response = llm(prompt)\n",
    "generated_answer = str(response)\n",
    "pp.pprint(generated_answer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Shows how specific guidelines are satisfied vs. not satisfied based on the response generated from the model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "for guideline, evaluator in zip(GUIDELINES, evaluators):\n",
    "    eval_result = evaluator.evaluate(\n",
    "        query=query,\n",
    "        response=generated_answer,\n",
    "    )\n",
    "    print(\"=====\")\n",
    "    print(f\"Guideline: {guideline}\")\n",
    "    print(f\"Pass: {eval_result.passing}\")\n",
    "    print(f\"Feedback: {eval_result.feedback}\")"
   ]
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 57,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.trn1.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 58,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1.32xlarge",
    "vcpuNum": 128
   },
   {
    "_defaultOrder": 59,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1n.32xlarge",
    "vcpuNum": 128
   }
  ],
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "kb-env",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}