From 9e46771fa5267b00e570846a6b85e76eb0612e49 Mon Sep 17 00:00:00 2001 From: Srinidhi Sathyamurthy Date: Tue, 7 Jul 2026 23:30:01 +0100 Subject: [PATCH 1/2] Hybrid search Agent memory --- notebooks/hybrid_search_agent_memory.ipynb | 1354 ++++++++++++++++++++ 1 file changed, 1354 insertions(+) create mode 100644 notebooks/hybrid_search_agent_memory.ipynb diff --git a/notebooks/hybrid_search_agent_memory.ipynb b/notebooks/hybrid_search_agent_memory.ipynb new file mode 100644 index 00000000..922cd55b --- /dev/null +++ b/notebooks/hybrid_search_agent_memory.ipynb @@ -0,0 +1,1354 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "a16ff7c8", + "metadata": {}, + "source": [ + "# Hybrid Search for Oracle AI Agent Memory: Combining Semantic Recall with Exact Match\n", + "\n", + "This notebook demonstrates how Oracle AI Agent Memory retrieves durable memories using both semantic meaning and exact text. It uses one support-and-finance scenario throughout: **Northstar Renewals**, invoice **INV-48291**, Oracle error **ORA-27102**, and a renewal blocker caused by a reconciliation failure.\n", + "\n", + "The notebook is designed for the Oracle AI Agent Memory 26.6 API. The hybrid-search section requires a database-resident embedding model that can be used by `OracleDBEmbedder`. The environment and database preflight cells can run before that model access is available; the notebook never substitutes mock retrieval results for Oracle hybrid search." + ] + }, + { + "cell_type": "markdown", + "id": "b3eda2ae", + "metadata": {}, + "source": [ + "## What You Will Learn\n", + "\n", + "- Configure Oracle AI Agent Memory with `SearchStrategy.HYBRID` and `OracleDBEmbedder`.\n", + "- Store memories that combine natural-language context with exact identifiers.\n", + "- Search the same scoped memories by invoice ID, error code, and semantic question.\n", + "- Understand how search scope and hybrid ranking solve different retrieval problems.\n", + "- Validate retrieval behavior without presenting a small demo as a formal benchmark.\n", + "- Apply the correct schema and index synchronization choices for production." + ] + }, + { + "cell_type": "markdown", + "id": "7a9c2169", + "metadata": {}, + "source": [ + "## Workflow\n", + "\n", + "1. Validate Python and package versions.\n", + "2. Load database configuration without displaying secrets.\n", + "3. Connect to Oracle AI Database.\n", + "4. Discover database-resident embedding models.\n", + "5. Configure Oracle AI Agent Memory for Oracle-managed hybrid search.\n", + "6. Add scoped memories with exact identifiers and descriptive context.\n", + "7. Search by exact identifier and semantic meaning.\n", + "8. Validate top-ranked results and review production considerations." + ] + }, + { + "cell_type": "markdown", + "id": "26e4db14", + "metadata": {}, + "source": [ + "## Prerequisites\n", + "\n", + "- Python 3.10-3.13. This project uses the dedicated **Python 3.11 - OAM Hybrid Search** kernel.\n", + "- Oracle AI Agent Memory 26.6.0 installed from the internal package source.\n", + "- Access to Oracle AI Database.\n", + "- A database-resident embedding model for the complete hybrid-search run.\n", + "- An `.env` file in the project folder:\n", + "\n", + "```env\n", + "ORACLE_USER=\n", + "ORACLE_PASSWORD=\n", + "ORACLE_DSN=\n", + "ORACLE_DB_EMBEDDING_MODEL=\n", + "ORACLE_DB_EMBEDDING_DIMENSION=384\n", + "```\n", + "\n", + "The first three settings are sufficient for connection and model discovery. The model name and its correct dimension are required before creating `OracleDBEmbedder`." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "91633643", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Python: 3.13.14\n", + "oracleagentmemory: 26.6.0\n", + "Runtime preflight: READY\n" + ] + } + ], + "source": [ + "import sys\n", + "from importlib.metadata import version\n", + "\n", + "python_supported = (3, 10) <= sys.version_info[:2] <= (3, 13)\n", + "package_version = version(\"oracleagentmemory\")\n", + "\n", + "assert python_supported, f\"Python 3.10-3.13 is required; found {sys.version.split()[0]}\"\n", + "assert package_version == \"26.6.0\", f\"Oracle AI Agent Memory 26.6.0 is required; found {package_version}\"\n", + "\n", + "print(f\"Python: {sys.version.split()[0]}\")\n", + "print(f\"oracleagentmemory: {package_version}\")\n", + "print(\"Runtime preflight: READY\")" + ] + }, + { + "cell_type": "markdown", + "id": "f4e3286a", + "metadata": {}, + "source": [ + "## Step 1 - Load Configuration Safely\n", + "\n", + "The cell below loads `.env` and reports only whether required values are present. It never prints usernames, passwords, DSNs, or model credentials." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "a8f027f6", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Configuration: READY\n" + ] + } + ], + "source": [ + "import os\n", + "from pathlib import Path\n", + "\n", + "import pandas as pd\n", + "from dotenv import load_dotenv\n", + "\n", + "\n", + "def find_project_dir(start: Path) -> Path:\n", + " for candidate in [start, *start.parents]:\n", + " if (candidate / \".env\").exists():\n", + " return candidate\n", + " raise FileNotFoundError(\"No .env file was found in the current directory or its parents.\")\n", + "\n", + "\n", + "PROJECT_DIR = find_project_dir(Path.cwd())\n", + "ENV_PATH = PROJECT_DIR / \".env\"\n", + "load_dotenv(ENV_PATH, override=False)\n", + "\n", + "CONFIG = {\n", + " \"ORACLE_USER\": os.getenv(\"ORACLE_USER\", \"\").strip(),\n", + " \"ORACLE_PASSWORD\": os.getenv(\"ORACLE_PASSWORD\", \"\"),\n", + " \"ORACLE_DSN\": os.getenv(\"ORACLE_DSN\", \"\").strip(),\n", + " \"ORACLE_DB_EMBEDDING_MODEL\": os.getenv(\"ORACLE_DB_EMBEDDING_MODEL\", \"\").strip(),\n", + " \"ORACLE_DB_EMBEDDING_DIMENSION\": int(os.getenv(\"ORACLE_DB_EMBEDDING_DIMENSION\", \"384\")),\n", + "}\n", + "\n", + "connection_keys = [\"ORACLE_USER\", \"ORACLE_PASSWORD\", \"ORACLE_DSN\"]\n", + "missing_connection_keys = [key for key in connection_keys if not CONFIG[key]]\n", + "\n", + "if missing_connection_keys:\n", + " raise RuntimeError(f\"Missing required database settings: {', '.join(missing_connection_keys)}\")\n", + "\n", + "print(\"Configuration: READY\")" + ] + }, + { + "cell_type": "markdown", + "id": "9c79e5f1", + "metadata": {}, + "source": [ + "## Step 2 - Connect to Oracle AI Database\n", + "\n", + "This notebook uses one thin-mode `python-oracledb` connection. It validates connectivity without displaying environment-specific database or schema details." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "641d420e", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Database connection: READY\n" + ] + } + ], + "source": [ + "import oracledb\n", + "\n", + "connection = None\n", + "DB_READY = False\n", + "DB_BLOCKER = \"\"\n", + "\n", + "try:\n", + " connection = oracledb.connect(\n", + " user=CONFIG[\"ORACLE_USER\"],\n", + " password=CONFIG[\"ORACLE_PASSWORD\"],\n", + " dsn=CONFIG[\"ORACLE_DSN\"],\n", + " )\n", + " DB_READY = True\n", + " print(\"Database connection: READY\")\n", + "except oracledb.Error as exc:\n", + " error = exc.args[0]\n", + " DB_BLOCKER = (\n", + " \"Oracle Database is not reachable with the current DSN. Start the local database/listener \"\n", + " \"or update ORACLE_DSN to a reachable database before running hybrid search.\"\n", + " )\n", + " print(\"Database connection: BLOCKED\")\n", + " print(f\"Oracle driver error code: {getattr(error, 'code', 'unknown')}\")\n", + " print(DB_BLOCKER)" + ] + }, + { + "cell_type": "markdown", + "id": "20773aee", + "metadata": {}, + "source": [ + "## Step 3 - Validate the Database Embedding Model\n", + "\n", + "Oracle-managed hybrid search requires `OracleDBEmbedder`, which in turn requires a database-resident embedding model. This read-only preflight checks whether the model configured in `.env` is available to the connected application.\n", + "\n", + "The validation deliberately reports only readiness. Model names, catalog views, algorithms, database versions, and schema names are environment-specific implementation details and are not displayed in this public notebook." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "8091333e", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Database embedding model: READY\n", + "Hybrid search prerequisites: READY\n" + ] + } + ], + "source": [ + "def discover_embedding_models(db_connection):\n", + " queries = [\n", + " (\n", + " \"USER_MINING_MODELS\",\n", + " \"\"\"SELECT model_name, mining_function, algorithm\n", + " FROM user_mining_models\n", + " WHERE UPPER(mining_function) = 'EMBEDDING'\n", + " ORDER BY model_name\"\"\",\n", + " ),\n", + " (\n", + " \"ALL_MINING_MODELS\",\n", + " \"\"\"SELECT owner || '.' || model_name AS model_name, mining_function, algorithm\n", + " FROM all_mining_models\n", + " WHERE UPPER(mining_function) = 'EMBEDDING'\n", + " ORDER BY owner, model_name\"\"\",\n", + " ),\n", + " ]\n", + "\n", + " errors = []\n", + " for source, query in queries:\n", + " try:\n", + " with db_connection.cursor() as cursor:\n", + " rows = cursor.execute(query).fetchall()\n", + " return source, pd.DataFrame(rows, columns=[\"model_name\", \"mining_function\", \"algorithm\"]), errors\n", + " except oracledb.DatabaseError as exc:\n", + " error = exc.args[0]\n", + " errors.append(f\"{source}: {getattr(error, 'code', 'unknown error')}\")\n", + "\n", + " return \"UNAVAILABLE\", pd.DataFrame(columns=[\"model_name\", \"mining_function\", \"algorithm\"]), errors\n", + "\n", + "\n", + "if DB_READY:\n", + " model_source, embedding_models, discovery_errors = discover_embedding_models(connection)\n", + "else:\n", + " model_source = \"NOT_CONNECTED\"\n", + " embedding_models = pd.DataFrame(columns=[\"model_name\", \"mining_function\", \"algorithm\"])\n", + " discovery_errors = []\n", + "\n", + "configured_model = CONFIG[\"ORACLE_DB_EMBEDDING_MODEL\"]\n", + "visible_models = set(embedding_models[\"model_name\"].astype(str))\n", + "\n", + "if not DB_READY:\n", + " HYBRID_READY = False\n", + " HYBRID_STATUS = DB_BLOCKER\n", + "elif configured_model and model_source == \"UNAVAILABLE\":\n", + " HYBRID_READY = True\n", + " HYBRID_STATUS = \"Model catalog is unavailable; the configured model will be validated by OracleDBEmbedder.\"\n", + "elif configured_model and configured_model in visible_models:\n", + " HYBRID_READY = True\n", + " HYBRID_STATUS = \"The configured database embedding model is available.\"\n", + "elif configured_model:\n", + " HYBRID_READY = False\n", + " HYBRID_STATUS = \"Configured model was not found among models visible to this schema.\"\n", + "else:\n", + " HYBRID_READY = False\n", + " HYBRID_STATUS = \"ORACLE_DB_EMBEDDING_MODEL is not configured.\"\n", + "\n", + "if not DB_READY:\n", + " HYBRID_BLOCKER = DB_BLOCKER\n", + "else:\n", + " HYBRID_BLOCKER = (\n", + " \"A database-resident embedding model is required. Ask the package team for the \"\n", + " \"Oracle Database/schema connection, model name, and embedding dimension to use with OracleDBEmbedder.\"\n", + " )\n", + "\n", + "if HYBRID_READY:\n", + " print(\"Database embedding model: READY\")\n", + " print(\"Hybrid search prerequisites: READY\")\n", + "else:\n", + " print(\"Database embedding model: BLOCKED\")\n", + " print(\"Hybrid search prerequisites: BLOCKED\")\n", + " print(HYBRID_STATUS)\n", + " print(HYBRID_BLOCKER)" + ] + }, + { + "cell_type": "markdown", + "id": "c680eb45", + "metadata": {}, + "source": [ + "### Why the Database Model Matters\n", + "\n", + "`OracleDBEmbedder` keeps embedding generation inside Oracle AI Database. The same database-resident model must be used for query embeddings and for the managed hybrid vector index so their vectors have compatible dimensions and semantics.\n", + "\n", + "- Connection credentials and model configuration are read from `.env` and are never printed.\n", + "- The model name and embedding dimension are deployment-specific.\n", + "- Model discovery is a prerequisite check, not part of the retrieval demonstration.\n", + "- A successful preflight confirms that the notebook can proceed with `OracleDBEmbedder` and managed hybrid indexing.\n", + "\n", + "If validation is blocked, confirm that the configured database user can access the intended embedding model and create or query the managed hybrid vector index." + ] + }, + { + "cell_type": "markdown", + "id": "242f1780", + "metadata": {}, + "source": [ + "## Step 4 - Configure Oracle AI Agent Memory for Hybrid Search\n", + "\n", + "`SearchStrategy.HYBRID` combines Oracle-managed text matching with vector-aware ranking. `OracleDBEmbedder` ensures the memory client and the managed hybrid vector index use the same in-database model. `ON_COMMIT` makes newly committed memories searchable immediately, which is appropriate for this interactive walkthrough." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "8dbf0b1e", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Oracle AI Agent Memory 26.6 hybrid-search imports: READY\n" + ] + } + ], + "source": [ + "from oracleagentmemory.apis.searchscope import SearchScope\n", + "from oracleagentmemory.core import (\n", + " MemoryExtractionConfig,\n", + " OracleAgentMemory,\n", + " SchemaPolicy,\n", + " SearchIndexSyncMode,\n", + " SearchStrategy,\n", + ")\n", + "from oracleagentmemory.core.embedders import OracleDBEmbedder\n", + "\n", + "\n", + "def require_hybrid_ready() -> None:\n", + " if not HYBRID_READY:\n", + " raise RuntimeError(HYBRID_BLOCKER)\n", + "\n", + "\n", + "print(\"Oracle AI Agent Memory 26.6 hybrid-search imports: READY\")" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "29cc7e8a", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Oracle-managed hybrid search client: READY\n" + ] + } + ], + "source": [ + "require_hybrid_ready()\n", + "\n", + "db_embedder = OracleDBEmbedder(\n", + " connection=connection,\n", + " model=CONFIG[\"ORACLE_DB_EMBEDDING_MODEL\"],\n", + " embedding_dimension=CONFIG[\"ORACLE_DB_EMBEDDING_DIMENSION\"],\n", + ")\n", + "\n", + "memory = OracleAgentMemory(\n", + " connection=connection,\n", + " embedder=db_embedder,\n", + " memory_extraction_config=MemoryExtractionConfig(extract_memories=False),\n", + " schema_policy=SchemaPolicy.CREATE_IF_NECESSARY,\n", + " search_strategy=SearchStrategy.HYBRID,\n", + " search_index_sync=SearchIndexSyncMode.ON_COMMIT,\n", + " memory_store_id=\"hybrid_blog\",\n", + ")\n", + "\n", + "print(\"Oracle-managed hybrid search client: READY\")" + ] + }, + { + "cell_type": "markdown", + "id": "a892757b", + "metadata": {}, + "source": [ + "`SchemaPolicy.CREATE_IF_NECESSARY` is intentional for the first setup. It may create or upgrade managed search objects and build the hybrid index. For a large existing schema, run this as a planned migration. After the schema is ready, production applications can use `SchemaPolicy.REQUIRE_EXISTING` for normal startup validation." + ] + }, + { + "cell_type": "markdown", + "id": "50f01b42", + "metadata": {}, + "source": [ + "## Step 5 - Register a Scoped User and Agent\n", + "\n", + "Scope determines which records are eligible for retrieval. Hybrid search then ranks only those eligible records. A unique run identifier keeps repeated notebook runs isolated while preserving one stable managed memory store." + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "e3974b5b", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Scoped demo user and agent: READY\n" + ] + } + ], + "source": [ + "import uuid\n", + "\n", + "require_hybrid_ready()\n", + "\n", + "RUN_ID = uuid.uuid4().hex[:8]\n", + "USER_ID = f\"finance_user_{RUN_ID}\"\n", + "AGENT_ID = f\"support_finance_{RUN_ID}\"\n", + "\n", + "memory.add_user(\n", + " user_id=USER_ID,\n", + " information=\"Finance user with access to renewal-related data.\",\n", + ")\n", + "memory.add_agent(\n", + " agent_id=AGENT_ID,\n", + " information=\"Support-finance agent for invoices, reconciliation failures, and renewal blockers.\",\n", + ")\n", + "\n", + "scope = SearchScope(user_id=USER_ID, agent_id=AGENT_ID)\n", + "print(\"Scoped demo user and agent: READY\")" + ] + }, + { + "cell_type": "markdown", + "id": "33706a04", + "metadata": {}, + "source": [ + "## Step 6 - Build a Deliberately Ambiguous Memory Set\n", + "\n", + "The dataset combines exact operational handles with natural-language context. It also includes carefully chosen distractors: a neighboring invoice, a different customer's renewal delay, a general error-code explanation, and a customer-alias record.\n", + "\n", + "This matters because a useful retrieval demo should not succeed merely because there is only one record. The target memory must outrank records that overlap on customer, invoice, renewal, or error vocabulary." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "5c8a8fdf", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
Memory inventory used by the retrieval evaluation
labelexact_signalsemantic_contextevaluation_role
renewal_blockerINV-48291 · ORA-27102 · Northstar RenewalsFailed reconciliation blocked a renewalPrimary target
similar_invoiceINV-48290 · Northstar RenewalsNeighboring invoice, already paidExact-ID distractor
milan_renewalMilan Office SuppliesRenewal delayed by approval workflowSemantic distractor
error_referenceORA-27102General error-code explanationExact-code target
customer_aliasNorthstar RenewalsEnterprise account mappingAlias distractor
\n" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "require_hybrid_ready()\n", + "\n", + "sample_memories = [\n", + " {\n", + " \"label\": \"renewal_blocker\",\n", + " \"content\": (\n", + " \"Northstar Renewals cannot proceed with its renewal because invoice INV-48291 \"\n", + " \"failed reconciliation after ORA-27102 during month-end processing. Finance must \"\n", + " \"rerun the reconciliation job before the renewal can continue.\"\n", + " ),\n", + " \"metadata\": {\n", + " \"customer_alias\": \"Northstar Renewals\",\n", + " \"invoice_id\": \"INV-48291\",\n", + " \"error_code\": \"ORA-27102\",\n", + " },\n", + " },\n", + " {\n", + " \"label\": \"similar_invoice\",\n", + " \"content\": (\n", + " \"Northstar Renewals also has invoice INV-48290 marked as paid. \"\n", + " \"No finance follow-up is required for that invoice.\"\n", + " ),\n", + " \"metadata\": {\"customer_alias\": \"Northstar Renewals\", \"invoice_id\": \"INV-48290\"},\n", + " },\n", + " {\n", + " \"label\": \"milan_renewal\",\n", + " \"content\": (\n", + " \"Milan Office Supplies experienced a renewal delay because its approval workflow \"\n", + " \"was waiting for a regional manager. No reconciliation error was reported.\"\n", + " ),\n", + " \"metadata\": {\"customer_alias\": \"Milan Office Supplies\"},\n", + " },\n", + " {\n", + " \"label\": \"error_reference\",\n", + " \"content\": (\n", + " \"ORA-27102 can indicate memory allocation pressure. Database memory settings \"\n", + " \"should be checked before rerunning affected batch processing.\"\n", + " ),\n", + " \"metadata\": {\"error_code\": \"ORA-27102\"},\n", + " },\n", + " {\n", + " \"label\": \"customer_alias\",\n", + " \"content\": (\n", + " \"Northstar Renewals maps to the enterprise account Northstar Renewal Services Ltd. \"\n", + " \"Finance operations owns escalations for that account.\"\n", + " ),\n", + " \"metadata\": {\"customer_alias\": \"Northstar Renewals\"},\n", + " },\n", + "]\n", + "\n", + "for item in sample_memories:\n", + " memory.add_memory(\n", + " item[\"content\"],\n", + " user_id=USER_ID,\n", + " agent_id=AGENT_ID,\n", + " metadata={**item[\"metadata\"], \"label\": item[\"label\"], \"demo\": \"hybrid-search\"},\n", + " )\n", + "\n", + "memory_inventory = pd.DataFrame(\n", + " [\n", + " {\n", + " \"label\": \"renewal_blocker\",\n", + " \"exact_signal\": \"INV-48291 · ORA-27102 · Northstar Renewals\",\n", + " \"semantic_context\": \"Failed reconciliation blocked a renewal\",\n", + " \"evaluation_role\": \"Primary target\",\n", + " },\n", + " {\n", + " \"label\": \"similar_invoice\",\n", + " \"exact_signal\": \"INV-48290 · Northstar Renewals\",\n", + " \"semantic_context\": \"Neighboring invoice, already paid\",\n", + " \"evaluation_role\": \"Exact-ID distractor\",\n", + " },\n", + " {\n", + " \"label\": \"milan_renewal\",\n", + " \"exact_signal\": \"Milan Office Supplies\",\n", + " \"semantic_context\": \"Renewal delayed by approval workflow\",\n", + " \"evaluation_role\": \"Semantic distractor\",\n", + " },\n", + " {\n", + " \"label\": \"error_reference\",\n", + " \"exact_signal\": \"ORA-27102\",\n", + " \"semantic_context\": \"General error-code explanation\",\n", + " \"evaluation_role\": \"Exact-code target\",\n", + " },\n", + " {\n", + " \"label\": \"customer_alias\",\n", + " \"exact_signal\": \"Northstar Renewals\",\n", + " \"semantic_context\": \"Enterprise account mapping\",\n", + " \"evaluation_role\": \"Alias distractor\",\n", + " },\n", + " ]\n", + ")\n", + "\n", + "display(\n", + " memory_inventory.style\n", + " .hide(axis=\"index\")\n", + " .set_caption(\"Memory inventory used by the retrieval evaluation\")\n", + " .set_properties(**{\"text-align\": \"left\", \"white-space\": \"normal\"})\n", + " .set_table_styles(\n", + " [\n", + " {\"selector\": \"caption\", \"props\": \"font-weight: 700; text-align: left; padding: 8px 0;\"},\n", + " {\"selector\": \"th\", \"props\": \"background-color: #312d2a; color: white; text-align: left; padding: 8px;\"},\n", + " {\"selector\": \"td\", \"props\": \"border-bottom: 1px solid #d9d4cf; padding: 8px;\"},\n", + " ]\n", + " )\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "36151b5f", + "metadata": {}, + "source": [ + "## Step 7 - Define One Reusable Search Path\n", + "\n", + "Every evaluation query uses the same documented `SearchScope` overload and the same Oracle-managed hybrid index. This keeps the comparison focused on the query rather than on changes in configuration.\n", + "\n", + "`OracleSearchResult.distance` is used for display: smaller values indicate a stronger match. The result metadata carries the human-readable label used by the validation summary." + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "9e31df8a", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Hybrid search helper: READY\n" + ] + } + ], + "source": [ + "async def search_memory(query: str, max_results: int = 5) -> pd.DataFrame:\n", + " require_hybrid_ready()\n", + " results = await memory.search_async(\n", + " query=query,\n", + " scope=scope,\n", + " max_results=max_results,\n", + " record_types=[\"memory\"],\n", + " )\n", + " return pd.DataFrame(\n", + " [\n", + " {\n", + " \"rank\": rank,\n", + " \"distance\": result.distance,\n", + " \"label\": (result.metadata or {}).get(\"label\", \"\"),\n", + " \"record_type\": result.record.record_type,\n", + " \"content\": result.content,\n", + " \"metadata\": result.metadata or {},\n", + " }\n", + " for rank, result in enumerate(results, start=1)\n", + " ]\n", + " )\n", + "\n", + "\n", + "print(\"Hybrid search helper: READY\")" + ] + }, + { + "cell_type": "markdown", + "id": "371b33bd", + "metadata": {}, + "source": [ + "## Step 8 - Run the Retrieval Evaluation\n", + "\n", + "### What to observe\n", + "\n", + "The five cases exercise three retrieval patterns:\n", + "\n", + "- **Exact identifier:** the query is only an invoice ID or error code.\n", + "- **Semantic:** the query describes the event without repeating its stored wording.\n", + "- **Mixed:** the query combines business meaning with a named customer or object.\n", + "\n", + "A case passes only when the expected memory is ranked first. The evaluation is intentionally small and deterministic; it validates the tutorial workflow, not general retrieval quality." + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "609474e5", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
Hybrid retrieval evaluation summary
categoryqueryexpectedtop_resultdistancestatus
Exact identifierINV-48291renewal_blockerrenewal_blocker0.4191PASS
Exact identifierORA-27102error_referenceerror_reference0.3757PASS
SemanticWhat blocked the Northstar renewal?renewal_blockerrenewal_blocker0.2534PASS
SemanticWhich issue delayed the Milan customer renewal?milan_renewalmilan_renewal0.2914PASS
MixedWhich invoice failed reconciliation for Northstar?renewal_blockerrenewal_blocker0.2152PASS
\n" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "evaluation_cases = [\n", + " {\n", + " \"case_id\": \"exact_invoice\",\n", + " \"category\": \"Exact identifier\",\n", + " \"query\": \"INV-48291\",\n", + " \"expected_label\": \"renewal_blocker\",\n", + " \"expected_text\": \"INV-48291\",\n", + " },\n", + " {\n", + " \"case_id\": \"exact_error\",\n", + " \"category\": \"Exact identifier\",\n", + " \"query\": \"ORA-27102\",\n", + " \"expected_label\": \"error_reference\",\n", + " \"expected_text\": \"ORA-27102\",\n", + " },\n", + " {\n", + " \"case_id\": \"semantic_blocker\",\n", + " \"category\": \"Semantic\",\n", + " \"query\": \"What blocked the Northstar renewal?\",\n", + " \"expected_label\": \"renewal_blocker\",\n", + " \"expected_text\": \"INV-48291\",\n", + " },\n", + " {\n", + " \"case_id\": \"semantic_milan\",\n", + " \"category\": \"Semantic\",\n", + " \"query\": \"Which issue delayed the Milan customer renewal?\",\n", + " \"expected_label\": \"milan_renewal\",\n", + " \"expected_text\": \"Milan Office Supplies\",\n", + " },\n", + " {\n", + " \"case_id\": \"mixed_invoice_context\",\n", + " \"category\": \"Mixed\",\n", + " \"query\": \"Which invoice failed reconciliation for Northstar?\",\n", + " \"expected_label\": \"renewal_blocker\",\n", + " \"expected_text\": \"INV-48291\",\n", + " },\n", + "]\n", + "\n", + "evaluation_results = {}\n", + "summary_rows = []\n", + "\n", + "for case in evaluation_cases:\n", + " ranked = await search_memory(case[\"query\"])\n", + " evaluation_results[case[\"case_id\"]] = ranked\n", + " assert not ranked.empty, f\"No results returned for {case['query']!r}\"\n", + "\n", + " top = ranked.iloc[0]\n", + " passed = (\n", + " top[\"label\"] == case[\"expected_label\"]\n", + " and case[\"expected_text\"] in str(top[\"content\"])\n", + " )\n", + " assert passed, (\n", + " f\"Unexpected top result for {case['query']!r}: \"\n", + " f\"expected {case['expected_label']!r}, received {top['label']!r}\"\n", + " )\n", + "\n", + " summary_rows.append(\n", + " {\n", + " \"category\": case[\"category\"],\n", + " \"query\": case[\"query\"],\n", + " \"expected\": case[\"expected_label\"],\n", + " \"top_result\": top[\"label\"],\n", + " \"distance\": float(top[\"distance\"]),\n", + " \"status\": \"PASS\",\n", + " }\n", + " )\n", + "\n", + "evaluation_summary = pd.DataFrame(summary_rows)\n", + "\n", + "def highlight_status(value):\n", + " return \"background-color: #e7f4e4; color: #1b5e20; font-weight: 700\" if value == \"PASS\" else \"\"\n", + "\n", + "display(\n", + " evaluation_summary.style\n", + " .hide(axis=\"index\")\n", + " .format({\"distance\": \"{:.4f}\"})\n", + " .map(highlight_status, subset=[\"status\"])\n", + " .set_caption(\"Hybrid retrieval evaluation summary\")\n", + " .set_properties(**{\"text-align\": \"left\", \"white-space\": \"normal\"})\n", + " .set_table_styles(\n", + " [\n", + " {\"selector\": \"caption\", \"props\": \"font-weight: 700; text-align: left; padding: 8px 0;\"},\n", + " {\"selector\": \"th\", \"props\": \"background-color: #312d2a; color: white; text-align: left; padding: 8px;\"},\n", + " {\"selector\": \"td\", \"props\": \"border-bottom: 1px solid #d9d4cf; padding: 8px;\"},\n", + " ]\n", + " )\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "9c819296", + "metadata": {}, + "source": [ + "## Step 9 - Compare Top-Result Strength\n", + "\n", + "The chart below makes the five successful cases easier to compare at a glance. Distance is shown only to explain this run: **smaller distance indicates a stronger match**. Different query shapes naturally produce different values, so this is not a cross-system benchmark." + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "bf0ba462", + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAA4QAAAHBCAYAAAAxRHXNAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjExLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlcelbwAAAAlwSFlzAAAPYQAAD2EBqD+naQAAb19JREFUeJzt3QncFfP////31b7v2hephBYqISFU9kIiW5IteyX7B0WkCBGy75G9FSmFSigqW6koLUr7vtf5357v33/Od85cZ5lzrlPXdZ153G+3i845c2be8545M+/Xe5usUCgUMgAAAACAwCmQ2wkAAAAAAOQOAkIAAAAACCgCQgAAAAAIKAJCAAAAAAgoAkIAAAAACCgCQgAAAAAIKAJCAAAAAAgoAkIAAAAACCgCQgAAfHj44YdNVlaW/bv44ov3e54dffTR4e298cYbuZaOTEP+AUAkAkIAyBD9+vULBwp+/q6//vrcTnJGmDlzZjhPS5UqZfKD/JhmAMD+QUAIAAAAAAFFQAgAGdRCGAqFwn8LFiyI+HzKlCkRn7/wwgu5llak7r777gsfwxEjRpCVAIAcISAEgIDatm2bGThwoB2rVrp0aVO0aFFz8MEHm+7du5s//vgj7rirpUuXmvPPP9+UKVPGVKhQwVx22WXm33//9b1t7/r++usv06FDB7u+o446Krzcjh07zODBg80xxxxjP1MaGzRoYPr06WPWrFmTbX8eeOAB06RJE1OyZElTokQJ06hRI9OtWzfz+eefm3379oWXVXdZZ/s333xzxHoOO+yw8Gdjx46Nux8nnHCCadmyZfj11q1bI7rlzp49O2FezJs3z+67joHy8qqrrjIbN270nXfJ7r/fNL/zzjvZuhmre+mRRx5p7r//frNly5a46Vq1apW59NJLTdmyZU3VqlVNly5dop4ju3fvNs8995w56aST7P4XL17c7sMjjzxiNm/eHLFsMueDX4nO5VtuuSW8X9qOm4LyGjVqhD//4YcfEm5v7ty5UY93rDGjqZ6ryeRVot9juvMAQB4TAgBkpAULFoR0mXf+pkyZEv5szZo1ocaNG0d87v4rWrRoaNSoUeHl+/fvH/6sXbt2odq1a2f7Tt26dUNr1671lTb3+tq2bRuqXr16+HWTJk3sMuvWrQs1a9YsZhpr1aoVWrx4cXidF1xwQcxl9Tdjxozwsj169Ai/f/PNN0ekrWHDhuHPxowZEzXNXbp0se+1bt067jZnzZoVNx8WLlwYqlChQrbvHXPMMTYfnNevv/563HQks/9+0/z222/HXU7HZseOHVHT1b59+1D9+vWjfmfv3r3h72zYsMHua6xtPProo+Flkz0f/Jx7fs7lefPmhbKysuz7lSpVCu3cuTO8rqlTp4a/c9hhhyXc9vz580Ply5dP6nincq4mm1eJfo/pzAMAeQ8thAAQQD179jS//fab/bdakfRvtRrceuut9r2dO3earl27mnXr1mX77sSJE23Ljz6bMWOGqV27tn1/0aJF5qGHHko6LV999ZVtHdL31drwyy+/2Pd79eplZs2aZf+tz9UFdsOGDfZ9p2VHrStOK9PIkSPtv+vWrWsWLlxoW0j+/PNPM3z4cNOuXTtToED0W16s9/2YOnWqzQOHWubc3XLdrZ3R3H777eE8btq0qW0tVMuaWn5+/fVX3+lIZv/9pvnyyy+PeF/r0/FQ653o3++9917U9EyYMMGcffbZ9pxS62ThwoXD33G3IPXu3dv8+OOP9t9qLVMX2PXr19tz4fnnnzeVKlUKL5vM+eCXn3O5YcOGpm3btvbf2h8nn+XDDz8M/1stsYnccccddv9SPd5+z9Wc5FW032M68wBAHpTbESkA4MC2EG7ZsiVUpEiR8Pvjxo0Lf2fXrl2hypUrhz97+eWXs7UgqHXB3cozbNiw8GdqWZDt27dHbZno27dvtvVVqVLFLu/mTeMff/wR/mz37t2h4sWLhz9btGiRTY/zXrVq1UKfffZZaPny5THzxt3q0rt375RbCEUtb877JUuW9H18tm3bFipcuHD4u2PHjg1/tn79ettK67eFMNn9TzXN8thjj4W/261bt6jp0nmg4+Q48sgjw5+99dZb9r2tW7dG7OOrr74ac5vJng/xJHsuy8iRIyNaP2Xfvn2hmjVr2vcKFCgQWrp06X453smeq6nkVaLfY7ryAEDeRAshAASMxgft2rUr/LpZs2bhf6slRy2G7vFOXmohcrdUaEyZQ2OvNJYtGc2bNzfFihWLm8YjjjgiPEZJady+fXv4s99//92mRy09smLFCnPWWWfZcU0aA6XWMY3JUmtHulsIc2LJkiW2Zc+hFiNHuXLlTJ06dXyvKyf7H4vGFqqVTOMNlZ6CBQva/L/zzjvDy/z3339Rv6tWxkKFCoVfKx3u9TrHWC3RDqUzlmTPB7/8nsvnnHNO+HioVXHx4sVm+vTpZtmyZeG016xZM+62/vnnnxwfbz/nak7zKtrvMV15ACBv+r+rNQAAPqhg6ZZsoOHlDhZS4Uxuoi6GCmY//vhj281NE5LoT13g9OfuMufmnmxGNm3aZA4Eb77lNF9T3f9YzjzzTDszbTzuAMdNE6YkCmRyet7E4p3sJh6/ea5gWJO73HPPPXaZV199NWI7frpKereVyvFO97kaLa9i/R7TkQcA8iZaCAEgYOrVq2eKFCkSfu2MNXIK+O4ZRjW2yUtjndwFU2fMn1SrVs3ObqkWBvf4M+dPj8ZIJY1q9Yi2Pv1p9krR8hqTpjFyKiir9WrQoEHhdXzwwQfhfyuNDveMnsuXL4/Z6hWLu2CfTJCj1hZnbJ3MmTMnIk1qQUxGMvufKM3z58+PCAZHjRplW5aSOYaJ1K9f38586VDQms7zwQ8/57LjmmuuCaf39ddfNx999FE4gNIspYlofGIqxzvZc3V/5VU68gBA3kRACAABo0lELrzwwvDru+++2waBmlhD3QGdQqYKeZ06dcr2fRVc9XgDtTj99NNP5tFHHw1/5l5vOtOoCW5+/vlnO7HJypUrzXfffWefx+dMdCEnn3yyeeqpp+xjE5Q2PSLBHXC4W7MUjDg06Ym6zqmLYI8ePbK1wiSixyo41MVQAbafwFCPV1DXTse9995rJ4FxJvdxd6f0I5n9T5Rmd+Ci4LF8+fL2/5MnT7aPiEgHBTqXXHJJ+LXOPQUYTnD07LPPmhdffDHl88GPZM5lTXDjBFAKxpyukhdddJE9lvvreCd7ru6vvEpHHgDIo3J7ECMAIHceO9GoUaO0PnaiTp06dr1+xJqgxU3ratq0acw0Ott0VKxYMe6ymjDEve5y5cplW8b7uAQ/k8poUhKlw7supcfPYwiipaNly5ZJP3Yimf33k+bjjz8+2+eaFOWSSy6JeESBn2Papk2bqOnQZCotWrTw9diJZM+HWHJyLv/444/ZltVjF/xK5Xincq4mm1d+fo/pygMAeQ8thAAQQBUrVrTT/w8YMMBOIqFWBbUKqVubxgKptaRjx44xv6vuhOedd54dK6bWJrX0TJs2zX6W7jQ+8cQTplWrVnY76gqnNLZp08YMHDjQtlg51PLRt29fc9xxx5nKlSvbMU9q0dBkF5oaX+Of3OtWF8VTTz3VTuhx0EEHmdtuu82MHj3afi8ZGh/36aef2hY6tcolQw8KV77pEQ36rlriunfvbr788suIbn9+JLP/ftKsRwuoi2D16tVt64/Wq8dJ6EHn6aK8V7qfeeYZ07p1a/tarZqNGze2E9rceOONKZ8PfiR7LmuCHff+q/VO6d6fxzuVc3V/5FW68gBA3pOlqDC3EwEAyNsefvhhc//999t/q8uYnhcHBP230L9/f9v9Mh2OPvpoWxHjjM+78sorTdDyAEDuoIUQAADABz064p133rH/1sRJV199deDyjTwAMg+PnQAAAIhj5syZtqukm7ptaibSoCAPgMxFQAgAAOCDxl7WqFHDjvvTeM0gIg+AzMMYQgAAAAAIKMYQAgAAAEBAERACAAAAQEAREAIAAABAQBEQAkCG2LBhg8nUR8tqvzJ1/9i3/CmTj1um7x/7lj9l8nHLbQSEAAAAABBQBIQAAAAAEFAEhAAAAAAQUASEAAAAABBQBIQAAAAAEFAEhAAAAAAQUASEAAAAABBQBIQAAAAAEFAEhAAAAAAQUASEAAAAABBQBIQAAAAAEFAEhAAAAAAQUASEAAAAABBQBIQAAAAAEFAEhAAAAAAQUASEAAAAABBQhXI7AQCA9Fgz7hqzbd8qk2VCGZelIZNlthWonJH7x74F+7hV6zolrekCgGTRQggAAAAAAUVACAAAAAABRUAIAAAAAAFFQAgAAAAAAUVACAAAAAABRUAIAAAAAAFFQAgAAAAAAUVACAAAAAABRUAIAAAAAAFFQAgAAAAAAUVACAAAAAABRUAIAAAAAAFFQAgAAAAAAUVACAAAAAABRUAIAAAAAAFFQAgAAAAAAUVACAAAAAABRUAIAAAAAAFFQAgAAAAAAUVACAAAAAABlScCwt27d5s9e/bkeJmcLJ/qd9LhQG93x44dJhQKpZS2dKU1t/L6QFI+79u3z/frdG1nf9u7d6/ZuXOnyRTJ/B4A4EDStenxxx83jRs3NrVq1TJdunQxS5YsifudKVOmmAsvvNDUq1fPHHXUUeauu+4yGzdujFhm06ZN5rnnnrOfV6pUyaxYsSLbejp37mzXcdBBB9ll9HfFFVckvS0AAQoIVbjftWtXtvf1ngr/0QphKljKmWeeae67776469cFp1evXr7T42edOd1GuhyI7SqvL7/8clOqVClTrlw5M2vWrJTSlq60du/e3Vx//fUmkxUvXtxMmjTJ9+tUg79U1pOK+fPnm5YtW5qSJUuaRo0apX39CjKjBZqxri2p8Obfli1bbP798MMPaVtnuhCkAnj00UfNI488YgYOHGi++OILe81q165dzEo5XadV9rn44ovNhAkTbND32WefmU6dOkUs16NHD/P777+ba665xqxduzZcHnNTYKd7/ty5c828efPsn9aX7LYABCggfP31103ZsmUjLlL6t9478sgjI5b9888/bSFMNUt+FSlSxBQuXDhdyc21beTWdkeNGmU+//xzs3jxYluAbd68eZ5JW6YqWrSoKViwYMqfRwuM9Lv59ttvc7SeVD388MOmdu3aZuvWrWbhwoVpX3+VKlVMsWLFzI8//hjx/t13321OOumkHK8/Vv7lpXVu377d9OvXz9StW9ce16ZNm5rJkyenZd0A8hdVpqt18H//+58555xzbEXcm2++af7++2/zwQcfRP1OgwYNzDfffGMuuOACc8ghh5jWrVubwYMH20rDpUuXhpd79913zfPPP29bHuPRNdlpHdRf6dKlk94WgLyvULpWdMopp9hAQzXtTuHt+++/NxUqVLAXr//++88W+OTrr7+2F5lWrVr5Xv/w4cNNVlZW1M9UO1+gQM5jW+82dDHW60KFCsUsDGrbCprcVNOmP+/7sdKZk33zs+/aDwXhderUsS2Eam1R2pR+p9umjkc08dKWbFrU4uF3Xd6812tvYOpnffGWcVpgon2uc1kFcn2W6nY2bNiQ7RyI97kqULQe5WG07zmVLTp+Sp+zXKztpHP/tG0Fge3bt7fHQt9x/y78bsuphY4VwJYoUcLceeed9hqRSLLbjJZ/0dYZLy+8n8c6JomOZaw0qpJMr2fOnGnz4pZbbjHnnXeeWbNmDRUzQMD88ssv9vquFkGHgrJmzZrZa0XXrl2zfSfa9UsVTd77vN978YgRI8xHH31ky29nnHGGuffee20vkWS2BSBALYT169c3NWvWjKjN1r91IVM3M3cBT+8ff/zxtjDkUMCorgkKIMuUKWOuu+66iDFm3q6KCj7UlUItFipwqSb9q6++ikhTonV6ebehbqfXXnttzHW88cYbpkaNGtnWedlll9l+/k6hT4U6ddNUQNOkSRPbUhdvu4n2TQXNAQMG2PzW56qZe/bZZ2PuV7du3cwDDzxgZs+ebdOhm4n07t3bvtafAsW2bduaP/74I27avPykZdu2bfbG5dQ0agxCojEGynt1ZVFhWAXjs88+276/atUqc+mll9r06qZ03HHHmenTp0d8V8uoe6xap7Wc1rF8+fKIbi6nnXaaPf/UutOhQ4eIMRlON0LVdB599NF2+9q/t99+O6ntJOrK6f28YcOG4WOhm++NN95o0+LQGA1RTbGW0+8q2nrStX9uFStWtBU8Otba9v3335/UtlTLrdptbWvYsGExt3PTTTeZ7777znY7iiXVbcbKP1HhKlZe6BxXDb3TgqlxMs5vONY6Ex3LWGnUfvXt29fmtz4//fTT7bL6DQEIln///df+36lMd+h1tDF/0ShA0zVF1xKNBUyGrmODBg2y10f1EFFgqGtdrO7sOdkWgAyaVEathN6A8OSTTzZt2rSJeF/BoZZ1UwFM3Q5Wrlxppk6daj788EPz2muvxdyWaqmGDBliXn75ZduFTS1ZY8aMydE6o4m3DgVLmzdvNuPHjw8vr9ejR4+2gYKTznHjxtl91mcKFs8999y4Xe4S7ZsK4++//74tNKsl4r333rMX61gFenUN6d+/v+0mqgBV4wZk6NCh9rX+FMioYHr++edHHfMZi5+0aJD5jBkzbECqLqvVqlUzY8eOTbjut956y6Zn/fr15ssvv7SBt240CgQXLVpk31dwqVpLJxBT2hXY6rVaWTQ24qqrrgrnn9ahAEJBnG6oSo9adzTmwXuTU/6/+uqrdvC9jokC1GXLlvnaTiqc7rxKj27AyjMF8g5n28oLLTdnzpxs60jX/nkpKDnmmGPszV7bVoVFMttS1yQdT3335ptvjpkHqlDQ2BZ1E402Li8n24yXf/HyQoWgl156yUycONEWePR7VBfseMck0bH0ky86t+655x6bH9pfL/3elF73387d2ccBAcifnGuat0eFemf4Gbes66UqUFUBq2E9yVLlrsorBx98sK2k1bVPZZlow31yui0AGRYQqhVBhRsVnNR91BsQqgVKLXennnpqxHdV8NfAZKdFTC1CaimIRkGSAia1VihAUEuBWt70XqrrjCXeOlRI02tdJB2ffPKJTY9q0ZQHGmStAEmtCmoFUEFXaX3mmWdS2jet88knn7StO0cccYS9KajFTxO0aGxBqtQaoXT+9ddf2VoJY/GTFrVsqDCtQfGHH364bTHRvmm2tEQU6Kl102lJVhCpgrYCWbW+qDueCu5qpfn000/tMgrG1T1WQbDGN6hFp2PHjuEJbPT5P//8Y1588UXbClO1alUbDPz888/Zxmop+NH4V7XsqoVH54CCPz/byWkhQK3Dffr0Ce+XX+nav3RvS0FWixYtfHVTUuCk46xA6UBtM15eaCyMegLod6hzThUnL7zwQlqOZaw0KhjUeBx10Y3V+q+gXNcg99+Lo//0lS4AeZ/TyrZ69eqI9/U6UQucuqGrAvqnn36y10ZVxOaUKpUVnGpymf29LQD5OCBUkKdaawVM+lO3Bk2OoO6hatFR9wddLBQUqLXB2zLgpgK/+s5Ho4uRtqMCUzzJrDPVdaglUK0FTncwBYeaqllBjMZOqoXA3TVNjj32WDtrVyr7pu8pEFMrifJRA7zVlVUzkDndS/xSa4a6qOn7ClYrV65sL+x+B4P7SYuTB+5JbHRDUYCciAJINxX6VfuoY6BtOdtTAOu01KgVUscs1g1JaVYAqS7ADhXYFVh4j4n32KvA7Rz7RNtJhQIgDfBXcKn9UutnsgPz07V/6d6W91jGo4LO7bffbgNDVS4diG3Gywv1BFi3bp0NGBU4qnY80Qygfo9lrDS+8sorNuBVN9JY43LVeqjfg/uvR8eGvvcZQN6mSmhdQ9Q7yaGyhu6FKkckmlVcwynUoqdyWDrofq51u4PR/bUtAPk4INSkJboY6KKgP7UOirr4qRbcef+EE07INlGL3wHO7mUTdZlIZp2prkMthAr+Ro4cabuVaiyX013UKch506nXsSbV8Ltvar1wuns6f35b9kQFWqVdBdIFCxbYLpBqzVMLSbLPB4yXFicPvAVoP91dos1uqoK7d1v6UxDq5F+8dSs90T6PdkziHftE20mWWtM1PlUzTKorrILojz/+OOpU4PGka//Sva1kZ6pVi5ryQK3BB2Kb8fJCrdlqDX7ooYds0HXJJZfYa1usR2EkcyxjpVGTyOh6Go+uOwo23X9FC+//2WYBHBgqO1199dX2/qZhJrrXaQiGKmB1HXKoW6fzfEBdC9WzZtq0aba85a3s8ktjtTV2WuUaUa8NDYvQdUm9l9K5LQAZ+GB6ZxyhM37QoW6jCpY0RbG3u2iyFMSoRUvrym3qWqYWQbUMajYudS1zZlnVxVFdMdWN1k01abGe45Zo35zP40264Ye6pOlPE96oJdeZ2TCZ8YN+0uLkgfs5b9qGajiTpUoF1VB6u6t4l1FrdKwH9yrftQ5NCONQN1m9TubZeom2kyyNMdPETDqXlKfinSxHAY8ConhBYrr2z4/9uS0VhNRCqG6RCqrSsU0/+ReLzmEVutR1W8dKj5nQOR1tnX6OZSLqiq3rCYBg05AMlZnU40CVPuqhoHuueso4VFGlMcTO9UflEY1rV08s9yMjFLg5dG3Ve7quOa2Req0x06LKffVSUC8iXY/VQ6J69er22udc1/xuC0CAHjvhDgidCUXcE4soIFTXK42R804ok0rhTFPT64GouvAo8FQNvgYya2KIA0195zXBiAqqGlTttDaoBl8zeaqWTRfXQw891LZ4qEUu1uQjifZNn2t9miRG3TbUyqcCs24QuimoVcIPdbFzuqRpHJPSrprIZPhJi7q7aEyWtqGB6apd1PJ+Z0hz07hMdZPReaTJODRuUenWGEUdA+WVltGNS7O8Kq/VuqNJf9TlT7OlKo2aOU01nRrHqdZQjUNUt+YTTzwxqbTE206yFMyoRlbjJPU4Fq3r6aefjlhG55XyTwUC5YMqI7yPNEjX/vmxv7elVjYFYCpwOF2Mc7LNaPnnh8a8qkeDCk7qVq2xgPquftPR1unnWCbSvXt32w3az+M3AGQu3UM1kZ3ucxpO4jzywU3DVpzeOKqs9I45dLgnp+rZs6edRd3Lec6gei/cdttttlyg65oTBLr53RaAgLYQqtbc6T7qUDdRUYHK+1D0aA8/12t3gc27jMbyKODQnwItBVDuCT38rNPL+x2/61CLoPZVk1043UUdDz74oLnyyittdw4VZFW7phkJNe4p1X1TcKUCpiab0IQmetyBxuzdeuutMfdNBVr3Yz70Wt3YnG4eeiyEAjcFie6ud4nyxE9aNKHMWWedZSddcboLK5CK16UvWt4rXco7TbShbipqhVELpwrdTjCgZSZMmGCDNRXgFUiotdp5XpM+16yvCmZVo6n06FhoMiCHCvnKK+/YLd2YnbxJtJ1oD4yP91q/G03qo5kmNUZOk6ZoBlf3MRPltWa61dhFZ2yqez3p2r9o9B13V++cbCsabd+9fv1bwZiz7XRs05t/fvJCvwtVdOj81TmniYQ0uY0eTxFtnX6OZaJ8iRbsAwguXQ+jBYNO8OUEclrO3VLn/nPfUxXgRVvGe88RXW9jpcnPtgDkfVmhRLMjAADyhYXDO5sS+1aZLJN5l/WQyTLbClTOyP1j34J93Kp1zf4Yh7xAxUP19lHAmY45GfIS9i1/yuTjlnEthAAAAACA/IGAEAAAAAACioAQAAAAAAKKgBAAAAAAAoqAEAAAAAACioAQAAAAAAKKgBAAAAAAAoqAEAAAAAACioAQAAAAAAKKgBAAAAAAAoqAEAAAAAACioAQAAAAAAKKgBAAAAAAAoqAEAAAAAACioAQAAAAAAKKgBAAAAAAAoqAEAAAAAACioAQAAAAAAKKgBAAAAAAAoqAEAAAAAACioAQAAAAAAKqUG4nAACQHpXOfsWULVvWZGVlZVyWhkIhs3HjxozcP/Ytf8rk4wYgWGghBAAAAICAIiAEAAAAgIAiIAQAAACAgCIgBAAAAICAIiAEAAAAgIAiIAQAAACAgCIgBAAAAICAIiAEAAAAgIAiIAQAAACAgCIgBAAAAICAIiAEAAAAgIAiIAQAAACAgCIgBAAAAICAKpTbCQAApMclg4eYlVu2mFAGZmiWMaZqqVIZuX/sW/6Uycct2v7NHjI4t5MEYD+hhRAAAAAAAoqAEAAAAAACioAQAAAAAAKKgBAAAAAAAoqAEAAAAAACioAQAAAAAAKKgBAAAAAAAoqAEAAAAAACioAQAAAAAAKKgBAAAAAAAoqAEAAAAAACioAQAAAAAAKKgBAAAAAAAoqAEAAAAAACioAQAAAAAAKKgBAAAAAAAoqAEAAAAAACioAQAAAAAAKKgBAAAAAAAoqAEAAAAAACioAww+zdu9ds2LAhz2zLWSYUCh3Q7aZi27ZtZvv27Ul9Z+vWrWbHjh0mU2Ta/uQHOpf37NmT28kAgBwZMWKEOeqoo0zFihVN69atzTfffBN3+d9//91ceeWV5pBDDjEHH3yw6dKli5k/f37EMv379zelSpWK+KtXr17EMmvWrDF33XWXadSokalWrZo5+eSTzbhx42Ju99lnnzWlS5c2t956K0cc+P8REObhQuLu3bsj3tNrva+AyE0F+E2bNtl/T58+3ZQvXz7HhXo/gZyfbf366692mbVr1+YoPcluNxXXXXedueWWW+IGS96A8dxzzzX9+vUz+yM4jRYopDPA3h/7E22diE/n8tdff53t/V27dpF1APKFL774wnTt2tXcfPPNZvbs2eaUU04xZ5xxhvnzzz9jfuf66683p556qpk4caL56quvzL59+2wwpwDPsXPnTtOsWTOzcuXK8J/KFW4DBgwwtWrVMiNHjjQ//fSTOfPMM03Hjh3NtGnTsm1TaXv88cdtUEnlJ/B/CAjzqCZNmphHHnkk4r2HH37YFh4//PDDiPe7detmzjvvvLRuf9asWXZbGzduTOt68zPVXt5zzz0HZFs33nijzf///e9/UYPh//77L0/uz4HMo0y0ZcsWc/fdd5saNWrYmvBDDz3UjB8/PreTBQBxDRw40JZDrrnmGhucqbxSt25dM2TIkJjfmTJlirniiitsC6ECtFdffdUGfJMmTYpYrmDBghEthCVKlIj4/Mknn7SBaIMGDUz16tVta6FaKadOnZqtwvLiiy82zz//vDnooIM4ooALAWEepdo1b6uBXuui6X1f3TK0fDTeVkZR65JamfSnC6SXaulUMBUFhFpOLVaJqCYvGaqd07biUQuZn+50WiZaq2qiGkC1wvhpbdP+axvaRyfvvGnXa2/rrVcyNZK68T3zzDNm6dKlCZfVtmO1zLm7gWof9Dod++NtwYq1TqXLeR1rfdHSmMpy2l60c17c24+1TKL1eLvUOq327rxzL6OWe33utOAn8sMPP9iAf968eXY9bdu2NZ07d6a1EECepevq999/b69Xbu3bt88WlMWzefNmez/2Bnxq9atataoNHC+99FKzaNGiuGlR11WtS9t3u+mmm2wazz77bN9pAoKCgDCPUoCnC6xTyNf/VVhUi9HkyZPDy/3xxx+2tUjdLtxeeeUV2ydfQYUuou4at9WrV9vP9KdassqVK5u+ffuGC7XLli0zHTp0sP9u2rSpXS5eV8phw4bZGkFtSzV0qvWL58svv7R9/dWHXxf+yy+/PFuBWS2U6jpSrFgxW9PXvXv3mIVqFbiVXxdeeKENRnRDeOCBB0ylSpVM2bJlbY3h4MGDI76jm4WW1/YrVKhgLrroIrN+/fqYab799tttl5bXXnstnHd//fWX/Wz58uU2v7Qe5YFqPN3Bkp/0RKMxGEceeaT9biwKwq699lqbl2XKlLH5P2rUqGzdQLWMuu9oGbXi5WR/PvroI1sxoXXp3Ln33nvt57HWqXPWea31ab/mzJmTMI3RxFpuxYoVplOnTvZ4Ko81jsU9fkUVHAq0VIvduHFj+10dD9VIuyVaz5133mnPV4eOo9b72Wefhd9r3ry5efPNN+2/W7ZsafdbLX5a31VXXRU3OFRhRbXbOp6FCxe257V++3RtApBXaUiI7r26H7jpta6pfvXp08fUrl07ojyjMooqRmfOnGk+/fRTW0l93HHHRXQrdcoMur8ULVrUXH311ebtt9+212LH8OHDzY8//ujr3gsEEQFhHqWCoC6w6iIo3333nR0sre4OixcvNv/++699X8FhyZIlzTHHHBPx/bFjx9qLnwIGFaLVt99pHdFF2mmx0ecK0FSIf+ONN+znuiA7Qec///xjl/MWnN3eeecdmz4FWdqWuo3EKvQq2NTnl1xyiS2kz5071/bpV3cPh1rEtP+HH364veirC0mbNm1sQOzlfKbaQw0i1w1BAYoK6EqT8vDzzz83Tz31lN1Hxx133GGD6QULFth1HHbYYRGFei91MTn99NPNDTfcEM47BV+i2kjdgNatW2cDHaXDnV9+0hPLY489Zt566y3z22+/Rf1cAYpajFWDqmOpGlC1KHnHbbz77rs2qNYx0rmR6v7ouKqGVuMMFahoO+XKlbP/j7VOdedxXmudCggvuOCCbC1w3jTG4l1O61FNsALtVatW2fc1WcA555xjlixZku1c/fjjj21eKW+VVmcZP+tRJYUCRKdVWXmvc89ptde5pEkRtJwoX7TfWpfGvajl77777jN+6Len46vfhgJYL51LOh7uv31MTgMglxQoUCDba7/j3VXxqeu5hsS4Wwh79uxpr/c1a9a0FaQffPCB7R2iSm83fabrr1oP77//fltx51yX//77b7seBYXFixdPy74CmYaAMI9yWlScwEz/VyFTFzMFf+73TzjhBNua4KaAQ4Gf+t6rdU8BpLdwLGp50HYUoI0ePTqltA4aNMi2EKo179FHHzWFChWyF95YrYnangrFqsnTGAMVzFVQVyHcWUatN5oJTMGG9lkzkXm7f6j1ScHFsccea95//31TpEgRW9BXbaJmJlPBXgVxbUMtM9qGaBkFYxqjqc+UDgU4derUSWn/1aKkIFc3P435UncUte4620qUnnhOOukkO0Be48q8FJC9+OKLdqyGAlqdA7169bID8IcOHRqxrNah1jSdDznZH9UEK3Bq1aqVycrKsq1jClo05jURFQz0XbUkqlJDM8ylkkbvcmPGjLGtmjqeSpPyXEGxglFNMuAtdDRs2NAup2Ogc1bBtN/16DeoPFCArkKJgny16Ll/jwoQtQ037beCOgWgfn5n+q3q3D7//PPtbzka/dbU6uj+++s7/92zACAddB/Q/Ue9j9z02ttqGI3uYao4VMWpt3LbSxXgui95ZyPV/UoVwiqL6H554okn2jKEqHJcPYD0njMOUcGiWhzVG8MpewBBVii3E4DY1G3CXdDULJiiFjG9VkuNWivU2uWloMvhtC44E8Soa6gCMgVFaoHTxVGtDerGmQp1q3PopqAuebFmFtP77m4cTrc6BQtqrdPNQy0puikkCgzUiqjg4KWXXgq/pxYYBblqSVWh3k2BmFNbqAK6O926maiGMRXuvHbyWy2hftOTiLo5Km061u48UUCsoKRFixbZ8tOb/94AJdX9UZq1Lzo+ClZ0DNSCpsA9FnX1ue222+xNWenXOaLWaq3TfQz8ptG7nFqYFWhHC+i9N/po++b8LvysR92X1HKtwoTTDVW11wqK1RKo953WQdG5qQoPBcCq9dZvz0/3T/02VYEQKxgUTd6jfHW7cMBjZhWPDQFwAOmarvvQt99+a7v0O1ROUffOeFQBp8ot9URRBWgiGp6ge7jKQfHofuv0ilLFnu5TbhoWoeu3uverDAQEHS2EeZgK2ypEq5ZtxowZ4YKmExAqcFJAF21CGW/w4aZWJV0EdQF2JsVQ4TLVZ6F5JwrRetRKGI3e927Hee18R8FZokk/RC08am1xWnjcNJDd6abo/GmMgThBVbR0pyJeXvtJTyIKsNXlV0GHm5Nf0fLTm//eFuSc7M97771nzz8FRi+//LIdo/rzzz9HXVYBkG68KhRorKsmSlELW7TzwG8aoy2nIM6bv/pTzXMy++ZnPfodav8V/Om3p5Y5tZCqMOQOCNV6qC6nmmXPeTSMursmmnhItKzyNR61bCugdf8ViPG7A4D9Sb1T1N1T3T5VwazWOfWkcA8H0VwF7mcIqneRgkG1DLor0tx071Nlne4XuodoFlNV3KkizqHuob/88kt4orHnnnvOjme/7LLL7Oe633ifZahygPO+n3s4kOkICPMwFTYVGOmiqUkpnJaL448/3o6z09gyFUa9LW6JKIBS66Nq9JwLofd5Pep+KX4Kr844R9HFWBdmBTHRqOCssYDu9SpYUiFf3UCcFi5nvF086uapAre6kjoByRFHHGEv8Or+F4sK2mqtcadbtY6xghp3nvjJDzc/6fFDXU6Vr+5Hjjj7obxyqKVVxzJW/ud0fxxqsVTXT7Vaqruq00XYu051Vda4jh49ethzVVTJkc4Hsau1UuNGvF1Q99d6nHGETjdu5z0Fyu7xgzqf1N1UNdNOgB7tuVjRKAD1jpEBgLxK3fjVm0Xjz3VfUhdQDeVw92DRPd09M7S6dqqyTMMS3MGavutQjxQ9r1BdO1VGUFCo66i7p4iWce4x6mWkCm+NNVTLIAB/CAjzMAWBKlBqsg537Zn60Osiq/fVxcLPuDA3zRyqiWR0UdXEFQqs9NpNE8uoEKuHzarvfbzHTmhmMAV1CxcutLV3uqDrAh2NxlCp9UO1hur2oRYVBRbOc/dEF3/ROhQEab1KY7SxVw8++KCdSKVdu3a2AK4xYaqFVDcUjUXUWCxNjKJaSA00d1pWFEiqVVSFeqVD21TgEo8CMAUz6uoY7TEN0fhJjx8aF6E8e+GFF8LvKfhSd2HN4qnjp26Jzvg81dYmksr+KPjUjV8BkfJLx09dfZ3uxt516uasP7WSaXyeZqBVy246nXXWWXYcrW7+EyZMCG9HNcvuGXnTtR79FvWbUA200zqv91QAcY8f1O9MXYY1K6vWpW6g3rGdsah7uLeLEwDkZbrvKGBTkKf7qsajuz300EP2fYda+nSfcD90Xn/uWc2dMezqoq/u/Xouq8bJu+laqQpelS0UcOoepInL4lElrd/rMRAE9C/K4zRGTrVdCnjcTjvtNDtDp2Z1dFMQp1oydxcIdcHUe07gqOBLQYO6WagFUgOtVVPnfl6QxoTpYqlARhdnjRfzzjTqbEtjpPR9jWlTYKALtgIh0Ta1jDP7mB5loMK1lldLp2r9VOBW0OTQhDIKVhXoqJCuZTTpjf4dbR/VeiYqyKvriYIijb9StxEFkvq3bhjusZa6MalVUI9U0Pp109G/vc8/cuvdu7ct4GsSG9101I1Xwa931jKtQ0G7w096vLQO77gGzVaqGlfd9Jz8VFCp/FCAqyBFQYgeMaLKBEe0NKa6P5pMRsGejpdaw6pUqWLT5QR50dapCVmUB2rJ1ky56vqqdLu7fsZKo1e05ZQXqrhQq5pmklM+KChTUOVUpOhc0Tnj7Uqr95zWcD/rccYRqhVb++eMSVTFjNal36tD7z3xxBN2vK4KPWpVHTBggD0HvGnwdoNVfjOuBUB+FKv7v661zvVWkrnG+an4TqZyXGWUZJ+dDGSyrJDfOYEBAHnamff1Myu3bDGZeFFX9U/VUqUycv/Yt/wpk49btP2bPSRznuGnoq9aHL0V6JmAfUMq6DIKAAAAAAFFQAgAAAAAAUVACAAAAAABRUAIAAAAAAFFQAgAAAAAAUVACAAAAAABRUAIAAAAAAFFQAgAAAAAAUVACAAAAAABRUAIAAAAAAFFQAgAAAAAAUVACAAAAAABRUAIAAAAAAFFQAgAAAAAAUVACAAAAAABRUAIAAAAAAFFQAgAAAAAAUVACAAAAAABRUAIAAAAAAFFQAgAAAAAAUVACAAAAAABVSi3EwAASI/3bu9lypYta7KysjIuS0OhkNm4cWNG7h/7lj9l8nELwv4B+D+0EAIAAABAQBEQAgAAAEBAERACAAAAQEAREAIAAABAQBEQAgAAAEBAERACAAAAQEAREAIAAABAQBEQAgAAAEBAERACAAAAQEAREAIAAABAQBEQAgAAAEBAERACAAAAQEAREAIAAABAQBXK7QQAANLjksFDzMotW0woAzM0yxhTtVSpjNw/9i1/yuTjlp/3b/aQwbmdBCDfoYUQAAAAAAKKgBAAAAAAAoqAEAAAAAACioAQAAAAAAKKgBAAAAAAAoqAEAAAAAACioAQAAAAAAKKgBAAAAAAAoqAEAAAAAACioAQAAAAAAKKgBAAAAAAAoqAEAAAAAACioAQAAAAAAKKgBAAAAAAAoqAEAAAAAACioAQAAAAAAKKgBAAAAAAAoqAEAAAAAACioAQAAAAAAKKgBAAAAAAAuqABISbNm0y27Zti7vM5s2bzdatW9O6zpxuIy/YsmWL/cuv+5lK+lM5tvuDNx/zUr4ic6xZs8bs3r07t5MBAIHw2muvmcMOO8zUqFHDtGzZ0kyaNCnu8jNnzjRdunQx1atXN9WqVTPnn3++mTt3bsQyGzZsMLfccos5+OCDTenSpc3JJ59sZs2aFbGMykIvvfSSadasmSlUqJAZPXp03O0+9thjdrmrr746B3sLpDkgVAF9/fr12d7XeyrARyvk7Ny50/67U6dO5qGHHoq7/q5du5q77rrLb3J8rTOn28gLrr/+etOrV698u5+J0q9zxxtkpXJs/dDFWOfl9u3bs32mc1Wfuc9lbz7mpXzNiWh5jtxz0EEHmW+++Sbb+9HOUwBA6j799FNbLnnggQfMnDlzzNlnn23/5s2bF/M7d955p+ncubP56aefzIwZM0zRokXNKaecYtauXRte5pJLLjHffvutGTNmjFmyZIk5/fTTzamnnmpWrFgRXubZZ5+133/66afN3r17zb59+2Ju84cffjDDhg0zjRo1sssCeSYgHDFihK0Z2bFjR/g9/VvvqYbF7Y8//rCFnB9//NF3QsqUKWNKlSpl9qcDsQ0k59JLLzX/+9//Dki23Xzzzfa87N69e7bP7rjjDvvZRRddFH4vU8+XA5nnSI5aoW+//XZ7XS1fvrw55JBDzGeffUY2AkAaPP744/Y+r/tgpUqVTL9+/Wyr3jPPPBPzO2pBvPDCC+11uWbNmubFF180//33n/n666/t5woMv/jiC1uR3aRJE3vtvueee0yVKlXM888/H17P3XffbV5++WXTvHnzuGncuHGjTd+rr75q1wXkqYBQtSFqRZk+fXr4ve+++85UrlzZ1ob8+++/4ff1IylRooQ59thjs61nz5499s/rueeeM/fff3/UbSdqzYi1Tr/biFcTH+0zdQ3wvq+CXLSukaFQKOkukwq049UcJbte5Z9av2J1TdPnugB5aR+9rcJ+tusn/aL1KD1a3kmftyYs0bFNNn/r169vRo0aZdatWxd+T+f18OHDzaGHHur7nPSTr97ur37PU+82cnL+eZdLlOdKn9blZ1927dqVbVvxajLjHSv3unU8ovU68LOcs2/RaF+d/I+1TKL1eLszKw+8eeheRueZPnefb/Go9liFjoULF9q8Ouuss2zhxZvXAIDk6Dqqa6zKs25t27a15Vm/nJZBdQ0Vp7xToEBkkbpgwYJm6tSpSR+m6667zvaUUgsjkOcCwrp165o6deqYyZMnh9/Tv9u1a2cDP6emxHm/devWpkiRIuH3li5das4880xbI1OyZElb++Eu5Hi746ngplaMihUr2j/VlKsp3i3ROr2823j//fftPmn9Woe6BTjdXON9pr7hChbcrr322oiukSog3nvvvaZChQo2aNbfwIEDbaE4FhUk1Tdd+6Lv6YLg7pKQ6nr79u1r+8vrTxcwHS9333YFSKoh8xaAL774YnPjjTf63q6f9LupBk3nyptvvhlO36JFi3wd21TyQerVq2f3XwGg4+OPP7ZjA7wt3Ym6iCbKV1Ee6NxI5jxNdO77Of9inb+x8lyBp1pOtR9aXr935Yt3X7p162batGljl1Glz5VXXmnHUpxwwgl2W/r+gw8+GPE9P8fKySddT7Qe5X00sZZbvny56dixo23R1fuNGzeOGBei4EotwAMGDLD7rGXKlStna3rdEq1H++FO2+DBg+163a14Rx99tHnnnXfsv0888US7PZ13WucVV1wRtfLFoQJAnz597HmisSP6vn6XiQJYAEB8TqWg7kFuuoavXLnSV/bpvtW7d29buax7ofN9lXn79+9v/vrrL1s5qhZH3RvdXUb90BjDP//80zzyyCMcTuTdSWVUq+INCFU41Y/CeV8/Fo2H8dbAfPjhh+amm26yLU7qUvrll1+aV155Jea2VChSoVWDbvXjUqHM3TqZyjrdVChTwU4FU9Xm//3336Zq1apmwYIFcT/zS4HExIkTbZ9zrUP5o0K8ugDEoq5iahlQAX316tXmqKOOst0QcrpeFVqd1iDtmwqdF1xwQTgo0b9Vs/XJJ5+Ev6PuECrkOoOZ/WzXT/rdhg4dak477TTTo0ePcPp0kfVzbFPJB4f2yb2c/p3KoO1E+bo/z/144p2/sfJcXWZVQ/rrr7/a4FBpUIWAd+C80qSgUvsybdo0G/goGFQQqG3pnNG/Z8+enfSxUrf0G264wVYsqJIiFu9yyu/27dvbIFYtcQr+FACfe+655p9//on4rgLlcePG2X3UDVvdiJ1l/KxH1ztd35xgVhVhqkxwKsR081c+azn5/fffbR4rv+bPn2+Phd/uuvodafs9e/a03Ze9nBZS99++JFugASDoVP5JVJns0L1R94CPPvrIjiV03+d1L1XFsLp5ajyhKlCzsrJ8p0OBoO6v7777bkSDCpAnA0KNC1ShTn/6tzcg/O2332ww4G3qVs3+OeecY38cqi1XVygNmo1GXdY0mFa1+ap10XfUgqXXqa7TSwU+dQtr0aKFfa0C12233WZbBOJ95ocKmxo8rEHLahVRYVDdwHRx0A891ndef/118/DDD5vatWubwoULm/vuu8/ud07W66ZuDVrHrbfearv5KjgRXdQuu+wyO/OW46233rIzcKkrhZ/t+kl/MuId25zmgwaHL1682AYoKqArsLn88stNqmLlq599SfXcjyfZ81e/ZY1rUO2mbmhqmVKgpO8rn737csYZZ9h06canAe9qUdN5IroeKKBSt5xkj5UG9juVE/F4l1PrqQIxtdgqqFOXWuV3gwYNsgWW6gKsYyBqrStevLj5+eeffa9H+6f8VeCsPFYQrXEhzvVP/9f+ebsfK3jTDV6tm2PHjk1wBI09P3X81V1UFQ/RPProo6Zs2bIRf399l3z3JAAIAvVsUTdOlVHdVq1aZcf7JaJrvSoyVal75JFHRnym677uaaoAVI8OBYwaSqUePn7pvqn7S9OmTe19WH8KPlUe07/j9bgCcqpQMgsryFNBSQVo1aZocK0K/2ouVxc//alApG5j6jblpu5rbiqkLlu2LGYtiQpbxx13XNz0JLNOLxVaFQQcc8wxtgVAwa4Ktiq0xvvMD6VfBcBotUOxAiQFJurK4L7IqMCrC0NO1isKPtTFQVMnqwCsYE3d+HS81IonKqhq22oJUb4quFN3QG3Hz3b9pD8Z8Y5tqvngUB6o9UsBsLoNdujQwd4okuUnXxPtS6rnfjzJnr/OsfMOdFdAqPS4efdFv/Vo7zndIpM5Vt4gKhbvcpopTq150b6vm3O89OtYOGn1sx6dJ0cccYRtEdSy6laqfVOtsYJdve+0DoqC+0GDBtmuqMoXnR9+xrzq91erVi07AUIsqklWoO924YDHzCq6lwJANqqU031NQZZ7cjn1wlH3/Hg0XEDX8/Hjx9t7ayLqgqrtxJusxksV8yqbuKmyVfdKBaIKCoH9JamzSwGgWhAU9CkgdAo+Kgxr/JXe199JJ51ka2Hckmk2d076RM/mSmad0bz99tu2S9eECRPsvzUGS13bFMzG+yzadt2TSjifKy/8BkQKJsQ76Yj7dSrrVeuVggJd/FSrpXFMSmuxYsUi0qyZsXQMVRDVdMmaglkBod/t+kl/MuId21TywUtdRNV1UuPg/HQzTTVf3elN17mf6PyTeOev323qtXNcU5XMsfJ7s/Mup20oeFJg6zc9sT7zsx5d97Q/CuwUbCuo1L6pi5ACQnWdFlWcqcJg5MiR9lxTBYlaGs8777yE6VTLqtOSGYta9t1dlqQABQYAiEmVdxpSofu3yjwvvPCCHeqi4QTuYQ56rZ4aTs8SDXPQvT7aZImiYFG9qnStV8W6yhiqGI42q3m8e1C0+1u094FcfzC9CkAq9HhrwtVtVLUsKhTldGakww8/3Bawv/rqq4j3/fbx9sNZl7q8qbCsbenfavKP95mopcXb5cA91krpV2tAtIeOxtoHtepoIgn3WDG1rKhbY07Wqy4LGg+oi5PzCIXvv/8+aqCmVsI33njDBkg6hk4Ljp/t+kl/rBq7ZJ+xk0o+eOlGoGOqvv66gCcrmXxN97mf6PxLdP5681xdWnTsFMC416HXqijI6f7k9Fglohu0btzqxpmTbfhdjzOOUNc75xqo/yt/3eMHNcGQKtDUxdbp3up3xjlNha4CBgAgfdQN/8knn7T3RpVbVGGqSjv3kArdH517uf6toTAazqFu/E5XTv1pPQ5V9GlMvXqNqLVRvaXUmuiuVNV9UN9TzyTR0Ae9VpAK5LZCqQSEakUSb0CoH4QmwfBOKJMs1Xpr/JlqZVSY1HbU9Uw/XPfskDmhwrvG5mgWzYYNG9oWMW1DQVG8z0QzHGpyDnU1VL9xPWdG48acbn5qJdLEGpo8QuN6NOZJ3ck04Yb6lkebPUr7rMkj1EddE4Coa5uWU992RyrrVfrUN14XLtV6qfVDD2WNRl0V1KKh4+vMkuh3u37SH40KzKpcUEFcgZCfZ+6kkg/RpDIddCr5mu5zP9H5l+j8jZbnmrxE29UEKeoyqXF/Gg/pnrk0Fek6VvEo4NL1R2NDhwwZYrt06nhotjZN3+33euR3PTomGl+ogNBpXdZ76pbrHj+o2mHluybBadWqlS0ceMdkxqJzSd2K3bM3AwByTpO86f6o4QK6L3l7jjz22GO2q7+ot1usHjvunnC69n/wwQdxt6t7drQZo+ONm1eFbk57wwH7LSBU7YYzftChmhN1ndKPwjvYVj84tUC4qSDqnjnP+xBwFbI1NbCa8zUhhdapWROTWaeXexsqoKm/tiZm0Ox/KtxrMgmnm2S8zxQ0KdBRVwClQY9a0Dgi55k0zjIqXKuwrnTr35qgQuN+YlHBWbVSKnxqXSo8O48CSGa97v3UBUvd1FTgP/744+3xcQIOb3czfUeFYdWWaZ/c/GzXT/q9FHAoj1UQVzc5BTN+jm0q+at0xHvmpD53X/i952Qq+ZrKeZro3E90/iU6t6PludKulkPtjwI2dYF0ZtCMty+6FqjbrZsCTPd7fo5VtHVHE2053UwVYGp/lXdKvwJhdxCnG6pqbr1dYNXaqqDV73qccYTKY03G44xJVDd5va9z3qHZV59++mlbu6wAUsdRBQ3vYzmULu+McjqW2lcAwIHlDdDS1V0zla6f3uFXwP6SFUpnP0zkeyrYKhjw25IBIO84875+ZuWWLSYTL+qqI69aqlRG7h/7lj9l8nHLz/s3e0j0mZndVPSN1UKY37FvSAWjVGFpqmNNOqJp9FOZZAUAAABA/kNAiPAYKHUfVDdFPXcNAAAAQOYjIIT1yy+/kBMAAABAwCT92AkAAAAAQGYgIAQAAACAgCIgBAAAAICAIiAEAAAAgIAiIAQAAACAgCIgBAAAAICAIiAEAAAAgIAiIAQAAACAgCIgBAAAAICAIiAEAAAAgIAiIAQAAACAgCIgBAAAAICAIiAEAAAAgIAiIAQAAACAgCIgBAAAAICAIiAEAAAAgIAiIAQAAACAgCIgBAAAAICAKpTbCQAApMd7t/cyZcuWNVlZWRmXpaFQyGzcuDEj9499y58y+bgFYf8A/B9aCAEAAAAgoAgIAQAAACCgCAgBAAAAIKAICAEAAAAgoAgIAQAAACCgCAgBAAAAIKAICAEAAAAgoAgIAQAAACCgCAgBAAAAIKAICAEAAAAgoAgIAQAAACCgCAgBAAAAIKAICAEAAAAgoArldgIAAOnx681dTaE1/5msUCjjsjSUlWX2VKqSkfvHvuVPmXzcMn3/2Lf0O278jP2wVhwotBACAAAAQEAREAIAAABAQBEQAgAAAEBAERACAAAAQEAREAIAAABAQBEQAgAAAEBAERACAAAAQEAREAIAAABAQBEQAgAAAEBAERACAAAAQEAREAIAAABAQBEQAgAAAEBAERACAAAAQEAREAIAAABAQBEQAgAAAEBAERACAAAAQEAREAIAAABAQBEQAgAAAEBAERACAAAAQEAREAIAAABAQBEQAgAAANgvxo0bZ5o0aWKKFi1qGjRoYN544424y69fv97cd9995tBDDzUlS5Y0jRo1MkOHDo1Ypl27diYrKyvb3zHHHBNeRtuJtsyWLVuSWk8QFMrtBAAInuXLl5vZs2ebvXv3miOOOMLUr18/4vPdu3ebjz/+OPy6bNmypmHDhuaQQw6Juj5n+YMPPtgcd9xxvtOxbNky88svv5hy5cqZ5s2bm2LFioU/27x5s72JRdOyZUtTr1698OtQKGS+/vpr899//5nOnTubQoWyX1p37Nhhpk2bZrZu3WrTWLly5aTSAwBAfqN7/fnnn28effRRc9VVV5mxY8fa/x900EHm7LPPjvqdt956y1SqVMmMHz/eVKlSxXzxxRfmsssuMyVKlDAXXHCBXWbixInZgshq1arZe7BbjRo17L01Fr/ryXQEhAAOGAVZPXr0MKNGjTKtW7e2tYVTp041Rx99tL0B6CIsCpouueQSc8IJJ9iL+YYNG2zAdemll5rXXnst23q1Pi2vZf/55x9TsGDBuOlYtWqVufbaa+2NqnHjxvY72sZ7771nTjzxxHBaR44cGfG9pUuXmu+++84Gik5A+Morr5hBgwbZoPCvv/4y55xzjilVqlTE9+bOnWtOP/10+75ucj/99JN56aWX7A3Ob3oAAMhvhgwZYo466ijTp08f+7pr167mk08+MYMHD44ZEPbs2TPidadOnUyzZs3M9OnTwwGh1zvvvGP27dtnunXrlqP0vpOm9eQ3dBkFcMDooq5gaN68eebLL780Y8aMMYsXL7YtfO3bt7etaG66gYwYMcLWDqqm8PXXX7ff8VJQpkBTQZyWTUQB59VXX20DLwV3v/76qznzzDNNly5dwstUr17dbtv9py4vCjoV3DnUteTzzz+3N7dYunfvbm+Iv/32m/n222/NI488YgPAFStW+E4PAAD5jSpRTz755Ij32rZta3744QcbeCWicoFaFXX/VEtjLK+++qrp2LGjbVF0U88d9bopX768OeWUU2xPnXhejbGeTEdACOCAmDBhgu2a8dRTT5latWpFdAdVa5mCxDfffDPm99u0aWOKFy9u5syZE/H+kiVL7Hp79eplWxAVHCZSt25de8F3B3W60ShAU2tdNNu2bbMtdurq4m6BVCDn7fLqtnDhQnvjU/oKFPh/l1wFr/q3aklTTQ8AAHndypUrbfdQNw2Z2L59u9m4cWPM76mXjO6Fuu+fd955pl+/fuass86KuqwqmlU2UEWrW5kyZcyzzz5rFixYYAPKpk2b2qBQvXGSWU8Q0GUUwAGhFkFd2KNd0DVwXK1vagVUsBTNv//+a28g3rF3ajU8/vjjzWGHHWa/q/F9qhFMtnZPQaXWHW1sn3z00Ud2ILoCwmSotU/UFdShfFAQ6XyWSnp27txp/9x27d3LRR0AkKdpiIUo4ItFrXpaTvdd9fxRF06NIbz44oujturVqVPH9jTy9kpye/rpp20vnWeeeSbq8JNXY6wnCGghBHDAagndLYNetWvXtsu4aXyhumq+8MILdqyBgkaNFXSou4kCwuuuu86+VrdM/cVraYzmq6++sjOYDRw4MOYyulHoJqGJa5Lh1IBWqFAh4v2KFSvaGtBU06MB+mpddf+98/vCpNIGAMD+VLVqVbN69eqI9/Rak6apBS8Rjb3XBC8aevH8889n+1wVxe+++67treP0wolHlbNqMczpejINLYQADghd1NeuXRvz8zVr1mSbjEVdLTU7mFoHNTHL+++/b0qXLh3RiqaJXvbs2WMDR6e1UcHbnXfeGV7HokWLwt/RgPTChQuHX3///fe2e6aW1w0nGt08VKuoVsJkaeIcUS2najwdeh0tQPaTHrnnnnvMbbfdFvHejB4XG7N+TdJpBABgf1APnsmTJ0e8N2nSJDvbdjKBl+YaiNaiqBnGk+m9o66jKifkdD2ZJnghMIBcoWf6KCCMVjOnyWB+//13c+yxx0adVEbBmLp6qKvIH3/8Ef5c4wU1JkDdSTQjqP70KAsFkFOmTLHL/Pzzz+HP9KebikPBoiaIufHGG+1EL7Goa4m6brrH+fnlzEaqsY5ueu19jIbf9DiBpmpX3X9FEsyuCgDAgaQZQzUu74knnrCPdNAsnpoczpl11LmXu58PqKBM8w6oF826detsrx/9XXnlldnWrwpgTcKmCd+81Nqn8oHWo3KB0qKhGrfccktS6wkCWggBHBAXXXSRfdCsWr40mYq7pu/hhx+2gZzT9TMajQ/UjUStYrrAq0VRj5tQ90o9nsJNj3PQDUaPbLjhhhvsn9eMGTNs8KXP4nXNVLqcG5G7ZdEvPU9Q4xk//PBDG7zKN998Y7vHuqfc9pseAADyCz0u4tNPPzV333237dmi4SGaSE6PaIpFgduDDz5oh4hoaIha9HRP173dPRGNHvWk+6n3EVHu9WgyGn1Pk8EdeeSRtoJZrZZufyVYTxAQEAI4IEqWLGkDOAVBmjFUF/oiRYrYxyyoJlDdQeONMZTHHnvMXsjV3UQ1juqC6b2wi2YkUwCncXjRxijo4n/aaafZFjqNOXS6m8oZZ5wR0bXzs88+s7N9XnPNNVHTpBbI+fPnm5kzZ4a7naj17qSTTrKPrtBD6p988kk7IF6tkwoO9YgKvW7RokXS6QEAID/RfT/WMwdF91f3PVaBmzMLd7TJaNw9cOI9ukKVsNHW41UvwXqCgIAQwAGjB9AreBo+fLid3lmtb61atbIDxTXw3KFAUc/gq1mzZsT3tewDDzxgW9M0APzee++NOgZB3T7UvVPdUPUdL3VLcZ4l6K0R1PLuAGz58uW2VbNBgwZR90mBqWZHFaVZzySUhg0b2oBQ9DgM1Yoq0NNjJAYMGGAfzptKegAAANIpK+QNtwEA+dKUyzuYQmv+04XdZJpQVpbZU6lKRu4f+5Y/ZfJxy/T9Y9/S77jxM8z+ppBFXUY1q3a8R1YgeUwqAwAAAAABRUAIAAAAAAFFQAgAAAAAAUVACAAAAAABRUAIAAAAAAFFQAgAAAAAAUVACAAAAAABRUAIAAAAAAFFQAgAAAAAAUVACAAAAAABRUAIAAAAAAFFQAgAAAAAAUVACAAAAAABRUAIAAAAAAFFQAgAAAAAAUVACAAAAAABRUAIAAAAAAFFQAgAAAAAAUVACAAAAAABRUAIAAAAAAFFQAgAAAAAAZUVCoVCuZ0IAEDObdiwwZQtW9ZkZWVlXHbqVrVx48aM3D/2LX/K5OOW6fvHvuVPmXzcchsthAAAAAAQUASEAAAAABBQBIQAAAAAEFAEhAAAAAAQUASEAAAAABBQBIQAAAAAEFAEhAAAAAAQUASEAAAAABBQBIQAAAAAEFAEhAAAAAAQUASEAAAAABBQBIQAAAAAEFAEhAAAAAAQUFmhUCiU24kAAOTclMs7mEJr/tOFPeOyM5SVZfZUqpKR+8e+5U+ZfNwyff/Yt/xpfx6348bPMEFGCyEAAAAABBQBIQAAAAAEFAEhAAAAAAQUASEAAAAABBQBIQAAAAAEFAEhAAAAAAQUASEAAAAABBQBIQAAAAAEFAEhAAAAAAQUASEAAAAABBQBIQAAAAAEFAEhAAAAAAQUASEAAAAABBQBIQAAAAAEFAEhAAAAAAQUASEAAAAABBQBIQAAAAAEFAEhAAAAAAQUASEAAAAABBQBIQAAAADEsW/fPpOKHTt2mN27d8f8fOfOnWbv3r053raf9cRCQAgAAAAAUbz33nvmkEMOMYUKFTIHH3ywefPNN41fr7zyiilevLg588wzI97funWreemll0yzZs1MsWLFzLBhw7J9d8uWLaZnz56mWrVqpkiRIuaII44w48aNS3o9fhAQAsh1uui98cYbZuXKlRGvR44cmW3ZL7/80kyfPt0sWbLELqOLYTQzZ840H374oa/tz5o1y67r77//zvbZihUrzNixY81HH31kfvvtt6jfX7hwofnkk0/MpEmTYqZn2bJlZvTo0ebTTz81a9eujbqMavY+//xzm5Y9e/b4SjsAANg/vv32W9O1a1dz33332bJJ//79zdVXX20mTJiQ8Ltz5841Dz74oDn55JOzfaaygMopr776qilbtmzU7994443ms88+s9vavn276devn7ngggvMzz//nNR6/CAgBJDr1qxZY7p3727mzZsX8fr88883M2bMiFj2scces7VzJUuWND169DAffPBB1HVee+212WrSolHAd+6559rtfffddxGfPfPMM7ZWUP8fMWKEad26tencuXM4WPvvv//M6aefbs466ywzfPhwc8cdd9jlFRi6DRo0yDRs2NC8+OKL5oUXXrDLjBkzJmKZ5557ztSrV8/06dPHpkVdTAAAQO4ZMmSIDeiuuuoqU6JECRsctmvXzr4fj+7hXbp0MU899ZSpUaNGts8vv/xy27LXvHnzmBXEqoi+7bbbTOPGjU3hwoXNRRddZNq0aWPX6Xc9fhEQAsizGjRoYO68886on1WsWNEGcqoVi9biN3v2bFuLl6hPvi6mCsKiXcxvv/12M2DAANsqqQuzAsaPP/7YjB8/3i6jGjt9d/78+fb9n376ydbeXXrppSYUCtll5syZY+655x7z7rvv2gBV31WA2a1bN7Nx48bw9sqUKWOmTJlitwcAAHLf9OnTbRDmdsopp9j34+ndu7dp0aKFrURORVZWlv3zjgnUa2/ldToQEALIs/r27WumTZtmu0xEc80119jPFZC5KUhUi9yJJ54Yd/0KvtTn/qabbsr2mdMKWL169fB7+rcu0M7gbo0lOO200yK+d84559iWw9WrV9vX6spRoEAB06FDh/Ay5513nlm/fn1EK6FqHWvVqhU3vQAA4MBZtWqVOeiggyLeq1y5sr2Hx5ooRkNI1M1Tlb+pUrlBweTgwYNtALhu3To7HvGbb74JD69JJwJCAHmWulCqW+jdd98ddYYtdduoU6eOee211yJa9tQal6h1UIGkBl+7v+tWqlQp8+yzz5oHHnjADBw40P5bwd6VV15pzj777Jjr/eKLL0zVqlXtDUMU5KlGz+kOK7///rv9/6+//mpSpdnENm3aFPG3K8XZxQAAQHTe8ofzWhXE0YahXHfddTZ4UzdPlUm0vP70b6f3kB8aSqJeR1dccYUtD6mXkXolFSxY0KQbASGAPO3+++83ixcvNm+99VbUGjSNt9OYQqdFTzVzmzdvthfQWFSzp26dGtNXpUqVmMvVrl3bzio2ceJE89VXX5nly5fblkdtNxpNCPP888+bxx9/PCJo1fgDBZGq6VNwqbEIlSpVskFcqh599FE7gNz9987vC1NeHwAAiKQKXrUSuum1hq2ofOC1YMECO/nMGWecYcqVK2f/3n//fduyp38nUxGsiuknnnjCTlyncosmkFHroILDdCMgBJCnqaVNNWJqqYs20YqCK12cFYyJWvzUPdMJ9KZOnWpn7XT+du3aZR566CE7Zk+T1+g9J9jUGD7NBCpLly61XTs1oFsBoS7E+r9mDNMEMl76rrp3qJurxiU6FDyq64hmB9M6NcOo1qULfenSpVPOF41L1BhE99/ljeqnvD4AABDphBNOMJMnT454TxXEmmTOoV5ATvnkpJNOsv92/11yySV23KH+3bRpU5MqVXZrqEmnTp1MumUPbQEgj1FAqO6dQ4cOzfaZumS2b9/eBoJNmjSxM3y6x+apq6aCQodm6WrUqJGtbfv666/te04XDk0RrUCyY8eOduyfumUqKHTUr1/fflfB32WXXRZ+X+vXTKOaZVQtml6qRdQkMvpzupSo1VPPDUpV0aJF7Z9bkf3QjQQAgKC67bbbbFComT0vvvhiO8Gcyg5O+UFUNtEkMsl0B9WyKmM41MtJAaO6g6qrqajyWT2JFACqMrlXr162xVJpSmY9ftBCCCDPU2uaWgjVTVKBXLTJZfSsQH2uiV/UVcP9mbuFUNNGe99zZipVv3+1HjrdRcX97EF1A1m0aFH4M9Fgbz1wVhdotQJG4x0Arkdn6EGz7mATAADkLccee6wdiqKeRBoyorGBesaxgkR3pa+3gtZND5XXn3c2dKdLqQI4zZWgfzsVx6Iywp9//mkrjzWBnbqK6rmIeuxWMuvxgxZCAPmCgjU990cPZG3ZsmXEZ3r8RPny5e2zePTw2HQMuNZ00XoOop4jdPPNN9supu+8844NKJUWZ6yAgkE9HqNu3bo2uHSolbFChQr236o51L/VgqlaRc2aqgC2ePHi4eV/+OEH20LpPHBWNYO6wWgMYs2aNXO8PwAAIHkdOnSImCncS2UE/cUSbfI6PTcw0fOGFfip3BPvmYd+1uMHASGAPNECqNosdYVwv3ZP9awaOM24pUCpVatWEd9Xtwi1Dqorp1r/kqVxftqed6C2ni2omsHvv//ePkpCy2iyGmfsny7CChrF3X1ETj311HBA+Pbbb9tgUc8pPOqoo8yTTz4Z8TgLJ7h01qHtOM84Us0gASEAANhfskLJdHgFAORZUy7vYAqt+U8XdpNpQllZZk+lKhm5f+xb/pTJxy3T9499y5/253E7bvwME2SMIQQAAACAgCIgBAAAAICAIiAEAAAAgIAiIAQAAACAgCIgBAAAAICAIiAEAAAAgIAiIAQAAACAgCIgBAAAAICAIiAEAAAAgIAiIAQAAACAgCIgBAAAAICAIiAEAAAAgIAiIAQAAACAgCIgBAAAAICAIiAEAAAAgIAiIAQAAACAgCIgBAAAAICAIiAEAAAAgIAiIAQAAACAgCIgBAAAAICAIiAEAAAAgIDKCoVCodxOBAAg5zZs2GDKli1rsrKyMi47davauHFjRu4f+5Y/ZfJxy/T9Y9/yp0w+brmNFkIAAAAACCgCQgAAAAAIKAJCAAAAAAgoAkIAAAAACCgCQgAAAAAIKAJCAAAAAAgoAkIAAAAACCgCQgAAAAAIKAJCAAAAAAgoAkIAAAAACCgCQgAAAAAIKAJCAAAAAAgoAkIAAAAACCgCQgAAAAAIqEK5nQAAQM6FQiGzadMmk5WVZf8yTSbvH/uWP2Xyccv0/WPf8qf8fNxKly6dp9OcFVLuAgDytdWrV5vKlSvndjIAAIDHxo0bTZkyZUxeRQshAGSAIkWK2P8vXbo0T990UqVa4Vq1amXk/rFv+VMmH7dM3z/2LX/Kz8etdOnSJi8jIASADOB0RdFNMr/dKJORyfvHvuVPmXzcMn3/2Lf8KZOPW25hUhkAAAAACCgCQgAAAAAIKAJCAMgARYsWNX379rX/z0SZvH/sW/6Uycct0/ePfcufMvm45TZmGQUAAACAgKKFEAAAAAACioAQAAAAAAKKgBAAAAAAAornEAJAHrVs2TKzcuVK06BBA1O2bNm0fSeV9abb+vXrzV9//WWqVatmatSokePvrF271sydOzfbd4499lhTuHBhcyDt2rXL/Pbbb6ZkyZKmYcOGvr4TCoXMTz/9ZCdLaNKkSdrWuz8on3fs2GEaN27sO2///PNPs3r1anPCCSdk+2zWrFlm69atEe/pGNerV88caMn+Nvbu3WsWLFhgChQoYA455BBTqFChjPrN6QHg+p6Ohc67TPrN6ZybP3++qVChgqlTp07a1psXfnN79uyxvzktW7du3Wzfyc+/OfdD6n/55RdTvXp1+9tL13oDKwQAyFN27NgR6ty5c6h48eKhww8/PFSsWLHQU089lePvpLLe/WHgwIF2204aLr300tCuXbty9J0PP/wwVLBgwVDr1q0j/tauXRs6kMaPHx+qVKlSqG7duqEKFSqEmjVrFlq2bFnM5ffs2RMaNGhQqF69eqGyZcvaNKdjvfvD4sWLQ02bNrXpOPjgg0NVqlQJTZ48Oe53Pvroo9Dxxx8fKl++fEhFjt27d2dbplGjRna/3Mft8ccfDx1Iqfw2HnnkkVDVqlVDhx12mE1/9erVQyNHjszxevPCb077oeNSs2bNUJMmTUIlS5YMPfjggxHL5Nff3Pr160PXXXedXbZ58+ahihUr2vP6t99+y9F688pvTsda56W+V7t2bfvvDz74ICN+c24dO3YMFShQIHTTTTeldb1BRUAIAHnMfffdZwuXTuFjzJgxtjA9bdq0HH0nlfWm28SJE+1NXP+XRYsW2cKOCtc5+Y4KpwqocpMKwkrDAw88EC6YqKDVvn37mN/ZvHlz6I477ggtXLgw1KNHj6gBYSrr3R9OOOGEUNu2bcOBRJ8+fWxheuPGjTG/89BDD4WmTJkSeu+99+IGhLldYEvlt3HvvfeGVq9eHX7dv39/W/h0Bw359Tc3ZMiQ0B9//BF+PWnSpFChQoVCH3/8cb7/zc2bNy80fPhwWxkjO3fuDJ1xxhmhY489NkfrzSu/OQWEmzZtivgNFi5cOOI7+fU353j66adD7dq1swG9NyDMC7+5/IiAEADyGN3MdFNzO+qoo0JXX311jr6TynrTTS0TKuS49erVy7aQ5eQ7KpyWKVMm9Pvvv4d+/fXX0Pbt20MH2ksvvWQDAgV57pYWFUaWLFmS8PuxAsKcrjcd5s+fb7fnBBWyZs0aGyS8/fbbCb+fKCBUwfvHH38M/fvvv6HckI7fhvJD+zhq1Ki0rjc3fnPRNG7cONS7d++M+s25gya1hqZ7vbn5m3N8++23dj2qdMqE39ysWbPsd5cvXx5q0aJFtoAwL/zm8iMmlQGAPGTVqlXm33//NS1atIh4/5hjjrHjPlL9Tirr3R+0rWhp0NimzZs35+g7GlNy7rnn2j+NCxowYIA5kJROjTMqVapURDpl9uzZeW69yaZB3MehYsWKduxOOs6fJ554wlx77bXm0EMPtWPQ5s2bZw6UdP02ZsyYYf/vjMPKz785L40XXLRokalfv37E+/n5N/frr7+ab775xrz44ovm+eefNw8++GBa1psXfnNLliwxU6dONR988IHp2bOnueqqq7KND8yPvzmNe+zSpYsZOnSoHTuYrvWCSWUAIE9Zt25d+MbvptfOZ6l8J5X17g/aVrQ0OJ+VLl06pe9oUoiff/7ZNGvWzL4/duxYc95555latWqZrl277sc98p/OvLbeZNNQsGDBbJMzpOP8ueOOO8zFF19sJ9TZuHGjufDCC02nTp1swbtIkSJmf0vHb0MB00033WTT3ahRo7StN7d+c27qTabA4aCDDor4LeX339ywYcPsRE6aWEYBUfv27dOy3rzwmxs/frx54403bGCoCXGuvPLKjPjN3XjjjebEE0+0aU3nesFjJwAgT3Fmg9OMcm7bt2+PeaP2851U1rs/KB3R0iDx9i/Rd1q2bBkumMo555xj/0aMGGHy8r7l5nqTTYNm1Ny9e3e2dOQ0Dd26dbMFU1Hh97HHHrOzKs6ZM8ccCDn9baiV7KyzzrKFztdffz1t602XnJ4/CnS//fZbM2bMmIjgMb//5tQq+MMPP9gWJe2XAkKd4zldb174zSmAnzZtmg0Ir7nmGtOuXTvz999/5+vf3GeffWZGjhxpOnfubFs/9bdlyxazYsUK+29VXOSV31x+RJdRAMhDNB28aoWXL18e8b5e165dO+XvpLLe/UGtCtHSoMJJ5cqV0/YdqVKlSrbv7U+x0ik5yeP9td5k0yAqPLvpdbrToOMmB+rY5eS3oS6XZ5xxhi24f/nll6ZMmTJpWW86pfr7kVtuucV2O/zqq6/sIw8SyY+/ueLFi5ubb77ZPqZB3WLTtd688JvLysoyvXr1Mvv27TOTJk3K1785BcZ6JM/DDz9s7r77bvun5adPn27/rc/zym8uPyIgBIA8pFixYrZLzOjRoyNqNydMmBDRpUnjf9Rdy+93/K53f9O21J1Jz/dyjBo1ypx66qn2Ri5r1qyxNb5Ozbif73ifqaXvanyQn0Jsuiid//zzj302ljud5cqVs60pTrq0b9rHdK53f2vVqpXteuY+f1QQ05gd9/mjc1Lnpl/btm2zNftuCqzE6XqZF39z7mBQ56WW1fFIZb158Tcnt956q3n33XfNxIkTzZFHHpltvfn1N+dNtyxcuNA+S7J8+fK+15sXf3P6PXlpP/RcQqcbZX79zWmcqtMy6PxpnKe6j+rfagHMK7+5fCm3Z7UBAESaOnWqnSZcjyPQjIWnnXaafQaVe9pwzUjZsGHDpL7jZ5n9TdO5aza/Dh06hEaPHh3q2bNnqGjRona2O++MlCtWrPD9nQsuuCB011132f3S7IennHKKfXbY3LlzQwfSmWeeaZ9Lp+d+DR061M5U+Mwzz4Q/1z5p37SPjpkzZ9pHM5x77rn2mW/6t/6SWe+BoOcl6nl0w4YNC40YMcLOUnn++edHLKNzUuemY8GCBXZf+vXrZ/f766+/tq83bNhgP//uu+/szKovv/yyfe6bHoVQqlSp0A033JCnf3OaLfXEE0+0j2/QtPbOMdOfc976XW9e/M3dc8899hmDOtfc+6ZHNuT339yAAQNCV1xxhX2tc06v9YgJ5Usy682Lvzkdo5NOOsn+niZMmGD/f+ihh4ZatmwZfnRFfv3NRRNtltG88JvLj7L0n9wOSgEAkVQT/Oyzz5qVK1faGnd1ialWrVr488GDB9tZ04YPH+77O36X2d+WLVtmBg4caMesaKY4tUS4a93VPa1v3762llczF/r5zs6dO+1sgZMnT7a1302bNrXLVKpU6YDum2qjn3zySdtSUqJECXPJJZfYWfEcmtigY8eOdkbDtm3b2vc0uYP2z0vrcFpwEq33QNH59v7779v8VguTuqM5Y5Hksssus+PKbr/9dvt60KBBduyZ15AhQ8zRRx9t/63z+OWXX7atNDVr1rQ1/hqLlpd/cxq7pNbBaDRhh1oz/K43L/7munfvbhYsWJBtPRqL1q9fv3z/m/v000/NJ598YlvbNAmOJlU5/fTTk1pvXv3NuX9PmgjolFNOMVdccUXEGLr8+JuLRuMj1arZu3fvpNaL7AgIAQAAACCgGEMIAAAAAAFFQAgAAAAAAUVACAAAAAABRUAIAAAAAAFFQAgAAAAAAUVACAAAAAABRUAIAAAAAAFFQAgAQA789ttv9sHemey7774z8+fPz1P5vD/zPV3r/vPPP82XX36Z9vUCyRo3bpz566+/4i4zcuRIs379ejI3gAgIAQDwUPAzYsQI+/fhhx+a8ePHm8WLF0fNp48++sj07dvXVx7++uuvZtKkSfkqv5cvX27OPfdcU7x48VxNhzefk8n3nG4rVWPGjDH33ntv2teb38+p/JjmvOSLL75IuoKmZ8+eZsKECXGX+eabb8z//ve/HKYO+VGh3E4AAAB5zWeffWbuvPNO06lTJxMKhWyt+c8//2yqVKliHn/8cXPWWWeFl23cuLHv9b7//vtm6tSp5tRTTzX5Rf/+/W0+1KpVywRFMsc0t9ebH8+p/JjmvOT2228311xzjTn00EPTut477rjD1K1bN/x/BAcBIQAAUZQoUcK2EDp27dplHn30UdOhQwfz+eefm9NOO82+f9hhh5ny5ctHfPfff/81s2fPNmXKlDEtWrSwrWuq0f/jjz/MqlWrwus9/vjjze7du82MGTPs69KlS5tGjRqZgw8+OGJ9WtfGjRvNcccdZ/+9bt0607JlS1OpUqVs6V6wYIHdTp06dcxRRx2V7fN58+bZv2rVqpnmzZubwoULxzz+mzZtMm+//baZOHFiwv1zp/OYY46x/16zZo1p3bq1qVChgtm2bZuZNm2aDbC136VKlQqvT13ZEuWBX/H2z50+dYPV/p1//vnZ1uE9pn7zf8+ePWbKlCn2/9HyPtq5Eu+YJcqXWOdU7dq1E+ZFvOPolerxSeZ3oDyNdWy2bt1q39M5dPTRR5saNWpk6w6pvC1btqyZNWuWKVasmD1W3v3V+aj1VK5c2eazzkflzRFHHBFeRsfuhx9+MGvXrjUNGza0f9G2peuD8kSVRMcee2y2fXeWU34pD4oUKWJOPPFEU7BgQdvq/tNPP0X9rlrpVqxYYbKysuznSme5cuXCn6tlVfmk/XTy76KLLjIFChQwe/futWlS3iq/vfnk5EGsPKpevbo5+eSTzYsvvmgGDhyY8PgicxAQAgDggwp06u6nAuV9990XDgjVDVABU9u2be3r559/3tx1113hYG/lypXmvffes4VjjSlTgUxjdUQBgAq7zmsVghVQ3HTTTREFsnfeeccWMAsVKmQLsBs2bLAFa41PUwFaduzYYbp162aDVRX0tJ2aNWuaUaNG2cLlzp07zRVXXGELnApm1AV23759tlvjIYccEnWfVfjUd51txNu/I4880qZz9OjRtmDaoEEDW/BdtmyZefrpp81DDz1k39N2FVyr4KpAURYtWpQwDxLxs39K39ixY23hWYVf5X+0gNB7TP3kv1qR1eK1evVq06RJE/PLL7/Y/Y233kTHLFG+KFCLdk4pkEiUF/GOo1cqxyfZ38HHH38c9dh8//33pmPHjjbfFYBPnz7d9OvXz7bgu7tDKkBVOhXczZkzxxx00EE24FPgI8p3rU8BnoI0BU3K/+7du4cDQqVL29JvXS1kP/74oz1WqhRRutzbWrhwoQ3WTj/99KgBoZbTcdBvQOeDgkzlfZcuXczQoUNtUK1riSqY3njjjfD3tH8KIFVxsmTJEpumd99915xxxhn2c7Wsbt682Xa71TkvF1xwgT0fzzvvPLN9+3bbEj137lzb0nfDDTdEnH/q4RArj0TnsPaXgDBgQgAAIMJTTz0VKlu2bNRcGTp0aCgrKyu0ZcsW+7pv376h1q1bhz+vVKlS6N133w2/XrJkSWjGjBn23//73/9Cbdq0iZvbCxYsCJUoUSI0c+bM8Ht9+vQJFSxYMPT999+H3+vcuXOoY8eO4dd33HFHqEaNGqHFixeH3xs9enRo79699t/33ntvqFWrVqGtW7eGP7/22mtDp59+esy03H///aHmzZtHvBdv/5x0Oq+17aZNm4aKFCkSmjNnjn1v9+7dofr164cGDRqUVB5489n72s/+KX0q+kyaNCnmtqOt20/+9+7dO9S4cePQpk2b7Ou///47VLp06VCLFi1irjfRMfOTL9HOKT95Ee84JhItHV7J/g6iHRudK4cddphNu2PkyJGhAgUKhH755Zfwe/Xq1QsdccQRoY0bN9rX+n/FihVDr732Wng9WqZnz57h77z++ut2e/3797ev9+3bZ4+fjpFjw4YN9nvDhg2L2FbdunVD69ati5tHWq5Ro0ahzZs329fz58+31w39npzjMmvWLJuGP//8M+Z6nnvuuVCtWrVs+hxar65RDp0vyqdOnTqFdu7cad/btWtXaNy4cb7zyKHvKE3btm2Lu3/ILLQQAgCQhKpVq9rae3UpK1myZLbP1S3u999/t13P1KKksXeJxt+pK5zGKKoVRd9Ty4JaJ9Tty6Euf+6WCHXtUkuD48033zS33XabbVlxqPXB8frrr9vWD42PVPr1p65zag1Q65HTAuKmVhxvF8dE+6c0q1ufaJ1qQVOXt6ZNm9r39B197p0Uw08exON3/9Rac8opp5hkJcr/Dz74wNxzzz229UnUwnThhRfalphYEh2zVPPFT14ke54mm45UfgfeY6O8U5dX7YdDExyphUuTPWl5x2WXXWa7por+36xZM9u6JuoiqdZUtZg51IKqFkyHWqw1C2yvXr1sS5qTb/Xr1zeTJ082119/fXjZyy+/PGrXXy+lyekardbiihUr2u+qu6mohdHpRuseD6hWVOWdutHK0qVLbb6rlTQatT4qn9TiqtZNUVdQ91jnRHnkcPZLv/0gjRsOOgJCAACSoO5aEi0YFHX/uvnmm81zzz1nTjrpJNO5c2dbEIsWcDld2S6++GLbTU5dytR9S9tQlzY3p3ulo2jRorbLm6jrmJaPNcmEumhqXJK6kam7o5sK2FqPU0h1U2FWXVqT2T9vQVnpjPaek/Zk8iCWZPYvVqE6kXj5r6BHhXjvuDoFhbECwkTHLNV88ZsXyZynqaQj2d9BtGPzzz//2OXdAbPUq1fPfub3+CigUoDkHlOn9TpjLUXdatVN1zsTpyoz1L0zXjpj8fNbUADn/i1ols8hQ4bYShN16VQAL8rrWNtV11KlXcFrPPHyyOH83p2KDQQDASEAAEnQGB+N81JtfzQag6NJM1TA1OMq1CqhwvmAAQOiLq8WIo3HevDBB8PvaTIKtU74pYKdWhrUahmNCsMqxGv80o033uh7vQpW3nrrrRztnx85zYNk9k8F53RTC5gmNPE+wy3eM90SHbNU88VvXiRzHFNJRyrniffYaMygAiJNouIOpNRypqDQLwVCGse4ZcuWiMmM3MdHLWbanyeffNIGvgf6HJK///7b5o9aYtV6J2r5+/TTT+PmtYJWZzbkaBNNJUPjMNWa7J7IBpmP5xACAODTzJkz7QQjsQramkzlv//+s/9Wa1GPHj1sq4gmxhAVRr018uoK5p7JUJORaNbJZLVv3952CXQXHNXtS69VgNXkF6+88opNo5smvYhFE2pof5wHWifav1TlNA9S3b90OuGEE8KTpDithppgJ9Vj5jdfvOeUn7xI9jgme3xS+R1Eo27GCtQ++eST8Htq7VMXSeW3X5ooRy367uOhLpkKwBya/EatYi+88ELEd3Us1OJ6ICifdfzcrcbqvurlzb9WrVrZ97yVN5rgKFmaZKZdu3ZJfw/5Gy2EAABEoRYFZ1p3db1TYVbPT+vatWvEDIfe76igqgK5avhVuNd4rkceecR+rm5gmiFRs25qDJYKoZoZUA8v1zbUXeupp56KaMXwa/DgwXZaexXm1D1PhUGNs1KXRRUy1fLRpk0bu02NY1JB99tvv7Vd1jSLYTTq8qigULNDambVRPuXqnTkQSr7l06aRVXb1qyhetSGxhRqf9RymOox85Mv0c6pRHmR7HFM9vik8juIRq1Ueg7mLbfcYrtFqgulxm0qz6LNDhuLWhfvvvtuOw5QLWAK/J555hkbbDqtffq3gsErr7zSdkfVNhSgqXWuT58+5tJLLzX7m8YUqhurzgWNP9Vso9HOXeXf8OHDbRdStTTrsRPKl+uuu87Ofqrxrqq80nHXIyT8UjdmzXDrDsARDASEAAB4qDVEk3uoxUfPDVMBUq0VmjTE+1wy98PG1VVPhThNFqKupfqeCvjOYwZU8NfYKk1SoQBTY6OeffZZ89JLL9lJLVQo1fIax+RerwrV3q5g6jLnnoBEk1ZoKnq1DGnd+lxjv5wxWxr7pc+VNnVJU2FbwYLGlcXzwAMP2OXU5S/R/kVLpyYd8T7kWpOzOGOjxE8eeB/q7n3tZ/+ipS8a77r95L+WUb4r/5VHepyBgkHlU6z1JjpmfvIl2jmlFqN4eZHoOHr5SYdbKr+DWMfm1ltvNYcffrgNzBQU9u7d2+at2znnnJNt/JzGLWoCKIcqNLSMuq+qS6QqOTSxjHusnII+BWUKtvRoDbVuKp3uyWuibSuaaMsp/72/BT0ywpm8RWM71UKncZdff/21Paf1Wo+7cXeZVbfSYcOG2fxTS6HWoUBW6dR+Kc91DrjzyU8eqbVa4yVTmXQJ+VuWphrN7UQAAIC8S2PHVIhXyxeQH2ncoXtSFbWkKdBU4MV5/f+o0kcVB9GeRYnMRkAIAACAjKaWM3UBP/PMM82mTZtsF0u1+o8bNy63kwbkOgJCAAAAZDw9z/CLL76wY+v0fMxEj8EAgoKAEAAAAAACimoRAAAAAAgoAkIAAAAACCgCQgAAAAAIKAJCAAAAAAgoAkIAAAAACCgCQgAAAAAIKAJCAAAAAAgoAkIAAAAACCgCQgAAAAAwwfT/Aeh0nPRYcJvqAAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "import matplotlib.pyplot as plt\n", + "\n", + "plot_data = evaluation_summary.sort_values(\"distance\", ascending=True)\n", + "colors = plot_data[\"category\"].map(\n", + " {\"Exact identifier\": \"#c74634\", \"Semantic\": \"#2f6f73\", \"Mixed\": \"#e6a23c\"}\n", + ")\n", + "\n", + "fig, ax = plt.subplots(figsize=(9, 4.6))\n", + "bars = ax.barh(plot_data[\"query\"], plot_data[\"distance\"], color=colors)\n", + "ax.invert_yaxis()\n", + "ax.set_title(\"Top-result distance by query\", loc=\"left\", fontsize=13, fontweight=\"bold\")\n", + "ax.set_xlabel(\"Distance (smaller indicates a stronger match)\")\n", + "ax.grid(axis=\"x\", alpha=0.2)\n", + "ax.spines[[\"top\", \"right\", \"left\"]].set_visible(False)\n", + "ax.bar_label(bars, labels=[f\"{value:.4f}\" for value in plot_data[\"distance\"]], padding=4)\n", + "plt.tight_layout()\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "36f395b1", + "metadata": {}, + "source": [ + "### Reading the results\n", + "\n", + "Every query returned its expected memory at rank 1. Exact queries preserved literal identifiers, while semantic queries recovered the correct event from business-language descriptions. The mixed query is the most representative enterprise case because it combines intent, customer context, and an implied operational object.\n", + "\n", + "To keep the notebook concise, only that mixed case is expanded below." + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "b4f6a8a7", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
Detailed ranking for: Which invoice failed reconciliation for Northstar?
rankdistancelabelcontent
10.2152renewal_blockerNorthstar Renewals cannot proceed with its renewal because invoice INV-48291 failed reconciliation after ORA-27102 during month-end processing. Finance must rerun the reconciliation job before the renewal can continue.
20.2974similar_invoiceNorthstar Renewals also has invoice INV-48290 marked as paid. No finance follow-up is required for that invoice.
30.4134customer_aliasNorthstar Renewals maps to the enterprise account Northstar Renewal Services Ltd. Finance operations owns escalations for that account.
40.4265milan_renewalMilan Office Supplies experienced a renewal delay because its approval workflow was waiting for a regional manager. No reconciliation error was reported.
50.5170error_referenceORA-27102 can indicate memory allocation pressure. Database memory settings should be checked before rerunning affected batch processing.
\n" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "detailed_results = evaluation_results[\"mixed_invoice_context\"][\n", + " [\"rank\", \"distance\", \"label\", \"content\"]\n", + "].copy()\n", + "\n", + "display(\n", + " detailed_results.style\n", + " .hide(axis=\"index\")\n", + " .format({\"distance\": \"{:.4f}\"})\n", + " .set_caption(\"Detailed ranking for: Which invoice failed reconciliation for Northstar?\")\n", + " .set_properties(**{\"text-align\": \"left\", \"white-space\": \"normal\"})\n", + " .set_table_styles(\n", + " [\n", + " {\"selector\": \"caption\", \"props\": \"font-weight: 700; text-align: left; padding: 8px 0;\"},\n", + " {\"selector\": \"th\", \"props\": \"background-color: #312d2a; color: white; text-align: left; padding: 8px;\"},\n", + " {\"selector\": \"td\", \"props\": \"border-bottom: 1px solid #d9d4cf; padding: 8px;\"},\n", + " {\"selector\": \"tbody tr:first-child\", \"props\": \"background-color: #e7f4e4;\"},\n", + " ]\n", + " )\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "6a52deee", + "metadata": {}, + "source": [ + "## Vector, Keyword, and Hybrid Search\n", + "\n", + "The comparison below is conceptual. This notebook executes the hybrid configuration only; it does not claim to benchmark three independently configured search stores." + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "50301865", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
Conceptual retrieval-mode comparison
modebest_atweak_spotexample
VectorMeaning, paraphrase, and conceptual similarityShort identifiers can be underweightedWhat blocked the renewal?
KeywordExact IDs, error codes, aliases, and filenamesDifferent wording can miss the stored recordINV-48291
HybridNatural-language intent plus exact operational handlesRequires database embedding and managed index setupWhy did INV-48291 block the renewal?
\n" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "search_mode_comparison = pd.DataFrame(\n", + " [\n", + " {\n", + " \"mode\": \"Vector\",\n", + " \"best_at\": \"Meaning, paraphrase, and conceptual similarity\",\n", + " \"weak_spot\": \"Short identifiers can be underweighted\",\n", + " \"example\": \"What blocked the renewal?\",\n", + " },\n", + " {\n", + " \"mode\": \"Keyword\",\n", + " \"best_at\": \"Exact IDs, error codes, aliases, and filenames\",\n", + " \"weak_spot\": \"Different wording can miss the stored record\",\n", + " \"example\": \"INV-48291\",\n", + " },\n", + " {\n", + " \"mode\": \"Hybrid\",\n", + " \"best_at\": \"Natural-language intent plus exact operational handles\",\n", + " \"weak_spot\": \"Requires database embedding and managed index setup\",\n", + " \"example\": \"Why did INV-48291 block the renewal?\",\n", + " },\n", + " ]\n", + ")\n", + "display(\n", + " search_mode_comparison.style\n", + " .hide(axis=\"index\")\n", + " .set_caption(\"Conceptual retrieval-mode comparison\")\n", + " .set_properties(**{\"text-align\": \"left\", \"white-space\": \"normal\"})\n", + " .set_table_styles(\n", + " [\n", + " {\"selector\": \"caption\", \"props\": \"font-weight: 700; text-align: left; padding: 8px 0;\"},\n", + " {\"selector\": \"th\", \"props\": \"background-color: #312d2a; color: white; text-align: left; padding: 8px;\"},\n", + " {\"selector\": \"td\", \"props\": \"border-bottom: 1px solid #d9d4cf; padding: 8px;\"},\n", + " ]\n", + " )\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "e084d799", + "metadata": {}, + "source": [ + "## Scoped Retrieval\n", + "\n", + "Scope and hybrid ranking are complementary:\n", + "\n", + "- `SearchScope` determines which memories are eligible.\n", + "- Hybrid retrieval ranks those eligible memories using semantic and exact-text signals.\n", + "\n", + "Every evaluation query above used one generated finance-user and support-agent scope. The scope selected eligible records before the hybrid index ranked them." + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "8d91ea61", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
scopeevaluated_querieseligible_demo_memoriestop_result_validations
Generated finance user + support-finance agent555/5 PASS
\n" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "scope_confirmation = pd.DataFrame(\n", + " [\n", + " {\n", + " \"scope\": \"Generated finance user + support-finance agent\",\n", + " \"evaluated_queries\": len(evaluation_cases),\n", + " \"eligible_demo_memories\": len(sample_memories),\n", + " \"top_result_validations\": f\"{(evaluation_summary['status'] == 'PASS').sum()}/{len(evaluation_summary)} PASS\",\n", + " }\n", + " ]\n", + ")\n", + "display(scope_confirmation.style.hide(axis=\"index\"))" + ] + }, + { + "cell_type": "markdown", + "id": "25163f3b", + "metadata": {}, + "source": [ + "## What this demonstration proves\n", + "\n", + "- Oracle AI Agent Memory 26.6 can configure Oracle-managed hybrid search with `OracleDBEmbedder`.\n", + "- One scoped memory store can retrieve exact identifiers and semantically described events through the same search API.\n", + "- In this controlled dataset, all five expected memories were ranked first despite overlapping distractors.\n", + "- Saved outputs are generated by the real database-resident embedding model and hybrid index, not by a local mock.\n", + "\n", + "## What this demonstration does not prove\n", + "\n", + "- It is not a benchmark of general retrieval accuracy or latency.\n", + "- It does not isolate the individual contribution of vector and keyword signals.\n", + "- It does not compare separate vector-only, keyword-only, and hybrid stores.\n", + "\n", + "Those questions require a larger labeled dataset, repeated runs, fixed evaluation metrics, and separately configured retrieval strategies." + ] + }, + { + "cell_type": "markdown", + "id": "fd84ffb1", + "metadata": {}, + "source": [ + "## Practical Production Notes\n", + "\n", + "- **Initial schema setup:** use `SchemaPolicy.CREATE_IF_NECESSARY` intentionally when enabling hybrid search or upgrading a managed schema.\n", + "- **First hybrid index build:** Oracle scans existing search text and builds managed text/vector state. Treat this as a migration or maintenance operation for a large schema.\n", + "- **Normal startup:** after setup, consider `SchemaPolicy.REQUIRE_EXISTING` so the application validates rather than modifies schema objects.\n", + "- **`ON_COMMIT`:** newly committed memories become searchable immediately; useful for notebooks and interactive applications.\n", + "- **`MANUAL`:** appropriate for bulk loads and controlled refresh workflows.\n", + "- **`AUTO`:** Oracle-managed background refresh for hybrid search when some freshness lag is acceptable. `AUTO` is not supported for keyword-only search.\n", + "- **Strategy transitions:** do not reopen a keyword/hybrid schema as vector-only without following the documented recreation or embedding-backfill guidance.\n", + "- **Advanced exact-text indexing:** store-level `index_texts`/`index_text` can index identifiers, aliases, and short phrases while keeping visible record content concise." + ] + }, + { + "cell_type": "markdown", + "id": "9aa333e3", + "metadata": {}, + "source": [ + "## Summary\n", + "\n", + "This notebook demonstrates Oracle-managed hybrid search with the Oracle AI Agent Memory 26.6 API and a database-resident embedding model. It stores durable scoped memories, searches by exact identifiers and semantic questions, and validates the expected top results against deliberately similar records.\n", + "\n", + "The consolidated evaluation keeps the output readable while preserving the evidence needed for review: five real queries, five rank-1 validations, one distance overview, and one representative ranking. The conceptual retrieval-mode table provides context without claiming an unexecuted vector-versus-keyword benchmark." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.14.4" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} From 3b723eb75b11c0991a23fe63175626c7d4473773 Mon Sep 17 00:00:00 2001 From: Srinidhi Sathyamurthy Date: Fri, 10 Jul 2026 13:18:14 +0100 Subject: [PATCH 2/2] Updated review comment changes --- ...hybrid_search_agent_memory_reviewed.ipynb} | 611 +++++++++++------- 1 file changed, 370 insertions(+), 241 deletions(-) rename notebooks/{hybrid_search_agent_memory.ipynb => hybrid_search_agent_memory_reviewed.ipynb} (82%) diff --git a/notebooks/hybrid_search_agent_memory.ipynb b/notebooks/hybrid_search_agent_memory_reviewed.ipynb similarity index 82% rename from notebooks/hybrid_search_agent_memory.ipynb rename to notebooks/hybrid_search_agent_memory_reviewed.ipynb index 922cd55b..5485a09b 100644 --- a/notebooks/hybrid_search_agent_memory.ipynb +++ b/notebooks/hybrid_search_agent_memory_reviewed.ipynb @@ -34,7 +34,7 @@ "source": [ "## Workflow\n", "\n", - "1. Validate Python and package versions.\n", + "1. Validate the Python runtime and install Oracle AI Agent Memory 26.6.0 from PyPI.\n", "2. Load database configuration without displaying secrets.\n", "3. Connect to Oracle AI Database.\n", "4. Discover database-resident embedding models.\n", @@ -51,8 +51,8 @@ "source": [ "## Prerequisites\n", "\n", - "- Python 3.10-3.13. This project uses the dedicated **Python 3.11 - OAM Hybrid Search** kernel.\n", - "- Oracle AI Agent Memory 26.6.0 installed from the internal package source.\n", + "- Python 3.10-3.13. Oracle AI Agent Memory 26.6.0 requires Python 3.10 or later and earlier than Python 3.14.\n", + "- Oracle AI Agent Memory 26.6.0 installed from PyPI by the setup cell below.\n", "- Access to Oracle AI Database.\n", "- A database-resident embedding model for the complete hybrid-search run.\n", "- An `.env` file in the project folder:\n", @@ -65,22 +65,37 @@ "ORACLE_DB_EMBEDDING_DIMENSION=384\n", "```\n", "\n", - "The first three settings are sufficient for connection and model discovery. The model name and its correct dimension are required before creating `OracleDBEmbedder`." + "The first three settings establish the database connection. The model name and its correct dimension are required before creating `OracleDBEmbedder`. Keep the real `.env` file private and do not commit it to source control." ] }, { - "cell_type": "code", - "execution_count": 5, - "id": "91633643", + "cell_type": "markdown", + "id": "daa684b6", "metadata": {}, + "source": [ + "## Install and Validate the Environment\n", + "\n", + "This single setup cell validates the active Python runtime, installs Oracle AI Agent Memory 26.6.0 and the notebook dependencies into the current kernel without displaying package indexes or local paths, and confirms the installed package version.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "9d5a8433", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-08T17:07:33.368778Z", + "iopub.status.busy": "2026-07-08T17:07:33.368778Z", + "iopub.status.idle": "2026-07-08T17:07:35.866405Z", + "shell.execute_reply": "2026-07-08T17:07:35.865393Z" + } + }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Python: 3.13.14\n", - "oracleagentmemory: 26.6.0\n", - "Runtime preflight: READY\n" + "Python and Oracle AI Agent Memory environment: READY\n" ] } ], @@ -88,15 +103,25 @@ "import sys\n", "from importlib.metadata import version\n", "\n", - "python_supported = (3, 10) <= sys.version_info[:2] <= (3, 13)\n", - "package_version = version(\"oracleagentmemory\")\n", + "from IPython.utils.capture import capture_output\n", + "\n", + "\n", + "assert (3, 10) <= sys.version_info[:2] <= (3, 13), (\n", + " f\"Python 3.10-3.13 is required; found {sys.version.split()[0]}\"\n", + ")\n", + "\n", + "with capture_output():\n", + " get_ipython().run_line_magic(\n", + " \"pip\",\n", + " \"install oracleagentmemory==26.6.0 oracledb pandas matplotlib python-dotenv --quiet\",\n", + " )\n", "\n", - "assert python_supported, f\"Python 3.10-3.13 is required; found {sys.version.split()[0]}\"\n", - "assert package_version == \"26.6.0\", f\"Oracle AI Agent Memory 26.6.0 is required; found {package_version}\"\n", + "package_version = version(\"oracleagentmemory\")\n", + "assert package_version == \"26.6.0\", (\n", + " f\"Oracle AI Agent Memory 26.6.0 is required; found {package_version}\"\n", + ")\n", "\n", - "print(f\"Python: {sys.version.split()[0]}\")\n", - "print(f\"oracleagentmemory: {package_version}\")\n", - "print(\"Runtime preflight: READY\")" + "print(\"Python and Oracle AI Agent Memory environment: READY\")" ] }, { @@ -104,22 +129,29 @@ "id": "f4e3286a", "metadata": {}, "source": [ - "## Step 1 - Load Configuration Safely\n", + "## Step 1 - Load Configuration and Connect to Oracle AI Database\n", "\n", - "The cell below loads `.env` and reports only whether required values are present. It never prints usernames, passwords, DSNs, or model credentials." + "This step loads the private `.env` file, validates the required settings, and opens one thin-mode `python-oracledb` connection. It reports only readiness and never displays usernames, passwords, DSNs, database versions, schema names, or model details.\n" ] }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 2, "id": "a8f027f6", - "metadata": {}, + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-08T17:07:35.866405Z", + "iopub.status.busy": "2026-07-08T17:07:35.866405Z", + "iopub.status.idle": "2026-07-08T17:07:37.199233Z", + "shell.execute_reply": "2026-07-08T17:07:37.197091Z" + } + }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Configuration: READY\n" + "Configuration and database connection: READY\n" ] } ], @@ -127,6 +159,7 @@ "import os\n", "from pathlib import Path\n", "\n", + "import oracledb\n", "import pandas as pd\n", "from dotenv import load_dotenv\n", "\n", @@ -135,60 +168,28 @@ " for candidate in [start, *start.parents]:\n", " if (candidate / \".env\").exists():\n", " return candidate\n", - " raise FileNotFoundError(\"No .env file was found in the current directory or its parents.\")\n", + " raise FileNotFoundError(\n", + " \"No .env file was found. Create it in the notebook folder before continuing.\"\n", + " )\n", "\n", "\n", "PROJECT_DIR = find_project_dir(Path.cwd())\n", - "ENV_PATH = PROJECT_DIR / \".env\"\n", - "load_dotenv(ENV_PATH, override=False)\n", + "load_dotenv(PROJECT_DIR / \".env\", override=False)\n", "\n", "CONFIG = {\n", " \"ORACLE_USER\": os.getenv(\"ORACLE_USER\", \"\").strip(),\n", " \"ORACLE_PASSWORD\": os.getenv(\"ORACLE_PASSWORD\", \"\"),\n", " \"ORACLE_DSN\": os.getenv(\"ORACLE_DSN\", \"\").strip(),\n", " \"ORACLE_DB_EMBEDDING_MODEL\": os.getenv(\"ORACLE_DB_EMBEDDING_MODEL\", \"\").strip(),\n", - " \"ORACLE_DB_EMBEDDING_DIMENSION\": int(os.getenv(\"ORACLE_DB_EMBEDDING_DIMENSION\", \"384\")),\n", + " \"ORACLE_DB_EMBEDDING_DIMENSION\": int(\n", + " os.getenv(\"ORACLE_DB_EMBEDDING_DIMENSION\", \"384\")\n", + " ),\n", "}\n", "\n", - "connection_keys = [\"ORACLE_USER\", \"ORACLE_PASSWORD\", \"ORACLE_DSN\"]\n", - "missing_connection_keys = [key for key in connection_keys if not CONFIG[key]]\n", - "\n", - "if missing_connection_keys:\n", - " raise RuntimeError(f\"Missing required database settings: {', '.join(missing_connection_keys)}\")\n", - "\n", - "print(\"Configuration: READY\")" - ] - }, - { - "cell_type": "markdown", - "id": "9c79e5f1", - "metadata": {}, - "source": [ - "## Step 2 - Connect to Oracle AI Database\n", - "\n", - "This notebook uses one thin-mode `python-oracledb` connection. It validates connectivity without displaying environment-specific database or schema details." - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "id": "641d420e", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Database connection: READY\n" - ] - } - ], - "source": [ - "import oracledb\n", - "\n", - "connection = None\n", - "DB_READY = False\n", - "DB_BLOCKER = \"\"\n", + "required_connection_keys = [\"ORACLE_USER\", \"ORACLE_PASSWORD\", \"ORACLE_DSN\"]\n", + "missing = [key for key in required_connection_keys if not CONFIG[key]]\n", + "if missing:\n", + " raise RuntimeError(f\"Missing required database settings: {', '.join(missing)}\")\n", "\n", "try:\n", " connection = oracledb.connect(\n", @@ -196,17 +197,16 @@ " password=CONFIG[\"ORACLE_PASSWORD\"],\n", " dsn=CONFIG[\"ORACLE_DSN\"],\n", " )\n", - " DB_READY = True\n", - " print(\"Database connection: READY\")\n", "except oracledb.Error as exc:\n", " error = exc.args[0]\n", - " DB_BLOCKER = (\n", - " \"Oracle Database is not reachable with the current DSN. Start the local database/listener \"\n", - " \"or update ORACLE_DSN to a reachable database before running hybrid search.\"\n", - " )\n", - " print(\"Database connection: BLOCKED\")\n", - " print(f\"Oracle driver error code: {getattr(error, 'code', 'unknown')}\")\n", - " print(DB_BLOCKER)" + " raise RuntimeError(\n", + " \"Oracle Database is not reachable with the configured connection. \"\n", + " f\"Oracle driver error code: {getattr(error, 'code', 'unknown')}\"\n", + " ) from exc\n", + "\n", + "DB_READY = True\n", + "DB_BLOCKER = \"\"\n", + "print(\"Configuration and database connection: READY\")" ] }, { @@ -214,7 +214,7 @@ "id": "20773aee", "metadata": {}, "source": [ - "## Step 3 - Validate the Database Embedding Model\n", + "## Step 2 - Validate the Database Embedding Model\n", "\n", "Oracle-managed hybrid search requires `OracleDBEmbedder`, which in turn requires a database-resident embedding model. This read-only preflight checks whether the model configured in `.env` is available to the connected application.\n", "\n", @@ -223,9 +223,16 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 3, "id": "8091333e", - "metadata": {}, + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-08T17:07:37.204101Z", + "iopub.status.busy": "2026-07-08T17:07:37.202984Z", + "iopub.status.idle": "2026-07-08T17:07:37.227853Z", + "shell.execute_reply": "2026-07-08T17:07:37.225578Z" + } + }, "outputs": [ { "name": "stdout", @@ -334,16 +341,23 @@ "id": "242f1780", "metadata": {}, "source": [ - "## Step 4 - Configure Oracle AI Agent Memory for Hybrid Search\n", + "## Step 3 - Configure Oracle AI Agent Memory for Hybrid Search\n", "\n", "`SearchStrategy.HYBRID` combines Oracle-managed text matching with vector-aware ranking. `OracleDBEmbedder` ensures the memory client and the managed hybrid vector index use the same in-database model. `ON_COMMIT` makes newly committed memories searchable immediately, which is appropriate for this interactive walkthrough." ] }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 4, "id": "8dbf0b1e", - "metadata": {}, + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-08T17:07:37.232678Z", + "iopub.status.busy": "2026-07-08T17:07:37.232135Z", + "iopub.status.idle": "2026-07-08T17:07:37.751839Z", + "shell.execute_reply": "2026-07-08T17:07:37.748428Z" + } + }, "outputs": [ { "name": "stdout", @@ -375,9 +389,16 @@ }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 5, "id": "29cc7e8a", - "metadata": {}, + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-08T17:07:37.757263Z", + "iopub.status.busy": "2026-07-08T17:07:37.757263Z", + "iopub.status.idle": "2026-07-08T17:07:39.138399Z", + "shell.execute_reply": "2026-07-08T17:07:39.138399Z" + } + }, "outputs": [ { "name": "stdout", @@ -422,16 +443,23 @@ "id": "50f01b42", "metadata": {}, "source": [ - "## Step 5 - Register a Scoped User and Agent\n", + "## Step 4 - Register a Scoped User and Agent\n", "\n", "Scope determines which records are eligible for retrieval. Hybrid search then ranks only those eligible records. A unique run identifier keeps repeated notebook runs isolated while preserving one stable managed memory store." ] }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 6, "id": "e3974b5b", - "metadata": {}, + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-08T17:07:39.138399Z", + "iopub.status.busy": "2026-07-08T17:07:39.138399Z", + "iopub.status.idle": "2026-07-08T17:07:39.399528Z", + "shell.execute_reply": "2026-07-08T17:07:39.399159Z" + } + }, "outputs": [ { "name": "stdout", @@ -468,7 +496,7 @@ "id": "33706a04", "metadata": {}, "source": [ - "## Step 6 - Build a Deliberately Ambiguous Memory Set\n", + "## Step 5 - Build a Deliberately Ambiguous Memory Set\n", "\n", "The dataset combines exact operational handles with natural-language context. It also includes carefully chosen distractors: a neighboring invoice, a different customer's renewal delay, a general error-code explanation, and a customer-alias record.\n", "\n", @@ -477,80 +505,87 @@ }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 7, "id": "5c8a8fdf", - "metadata": {}, + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-08T17:07:39.402851Z", + "iopub.status.busy": "2026-07-08T17:07:39.402851Z", + "iopub.status.idle": "2026-07-08T17:07:40.037637Z", + "shell.execute_reply": "2026-07-08T17:07:40.037637Z" + } + }, "outputs": [ { "data": { "text/html": [ "\n", - "\n", + "
\n", " \n", " \n", " \n", - " \n", - " \n", - " \n", - " \n", + " \n", + " \n", + " \n", + " \n", " \n", " \n", " \n", " \n", - " \n", - " \n", - " \n", - " \n", + " \n", + " \n", + " \n", + " \n", " \n", " \n", - " \n", - " \n", - " \n", - " \n", + " \n", + " \n", + " \n", + " \n", " \n", " \n", - " \n", - " \n", - " \n", - " \n", + " \n", + " \n", + " \n", + " \n", " \n", " \n", - " \n", - " \n", - " \n", - " \n", + " \n", + " \n", + " \n", + " \n", " \n", " \n", - " \n", - " \n", - " \n", - " \n", + " \n", + " \n", + " \n", + " \n", " \n", " \n", "
Memory inventory used by the retrieval evaluation
labelexact_signalsemantic_contextevaluation_rolelabelexact_signalsemantic_contextevaluation_role
renewal_blockerINV-48291 · ORA-27102 · Northstar RenewalsFailed reconciliation blocked a renewalPrimary targetrenewal_blockerINV-48291 · ORA-27102 · Northstar RenewalsFailed reconciliation blocked a renewalPrimary target
similar_invoiceINV-48290 · Northstar RenewalsNeighboring invoice, already paidExact-ID distractorsimilar_invoiceINV-48290 · Northstar RenewalsNeighboring invoice, already paidExact-ID distractor
milan_renewalMilan Office SuppliesRenewal delayed by approval workflowSemantic distractormilan_renewalMilan Office SuppliesRenewal delayed by approval workflowSemantic distractor
error_referenceORA-27102General error-code explanationExact-code targeterror_referenceORA-27102General error-code explanationExact-code target
customer_aliasNorthstar RenewalsEnterprise account mappingAlias distractorcustomer_aliasNorthstar RenewalsEnterprise account mappingAlias distractor
\n" ], "text/plain": [ - "" + "" ] }, "metadata": {}, @@ -671,7 +706,7 @@ "id": "36151b5f", "metadata": {}, "source": [ - "## Step 7 - Define One Reusable Search Path\n", + "## Step 6 - Define One Reusable Search Path\n", "\n", "Every evaluation query uses the same documented `SearchScope` overload and the same Oracle-managed hybrid index. This keeps the comparison focused on the query rather than on changes in configuration.\n", "\n", @@ -680,9 +715,16 @@ }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 8, "id": "9e31df8a", - "metadata": {}, + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-08T17:07:40.037637Z", + "iopub.status.busy": "2026-07-08T17:07:40.037637Z", + "iopub.status.idle": "2026-07-08T17:07:40.049280Z", + "shell.execute_reply": "2026-07-08T17:07:40.048516Z" + } + }, "outputs": [ { "name": "stdout", @@ -724,7 +766,7 @@ "id": "371b33bd", "metadata": {}, "source": [ - "## Step 8 - Run the Retrieval Evaluation\n", + "## Step 7 - Run the Retrieval Evaluation\n", "\n", "### What to observe\n", "\n", @@ -739,34 +781,41 @@ }, { "cell_type": "code", - "execution_count": 14, + "execution_count": 9, "id": "609474e5", - "metadata": {}, + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-08T17:07:40.053538Z", + "iopub.status.busy": "2026-07-08T17:07:40.053538Z", + "iopub.status.idle": "2026-07-08T17:07:40.465843Z", + "shell.execute_reply": "2026-07-08T17:07:40.465843Z" + } + }, "outputs": [ { "data": { "text/html": [ "\n", - "\n", + "
\n", " \n", " \n", " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", " \n", " \n", " \n", " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", " \n", " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", " \n", " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", " \n", " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", " \n", " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", " \n", " \n", "
Hybrid retrieval evaluation summary
categoryqueryexpectedtop_resultdistancestatuscategoryqueryexpectedtop_resultdistancestatus
Exact identifierINV-48291renewal_blockerrenewal_blocker0.4191PASSExact identifierINV-48291renewal_blockerrenewal_blocker0.4191PASS
Exact identifierORA-27102error_referenceerror_reference0.3757PASSExact identifierORA-27102error_referenceerror_reference0.3757PASS
SemanticWhat blocked the Northstar renewal?renewal_blockerrenewal_blocker0.2534PASSSemanticWhat blocked the Northstar renewal?renewal_blockerrenewal_blocker0.2534PASS
SemanticWhich issue delayed the Milan customer renewal?milan_renewalmilan_renewal0.2914PASSSemanticWhich issue delayed the Milan customer renewal?milan_renewalmilan_renewal0.2914PASS
MixedWhich invoice failed reconciliation for Northstar?renewal_blockerrenewal_blocker0.2152PASSMixedWhich invoice failed reconciliation for Northstar?renewal_blockerrenewal_blocker0.2152PASS
\n" ], "text/plain": [ - "" + "" ] }, "metadata": {}, @@ -933,16 +982,23 @@ "id": "9c819296", "metadata": {}, "source": [ - "## Step 9 - Compare Top-Result Strength\n", + "## Step 8 - Compare Top-Result Strength\n", "\n", "The chart below makes the five successful cases easier to compare at a glance. Distance is shown only to explain this run: **smaller distance indicates a stronger match**. Different query shapes naturally produce different values, so this is not a cross-system benchmark." ] }, { "cell_type": "code", - "execution_count": 15, + "execution_count": 10, "id": "bf0ba462", - "metadata": {}, + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-08T17:07:40.465843Z", + "iopub.status.busy": "2026-07-08T17:07:40.465843Z", + "iopub.status.idle": "2026-07-08T17:07:41.338959Z", + "shell.execute_reply": "2026-07-08T17:07:41.338959Z" + } + }, "outputs": [ { "data": { @@ -989,83 +1045,90 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 11, "id": "b4f6a8a7", - "metadata": {}, + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-08T17:07:41.338959Z", + "iopub.status.busy": "2026-07-08T17:07:41.338959Z", + "iopub.status.idle": "2026-07-08T17:07:41.359389Z", + "shell.execute_reply": "2026-07-08T17:07:41.359389Z" + } + }, "outputs": [ { "data": { "text/html": [ "\n", - "\n", + "
\n", " \n", " \n", " \n", - " \n", - " \n", - " \n", - " \n", + " \n", + " \n", + " \n", + " \n", " \n", " \n", " \n", " \n", - " \n", - " \n", - " \n", - " \n", + " \n", + " \n", + " \n", + " \n", " \n", " \n", - " \n", - " \n", - " \n", - " \n", + " \n", + " \n", + " \n", + " \n", " \n", " \n", - " \n", - " \n", - " \n", - " \n", + " \n", + " \n", + " \n", + " \n", " \n", " \n", - " \n", - " \n", - " \n", - " \n", + " \n", + " \n", + " \n", + " \n", " \n", " \n", - " \n", - " \n", - " \n", - " \n", + " \n", + " \n", + " \n", + " \n", " \n", " \n", "
Detailed ranking for: Which invoice failed reconciliation for Northstar?
rankdistancelabelcontentrankdistancelabelcontent
10.2152renewal_blockerNorthstar Renewals cannot proceed with its renewal because invoice INV-48291 failed reconciliation after ORA-27102 during month-end processing. Finance must rerun the reconciliation job before the renewal can continue.10.2152renewal_blockerNorthstar Renewals cannot proceed with its renewal because invoice INV-48291 failed reconciliation after ORA-27102 during month-end processing. Finance must rerun the reconciliation job before the renewal can continue.
20.2974similar_invoiceNorthstar Renewals also has invoice INV-48290 marked as paid. No finance follow-up is required for that invoice.20.2974similar_invoiceNorthstar Renewals also has invoice INV-48290 marked as paid. No finance follow-up is required for that invoice.
30.4134customer_aliasNorthstar Renewals maps to the enterprise account Northstar Renewal Services Ltd. Finance operations owns escalations for that account.30.4134customer_aliasNorthstar Renewals maps to the enterprise account Northstar Renewal Services Ltd. Finance operations owns escalations for that account.
40.4265milan_renewalMilan Office Supplies experienced a renewal delay because its approval workflow was waiting for a regional manager. No reconciliation error was reported.40.4265milan_renewalMilan Office Supplies experienced a renewal delay because its approval workflow was waiting for a regional manager. No reconciliation error was reported.
50.5170error_referenceORA-27102 can indicate memory allocation pressure. Database memory settings should be checked before rerunning affected batch processing.50.5170error_referenceORA-27102 can indicate memory allocation pressure. Database memory settings should be checked before rerunning affected batch processing.
\n" ], "text/plain": [ - "" + "" ] }, "metadata": {}, @@ -1106,68 +1169,75 @@ }, { "cell_type": "code", - "execution_count": 17, + "execution_count": 12, "id": "50301865", - "metadata": {}, + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-08T17:07:41.364286Z", + "iopub.status.busy": "2026-07-08T17:07:41.364286Z", + "iopub.status.idle": "2026-07-08T17:07:41.377992Z", + "shell.execute_reply": "2026-07-08T17:07:41.377992Z" + } + }, "outputs": [ { "data": { "text/html": [ "\n", - "\n", + "
\n", " \n", " \n", " \n", - " \n", - " \n", - " \n", - " \n", + " \n", + " \n", + " \n", + " \n", " \n", " \n", " \n", " \n", - " \n", - " \n", - " \n", - " \n", + " \n", + " \n", + " \n", + " \n", " \n", " \n", - " \n", - " \n", - " \n", - " \n", + " \n", + " \n", + " \n", + " \n", " \n", " \n", - " \n", - " \n", - " \n", - " \n", + " \n", + " \n", + " \n", + " \n", " \n", " \n", "
Conceptual retrieval-mode comparison
modebest_atweak_spotexamplemodebest_atweak_spotexample
VectorMeaning, paraphrase, and conceptual similarityShort identifiers can be underweightedWhat blocked the renewal?VectorMeaning, paraphrase, and conceptual similarityShort identifiers can be underweightedWhat blocked the renewal?
KeywordExact IDs, error codes, aliases, and filenamesDifferent wording can miss the stored recordINV-48291KeywordExact IDs, error codes, aliases, and filenamesDifferent wording can miss the stored recordINV-48291
HybridNatural-language intent plus exact operational handlesRequires database embedding and managed index setupWhy did INV-48291 block the renewal?HybridNatural-language intent plus exact operational handlesRequires database embedding and managed index setupWhy did INV-48291 block the renewal?
\n" ], "text/plain": [ - "" + "" ] }, "metadata": {}, @@ -1229,36 +1299,43 @@ }, { "cell_type": "code", - "execution_count": 18, + "execution_count": 13, "id": "8d91ea61", - "metadata": {}, + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-08T17:07:41.382520Z", + "iopub.status.busy": "2026-07-08T17:07:41.382520Z", + "iopub.status.idle": "2026-07-08T17:07:41.392855Z", + "shell.execute_reply": "2026-07-08T17:07:41.392855Z" + } + }, "outputs": [ { "data": { "text/html": [ "\n", - "\n", + "
\n", " \n", " \n", - " \n", - " \n", - " \n", - " \n", + " \n", + " \n", + " \n", + " \n", " \n", " \n", " \n", " \n", - " \n", - " \n", - " \n", - " \n", + " \n", + " \n", + " \n", + " \n", " \n", " \n", "
scopeevaluated_querieseligible_demo_memoriestop_result_validationsscopeevaluated_querieseligible_demo_memoriestop_result_validations
Generated finance user + support-finance agent555/5 PASSGenerated finance user + support-finance agent555/5 PASS
\n" ], "text/plain": [ - "" + "" ] }, "metadata": {}, @@ -1317,6 +1394,58 @@ "- **Advanced exact-text indexing:** store-level `index_texts`/`index_text` can index identifiers, aliases, and short phrases while keeping visible record content concise." ] }, + { + "cell_type": "markdown", + "id": "44a3dbf9", + "metadata": {}, + "source": [ + "## Resources\n", + "\n", + "- [Oracle AI Agent Memory 26.6.0 on PyPI](https://pypi.org/project/oracleagentmemory/26.6.0/)\n", + "- [Oracle AI Agent Memory documentation](https://docs.oracle.com/en/database/oracle/agent-memory/26.4/)\n", + "- [Oracle AI Vector Search User's Guide](https://docs.oracle.com/en/database/oracle/oracle-database/26/vecse/)" + ] + }, + { + "cell_type": "markdown", + "id": "ab262bb5", + "metadata": {}, + "source": [ + "## Optional Cleanup\n", + "\n", + "The demo uses unique user and agent identifiers so repeated runs do not collide. Run the following cell when you no longer need to inspect this run's records. It removes the generated demo scope and closes the client and database connection. Leave it unexecuted if you want to inspect the stored records after the walkthrough." + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "433dd548", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-08T17:07:41.397899Z", + "iopub.status.busy": "2026-07-08T17:07:41.396890Z", + "iopub.status.idle": "2026-07-08T17:07:41.488519Z", + "shell.execute_reply": "2026-07-08T17:07:41.488519Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Demo data removed and database connection closed.\n" + ] + } + ], + "source": [ + "deleted_user_records = memory.delete_user(USER_ID, cascade=True)\n", + "deleted_agent_records = memory.delete_agent(AGENT_ID, cascade=True)\n", + "memory.close()\n", + "connection.close()\n", + "\n", + "print(\"Demo data removed and database connection closed.\")" + ] + }, { "cell_type": "markdown", "id": "9aa333e3", @@ -1332,7 +1461,7 @@ ], "metadata": { "kernelspec": { - "display_name": "Python 3", + "display_name": ".venv (3.11.6.final.0)", "language": "python", "name": "python3" }, @@ -1346,7 +1475,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.14.4" + "version": "3.11.6" } }, "nbformat": 4,