sdk/python/foundation-models/system/inference/summarization/summarization-online-endpoint.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summarization Inference using Online Endpoints\n",
    "\n",
    "This sample shows how to deploy `summarization` type models to an online endpoint for inference.\n",
    "\n",
    "### Task\n",
    "`summarization` creates a shorter version of a document or an article that captures all the important information. Along with translation, it is another example of a task that can be formulated as a sequence-to-sequence task. \n",
    "`summarization` can be:\n",
    "\n",
    "* Extractive: extract the most relevant information from a document.\n",
    "* Abstractive: generate new text that captures the most relevant information.\n",
    "\n",
    "### Model\n",
    "Models that can perform the `summarization` task are tagged with `task: summarization`. We will use the `sshleifer-distilbart-cnn-12-6` model in this notebook. If you opened this notebook from a specific model card, remember to replace the specific model name. If you don't find a model that suits your scenario or domain, you can discover and [import models from HuggingFace hub](../../import/import_model_into_registry.ipynb) and then use them for inference. \n",
    "\n",
    "### Inference data\n",
    "We will use the [CNN DailyMail](https://huggingface.co/datasets/cnn_dailymail) dataset.\n",
    "\n",
    "### Outline\n",
    "* Set up pre-requisites.\n",
    "* Pick a model to deploy.\n",
    "* Download and prepare data for inference. \n",
    "* Deploy the model for real time inference.\n",
    "* Test the endpoint\n",
    "* Clean up resources."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Set up pre-requisites\n",
    "* Install dependencies\n",
    "* Connect to AzureML Workspace. Learn more at [set up SDK authentication](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-setup-authentication?tabs=sdk). Replace  `<WORKSPACE_NAME>`, `<RESOURCE_GROUP>` and `<SUBSCRIPTION_ID>` below.\n",
    "* Connect to `azureml` system registry"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.ai.ml import MLClient\n",
    "from azure.identity import (\n",
    "    DefaultAzureCredential,\n",
    "    InteractiveBrowserCredential,\n",
    "    ClientSecretCredential,\n",
    ")\n",
    "from azure.ai.ml.entities import AmlCompute\n",
    "import time\n",
    "\n",
    "try:\n",
    "    credential = DefaultAzureCredential()\n",
    "    credential.get_token(\"https://management.azure.com/.default\")\n",
    "except Exception as ex:\n",
    "    credential = InteractiveBrowserCredential()\n",
    "\n",
    "workspace_ml_client = MLClient(\n",
    "    credential,\n",
    "    subscription_id=\"<SUBSCRIPTION_ID>\",\n",
    "    resource_group_name=\"<RESOURCE_GROUP>\",\n",
    "    workspace_name=\"<WORKSPACE_NAME>\",\n",
    ")\n",
    "# the models, fine tuning pipelines and environments are available in the AzureML system registry, \"azureml\"\n",
    "registry_ml_client = MLClient(credential, registry_name=\"azureml\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Pick a model to deploy\n",
    "\n",
    "Browse models in the Model Catalog in the AzureML Studio, filtering by the `summarization` task. In this example, we use the `sshleifer-distilbart-cnn-12-6` model. If you have opened this notebook for a different model, replace the model name and version accordingly. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_name = \"sshleifer-distilbart-cnn-12-6\"\n",
    "version_list = list(registry_ml_client.models.list(model_name))\n",
    "if len(version_list) == 0:\n",
    "    print(\"Model not found in registry\")\n",
    "else:\n",
    "    model_version = version_list[0].version\n",
    "    foundation_model = registry_ml_client.models.get(model_name, model_version)\n",
    "    print(\n",
    "        \"\\n\\nUsing model name: {0}, version: {1}, id: {2} for inferencing\".format(\n",
    "            foundation_model.name, foundation_model.version, foundation_model.id\n",
    "        )\n",
    "    )"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Download and prepare data for inference.\n",
    "\n",
    "The next few cells show basic data preparation:\n",
    "* Visualize some data rows\n",
    "* Save few samples in the format that can be passed as input to the online-inference endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download a small sample of the dataset into the ./news-summary-dataset directory\n",
    "%run ./news-summary-dataset/download-dataset.py --download_dir ./news-summary-dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the ./news-summary-dataset/train.jsonl file into a pandas dataframe and show the first 5 rows\n",
    "import pandas as pd\n",
    "\n",
    "pd.set_option(\n",
    "    \"display.max_colwidth\", 0\n",
    ")  # set the max column width to 0 to display the full text\n",
    "train_df = pd.read_json(\"./news-summary-dataset/train.jsonl\", lines=True)\n",
    "train_df.head(2)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Deploy the model to an online endpoint\n",
    "Online endpoints give a durable REST API that can be used to integrate with applications that need to use the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time, sys\n",
    "from azure.ai.ml.entities import (\n",
    "    ManagedOnlineEndpoint,\n",
    "    ManagedOnlineDeployment,\n",
    "    OnlineRequestSettings,\n",
    ")\n",
    "\n",
    "# Create online endpoint - endpoint names need to be unique in a region, hence using timestamp to create unique endpoint name\n",
    "timestamp = int(time.time())\n",
    "online_endpoint_name = \"summarization-\" + str(timestamp)\n",
    "# create an online endpoint\n",
    "endpoint = ManagedOnlineEndpoint(\n",
    "    name=online_endpoint_name,\n",
    "    description=\"Online endpoint for \"\n",
    "    + foundation_model.name\n",
    "    + \", for summarization task\",\n",
    "    auth_mode=\"key\",\n",
    ")\n",
    "workspace_ml_client.begin_create_or_update(endpoint).wait()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a deployment\n",
    "demo_deployment = ManagedOnlineDeployment(\n",
    "    name=\"demo\",\n",
    "    endpoint_name=online_endpoint_name,\n",
    "    model=foundation_model.id,\n",
    "    instance_type=\"Standard_DS3_v2\",\n",
    "    instance_count=1,\n",
    "    request_settings=OnlineRequestSettings(\n",
    "        request_timeout_ms=60000,\n",
    "    ),\n",
    ")\n",
    "workspace_ml_client.online_deployments.begin_create_or_update(demo_deployment).wait()\n",
    "endpoint.traffic = {\"demo\": 100}\n",
    "workspace_ml_client.begin_create_or_update(endpoint).result()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Test the endpoint with sample data\n",
    "\n",
    "We will fetch some sample data from the test dataset and submit to online endpoint for inference. We will then show the display the scored labels alongside the ground truth labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import os\n",
    "\n",
    "# read the ./news-summary-dataset/train.jsonl file into a pandas dataframe\n",
    "df = pd.read_json(\"./news-summary-dataset/train.jsonl\", lines=True)\n",
    "# escape single and double quotes in the masked_text column\n",
    "df[\"article\"] = df[\"article\"].str.replace(\"'\", \"\\\\'\").str.replace('\"', '\\\\\"')\n",
    "# pick 1 random row\n",
    "sample_df = df.sample(1)\n",
    "# create a json object with the key as \"inputs\" and value as a list of values from the article column of the sample_df dataframe\n",
    "sample_json = {\"inputs\": sample_df[\"article\"].tolist()}\n",
    "# save the json object to a file named sample_score.json in the ./news-summary-dataset folder\n",
    "test_json = {\"inputs\": {\"input_string\": sample_df[\"article\"].tolist()}}\n",
    "# save the json object to a file named sample_score.json in the ./news-summary-dataset folder\n",
    "with open(os.path.join(\".\", \"news-summary-dataset\", \"sample_score.json\"), \"w\") as f:\n",
    "    json.dump(test_json, f)\n",
    "sample_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# score the sample_score.json file using the online endpoint with the azureml endpoint invoke method\n",
    "response = workspace_ml_client.online_endpoints.invoke(\n",
    "    endpoint_name=online_endpoint_name,\n",
    "    deployment_name=\"demo\",\n",
    "    request_file=\"./news-summary-dataset/sample_score.json\",\n",
    ")\n",
    "print(\"raw response: \\n\", response, \"\\n\")\n",
    "# convert the json response to a pandas dataframe\n",
    "response_df = pd.read_json(response)\n",
    "response_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# compare the true summary with the predicted summary\n",
    "response_df.rename(columns={0: \"predicted_summary\"}, inplace=True)\n",
    "response_df[\"ground_truth_summary\"] = [sample_df[\"highlights\"].tolist()[0]]\n",
    "response_df.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Delete the online endpoint\n",
    "Don't forget to delete the online endpoint, else you will leave the billing meter running for the compute used by the endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "workspace_ml_client.online_endpoints.begin_delete(name=online_endpoint_name).wait()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "2f394aca7ca06fed1e6064aef884364492d7cdda3614a461e02e6407fc40ba69"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}