bert_embeddings/bert_embeddings.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BERT Embeddings Serverless Function\n",
    "This notebook presents deployment of pretrained BERT model that outputs embeddings for given textual sequences as a serverless function. Embeddings are meaningful, contextual representations of text in the form of ndarrays that are used frequently as input to various learning tasks in the field of NLP."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Both server & client are aligned (0.6.5).\n"
     ]
    }
   ],
   "source": [
    "!/User/align_mlrun.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Embeddings without bert"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[One-Hot Encoding](https://en.wikipedia.org/wiki/One-hot) is a general method that can vectorize any categorical features. It is simple and fast to create and update the vectorization.<br>\n",
    "in case of <b>text</b> embeddings, each <b>row</b> is a <b>sentence</b> and each <b>column</b> is a <b>word/char/[n-gram](https://en.wikipedia.org/wiki/N-gram)</b>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# some sentences to do examine\n",
    "sentences = ['the quick brown fox jumps over the lazy dog',\n",
    "              'Hello I am Jacob',\n",
    "              'Daniel visited Tel-Aviv last month']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "lets see the difference between bert embeddings and one-hot encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['the', 'quick', 'brown', 'fox', 'jumps', 'over', 'lazy', 'dog', 'Hello', 'I', 'am', 'Jacob', 'Daniel', 'visited', 'Tel-Aviv', 'last', 'month']\n"
     ]
    }
   ],
   "source": [
    "# constructing a list of all the words (will be our columns) - make sure no duplicate words are set\n",
    "tokens = []\n",
    "for sentence in sentences:\n",
    "    for word in sentence.split():\n",
    "        tokens.append(word) if word not in tokens else \"\"\n",
    "print(tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# constructing the one hot vector\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "one_hot = pd.DataFrame(columns = range(len(tokens)))\n",
    "# filling our empty dataframe with each sentence encoding\n",
    "for sentence in sentences:\n",
    "    vector = np.zeros(len(tokens))\n",
    "    for word in sentence.split():\n",
    "        vector[tokens.index(word)]=1\n",
    "    one_hot = one_hot.append(pd.Series(vector),ignore_index=True)\n",
    "one_hot.columns = tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>the</th>\n",
       "      <th>quick</th>\n",
       "      <th>brown</th>\n",
       "      <th>fox</th>\n",
       "      <th>jumps</th>\n",
       "      <th>over</th>\n",
       "      <th>lazy</th>\n",
       "      <th>dog</th>\n",
       "      <th>Hello</th>\n",
       "      <th>I</th>\n",
       "      <th>am</th>\n",
       "      <th>Jacob</th>\n",
       "      <th>Daniel</th>\n",
       "      <th>visited</th>\n",
       "      <th>Tel-Aviv</th>\n",
       "      <th>last</th>\n",
       "      <th>month</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   the  quick  brown  fox  jumps  over  lazy  dog  Hello    I   am  Jacob  \\\n",
       "0  1.0    1.0    1.0  1.0    1.0   1.0   1.0  1.0    0.0  0.0  0.0    0.0   \n",
       "1  0.0    0.0    0.0  0.0    0.0   0.0   0.0  0.0    1.0  1.0  1.0    1.0   \n",
       "2  0.0    0.0    0.0  0.0    0.0   0.0   0.0  0.0    0.0  0.0  0.0    0.0   \n",
       "\n",
       "   Daniel  visited  Tel-Aviv  last  month  \n",
       "0     0.0      0.0       0.0   0.0    0.0  \n",
       "1     0.0      0.0       0.0   0.0    0.0  \n",
       "2     1.0      1.0       1.0   1.0    1.0  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "one_hot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The table above represents the one-hot encoding of our sentences, each row is a sentence and each column is a word.\n",
    "this representation is very slim and will be a very weak learning dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introducing Bert embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mlrun import import_function, auto_mount"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# importing the function from the hub\n",
    "fn = import_function(\"hub://bert_embeddings\").apply(auto_mount())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2021-08-11 14:49:02,307 [info] Starting remote function deploy\n",
      "2021-08-11 14:49:02  (info) Deploying function\n",
      "2021-08-11 14:49:02  (info) Building\n",
      "2021-08-11 14:49:03  (info) Staging files and preparing base images\n",
      "2021-08-11 14:49:03  (info) Building processor image\n",
      "2021-08-11 14:49:10  (info) Build complete\n",
      "2021-08-11 14:49:42  (info) Function deploy complete\n",
      "> 2021-08-11 14:49:43,742 [info] function deployed, address=default-tenant.app.dev39.lab.iguazeng.com:32500\n"
     ]
    }
   ],
   "source": [
    "# deploying the function\n",
    "addr = fn.deploy(project = \"function-marketplace\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import json\n",
    "# sending a request to the function endpoint to get the sentences' embeddings\n",
    "resp = requests.post(addr, json=json.dumps(sentences))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "output_embeddings = pickle.loads(resp.content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "embeddings per token shape: (3, 11, 768), pooled embeddings shape: (3, 768)\n"
     ]
    }
   ],
   "source": [
    "print(f'embeddings per token shape: {output_embeddings[0].shape}, pooled embeddings shape: {output_embeddings[1].shape}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>758</th>\n",
       "      <th>759</th>\n",
       "      <th>760</th>\n",
       "      <th>761</th>\n",
       "      <th>762</th>\n",
       "      <th>763</th>\n",
       "      <th>764</th>\n",
       "      <th>765</th>\n",
       "      <th>766</th>\n",
       "      <th>767</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.733322</td>\n",
       "      <td>-0.223540</td>\n",
       "      <td>0.342462</td>\n",
       "      <td>0.383463</td>\n",
       "      <td>-0.164796</td>\n",
       "      <td>0.040522</td>\n",
       "      <td>0.802845</td>\n",
       "      <td>0.152842</td>\n",
       "      <td>0.331639</td>\n",
       "      <td>-0.999779</td>\n",
       "      <td>...</td>\n",
       "      <td>0.206564</td>\n",
       "      <td>0.231415</td>\n",
       "      <td>0.196434</td>\n",
       "      <td>0.797908</td>\n",
       "      <td>0.435176</td>\n",
       "      <td>0.749370</td>\n",
       "      <td>0.246098</td>\n",
       "      <td>0.427603</td>\n",
       "      <td>-0.577384</td>\n",
       "      <td>0.842063</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.953005</td>\n",
       "      <td>-0.535132</td>\n",
       "      <td>-0.743821</td>\n",
       "      <td>0.893934</td>\n",
       "      <td>0.646276</td>\n",
       "      <td>-0.279388</td>\n",
       "      <td>0.943513</td>\n",
       "      <td>0.275504</td>\n",
       "      <td>-0.555108</td>\n",
       "      <td>-0.999992</td>\n",
       "      <td>...</td>\n",
       "      <td>0.582385</td>\n",
       "      <td>-0.004613</td>\n",
       "      <td>0.976079</td>\n",
       "      <td>0.931517</td>\n",
       "      <td>-0.391442</td>\n",
       "      <td>0.530384</td>\n",
       "      <td>0.675933</td>\n",
       "      <td>-0.682720</td>\n",
       "      <td>-0.746339</td>\n",
       "      <td>0.957809</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.843678</td>\n",
       "      <td>-0.453405</td>\n",
       "      <td>-0.826011</td>\n",
       "      <td>0.650805</td>\n",
       "      <td>0.494036</td>\n",
       "      <td>-0.154117</td>\n",
       "      <td>0.821642</td>\n",
       "      <td>0.349507</td>\n",
       "      <td>-0.650629</td>\n",
       "      <td>-0.999978</td>\n",
       "      <td>...</td>\n",
       "      <td>0.618286</td>\n",
       "      <td>-0.336700</td>\n",
       "      <td>0.936262</td>\n",
       "      <td>0.857577</td>\n",
       "      <td>-0.787489</td>\n",
       "      <td>0.246137</td>\n",
       "      <td>0.676243</td>\n",
       "      <td>-0.612531</td>\n",
       "      <td>-0.708786</td>\n",
       "      <td>0.840879</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 768 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        0         1         2         3         4         5         6    \\\n",
       "0 -0.733322 -0.223540  0.342462  0.383463 -0.164796  0.040522  0.802845   \n",
       "1 -0.953005 -0.535132 -0.743821  0.893934  0.646276 -0.279388  0.943513   \n",
       "2 -0.843678 -0.453405 -0.826011  0.650805  0.494036 -0.154117  0.821642   \n",
       "\n",
       "        7         8         9    ...       758       759       760       761  \\\n",
       "0  0.152842  0.331639 -0.999779  ...  0.206564  0.231415  0.196434  0.797908   \n",
       "1  0.275504 -0.555108 -0.999992  ...  0.582385 -0.004613  0.976079  0.931517   \n",
       "2  0.349507 -0.650629 -0.999978  ...  0.618286 -0.336700  0.936262  0.857577   \n",
       "\n",
       "        762       763       764       765       766       767  \n",
       "0  0.435176  0.749370  0.246098  0.427603 -0.577384  0.842063  \n",
       "1 -0.391442  0.530384  0.675933 -0.682720 -0.746339  0.957809  \n",
       "2 -0.787489  0.246137  0.676243 -0.612531 -0.708786  0.840879  \n",
       "\n",
       "[3 rows x 768 columns]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(output_embeddings[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we can see that the size of the first dimension of the outputs is three since we passed in three sequences. Also the intermediate dimension of the first output is the maximal number of tokens across all input sequences. Sequences with less tokens are padded with zero values.<br>\n",
    "Note that the first input has an intermediate dimension of size 11 that corresponds to the number of max tokens in the input sequence after addition of two special tokens marking beginning and end of a sequence by the tokenizer.<br>\n",
    "The last dimension for both is of size 768 which is the embedding dimension for this default configuration of bert.<br>\n",
    "Now you tell me, which encoding are you gonna use in your project ??"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}