04_ngram_cnn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# N-Gram detection with 1D Convolution\n",
    "\n",
    "In the previous examples, we have run tasks on embeddings in each word, but we sometimes should handle a set of ordered items.<br>\n",
    "For instance, \"hot dog\" won't be in the species of \"dog\", but it will be a part of food. \"Paris Hilton\" will also be far from \"Paris\" in language context. Even when you find \"good\" in the sentence, it might be a signal of negative sentiment in the context \"not good\".<br>\n",
    "Not only bi-grams, but the same is true for tri-grams and generic N-grams.\n",
    "\n",
    "The convolution network (CNN) is a today's widely used model in computer vision (such as, image classification, object detection, segmentation, etc). In NLP, this convolutional architecture can also be applied in N-gram detection.<br>\n",
    "In computer vision, 2D convolution (convolution by 2 dimensions of width and height) is generally used, but in N-gram detection, 1D convolution is applied as follows.\n",
    "\n",
    "![Bi-gram CNN](images/bigram_convolution.png)\n",
    "\n",
    "There exist several variations for N-gram detection in NLP by convolutions.<br>\n",
    "The **hierarchical convolutions** can capture patterns with gaps, such as, \"not --- good\" or \"see --- little\" where \"---\" stands for a short sequence of words.<br>\n",
    "Similar to image processing, **multiple channels** can also be applied in NLP convolution. For instance, when each word has multiple embeddings (such as, word embedding, POS-tag embedding, position-wise word embedding, etc), these embeddings can be manipulated as multiple channels in NLP.<br>\n",
    "Or, after applying multiple N-grams (such as, 2-gram, 4-gram, and 6-gram), the results can also be manipulated as multiple channels.\n",
    "\n",
    "In this example, I'll simply apply bi-gram detection using 1D convolution (with a single channel) for the purpose of your beginning.\n",
    "\n",
    "*back to [index](https://github.com/tsmatz/nlp-tutorials/)*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install required packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install torch pandas numpy nltk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "nltk.download(\"popular\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Same as in [previous example](./03_word2vec.ipynb), here I also use text in news papers dataset. (However, in this example, we use 2 columns of \"headline\" and \"short description\".)\n",
    "\n",
    "Before starting, please download [News_Category_Dataset_v2.json](https://www.kaggle.com/datasets/rmisra/news-category-dataset/versions/2) (collected by HuffPost) in Kaggle."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "data = pd.read_json(\"News_Category_Dataset_v2.json\",lines=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, we'll apply text classification task.\n",
    "\n",
    "The words appearing in the former part in sequence will be more indicative (topical) rather than the latter part. For this reason, in practical text classification, a long text will then be separated into **regions**. In each region, the convolution (with pooling) is then applied and concatenated. (See below.)<br>\n",
    "For instance, with RCV1 (Reuters Corpus Volume I) dataset, 20 equally sized regions has better performance in category classification. (See [Johnson and Zhang (2015)](https://arxiv.org/abs/1504.01255).)\n",
    "\n",
    "![region separation](images/region_separation.png)\n",
    "\n",
    "In this example, ```headline``` and ```short_description``` are both short text, and we then treat these features as regions, instead of separating a single text into regions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>headline</th>\n",
       "      <th>short_description</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>There Were 2 Mass Shootings In Texas Last Week...</td>\n",
       "      <td>She left her husband. He killed their children...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Will Smith Joins Diplo And Nicky Jam For The 2...</td>\n",
       "      <td>Of course it has a song.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Hugh Grant Marries For The First Time At Age 57</td>\n",
       "      <td>The actor and his longtime girlfriend Anna Ebe...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Jim Carrey Blasts 'Castrato' Adam Schiff And D...</td>\n",
       "      <td>The actor gives Dems an ass-kicking for not fi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Julianna Margulies Uses Donald Trump Poop Bags...</td>\n",
       "      <td>The \"Dietland\" actress said using the bags is ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200848</th>\n",
       "      <td>RIM CEO Thorsten Heins' 'Significant' Plans Fo...</td>\n",
       "      <td>Verizon Wireless and AT&amp;T are already promotin...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200849</th>\n",
       "      <td>Maria Sharapova Stunned By Victoria Azarenka I...</td>\n",
       "      <td>Afterward, Azarenka, more effusive with the pr...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200850</th>\n",
       "      <td>Giants Over Patriots, Jets Over Colts Among  M...</td>\n",
       "      <td>Leading up to Super Bowl XLVI, the most talked...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200851</th>\n",
       "      <td>Aldon Smith Arrested: 49ers Linebacker Busted ...</td>\n",
       "      <td>CORRECTION: An earlier version of this story i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200852</th>\n",
       "      <td>Dwight Howard Rips Teammates After Magic Loss ...</td>\n",
       "      <td>The five-time all-star center tore into his te...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>200853 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 headline  \\\n",
       "0       There Were 2 Mass Shootings In Texas Last Week...   \n",
       "1       Will Smith Joins Diplo And Nicky Jam For The 2...   \n",
       "2         Hugh Grant Marries For The First Time At Age 57   \n",
       "3       Jim Carrey Blasts 'Castrato' Adam Schiff And D...   \n",
       "4       Julianna Margulies Uses Donald Trump Poop Bags...   \n",
       "...                                                   ...   \n",
       "200848  RIM CEO Thorsten Heins' 'Significant' Plans Fo...   \n",
       "200849  Maria Sharapova Stunned By Victoria Azarenka I...   \n",
       "200850  Giants Over Patriots, Jets Over Colts Among  M...   \n",
       "200851  Aldon Smith Arrested: 49ers Linebacker Busted ...   \n",
       "200852  Dwight Howard Rips Teammates After Magic Loss ...   \n",
       "\n",
       "                                        short_description  \n",
       "0       She left her husband. He killed their children...  \n",
       "1                                Of course it has a song.  \n",
       "2       The actor and his longtime girlfriend Anna Ebe...  \n",
       "3       The actor gives Dems an ass-kicking for not fi...  \n",
       "4       The \"Dietland\" actress said using the bags is ...  \n",
       "...                                                   ...  \n",
       "200848  Verizon Wireless and AT&T are already promotin...  \n",
       "200849  Afterward, Azarenka, more effusive with the pr...  \n",
       "200850  Leading up to Super Bowl XLVI, the most talked...  \n",
       "200851  CORRECTION: An earlier version of this story i...  \n",
       "200852  The five-time all-star center tore into his te...  \n",
       "\n",
       "[200853 rows x 2 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_data = data[[\"headline\", \"short_description\"]]\n",
    "label_data = data[\"category\"]\n",
    "text_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get the better performance (accuracy), we standarize the input text as follows.\n",
    "- Make all words to lowercase in order to reduce words\n",
    "- Make \"-\" (hyphen) to space\n",
    "- Remove stop words\n",
    "- Remove all punctuation\n",
    "\n",
    "> Note : Here I have removed stop words, but please take care if you train model for other tasks (such as, sentiment detection), since it might include important words for n-gram detection (such as, \"not\", \"don't\", \"isn't\", etc).\n",
    "\n",
    "> Note : Lemmatization (standardization for such as \"have\", \"had\" or \"having\") should be dealed with, but here I have skipped these pre-processing.<br>\n",
    "> In the strict pre-processing, we should also care about the polysemy. (The different meanings in the same word should have different tokens.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "from nltk.corpus import stopwords\n",
    "import re\n",
    "import string\n",
    "\n",
    "# to lowercase\n",
    "text_data = text_data.apply(lambda x: x.str.lower())\n",
    "\n",
    "# replace hyphen\n",
    "text_data = text_data.apply(lambda x: x.str.replace(\"-\",\" \"))\n",
    "\n",
    "# remove stop words (only when it includes punctuation)\n",
    "for w in stopwords.words(\"english\"):\n",
    "    if re.match(r\"(^|\\w+)[%s](\\w+|$)\" % re.escape(string.punctuation), w):\n",
    "        text_data = text_data.apply(lambda x: x.str.replace(r\"(^|\\s+)%s(\\s+|$)\" % re.escape(w),\" \",regex=True))\n",
    "text_data = text_data.apply(lambda x: x.str.strip())\n",
    "\n",
    "# remove punctuation\n",
    "text_data = text_data.apply(lambda x: x.str.replace(\"[%s]\" % re.escape(string.punctuation),\"\",regex=True))\n",
    "text_data = text_data.apply(lambda x: x.str.strip())\n",
    "\n",
    "# remove stop words (only when it doesn't include punctuation)\n",
    "for w in stopwords.words(\"english\"):\n",
    "    if not re.match(r\"(^|\\w+)[%s](\\w+|$)\" % re.escape(string.punctuation), w):\n",
    "        text_data = text_data.apply(lambda x: x.str.replace(r\"(^|\\s+)%s(\\s+|$)\" % re.escape(w),\" \",regex=True))\n",
    "text_data = text_data.apply(lambda x: x.str.strip())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>headline</th>\n",
       "      <th>short_description</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2 mass shootings texas last week 1 tv</td>\n",
       "      <td>left husband killed children another day america</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>smith joins diplo nicky jam 2018 world cups of...</td>\n",
       "      <td>course song</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>hugh grant marries first time age 57</td>\n",
       "      <td>actor longtime girlfriend anna eberstein tied ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>jim carrey blasts castrato adam schiff democra...</td>\n",
       "      <td>actor gives dems ass kicking fighting hard eno...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>julianna margulies uses donald trump poop bags...</td>\n",
       "      <td>dietland actress said using bags really cathar...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200848</th>\n",
       "      <td>rim ceo thorsten heins significant plans black...</td>\n",
       "      <td>verizon wireless att already promoting lte dev...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200849</th>\n",
       "      <td>maria sharapova stunned victoria azarenka aust...</td>\n",
       "      <td>afterward azarenka effusive press normal credi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200850</th>\n",
       "      <td>giants patriots jets colts among improbable su...</td>\n",
       "      <td>leading super bowl xlvi talked game could end ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200851</th>\n",
       "      <td>aldon smith arrested 49ers linebacker busted dui</td>\n",
       "      <td>correction earlier version story incorrectly s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200852</th>\n",
       "      <td>dwight howard rips teammates magic loss hornets</td>\n",
       "      <td>five time star center tore teammates friday ni...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>200853 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 headline  \\\n",
       "0                   2 mass shootings texas last week 1 tv   \n",
       "1       smith joins diplo nicky jam 2018 world cups of...   \n",
       "2                    hugh grant marries first time age 57   \n",
       "3       jim carrey blasts castrato adam schiff democra...   \n",
       "4       julianna margulies uses donald trump poop bags...   \n",
       "...                                                   ...   \n",
       "200848  rim ceo thorsten heins significant plans black...   \n",
       "200849  maria sharapova stunned victoria azarenka aust...   \n",
       "200850  giants patriots jets colts among improbable su...   \n",
       "200851   aldon smith arrested 49ers linebacker busted dui   \n",
       "200852    dwight howard rips teammates magic loss hornets   \n",
       "\n",
       "                                        short_description  \n",
       "0        left husband killed children another day america  \n",
       "1                                             course song  \n",
       "2       actor longtime girlfriend anna eberstein tied ...  \n",
       "3       actor gives dems ass kicking fighting hard eno...  \n",
       "4       dietland actress said using bags really cathar...  \n",
       "...                                                   ...  \n",
       "200848  verizon wireless att already promoting lte dev...  \n",
       "200849  afterward azarenka effusive press normal credi...  \n",
       "200850  leading super bowl xlvi talked game could end ...  \n",
       "200851  correction earlier version story incorrectly s...  \n",
       "200852  five time star center tore teammates friday ni...  \n",
       "\n",
       "[200853 rows x 2 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we convert a category name (e.g, \"WORLD NEWS\") to label ID (e.g, 2).<br>\n",
    "First we build functions for convertions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'CRIME': 0,\n",
       " 'ENTERTAINMENT': 1,\n",
       " 'WORLD NEWS': 2,\n",
       " 'IMPACT': 3,\n",
       " 'POLITICS': 4,\n",
       " 'WEIRD NEWS': 5,\n",
       " 'BLACK VOICES': 6,\n",
       " 'WOMEN': 7,\n",
       " 'COMEDY': 8,\n",
       " 'QUEER VOICES': 9,\n",
       " 'SPORTS': 10,\n",
       " 'BUSINESS': 11,\n",
       " 'TRAVEL': 12,\n",
       " 'MEDIA': 13,\n",
       " 'TECH': 14,\n",
       " 'RELIGION': 15,\n",
       " 'SCIENCE': 16,\n",
       " 'LATINO VOICES': 17,\n",
       " 'EDUCATION': 18,\n",
       " 'COLLEGE': 19,\n",
       " 'PARENTS': 20,\n",
       " 'ARTS & CULTURE': 21,\n",
       " 'STYLE': 22,\n",
       " 'GREEN': 23,\n",
       " 'TASTE': 24,\n",
       " 'HEALTHY LIVING': 25,\n",
       " 'THE WORLDPOST': 26,\n",
       " 'GOOD NEWS': 27,\n",
       " 'WORLDPOST': 28,\n",
       " 'FIFTY': 29,\n",
       " 'ARTS': 30,\n",
       " 'WELLNESS': 31,\n",
       " 'PARENTING': 32,\n",
       " 'HOME & LIVING': 33,\n",
       " 'STYLE & BEAUTY': 34,\n",
       " 'DIVORCE': 35,\n",
       " 'WEDDINGS': 36,\n",
       " 'FOOD & DRINK': 37,\n",
       " 'MONEY': 38,\n",
       " 'ENVIRONMENT': 39,\n",
       " 'CULTURE & ARTS': 40}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "category_array = label_data.unique()\n",
    "category_dic = {c: i for i, c in enumerate(category_array)}\n",
    "itoc = list(category_dic.keys())\n",
    "ctoi = category_dic\n",
    "category_dic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ctoi[\"WORLD NEWS\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'WORLD NEWS'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "itoc[2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we convert all label to label IDs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0          0\n",
       "1          1\n",
       "2          1\n",
       "3          1\n",
       "4          1\n",
       "          ..\n",
       "200848    14\n",
       "200849    10\n",
       "200850    10\n",
       "200851    10\n",
       "200852    10\n",
       "Name: category, Length: 200853, dtype: int64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "label_data = label_data.apply(lambda y: ctoi[y])\n",
    "label_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build data loader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Same as in previous examples, we will tokenize, in which it converts each text to the sequence of word's indices as follows.<br>\n",
    "In this example, each text will be padded by the padding index (here, 50000) when the the length of text is smaller than 140.\n",
    "\n",
    "![Index vectorize](images/index_vectorize2.png)\n",
    "\n",
    "First we create a list of vocabulary (```vocab```)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.tokenize import SpaceTokenizer\n",
    "\n",
    "###\n",
    "# define Vocab\n",
    "###\n",
    "class Vocab:\n",
    "    def __init__(self, list_of_sentence, tokenization, special_token, max_tokens=None):\n",
    "        # count vocab frequency\n",
    "        vocab_freq = {}\n",
    "        tokens = tokenization(list_of_sentence)\n",
    "        for t in tokens:\n",
    "            for vocab in t:\n",
    "                if vocab not in vocab_freq:\n",
    "                    vocab_freq[vocab] = 0 \n",
    "                vocab_freq[vocab] += 1\n",
    "        # sort by frequency\n",
    "        vocab_freq = {k: v for k, v in sorted(vocab_freq.items(), key=lambda i: i[1], reverse=True)}\n",
    "        # create vocab list\n",
    "        self.vocabs = [special_token] + list(vocab_freq.keys())\n",
    "        if max_tokens:\n",
    "            self.vocabs = self.vocabs[:max_tokens]\n",
    "        self.stoi = {v: i for i, v in enumerate(self.vocabs)}\n",
    "\n",
    "    def _get_tokens(self, list_of_sentence):\n",
    "        for sentence in list_of_sentence:\n",
    "            tokens = tokenizer.tokenize(sentence)\n",
    "            yield tokens\n",
    "\n",
    "    def get_itos(self):\n",
    "        return self.vocabs\n",
    "\n",
    "    def get_stoi(self):\n",
    "        return self.stoi\n",
    "\n",
    "    def append_token(self, token):\n",
    "        self.vocabs.append(token)\n",
    "        self.stoi = {v: i for i, v in enumerate(self.vocabs)}\n",
    "\n",
    "    def __call__(self, list_of_tokens):\n",
    "        def get_token_index(token):\n",
    "            if token in self.stoi:\n",
    "                return self.stoi[token]\n",
    "            else:\n",
    "                return 0\n",
    "        return [get_token_index(t) for t in list_of_tokens]\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.vocabs)\n",
    "\n",
    "###\n",
    "# generate Vocab\n",
    "###\n",
    "vocab_size = 50000\n",
    "max_seq_len = 140\n",
    "\n",
    "# create tokenizer\n",
    "tokenizer = SpaceTokenizer()\n",
    "\n",
    "# define tokenization function\n",
    "def yield_tokens(text_data):\n",
    "    for text in text_data:\n",
    "        tokens = tokenizer.tokenize(text)\n",
    "        tokens = tokens[:max_seq_len]\n",
    "        yield tokens\n",
    "\n",
    "# union headline and short_description\n",
    "text_all = pd.concat([text_data[\"headline\"], text_data[\"short_description\"]])\n",
    "\n",
    "# build vocabulary list\n",
    "vocab = Vocab(\n",
    "    text_all,\n",
    "    tokenization=yield_tokens,\n",
    "    special_token=\"<unk>\",\n",
    "    max_tokens=vocab_size,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The generated token index is ```0, 1, ... , vocab_size - 1```.<br>\n",
    "Now I set ```vocab_size``` (here 50000) as a token id in padded positions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "pad_index = vocab.__len__()\n",
    "vocab.append_token(\"<pad>\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get list for both index-to-word and word-to-index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "itos = vocab.get_itos()\n",
    "stoi = vocab.get_stoi()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The number of token index is 50001.\n",
      "The padded index is 50000.\n"
     ]
    }
   ],
   "source": [
    "# test\n",
    "print(\"The number of token index is {}.\".format(vocab.__len__()))\n",
    "print(\"The padded index is {}.\".format(stoi[\"<pad>\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we build a collator function, which is used for pre-processing in data loader."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch.utils.data import DataLoader\n",
    "\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "def collate_batch(batch):\n",
    "    label_list, head_token_list, desc_token_list = [], [], []\n",
    "    for label, head, desc in batch:\n",
    "        # 1. skip None data\n",
    "        if head is None or desc is None:\n",
    "            continue\n",
    "        # 2. generate word's index vector\n",
    "        head_token = vocab(tokenizer.tokenize(head))\n",
    "        desc_token = vocab(tokenizer.tokenize(desc))\n",
    "        # 3. limit token length to max_seq_len\n",
    "        head_token = head_token[:max_seq_len]\n",
    "        desc_token = desc_token[:max_seq_len]\n",
    "        # 4. pad sequence\n",
    "        head_token += [pad_index] * (max_seq_len - len(head_token))\n",
    "        desc_token += [pad_index] * (max_seq_len - len(desc_token))\n",
    "        # add to list\n",
    "        label_list.append(label)\n",
    "        head_token_list.append(head_token)\n",
    "        desc_token_list.append(desc_token)\n",
    "    # convert to tensor\n",
    "    label_list = torch.tensor(label_list, dtype=torch.int64).to(device)\n",
    "    head_token_list = torch.tensor(head_token_list, dtype=torch.int64).to(device)\n",
    "    desc_token_list = torch.tensor(desc_token_list, dtype=torch.int64).to(device)\n",
    "    return label_list, head_token_list, desc_token_list\n",
    "\n",
    "dataloader = DataLoader(\n",
    "    list(zip(label_data, text_data[\"headline\"], text_data[\"short_description\"])),\n",
    "    batch_size=512,\n",
    "    shuffle=True,\n",
    "    collate_fn=collate_batch\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "label shape in batch : torch.Size([512])\n",
      "headline token shape in batch : torch.Size([512, 140])\n",
      "short_desc token shape in batch : torch.Size([512, 140])\n",
      "***** label sample *****\n",
      "tensor(9, device='cuda:0')\n",
      "***** headline token sample *****\n",
      "tensor([14793, 23265,  1982,  2237,  8315,   312,  1229,   332,  1387,  1622,\n",
      "        50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000,\n",
      "        50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000,\n",
      "        50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000,\n",
      "        50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000,\n",
      "        50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000,\n",
      "        50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000,\n",
      "        50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000,\n",
      "        50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000,\n",
      "        50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000,\n",
      "        50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000,\n",
      "        50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000,\n",
      "        50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000,\n",
      "        50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000],\n",
      "       device='cuda:0')\n",
      "***** short_desc token sample *****\n",
      "tensor([ 1600,  2237,   936,  1037,  3553,   502,   264,    21,  1390, 50000,\n",
      "        50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000,\n",
      "        50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000,\n",
      "        50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000,\n",
      "        50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000,\n",
      "        50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000,\n",
      "        50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000,\n",
      "        50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000,\n",
      "        50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000,\n",
      "        50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000,\n",
      "        50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000,\n",
      "        50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000,\n",
      "        50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000,\n",
      "        50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000, 50000],\n",
      "       device='cuda:0')\n"
     ]
    }
   ],
   "source": [
    "# test\n",
    "for labels, heads, descs in dataloader:\n",
    "    break\n",
    "\n",
    "print(\"label shape in batch : {}\".format(labels.size()))\n",
    "print(\"headline token shape in batch : {}\".format(heads.size()))\n",
    "print(\"short_desc token shape in batch : {}\".format(descs.size()))\n",
    "print(\"***** label sample *****\")\n",
    "print(labels[0])\n",
    "print(\"***** headline token sample *****\")\n",
    "print(heads[0])\n",
    "print(\"***** short_desc token sample *****\")\n",
    "print(descs[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's build network.<br>\n",
    "In this neural network,\n",
    "\n",
    "(1) As you saw in previous examples, we build embedding vectors (dense vectors) $ \\{ \\mathbf{w}_1, \\mathbf{w}_2, \\ldots, \\mathbf{w}_m \\} $ from text for both ```headline``` and ```short_description``` respectively.\n",
    "\n",
    "(2) For these embedding vectors, we apply 1D convolution $ \\mathbf{p}_i = g(U (\\mathbf{x}_i) + \\mathbf{b}) $ where $ \\mathbf{x}_i = [\\mathbf{w}_i, \\mathbf{w}_{i+1}] $, $U$ is a weight matrix, $\\mathbf{b}$ is a bias vector, and $ g() $ is RELU activaiton. (i.e, In convolutions, the size of window is 2 (bi-gram) and the size of stride is 1.)<br>\n",
    "In this example, we apply half padding convolution (i.e, apply $ \\mathbf{x}_i = [\\mathbf{w}_i, \\mathbf{w}_{i+1}] $ for $ i=1,\\ldots,m $ where $\\mathbf{w}_{m+1}$ is zero) and the number of outputs will then also be $m$.<br>\n",
    "I assume that the result is $n$-dimensional vectors $ \\mathbf{p}_1, \\mathbf{p}_2, \\cdots, \\mathbf{p}_m $ .\n",
    "\n",
    "(3) Next we get $n$-dimensional vector $\\mathbf{c}$ by applying $\\mathbf{c}_{[j]} = \\max_{1 \\leq i \\leq m} \\mathbf{p}_{i [j]} \\forall j \\in [1,n]$. (i.e, max pooling)<br>\n",
    "Here I have denoted $j$-th element of vecotr $\\mathbf{p}_i$ by $\\mathbf{p}_{i [j]}$. ($i \\in [1,m], j \\in [1,n]$)\n",
    "\n",
    "(4) We concatenate the result's vectors $\\mathbf{c}$ and $\\mathbf{d}$, each of which is corresponing to ```headline``` and ```short_description```.\n",
    "\n",
    "(5) Finally, we apply fully-connected feed-forward network (i.e, Dense Net) for predicting class label.\n",
    "\n",
    "![composing network](images/1d_conv_net.png)\n",
    "\n",
    "Before max pooling, the values in padding positions (which index of token is ```pad_index```) are converted into zero, and these will then be underestimated in gradient descent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "\n",
    "embedding_dim = 200\n",
    "\n",
    "class BigramClassifier(nn.Module):\n",
    "    def __init__(self, vocab_size, embedding_dim, class_num, padding_idx, conv_channel=256, hidden_dim=128):\n",
    "        super().__init__()\n",
    "\n",
    "        self.padding_idx = padding_idx\n",
    "        self.conv_channel = conv_channel\n",
    "\n",
    "        self.embedding01 = nn.Embedding(\n",
    "            vocab_size,\n",
    "            embedding_dim,\n",
    "            padding_idx=padding_idx,\n",
    "        )\n",
    "        self.embedding02 = nn.Embedding(\n",
    "            vocab_size,\n",
    "            embedding_dim,\n",
    "            padding_idx=padding_idx,\n",
    "        )\n",
    "        self.conv01 = torch.nn.Conv1d(\n",
    "            in_channels=embedding_dim,\n",
    "            out_channels=conv_channel,\n",
    "            kernel_size=2,\n",
    "            stride=1,\n",
    "            padding=\"same\",\n",
    "        )\n",
    "        self.conv02 = torch.nn.Conv1d(\n",
    "            in_channels=embedding_dim,\n",
    "            out_channels=conv_channel,\n",
    "            kernel_size=2,\n",
    "            stride=1,\n",
    "            padding=\"same\",\n",
    "        )\n",
    "        self.relu = nn.ReLU()\n",
    "        self.max_pool = torch.nn.MaxPool1d(\n",
    "            kernel_size=max_seq_len,\n",
    "        )\n",
    "        self.hidden = nn.Linear(conv_channel*2, hidden_dim)\n",
    "        self.classify = nn.Linear(hidden_dim, class_num)\n",
    "\n",
    "    def forward(self, region01, region02):\n",
    "        # Get padding masks (in which, element is 0.0 when it's in padded position, otherwise 1.0)\n",
    "        mask01 = torch.ones(region01.size()).to(device)\n",
    "        mask01 = mask01.masked_fill(region01 == self.padding_idx, 0)\n",
    "        mask02 = torch.ones(region02.size()).to(device)\n",
    "        mask02 = mask02.masked_fill(region02 == self.padding_idx, 0)\n",
    "        # Embedding\n",
    "        #   --> [batch_size, max_seq_len, embedding_dim]\n",
    "        out01 = self.embedding01(region01)\n",
    "        out02 = self.embedding02(region02)\n",
    "        # Apply convolution on dimension=1\n",
    "        #   --> [batch_size, max_seq_len, conv_channel]\n",
    "        out01 = self.conv01(out01.transpose(1,2)).transpose(1,2)\n",
    "        out02 = self.conv02(out02.transpose(1,2)).transpose(1,2)\n",
    "        # Apply masking (In padded position, it will then be 0.0)\n",
    "        extend_mask01 = mask01.unsqueeze(dim=2)\n",
    "        extend_mask01 = extend_mask01.expand(-1, -1, self.conv_channel)\n",
    "        out01 = out01 * extend_mask01\n",
    "        extend_mask02 = mask02.unsqueeze(dim=2)\n",
    "        extend_mask02 = extend_mask02.expand(-1, -1, self.conv_channel)\n",
    "        out02 = out02 * extend_mask02\n",
    "        # Apply relu\n",
    "        out01 = self.relu(out01)\n",
    "        out02 = self.relu(out02)\n",
    "        # Apply max pooling on dimension=1\n",
    "        #   --> [batch_size, 1, conv_channel]\n",
    "        out01 = self.max_pool(out01.transpose(1,2)).transpose(1,2)\n",
    "        out02 = self.max_pool(out02.transpose(1,2)).transpose(1,2)\n",
    "        # Flatten\n",
    "        #   --> [batch_size, conv_channel]\n",
    "        out01 = out01.squeeze(dim=1)\n",
    "        out02 = out02.squeeze(dim=1)\n",
    "\n",
    "        # Concat outputs of head and short_description\n",
    "        #   --> [batch_size, conv_channel * 2]\n",
    "        out = torch.concat((out01, out02), dim=-1)\n",
    "\n",
    "        # Apply classification head\n",
    "        #   --> [batch_size, hidden_dim]\n",
    "        out = self.hidden(out)\n",
    "        out = self.relu(out)\n",
    "        #   --> [batch_size, class_num]\n",
    "        logits = self.classify(out)\n",
    "        \n",
    "        return logits\n",
    "\n",
    "model = BigramClassifier(vocab.__len__(), embedding_dim, len(ctoi), pad_index).to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train model\n",
    "\n",
    "Now let's train our network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/tsmatsuz/.local/lib/python3.8/site-packages/torch/nn/modules/conv.py:309: UserWarning: Using padding='same' with even kernel lengths and odd dilation may require a zero-padded copy of the input be created (Triggered internally at ../aten/src/ATen/native/Convolution.cpp:895.)\n",
      "  return F.conv1d(input, weight, bias, self.stride,\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1 - loss: 2.0183 - accuracy: 0.4295\n",
      "Epoch 2 - loss: 1.6093 - accuracy: 0.5235\n",
      "Epoch 3 - loss: 1.2186 - accuracy: 0.6443\n",
      "Epoch 4 - loss: 0.9033 - accuracy: 0.7181\n",
      "Epoch 5 - loss: 0.6881 - accuracy: 0.8188\n",
      "Epoch 6 - loss: 0.4761 - accuracy: 0.8792\n",
      "Epoch 7 - loss: 0.4410 - accuracy: 0.8792\n",
      "Epoch 8 - loss: 0.3817 - accuracy: 0.8658\n",
      "Epoch 9 - loss: 0.2463 - accuracy: 0.9195\n",
      "Epoch 10 - loss: 0.1037 - accuracy: 0.9664\n",
      "Epoch 11 - loss: 0.0286 - accuracy: 1.0000\n",
      "Epoch 12 - loss: 0.0230 - accuracy: 1.0000\n",
      "Epoch 13 - loss: 0.0252 - accuracy: 0.9933\n",
      "Epoch 14 - loss: 0.0098 - accuracy: 1.0000\n",
      "Epoch 15 - loss: 0.0063 - accuracy: 1.0000\n"
     ]
    }
   ],
   "source": [
    "from torch.nn import functional as F\n",
    "\n",
    "num_epochs = 15\n",
    "\n",
    "optimizer = torch.optim.AdamW(model.parameters(), lr=0.001)\n",
    "for epoch in range(num_epochs):\n",
    "    for labels, heads, descs in dataloader:\n",
    "        # optimize\n",
    "        optimizer.zero_grad()\n",
    "        logits = model(heads, descs)\n",
    "        loss = F.cross_entropy(logits, labels)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        # calculate accuracy\n",
    "        pred_labels = logits.argmax(dim=1)\n",
    "        num_correct = (pred_labels == labels).float().sum()\n",
    "        accuracy = num_correct / len(labels)\n",
    "        print(\"Epoch {} - loss: {:2.4f} - accuracy: {:2.4f}\".format(epoch+1, loss.item(), accuracy), end=\"\\r\")\n",
    "    print(\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classify text\n",
    "\n",
    "Now we classify 3 text about \"```Michael Jackson```\", \"```Michael Avenatti```\", and \"```Ronny Jackson```\".<br>\n",
    "All of them has high probabilities to each categories. (See the logits output.)\n",
    "\n",
    "In this experiment, \"```Michael Jackson```\" is strongly categorized to ```ENTERTAINMENT```. But neither \"```Michael```\" nor \"```Jackson```\" affects the result, because both \"```Michael Avenatti```\" and \"```Ronny Jackson```\" are classified to different categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ENTERTAINMENT\n",
      "MEDIA\n",
      "MEDIA\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "def classify_text(headline, description):\n",
    "    test_list = [\n",
    "        [1, headline, description],\n",
    "    ]\n",
    "    _, test_heads, test_descs = collate_batch(test_list)\n",
    "    pred_logits = model(test_heads, test_descs)\n",
    "    pred_index = pred_logits.argmax()\n",
    "    return itoc[pred_index.item()]\n",
    "\n",
    "print(classify_text(\n",
    "    \"report about michael jackson\",\n",
    "    \"michael jackson is wise and honest\"\n",
    "))\n",
    "print(classify_text(\n",
    "    \"report about michael avenatti\",\n",
    "    \"michael avenatti is wise and honest\"\n",
    "))\n",
    "print(classify_text(\n",
    "    \"report about ronny jackson\",\n",
    "    \"ronny jackson is wise and honest\"\n",
    "))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}