02_custom_embedding.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Custom Embedding (Dense Vector)\n",
    "\n",
    "As you saw in the [previous example](./01_sparse_vector.ipynb), the generated count vectors are sparse and a lot of algorithms won't work well with this high-dimensional vectors.<br>\n",
    "For this reason, the today's refined trainers will transform sparse vectors into non-sparse forms (dense vectors) and process some tasks (such as, NLP classification, etc) againt these dense vectors in practice. (See below.)\n",
    "\n",
    "![Dense vectorize](images/dense_vectorize.png)\n",
    "\n",
    "In today's advanced embedding, the embedding layer can sometimes be the model of non-linear neural networks.<br>\n",
    "But, in most cases, word embedding is essentially a lookup table which maps a sparse vector into a dense vector. Assuming $\\mathbf{w}$ is a word index vector (i.e, sparse vector) with voculabrary size $|V|$, in which the i-th element of $\\mathbf{w}$ is $1$ and other elements are $0$, the embedding table $\\mathbf{E}$ will then be a $ |V| \\times d $ matrix which converts a sparse vector $\\mathbf{w}$ to a $d$-dimensional dense vector by $ \\mathbf{w} \\mathbf{E} $. (i.e, The i-th row of $\\mathbf{E}$ is a dense vector for a word $\\mathbf{w}$.)<br>\n",
    "The $ |V| \\times d $ parameters will then be trained by some task.\n",
    "\n",
    "![Embedding](images/embedding_matrix.png)\n",
    "\n",
    "> Note : A lookup table cannot be used in sentence embedding, because the number of sentences is not finite.<br>\n",
    "> See note in [Transformer](./09_transformer.ipynb) for sentence embedding.\n",
    "\n",
    "The generated dense vector (i.e, non-sparse form) will represent some aspects (meaning) for words or documents.\n",
    "\n",
    "In order to get the trained (optimal) parameters of $\\mathbf{E}$ (and optimal dense vectors), you can take either of the following 3 options :\n",
    "\n",
    "1. Train embeddings $\\mathbf{E}$ from the beginning.\n",
    "2. Use existing pre-trained embeddings $\\mathbf{E_0}$ trained by a large text corpus. (For instance, see Hugging Face hub for a lot of pre-trained SOTA models.)\n",
    "3. Download pre-trained embeddings $\\mathbf{E_0}$ and train (fine-tune) $\\mathbf{E_0}$ furthermore to get new optimal $\\mathbf{E_1}$.\n",
    "\n",
    "> Note : In order to fine-tune the pre-trained vectors, there also exists the following approaches :<br>\n",
    "> - Find an additional matrix $\\mathbf{T} \\in \\mathbb{R}^{d \\times d} $, with which we can obtain new embedding $\\mathbf{E} \\mathbf{T}$\n",
    "> - Find an additional matrix $\\mathbf{A} \\in \\mathbb{R}^{|V| \\times d} $, with which we can obtain new embedding $\\mathbf{E} + \\mathbf{A}$\n",
    "> - Hybrid of 1 and 2\n",
    "\n",
    "In this exercise, here I'll show you the brief example for self-trained embeddings.\n",
    "\n",
    "In a lot of today's NLP models, the word is embedded into dense vectors and the sequence of words in document is trained by [RNN-based learners](./06_language_model_rnn.ipynb), [Attention-based learners](./08_attention.ipynb), or [Transformers](./09_transformer.ipynb) with a large corpus. However, for the purpose of your beginning, here I'll introduce a simple classification trainer, in which the word is embedded and the sequence is combined by using primitive continuos bag-of-words (CBOW) representation.<br>\n",
    "In this example, we'll train a model (which includes custom embedding) to detect sentiment with movie review dataset (IMDB) for natural language processing.\n",
    "\n",
    "CBOW (continuos bag-of-words) representation is a combination of vectors, which is obtained by the mean (average) of vectors as follows. (The magnitude of vector doesn't then depend on the length of sentence.)\n",
    "\n",
    "$ \\frac{1}{k} \\sum_{i=1}^{k} v(w_i) $ &nbsp;&nbsp;&nbsp; where $v(\\cdot)$ is dense vector.\n",
    "\n",
    "> Note : As I have mentioned in \"[Sparse Vector](01_sparse_vector.ipynb)\", you can also use weighted coefficients (such as, position weighting, TF-IDF weighting, etc) in CBOW representation. This is called weighted CBOW or WCBOW shortly.\n",
    "\n",
    "*back to [index](https://github.com/tsmatz/nlp-tutorials/)*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install required packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install torch nltk numpy pandas tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "nltk.download(\"popular\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, we build IMDB dataset (movie review dataset).<br>\n",
    "In this dataset, it includes the review text and 2-class flag 1 or 2 for dissatisfied/satisfied respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tar -xvzf aclImdb_v1.tar.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from torch.utils.data import Dataset\n",
    "import tqdm\n",
    "\n",
    "class IMDBDataset(Dataset):\n",
    "    def __init__(self, data_folder):\n",
    "        pos_folder = os.path.join(data_folder, \"pos\")\n",
    "        pos_files = os.listdir(pos_folder)\n",
    "        pos_files = [os.path.join(pos_folder, f) for f in pos_files]\n",
    "        pos_data = [(f, 2) for f in pos_files]\n",
    "        neg_folder = os.path.join(data_folder, \"neg\")\n",
    "        neg_files = os.listdir(neg_folder)\n",
    "        neg_files = [os.path.join(neg_folder, f) for f in neg_files]\n",
    "        neg_data = [(f, 1) for f in neg_files]\n",
    "        all_data = pos_data + neg_data\n",
    "\n",
    "        # load all data (because it's only 25000 records)\n",
    "        self.buffer = []\n",
    "        for i in tqdm.tqdm(range(len(all_data)), total=len(all_data)):\n",
    "            filename = all_data[i][0]\n",
    "            label = all_data[i][1]\n",
    "            with open(filename, \"r\") as f:\n",
    "                content = f.read()\n",
    "            self.buffer.append((content, label))\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.buffer)\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        item = self.buffer[idx]\n",
    "        return item[1], item[0]\n",
    "\n",
    "train_iter = IMDBDataset(\"aclImdb/test\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The record number is as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25000"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(list(train_iter))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we pick up and see the first row of records."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "***** text *****\n",
      "I love sci-fi and am willing to put up with a lot. Sci-fi movies/TV are usually underfunded, under-appreciated and misunderstood. I tried to like this, I really did, but it is to good TV sci-fi as Babylon 5 is to Star Trek (the original). Silly prosthetics, cheap cardboard sets, stilted dialogues, CG that doesn't match the background, and painfully one-dimensional characters cannot be overcome with a 'sci-fi' setting. (I'm sure there are those of you out there who think Babylon 5 is good sci-fi TV. It's not. It's clichéd and uninspiring.) While US viewers might like emotion and character development, sci-fi is a genre that does not take itself seriously (cf. Star Trek). It may treat important issues, yet not as a serious philosophy. It's really difficult to care about the characters here as they are not simply foolish, just missing a spark of life. Their actions and reactions are wooden and predictable, often painful to watch. The makers of Earth KNOW it's rubbish as they have to always say \"Gene Roddenberry's Earth...\" otherwise people would not continue watching. Roddenberry's ashes must be turning in their orbit as this dull, cheap, poorly edited (watching it without advert breaks really brings this home) trudging Trabant of a show lumbers into space. Spoiler. So, kill off a main character. And then bring him back as another actor. Jeeez! Dallas all over again.\n",
      "***** label *****\n",
      "1\n"
     ]
    }
   ],
   "source": [
    "# test (print first row)\n",
    "for label, text in train_iter:\n",
    "    print(\"***** text *****\")\n",
    "    print(text)\n",
    "    print(\"***** label *****\")\n",
    "    print(label)\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get the better performance (accuracy), we standarize the input's review text as follows.\n",
    "\n",
    "(1) Make all words to lowercase<br>\n",
    "  (ex. \"I am a Greatest Showman !\" -> \"i am a greatest showman !\")\n",
    "\n",
    "(2) Remove all stop words, such as, \"a\", \"the\", \"is\", \"i\", etc<br>\n",
    "  (ex. \"i am a greatest showman !\" -> \"greatest showman !\")\n",
    "\n",
    "(3) Remove all punctuation, such as, \"!\", \"?\", \"#\", etc<br>\n",
    "  (ex. \"greatest showman !\" -> \"greatest showman\")\n",
    "\n",
    "> Note : Some normalization - such as, changing to lower case - is also done in the following tokenizer.<br>\n",
    "> N-gram words (such as, \"New York\", \"Barack Obama\") and lemmatization (standardization for such as \"have\", \"had\" or \"having\") should be dealed with, but here I have skipped these pre-processing. For N-gram detection, see \"[N-Gram detection with 1D Convolution](./04_ngram_cnn.ipynb)\" example.<br>\n",
    "> In the strict pre-processing, we should also care about the polysemy. (The different meanings in the same word should have different tokens.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "from nltk.corpus import stopwords\n",
    "import string\n",
    "\n",
    "def standarize_text(text):\n",
    "    new_text = text\n",
    "\n",
    "    # 1. To lowercase\n",
    "    new_text = new_text.lower()\n",
    "\n",
    "    # 2. Remove stop words\n",
    "    for w in stopwords.words(\"english\"):\n",
    "        new_text = re.sub(\n",
    "            r\"(^|\\s+)%s(\\s+|$)\" % re.escape(w),\n",
    "            \" \",\n",
    "            new_text)\n",
    "    new_text = new_text.strip()\n",
    "\n",
    "    # 3. Remove punctuation\n",
    "    new_text = new_text.translate(str.maketrans(\"\", \"\", string.punctuation))\n",
    "    new_text = new_text.strip()\n",
    "\n",
    "    return new_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'greatest showman'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# test\n",
    "standarize_text(\"I am a Greatest Showman !\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to create a word's index vector as follows, first we create a list for words (```vocab```) used in the training set.\n",
    "\n",
    "![Index vectorize](images/index_vectorize.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.tokenize import SpaceTokenizer\n",
    "\n",
    "###\n",
    "# define Vocab\n",
    "###\n",
    "class Vocab:\n",
    "    def __init__(self, list_of_sentence, tokenization, special_token, max_tokens=None):\n",
    "        # count vocab frequency\n",
    "        vocab_freq = {}\n",
    "        tokens = tokenization(list_of_sentence)\n",
    "        for t in tokens:\n",
    "            for vocab in t:\n",
    "                if vocab not in vocab_freq:\n",
    "                    vocab_freq[vocab] = 0 \n",
    "                vocab_freq[vocab] += 1\n",
    "        # sort by frequency\n",
    "        vocab_freq = {k: v for k, v in sorted(vocab_freq.items(), key=lambda i: i[1], reverse=True)}\n",
    "        # create vocab list\n",
    "        self.vocabs = [special_token] + list(vocab_freq.keys())\n",
    "        if max_tokens:\n",
    "            self.vocabs = self.vocabs[:max_tokens]\n",
    "        self.stoi = {v: i for i, v in enumerate(self.vocabs)}\n",
    "\n",
    "    def _get_tokens(self, list_of_sentence):\n",
    "        for sentence in list_of_sentence:\n",
    "            tokens = tokenizer.tokenize(sentence)\n",
    "            yield tokens\n",
    "\n",
    "    def get_itos(self):\n",
    "        return self.vocabs\n",
    "\n",
    "    def get_stoi(self):\n",
    "        return self.stoi\n",
    "\n",
    "    def append_token(self, token):\n",
    "        self.vocabs.append(token)\n",
    "        self.stoi = {v: i for i, v in enumerate(self.vocabs)}\n",
    "\n",
    "    def __call__(self, list_of_tokens):\n",
    "        def get_token_index(token):\n",
    "            if token in self.stoi:\n",
    "                return self.stoi[token]\n",
    "            else:\n",
    "                return 0\n",
    "        return [get_token_index(t) for t in list_of_tokens]\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.vocabs)\n",
    "\n",
    "###\n",
    "# generate Vocab\n",
    "###\n",
    "vocab_size = 10000\n",
    "\n",
    "# create tokenizer\n",
    "tokenizer = SpaceTokenizer()\n",
    "\n",
    "# define tokenization function\n",
    "def yield_tokens(data_iter):\n",
    "    for _, text in data_iter:\n",
    "        text = standarize_text(text)\n",
    "        yield tokenizer.tokenize(text)\n",
    "\n",
    "# build vocabulary list\n",
    "vocab = Vocab(\n",
    "    train_iter,\n",
    "    tokenization=yield_tokens,\n",
    "    special_token=\"<unk>\",\n",
    "    max_tokens=vocab_size,\n",
    ")\n",
    "\n",
    "# get list for index-to-word, and word-to-index\n",
    "itos = vocab.get_itos()\n",
    "stoi = vocab.get_stoi()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[681, 2, 41, 0]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# test\n",
    "vocab([\"greatest\", \"movie\", \"show\", \"abcdefghijk\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we build data loader with a collator function, in which data is pre-processed.\n",
    "\n",
    "In this collator,\n",
    "\n",
    "1. The input's text is standarized with previous ```standarize_text``` function.\n",
    "2. The input's text is then tokenized into word's index (integer's list).\n",
    "3. Limit to 256 tokens.\n",
    "4. Generate mask array. For instance, if the length of token is 3, it will become ```[1.0, 1.0, 1.0, 0.0, 0.0, ..., 0.0]```.\n",
    "5. Pad all sequence by zero, if the length of sequence is shorter than max tokens.\n",
    "6. Convert curent label 1 or 2 into 0 or 1 respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch.utils.data import DataLoader\n",
    "\n",
    "seq_len = 256\n",
    "\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "def collate_batch(batch):\n",
    "    label_list, token_list, mask_list = [], [], []\n",
    "    for (label, text) in batch:\n",
    "        # 1. standarize text\n",
    "        text = standarize_text(text)\n",
    "        # 2. generate word's index vector\n",
    "        tokens = vocab(tokenizer.tokenize(text))\n",
    "        # 3. limit to first tokens\n",
    "        tokens = tokens[:seq_len]\n",
    "        # 4. generate mask array\n",
    "        length = len(tokens)\n",
    "        mask_array = [float(i < length) for i in range(seq_len)]\n",
    "        # 5. pad sequence\n",
    "        tokens += [0] * (seq_len - len(tokens))\n",
    "        # 6. convert label into 0 or 1\n",
    "        label = label - 1\n",
    "        # add to list\n",
    "        label_list.append(label)\n",
    "        token_list.append(tokens)\n",
    "        mask_list.append(mask_array)\n",
    "    # convert to tensor\n",
    "    label_list = torch.tensor(label_list, dtype=torch.int64).to(device)\n",
    "    token_list = torch.tensor(token_list, dtype=torch.int64).to(device)\n",
    "    mask_list = torch.tensor(mask_list, dtype=torch.float).to(device)\n",
    "    return label_list, token_list, mask_list\n",
    "\n",
    "dataloader = DataLoader(\n",
    "    train_iter,\n",
    "    batch_size=128,\n",
    "    shuffle=True,\n",
    "    collate_fn=collate_batch\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "label shape in batch : torch.Size([128])\n",
      "token shape in batch : torch.Size([128, 256])\n",
      "mask  shape in batch : torch.Size([128, 256])\n",
      "***** label sample *****\n",
      "tensor(0, device='cuda:0')\n",
      "***** token sample *****\n",
      "tensor([8261,    4, 1440,   23,    5,   17,   27,    0,    0, 2639,  232, 3768,\n",
      "        3301,  852,    0,  226,    0,    0, 6659,    0,  343,   87,  310,  224,\n",
      "        1476, 4918,    0,    0,    0,    0,    0, 4035,  680,  138, 3386,    0,\n",
      "          62,  859, 1068,   11,    0,  379, 3355,    0,  105,    5,   10,   71,\n",
      "         530,  200,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
      "           0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
      "           0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
      "           0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
      "           0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
      "           0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
      "           0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
      "           0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
      "           0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
      "           0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
      "           0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
      "           0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
      "           0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
      "           0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
      "           0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
      "           0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
      "           0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
      "           0,    0,    0,    0], device='cuda:0')\n",
      "***** input text *****\n",
      "['file', 'one', 'how', 'movies', 'like', 'get', 'made', '<unk>', '<unk>', 'indie', 'version', 'macbeth', 'adapted', 'fairly', '<unk>', 'but', '<unk>', '<unk>', 'unconventional', '<unk>', 'style', 'cast', 'gives', 'shot', 'christopher', 'walken', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', 'rising', 'dull', 'script', 'pat', '<unk>', 'actors', 'wasted', 'audiences', 'time', '<unk>', 'fans', 'brand', '<unk>', 'may', 'like', 'it', 'though', '4', '10', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>', '<unk>']\n",
      "***** mask *****\n",
      "tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
      "        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
      "        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 0., 0., 0.,\n",
      "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
      "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
      "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
      "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
      "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
      "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
      "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
      "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
      "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
      "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
      "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
      "        0., 0., 0., 0.], device='cuda:0')\n"
     ]
    }
   ],
   "source": [
    "# test\n",
    "for labels, tokens, masks in dataloader:\n",
    "    break\n",
    "\n",
    "print(\"label shape in batch : {}\".format(labels.size()))\n",
    "print(\"token shape in batch : {}\".format(tokens.size()))\n",
    "print(\"mask  shape in batch : {}\".format(masks.size()))\n",
    "print(\"***** label sample *****\")\n",
    "print(labels[0])\n",
    "print(\"***** token sample *****\")\n",
    "print(tokens[0])\n",
    "print(\"***** input text *****\")\n",
    "print([itos[i] for i in tokens[0]])\n",
    "print(\"***** mask *****\")\n",
    "print(masks[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we'll build the embedding module.\n",
    "\n",
    "![Embedding module](images/embedding_layer.png)\n",
    "\n",
    "This module converts each word's index into corresponding embedded vector (dense vector) as follows.<br>\n",
    "If the size of inputs is ```[128, 256]``` and embedding dimension is ```16```, the size of inputs will then become ```[128, 256, 16]```\n",
    "\n",
    "![Word embeddings](images/word_embedding.png)\n",
    "\n",
    "> Note : For the purpose of your learning, here I build custom embedding module from scratch, but in other exercises, I'll use ```torch.nn.Embedding``` module in PyTorch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "\n",
    "embedding_dim = 16\n",
    "\n",
    "class Embedding(nn.Module):\n",
    "    def __init__(self, num_embeddings, embedding_dim):\n",
    "        super(Embedding, self).__init__()\n",
    "        self.weight = nn.Parameter(torch.empty((num_embeddings, embedding_dim)))\n",
    "        self.reset_parameters()\n",
    "    def reset_parameters(self):\n",
    "        nn.init.uniform_(self.weight, -0.1, 0.1)\n",
    "    def forward(self, input: torch.Tensor):\n",
    "        return self.weight[input]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "output shape : torch.Size([128, 256, 16])\n",
      "***** output *****\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor([[[ 0.0537,  0.0995, -0.0639,  ..., -0.0117, -0.0210, -0.0004],\n",
       "         [ 0.0994, -0.0006,  0.0512,  ..., -0.0381,  0.0587, -0.0776],\n",
       "         [-0.0408, -0.0242, -0.0683,  ..., -0.0763, -0.0033,  0.0752],\n",
       "         ...,\n",
       "         [ 0.0227,  0.0767,  0.0386,  ..., -0.0433,  0.0949, -0.0391],\n",
       "         [ 0.0227,  0.0767,  0.0386,  ..., -0.0433,  0.0949, -0.0391],\n",
       "         [ 0.0227,  0.0767,  0.0386,  ..., -0.0433,  0.0949, -0.0391]],\n",
       "\n",
       "        [[-0.0635, -0.0664,  0.0637,  ..., -0.0536, -0.0019,  0.0504],\n",
       "         [ 0.0791,  0.0604,  0.0386,  ...,  0.0170,  0.0313, -0.0888],\n",
       "         [-0.0570, -0.0719, -0.0392,  ..., -0.0905,  0.0871, -0.0014],\n",
       "         ...,\n",
       "         [ 0.0227,  0.0767,  0.0386,  ..., -0.0433,  0.0949, -0.0391],\n",
       "         [ 0.0227,  0.0767,  0.0386,  ..., -0.0433,  0.0949, -0.0391],\n",
       "         [ 0.0227,  0.0767,  0.0386,  ..., -0.0433,  0.0949, -0.0391]],\n",
       "\n",
       "        [[ 0.0164,  0.0208, -0.0965,  ..., -0.0159, -0.0105, -0.0977],\n",
       "         [-0.0785,  0.0128,  0.0330,  ..., -0.0051,  0.0685, -0.0039],\n",
       "         [ 0.0897, -0.0164,  0.0486,  ...,  0.0224, -0.0740,  0.0836],\n",
       "         ...,\n",
       "         [ 0.0227,  0.0767,  0.0386,  ..., -0.0433,  0.0949, -0.0391],\n",
       "         [ 0.0227,  0.0767,  0.0386,  ..., -0.0433,  0.0949, -0.0391],\n",
       "         [ 0.0227,  0.0767,  0.0386,  ..., -0.0433,  0.0949, -0.0391]],\n",
       "\n",
       "        ...,\n",
       "\n",
       "        [[-0.0767,  0.0026,  0.0898,  ...,  0.0570, -0.0813,  0.0734],\n",
       "         [ 0.0147,  0.0882, -0.0059,  ..., -0.0005,  0.0965,  0.0330],\n",
       "         [-0.0840,  0.0118, -0.0646,  ...,  0.0153,  0.0669, -0.0279],\n",
       "         ...,\n",
       "         [ 0.0227,  0.0767,  0.0386,  ..., -0.0433,  0.0949, -0.0391],\n",
       "         [ 0.0227,  0.0767,  0.0386,  ..., -0.0433,  0.0949, -0.0391],\n",
       "         [ 0.0227,  0.0767,  0.0386,  ..., -0.0433,  0.0949, -0.0391]],\n",
       "\n",
       "        [[ 0.0865, -0.0626, -0.0700,  ..., -0.0905,  0.0485, -0.0487],\n",
       "         [-0.0176,  0.0219, -0.0390,  ..., -0.0985, -0.0119, -0.0879],\n",
       "         [-0.0978, -0.0426,  0.0983,  ...,  0.0707,  0.0164, -0.0214],\n",
       "         ...,\n",
       "         [ 0.0227,  0.0767,  0.0386,  ..., -0.0433,  0.0949, -0.0391],\n",
       "         [ 0.0227,  0.0767,  0.0386,  ..., -0.0433,  0.0949, -0.0391],\n",
       "         [ 0.0227,  0.0767,  0.0386,  ..., -0.0433,  0.0949, -0.0391]],\n",
       "\n",
       "        [[-0.0801, -0.0978, -0.0325,  ...,  0.0299,  0.0905, -0.0513],\n",
       "         [ 0.0348, -0.0246,  0.0625,  ...,  0.0199, -0.0878, -0.0442],\n",
       "         [-0.0831,  0.0064,  0.0506,  ...,  0.0753, -0.0491,  0.0541],\n",
       "         ...,\n",
       "         [ 0.0227,  0.0767,  0.0386,  ..., -0.0433,  0.0949, -0.0391],\n",
       "         [ 0.0227,  0.0767,  0.0386,  ..., -0.0433,  0.0949, -0.0391],\n",
       "         [ 0.0227,  0.0767,  0.0386,  ..., -0.0433,  0.0949, -0.0391]]],\n",
       "       device='cuda:0', grad_fn=<IndexBackward0>)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# test\n",
    "e = Embedding(vocab_size, embedding_dim).to(device)\n",
    "embs = e(tokens)\n",
    "\n",
    "print(\"output shape : {}\".format(embs.size()))\n",
    "print(\"***** output *****\")\n",
    "embs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we get CBOW (continuous bag-of-words) representation for word's embedded vectors as follows.\n",
    "\n",
    "$$ \\frac{1}{k} \\sum_{i=1}^{k} v(w_i) $$\n",
    "\n",
    "where $w_i$ is a word vector (in this case, the scalar number representing a word) and $v(\\cdot)$ is embedding function.\n",
    "\n",
    "![CBOW](images/continuous_bow.png)\n",
    "\n",
    "In this CBOW representation, the order of words in the sentence will be ignored, and it won't then capture contexts, such as :\n",
    "\n",
    "\"it's exciting, but it's unfavorable.\"\n",
    "\n",
    "Furthermore it won't understand n-grams, such as, \"don't like\".<br>\n",
    "In exercise 04, I'll discuss about n-gram detection. In the latter exercises, I'll also discuss how to capture contexts.\n",
    "\n",
    "In the following CBOW representation's implementation, I use mask (in which 0 is assigned in padded positions, and otherwise 1) in order to skip computation in padded positions.\n",
    "\n",
    "> Note : In PyTorch, you can use ```torch.nn.functional.avg_pool1d()``` for averaging globally. You can also use ```torch.nn.EmbeddingBag``` to get the mean of embedded vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CBOW(nn.Module):\n",
    "    def forward(self, embedded, masks):\n",
    "        # generate mask with embedding --> [batch_size, seq_len, embedding_dim]\n",
    "        extend_masks = masks.unsqueeze(dim=2)\n",
    "        extend_masks = extend_masks.expand(-1, -1, embedding_dim)\n",
    "        # filter embedding by multiplication\n",
    "        masked_embedded = embedded * extend_masks\n",
    "        # sum all embedding in each sequence --> [batch_size, embedding_dim]\n",
    "        embedded_sum = masked_embedded.sum(dim=1)\n",
    "        # compute token length --> [batch_size]\n",
    "        token_length = masks.sum(dim=1)\n",
    "        # divide by token length\n",
    "        # [batch_size, embedding_dim] / [batch_size] --> [batch_size, embedding_dim]\n",
    "        embedded_sum = embedded_sum.transpose(0,1) / token_length\n",
    "        return embedded_sum.transpose(0,1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "output shape : torch.Size([128, 16])\n",
      "***** output *****\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.0109,  0.0271,  0.0090,  ..., -0.0210,  0.0267, -0.0084],\n",
       "        [ 0.0064,  0.0199,  0.0137,  ..., -0.0186,  0.0045, -0.0095],\n",
       "        [ 0.0061,  0.0170, -0.0031,  ..., -0.0107,  0.0222, -0.0050],\n",
       "        ...,\n",
       "        [ 0.0050,  0.0042, -0.0062,  ..., -0.0173,  0.0200, -0.0131],\n",
       "        [-0.0021,  0.0077, -0.0024,  ..., -0.0098,  0.0032, -0.0044],\n",
       "        [ 0.0041,  0.0017,  0.0063,  ..., -0.0084,  0.0047, -0.0023]],\n",
       "       device='cuda:0', grad_fn=<TransposeBackward0>)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# test\n",
    "c = CBOW().to(device)\n",
    "cbow = c(embs, masks)\n",
    "\n",
    "print(\"output shape : {}\".format(cbow.size()))\n",
    "print(\"***** output *****\")\n",
    "cbow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally we'll build the task layer.\n",
    "\n",
    "![Task layer](images/task_layer.png)\n",
    "\n",
    "In our network, we just use fully connected feed-forward network (DenseNet), in which the final output is one-hot logits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "output shape : torch.Size([128, 2])\n",
      "***** output *****\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.1841, -0.0176],\n",
       "        [ 0.1860, -0.0200],\n",
       "        [ 0.1846, -0.0180],\n",
       "        [ 0.1830, -0.0159],\n",
       "        [ 0.1825, -0.0143],\n",
       "        [ 0.1820, -0.0134],\n",
       "        [ 0.1762, -0.0149],\n",
       "        [ 0.1812, -0.0116],\n",
       "        [ 0.1811, -0.0171],\n",
       "        [ 0.1789, -0.0131],\n",
       "        [ 0.1859, -0.0158],\n",
       "        [ 0.1821, -0.0178],\n",
       "        [ 0.1768, -0.0121],\n",
       "        [ 0.1690, -0.0115],\n",
       "        [ 0.1768, -0.0138],\n",
       "        [ 0.1794, -0.0155],\n",
       "        [ 0.1816, -0.0166],\n",
       "        [ 0.1573, -0.0196],\n",
       "        [ 0.1725, -0.0146],\n",
       "        [ 0.1761, -0.0134],\n",
       "        [ 0.1796, -0.0169],\n",
       "        [ 0.1806, -0.0131],\n",
       "        [ 0.1821, -0.0139],\n",
       "        [ 0.1806, -0.0194],\n",
       "        [ 0.1818, -0.0148],\n",
       "        [ 0.1746, -0.0088],\n",
       "        [ 0.1834, -0.0149],\n",
       "        [ 0.1837, -0.0178],\n",
       "        [ 0.1864, -0.0232],\n",
       "        [ 0.1800, -0.0160],\n",
       "        [ 0.1772, -0.0126],\n",
       "        [ 0.1855, -0.0177],\n",
       "        [ 0.1804, -0.0155],\n",
       "        [ 0.1908, -0.0187],\n",
       "        [ 0.1863, -0.0173],\n",
       "        [ 0.1746, -0.0128],\n",
       "        [ 0.1857, -0.0180],\n",
       "        [ 0.1756, -0.0130],\n",
       "        [ 0.1836, -0.0153],\n",
       "        [ 0.1831, -0.0137],\n",
       "        [ 0.1800, -0.0171],\n",
       "        [ 0.1820, -0.0110],\n",
       "        [ 0.1765, -0.0135],\n",
       "        [ 0.1839, -0.0156],\n",
       "        [ 0.1767, -0.0137],\n",
       "        [ 0.1806, -0.0149],\n",
       "        [ 0.1824, -0.0170],\n",
       "        [ 0.1720, -0.0106],\n",
       "        [ 0.1751, -0.0115],\n",
       "        [ 0.1871, -0.0164],\n",
       "        [ 0.1848, -0.0188],\n",
       "        [ 0.1736, -0.0184],\n",
       "        [ 0.1861, -0.0167],\n",
       "        [ 0.1811, -0.0149],\n",
       "        [ 0.1813, -0.0125],\n",
       "        [ 0.1873, -0.0162],\n",
       "        [ 0.1753, -0.0105],\n",
       "        [ 0.1823, -0.0159],\n",
       "        [ 0.1818, -0.0155],\n",
       "        [ 0.1724, -0.0100],\n",
       "        [ 0.1878, -0.0152],\n",
       "        [ 0.1823, -0.0155],\n",
       "        [ 0.1781, -0.0126],\n",
       "        [ 0.1825, -0.0147],\n",
       "        [ 0.1840, -0.0126],\n",
       "        [ 0.1821, -0.0114],\n",
       "        [ 0.1826, -0.0158],\n",
       "        [ 0.1797, -0.0110],\n",
       "        [ 0.1854, -0.0164],\n",
       "        [ 0.1795, -0.0169],\n",
       "        [ 0.1912, -0.0220],\n",
       "        [ 0.1789, -0.0136],\n",
       "        [ 0.1828, -0.0167],\n",
       "        [ 0.1779, -0.0124],\n",
       "        [ 0.1877, -0.0150],\n",
       "        [ 0.1755, -0.0172],\n",
       "        [ 0.1816, -0.0126],\n",
       "        [ 0.1829, -0.0150],\n",
       "        [ 0.1822, -0.0157],\n",
       "        [ 0.1731, -0.0136],\n",
       "        [ 0.1791, -0.0172],\n",
       "        [ 0.1860, -0.0201],\n",
       "        [ 0.1766, -0.0163],\n",
       "        [ 0.1737, -0.0159],\n",
       "        [ 0.1795, -0.0175],\n",
       "        [ 0.1859, -0.0173],\n",
       "        [ 0.1768, -0.0142],\n",
       "        [ 0.1850, -0.0175],\n",
       "        [ 0.1801, -0.0129],\n",
       "        [ 0.1897, -0.0168],\n",
       "        [ 0.1747, -0.0095],\n",
       "        [ 0.1874, -0.0193],\n",
       "        [ 0.1769, -0.0137],\n",
       "        [ 0.1797, -0.0186],\n",
       "        [ 0.1820, -0.0155],\n",
       "        [ 0.1803, -0.0138],\n",
       "        [ 0.1813, -0.0164],\n",
       "        [ 0.1699, -0.0124],\n",
       "        [ 0.1863, -0.0180],\n",
       "        [ 0.1835, -0.0118],\n",
       "        [ 0.1832, -0.0149],\n",
       "        [ 0.1777, -0.0148],\n",
       "        [ 0.1844, -0.0144],\n",
       "        [ 0.1723, -0.0120],\n",
       "        [ 0.1764, -0.0115],\n",
       "        [ 0.1775, -0.0117],\n",
       "        [ 0.1815, -0.0141],\n",
       "        [ 0.1831, -0.0192],\n",
       "        [ 0.1838, -0.0144],\n",
       "        [ 0.1808, -0.0142],\n",
       "        [ 0.1800, -0.0151],\n",
       "        [ 0.1744, -0.0137],\n",
       "        [ 0.1829, -0.0149],\n",
       "        [ 0.1829, -0.0156],\n",
       "        [ 0.1799, -0.0149],\n",
       "        [ 0.1790, -0.0162],\n",
       "        [ 0.1760, -0.0121],\n",
       "        [ 0.1806, -0.0144],\n",
       "        [ 0.1884, -0.0207],\n",
       "        [ 0.1837, -0.0169],\n",
       "        [ 0.1841, -0.0180],\n",
       "        [ 0.1878, -0.0178],\n",
       "        [ 0.1782, -0.0112],\n",
       "        [ 0.1881, -0.0163],\n",
       "        [ 0.1762, -0.0126],\n",
       "        [ 0.1819, -0.0160],\n",
       "        [ 0.1727, -0.0174],\n",
       "        [ 0.1792, -0.0140]], device='cuda:0', grad_fn=<AddmmBackward0>)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# test\n",
    "l = nn.Linear(embedding_dim, 2).to(device)\n",
    "logits = l(cbow)\n",
    "\n",
    "print(\"output shape : {}\".format(logits.size()))\n",
    "print(\"***** output *****\")\n",
    "logits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we put it all together and build a model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CBOWClassifier(nn.Module):\n",
    "    def __init__(self, vocab_size, embedding_dim, class_num):\n",
    "        super().__init__()\n",
    "        self.embedding = Embedding(vocab_size, embedding_dim)\n",
    "        self.cbow = CBOW()\n",
    "        self.linear = nn.Linear(embedding_dim, class_num)\n",
    "    def forward(self, tokens, masks):\n",
    "        embs = self.embedding(tokens)\n",
    "        output = self.cbow(embs, masks)\n",
    "        logits = self.linear(output)\n",
    "        return logits\n",
    "\n",
    "model = CBOWClassifier(vocab_size, embedding_dim, 2).to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try to predict sentiment (0 - negative, 1 - positive) before training.<br>\n",
    "As you can see, the result is not correct."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Text</th>\n",
       "      <th>True</th>\n",
       "      <th>Pred</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>I love this movie.</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>It's so disappointed.</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>I have recognized that a lot of people liked t...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>I agree with the reviewer who said this work i...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>I found it so turgid and poorly expressed by c...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>It helps you put into words what you want from...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>I think the ending is illogical at least and i...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                Text  True  Pred\n",
       "0                                 I love this movie.     1     1\n",
       "1                              It's so disappointed.     0     1\n",
       "2  I have recognized that a lot of people liked t...     1     1\n",
       "3  I agree with the reviewer who said this work i...     0     1\n",
       "4  I found it so turgid and poorly expressed by c...     0     1\n",
       "5  It helps you put into words what you want from...     1     1\n",
       "6  I think the ending is illogical at least and i...     0     1"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "text = [\n",
    "    [2,\"I love this movie.\"],\n",
    "    [1,\"It's so disappointed.\"],\n",
    "    [2,\"I have recognized that a lot of people liked this story.\"],\n",
    "    [1,\"I agree with the reviewer who said this work is boring.\"],\n",
    "    [1,\"I found it so turgid and poorly expressed by casts.\"],\n",
    "    [2,\"It helps you put into words what you want from main character.\"],\n",
    "    [1,\"I think the ending is illogical at least and is fiction.\"],\n",
    "]\n",
    "true_labels, tokens, masks = collate_batch(text)\n",
    "pred_logits = model(tokens, masks)\n",
    "pred_labels = pred_logits.argmax(dim=1)\n",
    "df = pd.DataFrame(\n",
    "    list(zip(\n",
    "        [t[1] for t in text],\n",
    "        true_labels.tolist(),\n",
    "        pred_labels.tolist())),\n",
    "    columns =[\"Text\", \"True\", \"Pred\"]\n",
    ")\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train model and Evaluate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's train our network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1 - loss: 0.6533 - accuracy: 0.6250\n",
      "Epoch 2 - loss: 0.5507 - accuracy: 0.7750\n",
      "Epoch 3 - loss: 0.3829 - accuracy: 0.8750\n",
      "Epoch 4 - loss: 0.3148 - accuracy: 0.9250\n",
      "Epoch 5 - loss: 0.3395 - accuracy: 0.8500\n",
      "Epoch 6 - loss: 0.3428 - accuracy: 0.8000\n",
      "Epoch 7 - loss: 0.2202 - accuracy: 0.9250\n",
      "Epoch 8 - loss: 0.1837 - accuracy: 0.9500\n",
      "Epoch 9 - loss: 0.2222 - accuracy: 0.9250\n",
      "Epoch 10 - loss: 0.1923 - accuracy: 0.9000\n",
      "Epoch 11 - loss: 0.2358 - accuracy: 0.9250\n",
      "Epoch 12 - loss: 0.1134 - accuracy: 1.0000\n",
      "Epoch 13 - loss: 0.1036 - accuracy: 0.9750\n",
      "Epoch 14 - loss: 0.1339 - accuracy: 0.9500\n",
      "Epoch 15 - loss: 0.2219 - accuracy: 0.9000\n"
     ]
    }
   ],
   "source": [
    "from torch.nn import functional as F\n",
    "\n",
    "num_epochs = 15\n",
    "\n",
    "optimizer = torch.optim.AdamW(model.parameters(), lr=0.001)\n",
    "for epoch in range(num_epochs):\n",
    "    for labels, tokens, masks in dataloader:\n",
    "        # optimize\n",
    "        optimizer.zero_grad()\n",
    "        logits = model(tokens, masks)\n",
    "        loss = F.cross_entropy(logits, labels)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        # calculate accuracy\n",
    "        pred_labels = logits.argmax(dim=1)\n",
    "        num_correct = (pred_labels == labels).float().sum()\n",
    "        accuracy = num_correct / len(labels)\n",
    "        print(\"Epoch {} - loss: {:2.4f} - accuracy: {:2.4f}\".format(epoch+1, loss.item(), accuracy), end=\"\\r\")\n",
    "    print(\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the training has completed, predict sample text again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Text</th>\n",
       "      <th>True</th>\n",
       "      <th>Pred</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>I love this movie.</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>It's so disappointed.</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>I have recognized that a lot of people liked t...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>I agree with the reviewer who said this work i...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>I found it so turgid and poorly expressed by c...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>It helps you put into words what you want from...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>I think the ending is illogical at least and i...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                Text  True  Pred\n",
       "0                                 I love this movie.     1     1\n",
       "1                              It's so disappointed.     0     0\n",
       "2  I have recognized that a lot of people liked t...     1     1\n",
       "3  I agree with the reviewer who said this work i...     0     0\n",
       "4  I found it so turgid and poorly expressed by c...     0     0\n",
       "5  It helps you put into words what you want from...     1     1\n",
       "6  I think the ending is illogical at least and i...     0     0"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = [\n",
    "    [2,\"I love this movie.\"],\n",
    "    [1,\"It's so disappointed.\"],\n",
    "    [2,\"I have recognized that a lot of people liked this story.\"],\n",
    "    [1,\"I agree with the reviewer who said this work is boring.\"],\n",
    "    [1,\"I found it so turgid and poorly expressed by casts.\"],\n",
    "    [2,\"It helps you put into words what you want from main character.\"],\n",
    "    [1,\"I think the ending is illogical at least and is fiction.\"],\n",
    "]\n",
    "true_labels, tokens, masks = collate_batch(text)\n",
    "pred_logits = model(tokens, masks)\n",
    "pred_labels = pred_logits.argmax(dim=1)\n",
    "df = pd.DataFrame(\n",
    "    list(zip(\n",
    "        [t[1] for t in text],\n",
    "        true_labels.tolist(),\n",
    "        pred_labels.tolist())),\n",
    "    columns =[\"Text\", \"True\", \"Pred\"]\n",
    ")\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The word embedding is a byproduct in this example.\n",
    "\n",
    "For instance, let's get top 10 words similar to the word \"```great```\" with trained embedding.<br>\n",
    "I note that this embedding is trained to capture the tone for sentiment, and it won't then detect other contexts of similarity, such like, \"```dog```\" and \"```puppy```\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['great',\n",
       " 'different',\n",
       " 'gem',\n",
       " 'bravo',\n",
       " 'refreshing',\n",
       " '710',\n",
       " 'excellent',\n",
       " 'perfect',\n",
       " 'beauty',\n",
       " 'ward']"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "from numpy.linalg import norm\n",
    "\n",
    "# create new embedding and restore weight from trained model\n",
    "e = model.embedding\n",
    "# get embedding vector for the word \"great\"\n",
    "token = stoi[\"great\"]\n",
    "# get embedded vector for \"great\"\n",
    "vec_t = e(token).tolist()\n",
    "# get vector list for all words (10,000 words)\n",
    "all_vec = [e(i).tolist() for i in range(vocab.__len__())]\n",
    "# get L2 distance in all words\n",
    "vec_t = np.array(vec_t)\n",
    "all_vec = np.array(all_vec)\n",
    "all_distance = np.array([np.dot(vec_t,v)/(norm(vec_t)*norm(v)) for v in all_vec])\n",
    "# get top 10 words similar to the word \"great\"\n",
    "indices_list = np.argsort(-all_distance)\n",
    "[itos[i] for i in indices_list[:10]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}