01_sparse_vector.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Primitive Embeddings (Sparse Vector)\n",
    "\n",
    "For the first tutorial, here I show you primitive embeddings (preprocessing, featurizing, or vectorizing) for languages.\n",
    "\n",
    "As you can see in the later tutorials, embeddings in this example is very beginning and will not be used in practices. But it will be a good example for your first understanding NLP.\n",
    "\n",
    "There are many types of embeddings - such as, character embedding, word embedding, sentence embedding, or document embedding, and I'll show you sentence vectorization in this notebook.\n",
    "\n",
    "*back to [index](https://github.com/tsmatz/nlp-tutorials/)*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install required packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install scikit-learn nltk pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "nltk.download(\"popular\")\n",
    "nltk.download('punkt_tab')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Count Vectorize\n",
    "\n",
    "One of primitive method to vectorize a text is count vectorization.<br>\n",
    "This method is based on one hot vectorizing and each element represents the count of that word in a document as follows.\n",
    "\n",
    "![Count vectorize](images/count_vectorize.png)\n",
    "\n",
    "Count vectorization is very straighforward and comprehensive for humans, but it'll build sparse vectors (in which, almost elements are zero) and also resource-intensive. I note that it will then waste a lot of time and resources for large data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>and</th>\n",
       "      <th>are</th>\n",
       "      <th>book</th>\n",
       "      <th>here</th>\n",
       "      <th>is</th>\n",
       "      <th>my</th>\n",
       "      <th>pen</th>\n",
       "      <th>these</th>\n",
       "      <th>this</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   a  and  are  book  here  is  my  pen  these  this\n",
       "0  1    0    0     1     0   1   0    0      0     1\n",
       "1  0    1    1     0     1   1   1    2      1     0"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from nltk import word_tokenize\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "import pandas as pd\n",
    "\n",
    "lemmatizer = WordNetLemmatizer()\n",
    "\n",
    "# Convert :\n",
    "# \"pens\" -> \"pen\"\n",
    "# \"wolves\" -> \"wolf\"\n",
    "def my_lemmatizer(text):\n",
    "    return [lemmatizer.lemmatize(t) for t in word_tokenize(text)]\n",
    "\n",
    "vectorizer = CountVectorizer(\n",
    "    tokenizer=my_lemmatizer)\n",
    "texts = [\n",
    "    \"This is a book\",\n",
    "    \"These are pens and my pen is here\"\n",
    "]\n",
    "vectors = vectorizer.fit_transform(texts)\n",
    "\n",
    "cols = [k for k, v in sorted(vectorizer.vocabulary_.items(), key=lambda item: item[1])]\n",
    "df = pd.DataFrame(vectors.toarray(), columns=cols)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hence this vectorization often results into low performance (low accuracy) in several ML use-cases. (Since the neural network won't work well with very high-dimensional and sparse vectors.)<br>\n",
    "The following is the example for classifying document into 20 e-mail groups.\n",
    "\n",
    "> Note : In the real usage, please train with unknown words with a specific symbol, such as \"[UNK]\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/tsmatsuz/.local/lib/python3.6/site-packages/sklearn/feature_extraction/text.py:489: UserWarning: The parameter 'token_pattern' will not be used since 'tokenizer' is not None'\n",
      "  warnings.warn(\"The parameter 'token_pattern' will not be used\"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "classification accuracy: 0.6240042485395645\n"
     ]
    }
   ],
   "source": [
    "from sklearn.datasets import fetch_20newsgroups\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn import metrics\n",
    "\n",
    "# Load train dataset\n",
    "train = fetch_20newsgroups(\n",
    "    subset=\"train\",\n",
    "    remove=(\"headers\", \"footers\", \"quotes\"))\n",
    "\n",
    "# Count vectorize\n",
    "vectorizer.fit(train.data)\n",
    "X_trian = vectorizer.transform(train.data)\n",
    "y_train = train.target\n",
    "\n",
    "# Train\n",
    "clf = MultinomialNB(alpha=.01)\n",
    "clf.fit(X_trian, y_train)\n",
    "\n",
    "# Evaluate accuracy\n",
    "test = fetch_20newsgroups(\n",
    "    subset=\"test\",\n",
    "    remove=(\"headers\", \"footers\", \"quotes\"))\n",
    "X_test = vectorizer.transform(test.data)\n",
    "y_test = test.target\n",
    "y_pred = clf.predict(X_test)\n",
    "score = metrics.accuracy_score(y_test, y_pred)\n",
    "print(\"classification accuracy: {}\".format(score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TF-IDF weighting\n",
    "\n",
    "In above example, the weight of word \"book\" or \"pen\" is the same as the weight of words \"a\", \"for\", \"the\", etc.<br>\n",
    "Using TF-IDF, you can prioritize the words that rarely appear in the given corpus.\n",
    "\n",
    "TF (=**T**erm **F**requency) is\n",
    "\n",
    "$$ \\frac{\\#d(w)}{\\sum_{w^{\\prime} \\in d} \\#d(w^{\\prime})} $$\n",
    "\n",
    "in which, $ \\#d(w) $ means the count of word $w$ in document $d$.<br>\n",
    "TF is the normalized value of the count of word $w$ in document $d$. \n",
    "\n",
    "TF-IDF (=**I**nverse **D**ocument **F**requency) is\n",
    "\n",
    "$$ \\frac{\\#d(w)}{\\sum_{w^{\\prime} \\in d} \\#d(w^{\\prime})} \\times \\log{\\frac{|D|}{|\\{d \\in D:w\\in d\\}|}}$$\n",
    "\n",
    "where $D$ is large corpus (a set of documents).\n",
    "\n",
    "If some word $w$ (such like, \"a\", \"the\") is included in all document $d \\in D$, the second term will be relatively small. If some word is rarely included in $d \\in D$, the second term will be relatively large."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see the following example.<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer\n",
    "from nltk import word_tokenize\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "\n",
    "lemmatizer = WordNetLemmatizer()\n",
    "\n",
    "# Convert :\n",
    "# \"pens\" -> \"pen\"\n",
    "# \"wolves\" -> \"wolf\"\n",
    "def my_lemmatizer(text):\n",
    "    return [lemmatizer.lemmatize(t) for t in word_tokenize(text)]\n",
    "\n",
    "# Count vectorize\n",
    "count_vectorizer = CountVectorizer(tokenizer=my_lemmatizer)\n",
    "texts = [\n",
    "    \"This is a book\",\n",
    "    \"These are pens and my pen is here\"\n",
    "]\n",
    "count_vectors = count_vectorizer.fit_transform(texts)\n",
    "\n",
    "# TF-IDF weighting\n",
    "tfidf_trans = TfidfTransformer(use_idf=True).fit(count_vectors)\n",
    "tfidf_vectors = tfidf_trans.transform(count_vectors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see above, only the word \"is\" is included in both documents. The word \"pen\" is also used twice, however, this word is not used in the first document.<br>\n",
    "As a result, only the word \"is\" has small value for IDF weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>and</th>\n",
       "      <th>are</th>\n",
       "      <th>book</th>\n",
       "      <th>here</th>\n",
       "      <th>is</th>\n",
       "      <th>my</th>\n",
       "      <th>pen</th>\n",
       "      <th>these</th>\n",
       "      <th>this</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.405465</td>\n",
       "      <td>1.405465</td>\n",
       "      <td>1.405465</td>\n",
       "      <td>1.405465</td>\n",
       "      <td>1.405465</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.405465</td>\n",
       "      <td>1.405465</td>\n",
       "      <td>1.405465</td>\n",
       "      <td>1.405465</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          a       and       are      book      here   is        my       pen  \\\n",
       "0  1.405465  1.405465  1.405465  1.405465  1.405465  1.0  1.405465  1.405465   \n",
       "\n",
       "      these      this  \n",
       "0  1.405465  1.405465  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cols = [k for k, v in sorted(count_vectorizer.vocabulary_.items(), key=lambda item: item[1])]\n",
    "df = pd.DataFrame([tfidf_trans.idf_], columns=cols)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The generated vectors has the following values.<br>\n",
    "As you can see below, the word \"is\" has relatively small value compared with other words in the same document.<br>\n",
    "The second document (\"These are pens and my pen is here\") has more words than the first document (\"This is a book\"), and then TF values (normalized values) in the second document are small rather than ones in the first document.<br>\n",
    "The word \"pen\" appears in the second documnt twice, and it then has 2x values compared to other words in this document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>and</th>\n",
       "      <th>are</th>\n",
       "      <th>book</th>\n",
       "      <th>here</th>\n",
       "      <th>is</th>\n",
       "      <th>my</th>\n",
       "      <th>pen</th>\n",
       "      <th>these</th>\n",
       "      <th>this</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.534046</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.534046</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.379978</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.534046</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.324336</td>\n",
       "      <td>0.324336</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.324336</td>\n",
       "      <td>0.230768</td>\n",
       "      <td>0.324336</td>\n",
       "      <td>0.648673</td>\n",
       "      <td>0.324336</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          a       and       are      book      here        is        my  \\\n",
       "0  0.534046  0.000000  0.000000  0.534046  0.000000  0.379978  0.000000   \n",
       "1  0.000000  0.324336  0.324336  0.000000  0.324336  0.230768  0.324336   \n",
       "\n",
       "        pen     these      this  \n",
       "0  0.000000  0.000000  0.534046  \n",
       "1  0.648673  0.324336  0.000000  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame(tfidf_vectors.toarray(), columns=cols)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see the example for classifying text into 20 e-mail groups. (Compare the result with the previous one.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/tsmatsuz/.local/lib/python3.6/site-packages/sklearn/feature_extraction/text.py:489: UserWarning: The parameter 'token_pattern' will not be used since 'tokenizer' is not None'\n",
      "  warnings.warn(\"The parameter 'token_pattern' will not be used\"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "classification accuracy: 0.6964949548592672\n"
     ]
    }
   ],
   "source": [
    "from sklearn.datasets import fetch_20newsgroups\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn import metrics\n",
    "\n",
    "# Load train dataset\n",
    "train = fetch_20newsgroups(\n",
    "    subset=\"train\",\n",
    "    remove=(\"headers\", \"footers\", \"quotes\"))\n",
    "\n",
    "# Count vectorize\n",
    "count_vectorizer.fit(train.data)\n",
    "X_train_count = count_vectorizer.transform(train.data)\n",
    "\n",
    "# TF-IDF weighting\n",
    "tfidf_trans = TfidfTransformer(use_idf=True).fit(X_train_count)\n",
    "X_train_tfidf = tfidf_trans.transform(X_train_count)\n",
    "\n",
    "# Train\n",
    "y_train = train.target\n",
    "clf = MultinomialNB(alpha=.01)\n",
    "clf.fit(X_train_tfidf, y_train)\n",
    "\n",
    "# Evaluate accuracy\n",
    "test = fetch_20newsgroups(\n",
    "    subset=\"test\",\n",
    "    remove=(\"headers\", \"footers\", \"quotes\"))\n",
    "X_test_count = count_vectorizer.transform(test.data)\n",
    "X_test_tfidf = tfidf_trans.transform(X_test_count)\n",
    "y_pred = clf.predict(X_test_tfidf)\n",
    "y_test = test.target\n",
    "score = metrics.accuracy_score(y_test, y_pred)\n",
    "print(\"classification accuracy: {}\".format(score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TF-IDF can also be applied to dense vectors** as follows :\n",
    "\n",
    "$$ \\frac{1}{\\sum_{i=1}^{k} \\verb|tfidf|(w_i)} \\sum_{i=1}^{k} \\verb|tfidf|(w_i) v(w_i) $$\n",
    "\n",
    "where $v(\\cdot)$ is word's vectorization (dense vector) and $\\verb|tfidf|(\\cdot)$ is TF-IDF weighting.\n",
    "\n",
    "See the next exercise for dense vector representation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}