lecture_21_natural_language_processing_1.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The autoreload extension is already loaded. To reload it, use:\n",
      "  %reload_ext autoreload\n"
     ]
    }
   ],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Natural Language Processing?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![title](images/nlp1/drake_reaction_nlp.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Many times there is a confusion with the \"NLP\" abbreviation. Neural Linguistic Programming is also well known scientific field connected with human communication and body language. But anyway it doesn't have any connection with Natural Language Processing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# What is Natural Language Processing?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![title](https://mildaintrainings.com/wp-content/uploads/2019/09/natural-language-processing-training.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "[Natural Language Processing](https://en.wikipedia.org/wiki/Natural_language_processing)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Natural Language Processing actually started very early in 1950s. Alan Turing published an article titled \"Computing Machinery and Intelligence\" which proposed what is now called the Turing test as a criterion of intelligence. But it took long to achieve some promising results. But Turing Test is still remaining unbeatable by AI models and hence there is a huge space for improvement :)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# How does it look like?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![title](https://2s7gjr373w3x22jf92z99mgm5w-wpengine.netdna-ssl.com/wp-content/uploads/2014/06/text-analytics.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "The main data source for natural language processing is text. Text is unstructured information which needs to be structured (classified, extracted some data, etc) and most of the NLP problems are trying to do that. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![title](https://machinelearning.technicacuriosa.com/wp-content/uploads/sites/7/2019/03/Speech_Recognition.jpeg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Another data source for NLP is sound which is again unstructured data and is being converted into textual format. It's actually called Speech recognition. Recorded sound is transformed into text (or vice versa). Finally text is further processed to get get useful insights from it (user commands for google assistant, controlling devices, etc)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Applications"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![title](https://i.pinimg.com/originals/45/da/2d/45da2d4d7333a5ccb9598cf08b61cd80.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Here are the wide range of NLP applications and but incomplete (you may find more). Important applications are:\n",
    "- **Information retrieval**: searching through huge dataset and retrieving relevant information (search engines)\n",
    "- **Keyword Extraction**: extracting important words from text which can be used for indexing documents (search engines)\n",
    "- **Text Summarization**: to get short summary of huge text content. Used in Google search results and Google assistant uses it for answering non-abstract questions (extractive and abstractive text summarizations are present)\n",
    "- **Named entity recognition**: recognizing entities like Person, Company, Date, Money, Email, Phone, etc.\n",
    "- **Sentiment Analysis**: analyzing sentiment of text. (Used in social media analysis, comments on products, etc)\n",
    "- **Text Similarity**: good text representation is the best way to make NLP work actually, it's more fundamental problem.\n",
    "- **Machine Translation**: translation is one of the most popular and important feature of NLP. \n",
    "- **Question Answering**: useful in some cases, can be widely used in chatbot like systems for answering simple (and sometimes hard) questions.\n",
    "- **Spell Checking**: very useful in text editors for correcting spelling mistakes and also detect typos.  \n",
    "- etc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![title](https://cdn.setapp.com/blog/images/prizmo-text-from-image.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Language models are used for OCR (Optical character recognition) for checking typos created by OCR errors (incorrect prediction of character because of noise). Having language model means to have understanding the language generally. Simple example to understand what it means is to consider text generation. Given some sentences, model have to generate next sentences as good as person could do/imagine."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "The best OCR comes from Google. It has multilingual and handwritten text recognition capabilities on a very low price per page of document.\n",
    "[OCR APIs](https://source.opennews.org/articles/so-many-ocr-options/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![title](https://thumbs.gfycat.com/ImperfectDarkKarakul-size_restricted.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Google assistant is another good example of real life NLP application. User's speech is converted into text, then text is processed to get insights/commands from it, then model generates answer (search result, command, etc) and that answer is converted back to speech to for good user experience (play music, ask something, ask about weather, etc).  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "[chatbots, assistants, etc](https://medium.com/@takashimokobe/the-evolution-of-conversation-design-chatbots-voice-user-interfaces-and-google-duplex-815d7bee2233)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![title](https://prishtinainsight.com/wp-content/uploads/2017/12/Google-Translate-GIF-highres.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Google translate is also the best translation engine. It converts text from one language into another. It's superior performance is achieved by huge amount of static (wikipedia contents in different language) and dymanic (user actions, searches, etc) contents available in Google. In order to achieve good translation you need to have huge amount of sentence pairs in two different languages. As new terms and words are created Google automatically updates it's engine to be able to track them and to be consistent with translations.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "[best machine translations for 2019](https://analyticsprofile.com/machine-learning/best-language-translation-apis-available-2019/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![title](https://media1.giphy.com/media/bdtCfNg4RK8oM/giphy.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "SEO (Search Engine Optimization) is very popular and important structural part for building websites in an optimized way. The idea is very simple. New website has to be easily reachable and must appear on the first page when user searches in Google. In order to achieve that you need to know important keywords and terms that are used for searching relevant information in search engine. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "[google semantic search ans SEO](https://blog.alexa.com/semantic-search/)\n",
    "\n",
    "[semantic search engine explained](https://www.xenonstack.com/blog/semantic-search-engine/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![title](https://hips.hearstapps.com/digitalspyuk.cdnds.net/16/32/1470838492-donald-trump-gif.gif?resize=480:*)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "[Sentiment Analysis](https://monkeylearn.com/sentiment-analysis/) is actually very useful \"feature\" of NLP. New state of the art Deep learning models are very good at classifying sentiments. Social media is full of comments, posts, actions, etc. It's very important to analyze client reactions, find problems with your product/service and optimize it for maximum profit.  \n",
    "\n",
    "Recently it was amazingly used for Trump Elections. See more [here](https://aisel.aisnet.org/pacis2017/48/)\n",
    "\n",
    "\n",
    "There is very interesting extension of sentiment analysis: Aspect Based Sentiment Analysis. Using that approach it's possible to detect sentiments in more granular way which is very important. see more [here](https://monkeylearn.com/blog/aspect-based-sentiment-analysis/) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Short History"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Here are some highlights from history of NLP. (You may find much more :), History is my weakness :D )\n",
    "\n",
    "- [chomsky-syntactic-structures (1957)](https://doubleoperative.files.wordpress.com/2009/12/chomsky-syntactic-structures-2ed.pdf)\n",
    "\n",
    "- [rule based systems (until 1980)]()\n",
    "\n",
    "- [machine learning approaches (like decision trees) replaced handwritten rules (after 1980)]()\n",
    "\n",
    "- [n-grams]()\n",
    "\n",
    "- [recurrent networks (started in 1997 and become successful in 2007)]()\n",
    "\n",
    "- [feed forward network by yoshua bengio (2001)]()\n",
    "\n",
    "- [siri: first successful voice assistant 2011]()\n",
    "\n",
    "\n",
    "References taken from [here](https://www.dataversity.net/a-brief-history-of-natural-language-processing-nlp/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Datasets & Tasks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "[squad 2.0](https://rajpurkar.github.io/SQuAD-explorer/explore/v2.0/dev/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "[Yelp Reviews](https://www.yelp.com/dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "[WordNet](https://wordnet.princeton.edu/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "[LibriSpeech](http://www.openslr.org/12/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "[Spam](http://www.dt.fee.unicamp.br/~tiago/smsspamcollection/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "[NLP Progress](http://nlpprogress.com/) is very good source for getting an idea of nlp tasks and datasets.\n",
    "\n",
    "[top 25 datasets](https://lionbridge.ai/datasets/the-best-25-datasets-for-natural-language-processing/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Part-of-speech Tagging (POS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![title](https://blog.aaronccwong.com/assets/images/bigram-hmm/pos-title.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Dependency Parsing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![title](https://achyutjoshi.github.io/aspect_extraction/assets/images/dependency.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Named Entity Recognition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![title](https://miro.medium.com/max/1000/1*0BuSAIQOLNQGVWJIxZhFuA.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Representation Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![title](https://mymightyvoice.com/wp-content/uploads/2020/02/what-does-it-mean-to-know-a-word_-1024x1024.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Learning text representations is a goal of natural language processing. Representation learning techniques are improving through time. The idea is to have numerical representation of words and that representation should encode it's meaning. You see the questions above that could arise while trying to explain/describe the meaning of word.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Wordnets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![title](https://www.researchgate.net/profile/Jaap_Kamps/publication/228729013/figure/fig4/AS:301958960304141@1449004034445/The-WordNet-database-from-the-vista-point-of-verb-be-and-maximal-MPL-of-3-and-polysemy.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "It's the most intuitive representation of words. It's an intuitive try to encode similarity of words. But there are many problems with using WordNet. Similarities of words aren't fine-grained. Two neighbours of the same word can be considered equally similar. If you want to measure similarity of between two terms you may calculate graph-wise distance between them but it's just natural number not float. Similarity between words can be different in different contexts and that can cause errors in analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "[wordnet](https://moz.com/blog/what-can-wordnet-be-used-for?fbclid=IwAR0OfJ_nD8RxX0Je1i5myp2zCksTdbngrt98Zmrsl0SUTD3jsDueIb7phv8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# One-Hot Encoded Vectors "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![title](https://miro.medium.com/max/674/1*YEJf9BQQh0ma1ECs6x_7yQ.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "There is no chance to use vectors for comparing different words (calculate similarity) since vectors are orthogonal. Also we have 0 or 1 values instead of counting actual number of occurrence of each word inside document. It's kind of a Bag of Word (BOW) representation which doesn't preserve order of words hence loosing context.\n",
    "\n",
    "BOW representation still works fine in information retrieval cases when you represent document by keywords only. Loosing context isn't a big problem here since speed and simplicity of model is more important."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Better Text Representations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![title](https://media.makeameme.org/created/wow-come-on.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# TF-IDF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![title](https://miro.medium.com/max/2514/1*vskqlnEhoNooBTpkoFobYg.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![title](https://miro.medium.com/max/3604/0*56JnM18OAx1lhIQb.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "[tf-idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf#:~:text=In%20information%20retrieval%2C%20tf%E2%80%93idf,in%20a%20collection%20or%20corpus.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Word2Vec (Mikolov et al. 2013)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**Word2vec is a framework for learning word vectors.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- We have a large corpus of text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Every word in a fixed vocabulary is represented by a vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Go through each position t in the text, which has a center word c and context (“outside”) words o"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Use the similarity of the word vectors for c and o to calculate the probability of o given c (or vice versa)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Keep adjusting the word vectors to maximize this probability"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![title](https://cdn-images-1.medium.com/max/1000/1*628Ck-nQpr2WP3G3KxQKNg.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![title](https://www.researchgate.net/profile/Wang_Ling/publication/281812760/figure/fig1/AS:613966665486361@1523392468791/Illustration-of-the-Skip-gram-and-Continuous-Bag-of-Word-CBOW-models.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![title](https://i.stack.imgur.com/nz4CI.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![title](https://sangminwoo.github.io/img/cs224n/lec1/3.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![title](https://aegis4048.github.io/images/featured_images/negative_sampling.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![title](https://miro.medium.com/max/1200/1*2r1yj0zPAuaSGZeQfG6Wtw.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "[word2vec explained](https://israelg99.github.io/2017-03-23-Word2Vec-Explained/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Glove (Improvement over word2vec)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![title](https://www.erogol.com/wp-content/uploads/2016/08/cooccurance_matrix.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Idea of improvement word2vec performance is to use global information of word co occurences (context). Since word2vec is trained on only local context information it has bad performance on small datasets (the smaller is the dataset the lower is the chance to approximate global context using many local context information). \n",
    "\n",
    "Co-occurence matrix X has counts of occuring i and j words from vocabulary in content through full dataset and hence it contains global information of word co-occurences of word pairs. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![title](https://theworldofdatascience.files.wordpress.com/2017/04/word2vec-vs-svd.png?w=1108)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "We didn't much consider many approaches of vectorization therefore it's little bit hard to explain what is shown on table above. Simply, I would say that we have two approaches: local and global context. Local context is something similar to word2vec approach which directly is trained for doing predictions of local context. Global context is more like co-occurence matrix analysis which generally uses global information to compare words. \n",
    "Local approach is slower to train and doesn't that much use of global statistics but it better encodes the information of complex local context patterns. In contrast, global approach is faster to train, efficiently uses global statistics about words and their similarity but lacks of local context information and doesn't perform well in complex scenarios since it knows more about words in general but not in concrete cases."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![title](https://sangminwoo.github.io/img/cs224n/lec2/20.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Idea of GloVe algorithm is to use both global and local context information effectively. It tries to encode similarity of words by their vector multiple. It has more specific details and would be good to consider it in cs224N course to understand the intuition better."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![title](https://adriancolyer.files.wordpress.com/2016/04/glove-vs-word2vec.png?w=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Here we see that GloVe outperforms both word2vec models and trains faster. It's because it uses global contextual information all the time plus encodes word similarity with their scalar multiple."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "[glove explained](https://towardsdatascience.com/light-on-math-ml-intuitive-guide-to-understanding-glove-embeddings-b13b4f19c010)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# PCA & Vector Space"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![title](https://ruder.io/content/images/size/w2000/2016/09/merge_from_ofoct--2-.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "We explore vector space of word embeddings. It's actually very intuitive and encodes conceptual information like word analogies, relationship between similar word pairs, clusters similar words, etc. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Weaknesses of Non-Contextual Embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Embedding of the word is static"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Averages all contexual embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Words with different meanings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Fixed vocabulary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- etc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Standard Libraries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![title](https://spacy.io/static/social_default-1d3b50b1eba4c2b06244425ff0c49570.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![title](https://data-flair.training/blogs/wp-content/uploads/sites/2/2018/08/NLTK-NLP-with-Python.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![title](https://pbs.twimg.com/profile_images/1232743999837888512/QI2OmtDZ.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![title](https://miro.medium.com/max/4000/0*Hm61mNBHsmocinMA.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Georgian NLP ?!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![title](https://memegenerator.net/img/instances/80371197.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Conclusions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "There are many libraries for doing very interesting stuff in NLP. Classical algorithms are useful in many cases  (recognizing entities, analyzing sentiments, searching similar content, extracting keywords for indexing documents, correct typos, classify texts, text similarity, etc) but Deep Learning models have more promising results that we will see in the next lecture."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Useful Links"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "[spacy advanced course](https://course.spacy.io/en/)\n",
    "\n",
    "[NLP Papers with code](https://paperswithcode.com/area/natural-language-processing)\n",
    "\n",
    "[mit nlp](https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-864-advanced-natural-language-processing-fall-2005/)\n",
    "\n",
    "[10 free nlp datasets](https://analyticsindiamag.com/10-nlp-open-source-datasets-to-start-your-first-nlp-project/)\n",
    "\n",
    "[spacy sample projects](https://github.com/explosion/projects)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "[cs224N course materials](http://web.stanford.edu/class/cs224n/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Questions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![title](https://www.memecreator.org/static/images/memes/4589248.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![title](https://66.media.tumblr.com/c921909d1c1b24b64f99c59e9793c5c3/tumblr_inline_p7bl0hstyg1vi5k5n_1280.jpg)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}