05-Getting Started With NLP.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting Started with NLP\n",
    "\n",
    "Some common applications of NLP:\n",
    "\n",
    "* Text Summarization\n",
    "* Text Tagging (topic tagging)\n",
    "* Named Entity Recognition\n",
    "* Chatbot\n",
    "* Speech Recognition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some terms:\n",
    "\n",
    "* Phonetics/phonology:\n",
    "The study of linguistic sounds and their relations to written words\n",
    "* Morphology:\n",
    "The study of internal structures of words/composition of words\n",
    "* Syntax:\n",
    "The study of the structural relationships among words in a sentence\n",
    "* Semantics:\n",
    "The study of the meaning of words and how these combine to form the meaning of sentences\n",
    "* Pragmatics:\n",
    "Situational use of language sentences\n",
    "* Discourse:\n",
    "A linguistic unit that is larger than a single sentence (context)\n",
    "* Tokenization:\n",
    "A step which splits longer strings of text into smaller pieces, or tokens. Larger chunks of text can be tokenized into sentences, sentences can be tokenized into words.\n",
    "* Normalization:\n",
    "Before further processing, text needs to be normalized. Normalization generally refers to a series of related tasks meant to put all text on a level playing field: converting all text to the same case (upper or lower), removing punctuation, expanding contractions, converting numbers to their word equivalents, and so on. Normalization puts all words on equal footing, and allows processing to proceed uniformly.\n",
    "* Stemming:\n",
    "is the process of eliminating affixes (suffixed, prefixes, infixes, postfix, circumfixes) from a word in order to obtain a word stem. ex: running → run\n",
    "* Lemmatization:\n",
    "lemmatization is able to capture canonical forms based on a word's lemma.\n",
    "For example, stemming the word \"better\" would fail to return its citation form (another word for lemma); however, lemmatization would result in the following:\n",
    "better → good\n",
    "* Corpus:\n",
    "(literally Latin for body) refers to a collection of texts. Such collections may be formed of a single language of texts, or can span multiple languages.\n",
    "* Stop words:\n",
    "words which are filtered out before further processing of text, since these words contribute little to overall meaning, given that they are generally the most common words in a language. For instance, \"the,\" \"and,\" and \"a,\" while all required words in a particular passage, don't generally contribute greatly to one's understanding of content.\n",
    "* Parts-of-speech (POS) Tagging:\n",
    "POS tagging consists of assigning a category tag to the tokenized parts of a sentence. The most popular POS tagging would be identifying words as nouns, verbs, adjectives, etc.\n",
    "* Bag of Words:\n",
    "representation model used to simplify the contents of a selection of text. The bag of words model omits grammar and word order, but is interested in the number of occurrences of words within the text. The ultimate representation of the text selection is that of a bag of words.\n",
    "\n",
    "  Ex:\n",
    "  \"Well, well, well,\" said John.\n",
    "  \"There, there,\" said James. \"There, there.\"\n",
    "  \n",
    "  The resulting bag of words representation as a dictionary:\n",
    "     {\n",
    "      'well': 3,\n",
    "      'said': 2,\n",
    "      'john': 1,\n",
    "      'there': 4,\n",
    "      'james': 1\n",
    "     }\n",
    "     \n",
    "     \n",
    "* N-grams:\n",
    "representation model for simplifying text selection contents. As opposed to the orderless representation of bag of words, n-grams modeling is interested in preserving contiguous sequences of N items from the text selection.\n",
    "\n",
    "  An example of trigram (3-gram) model of the second sentence of the above example (\"There, there,\" said James.     \"There, there.\") appears as a list representation below:\n",
    "\n",
    "   [\n",
    "      \"there there said\",\n",
    "      \"there said james\",\n",
    "      \"said james there\",\n",
    "      \"james there there\",\n",
    "   ]\n",
    "   \n",
    "* Statistical Language Modeling:\n",
    "the process of building a statistical language model which is meant to provide an estimate of a natural language. For a sequence of input words, the model would assign a probability to the entire sequence, which contributes to the estimated likelihood of various possible sequences. This can be especially useful for NLP applications which generate text.\n",
    "\n",
    "* Syntactic Analysis:\n",
    "Also referred to as parsing, syntactic analysis is the task of analyzing strings as symbols, and ensuring their conformance to a established set of grammatical rules.\n",
    "\n",
    "* Semantic Analysis:\n",
    "Also known as meaning generation, semantic analysis is interested in determining the meaning of text selections (either character or word sequences). After an input selection of text is read and parsed (analyzed syntactically), the text selection can then be interpreted for meaning. Simply put, syntactic analysis is concerned with what words a text selection was made up of, while semantic analysis wants to know what the collection of words actually means. \n",
    "\n",
    "* Sentiment analysis:\n",
    "the process of evaluating and determining the sentiment captured in a selection of text, with sentiment defined as feeling or emotion.\n",
    "   \n",
    "* Entity recognition:\n",
    "is the process used to classify multiple entities found in a text in predefined categories, such as a person,\n",
    "objects, location, organizations, dates, events, etc.\n",
    "\n",
    "* Word vector:\n",
    "refers to the mapping of the words or phrases from vocabulary to a vector of real numbers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Virtual Environment\n",
    "The main purpose of Python virtual environments is to create an isolated environment for Python projects. This means that each project can have its own dependencies, regardless of what dependencies every other project has."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# In python 2\n",
    "!pip install virtualenv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you are using Python 3, then you should already have the venv module from the standard library installed.\n",
    "# To create a new virtual environment inside the directory\n",
    "# python3 -m venv (env_name)\n",
    "!python3 -m venv handsonnlp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# On Debian/Ubuntu systems, you need to install the python3-venv package using the following command.\n",
    "# sudo apt-get install python3-venv\n",
    "# since it's not installed by default"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Activate / Deactivate your environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!source handsonnlp/bin/activate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!deactivate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## List Installed Packages With Pip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create requirements.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip freeze > requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Natural Language Processing Libraries\n",
    "## 1. NLTK  (www.nltk.org/)\n",
    "Most common package you will encounter working with corpora, categorizing text, analyzing linguistic structure, and more.\n",
    "\n",
    "It was developed by Steven Bird and Edward Loper in the Department of Computer and Information Science at the University of Pennsylvania."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# install NLTK with pip\n",
    "# activate your env first\n",
    "#!source handsonnlp/bin/activate\n",
    "!pip install nltk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenization in NLTK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "ename": "LookupError",
     "evalue": "\n**********************************************************************\n  Resource \u001b[93mpunkt\u001b[0m not found.\n  Please use the NLTK Downloader to obtain the resource:\n\n  \u001b[31m>>> import nltk\n  >>> nltk.download('punkt')\n  \u001b[0m\n  For more information see: https://www.nltk.org/data.html\n\n  Attempted to load \u001b[93mtokenizers/punkt/PY3/english.pickle\u001b[0m\n\n  Searched in:\n    - '/home/muhammad/nltk_data'\n    - '/home/muhammad/anaconda3/nltk_data'\n    - '/home/muhammad/anaconda3/share/nltk_data'\n    - '/home/muhammad/anaconda3/lib/nltk_data'\n    - '/usr/share/nltk_data'\n    - '/usr/local/share/nltk_data'\n    - '/usr/lib/nltk_data'\n    - '/usr/local/lib/nltk_data'\n    - ''\n**********************************************************************\n",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mLookupError\u001b[0m                               Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-8-5906727932bd>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;31m# Tokenization\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0msent_\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"I am almost dead this time\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0mtokens_\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnltk\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mword_tokenize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msent_\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/tokenize/__init__.py\u001b[0m in \u001b[0;36mword_tokenize\u001b[0;34m(text, language, preserve_line)\u001b[0m\n\u001b[1;32m    127\u001b[0m     \u001b[0;34m:\u001b[0m\u001b[0mtype\u001b[0m \u001b[0mpreserve_line\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbool\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    128\u001b[0m     \"\"\"\n\u001b[0;32m--> 129\u001b[0;31m     \u001b[0msentences\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mtext\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mpreserve_line\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0msent_tokenize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtext\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlanguage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    130\u001b[0m     return [\n\u001b[1;32m    131\u001b[0m         \u001b[0mtoken\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0msent\u001b[0m \u001b[0;32min\u001b[0m \u001b[0msentences\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mtoken\u001b[0m \u001b[0;32min\u001b[0m \u001b[0m_treebank_word_tokenizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtokenize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msent\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/tokenize/__init__.py\u001b[0m in \u001b[0;36msent_tokenize\u001b[0;34m(text, language)\u001b[0m\n\u001b[1;32m    104\u001b[0m     \u001b[0;34m:\u001b[0m\u001b[0mparam\u001b[0m \u001b[0mlanguage\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mthe\u001b[0m \u001b[0mmodel\u001b[0m \u001b[0mname\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mthe\u001b[0m \u001b[0mPunkt\u001b[0m \u001b[0mcorpus\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    105\u001b[0m     \"\"\"\n\u001b[0;32m--> 106\u001b[0;31m     \u001b[0mtokenizer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"tokenizers/punkt/{0}.pickle\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlanguage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    107\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mtokenizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtokenize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtext\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    108\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/data.py\u001b[0m in \u001b[0;36mload\u001b[0;34m(resource_url, format, cache, verbose, logic_parser, fstruct_reader, encoding)\u001b[0m\n\u001b[1;32m    750\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    751\u001b[0m     \u001b[0;31m# Load the resource.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 752\u001b[0;31m     \u001b[0mopened_resource\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_open\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresource_url\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    753\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    754\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mformat\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"raw\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/data.py\u001b[0m in \u001b[0;36m_open\u001b[0;34m(resource_url)\u001b[0m\n\u001b[1;32m    875\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    876\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mprotocol\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mprotocol\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlower\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"nltk\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 877\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mfind\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpath_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpath\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m\"\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    878\u001b[0m     \u001b[0;32melif\u001b[0m \u001b[0mprotocol\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlower\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"file\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    879\u001b[0m         \u001b[0;31m# urllib might not use mode='rb', so handle this one ourselves:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/data.py\u001b[0m in \u001b[0;36mfind\u001b[0;34m(resource_name, paths)\u001b[0m\n\u001b[1;32m    583\u001b[0m     \u001b[0msep\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"*\"\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0;36m70\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    584\u001b[0m     \u001b[0mresource_not_found\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"\\n%s\\n%s\\n%s\\n\"\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0msep\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmsg\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msep\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 585\u001b[0;31m     \u001b[0;32mraise\u001b[0m \u001b[0mLookupError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresource_not_found\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    586\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    587\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mLookupError\u001b[0m: \n**********************************************************************\n  Resource \u001b[93mpunkt\u001b[0m not found.\n  Please use the NLTK Downloader to obtain the resource:\n\n  \u001b[31m>>> import nltk\n  >>> nltk.download('punkt')\n  \u001b[0m\n  For more information see: https://www.nltk.org/data.html\n\n  Attempted to load \u001b[93mtokenizers/punkt/PY3/english.pickle\u001b[0m\n\n  Searched in:\n    - '/home/muhammad/nltk_data'\n    - '/home/muhammad/anaconda3/nltk_data'\n    - '/home/muhammad/anaconda3/share/nltk_data'\n    - '/home/muhammad/anaconda3/lib/nltk_data'\n    - '/usr/share/nltk_data'\n    - '/usr/local/share/nltk_data'\n    - '/usr/lib/nltk_data'\n    - '/usr/local/lib/nltk_data'\n    - ''\n**********************************************************************\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "\n",
    "# Tokenization\n",
    "sent_ = \"I am almost dead this time\"\n",
    "tokens_ = nltk.word_tokenize(sent_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /home/muhammad/nltk_data...\n",
      "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# So we need to install punkt\n",
    "nltk.download('punkt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's run word_tokenize() now\n",
    "sents = \"I am almost dead this time\"\n",
    "tokens = nltk.word_tokenize(sents)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['I', 'am', 'almost', 'dead', 'this', 'time']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting a synonym of a word in NLTK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package wordnet to /home/muhammad/nltk_data...\n",
      "[nltk_data]   Unzipping corpora/wordnet.zip.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Getting a synonym of a word.\n",
    "# you need to download wordnet first\n",
    "nltk.download('wordnet')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Synset('spectacular.n.01'), Synset('dramatic.s.02'), Synset('spectacular.s.02'), Synset('outstanding.s.02')]\n"
     ]
    }
   ],
   "source": [
    "from nltk.corpus import wordnet\n",
    "words = wordnet.synsets(\"spectacular\")\n",
    "print(words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'a lavishly produced performance'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Access definition of first synonym\n",
    "words[0].definition()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Synset('spectacular.n.01') : a lavishly produced performance\n",
      "Synset('dramatic.s.02') : sensational in appearance or thrilling in effect\n",
      "Synset('spectacular.s.02') : characteristic of spectacles or drama\n",
      "Synset('outstanding.s.02') : having a quality that thrusts itself into attention\n"
     ]
    }
   ],
   "source": [
    "# Access definition of each synonym\n",
    "for word in words:\n",
    "    print(word, \":\", word.definition())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stemming in NLTK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.stem import PorterStemmer\n",
    "stemmer = PorterStemmer() # Create the stemmer object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'decreas'"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stemmer.stem(\"decreases\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'run'"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stemmer.stem(\"running\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'cat'"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stemmer.stem(\"cats\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'sing'"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stemmer.stem(\"singing\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'shelv'"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stemmer.stem(\"shelves\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'better'"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stemmer.stem(\"better\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lemmatization in NLTK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.stem import WordNetLemmatizer\n",
    "lemmatizer = WordNetLemmatizer() # Create the Lemmatizer object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'decrease'"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lemmatizer.lemmatize(\"decreases\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'rock'"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lemmatizer.lemmatize(\"rocks\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'corpus'"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lemmatizer.lemmatize(\"corpora\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'better'"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lemmatizer.lemmatize(\"better\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'good'"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# What?! better is supposed to lemmatized to good \n",
    "# That's right.\n",
    "# But, one major difference with stemming is that lemmatize takes a part of speech parameter,\n",
    "# “pos” If not supplied, the default is “noun.”\n",
    "# so above it considers better to be a Noun not Adjective\n",
    "# to fix that supply pos parameter like so\n",
    "lemmatizer.lemmatize(\"better\", pos=\"a\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "sentence = \"\"\"At eight o'clock on Thursday morning\n",
    "... Arthur didn't feel very good.\"\"\"\n",
    "tokens = nltk.word_tokenize(sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['At',\n",
       " 'eight',\n",
       " \"o'clock\",\n",
       " 'on',\n",
       " 'Thursday',\n",
       " 'morning',\n",
       " '...',\n",
       " 'Arthur',\n",
       " 'did',\n",
       " \"n't\",\n",
       " 'feel',\n",
       " 'very',\n",
       " 'good',\n",
       " '.']"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## POS tagging in NLTK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "ename": "LookupError",
     "evalue": "\n**********************************************************************\n  Resource \u001b[93maveraged_perceptron_tagger\u001b[0m not found.\n  Please use the NLTK Downloader to obtain the resource:\n\n  \u001b[31m>>> import nltk\n  >>> nltk.download('averaged_perceptron_tagger')\n  \u001b[0m\n  For more information see: https://www.nltk.org/data.html\n\n  Attempted to load \u001b[93mtaggers/averaged_perceptron_tagger/averaged_perceptron_tagger.pickle\u001b[0m\n\n  Searched in:\n    - '/home/muhammad/nltk_data'\n    - '/home/muhammad/anaconda3/nltk_data'\n    - '/home/muhammad/anaconda3/share/nltk_data'\n    - '/home/muhammad/anaconda3/lib/nltk_data'\n    - '/usr/share/nltk_data'\n    - '/usr/local/share/nltk_data'\n    - '/usr/lib/nltk_data'\n    - '/usr/local/lib/nltk_data'\n**********************************************************************\n",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mLookupError\u001b[0m                               Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-47-2801749a9f40>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mtagged\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnltk\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpos_tag\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtokens\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/tag/__init__.py\u001b[0m in \u001b[0;36mpos_tag\u001b[0;34m(tokens, tagset, lang)\u001b[0m\n\u001b[1;32m    158\u001b[0m     \u001b[0;34m:\u001b[0m\u001b[0mrtype\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtuple\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    159\u001b[0m     \"\"\"\n\u001b[0;32m--> 160\u001b[0;31m     \u001b[0mtagger\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_get_tagger\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlang\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    161\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0m_pos_tag\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtokens\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtagset\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtagger\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlang\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    162\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/tag/__init__.py\u001b[0m in \u001b[0;36m_get_tagger\u001b[0;34m(lang)\u001b[0m\n\u001b[1;32m    104\u001b[0m         \u001b[0mtagger\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0map_russian_model_loc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    105\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 106\u001b[0;31m         \u001b[0mtagger\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mPerceptronTagger\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    107\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mtagger\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    108\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/tag/perceptron.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, load)\u001b[0m\n\u001b[1;32m    166\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mload\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    167\u001b[0m             AP_MODEL_LOC = \"file:\" + str(\n\u001b[0;32m--> 168\u001b[0;31m                 \u001b[0mfind\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"taggers/averaged_perceptron_tagger/\"\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mPICKLE\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    169\u001b[0m             )\n\u001b[1;32m    170\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mAP_MODEL_LOC\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/data.py\u001b[0m in \u001b[0;36mfind\u001b[0;34m(resource_name, paths)\u001b[0m\n\u001b[1;32m    583\u001b[0m     \u001b[0msep\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"*\"\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0;36m70\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    584\u001b[0m     \u001b[0mresource_not_found\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"\\n%s\\n%s\\n%s\\n\"\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0msep\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmsg\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msep\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 585\u001b[0;31m     \u001b[0;32mraise\u001b[0m \u001b[0mLookupError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresource_not_found\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    586\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    587\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mLookupError\u001b[0m: \n**********************************************************************\n  Resource \u001b[93maveraged_perceptron_tagger\u001b[0m not found.\n  Please use the NLTK Downloader to obtain the resource:\n\n  \u001b[31m>>> import nltk\n  >>> nltk.download('averaged_perceptron_tagger')\n  \u001b[0m\n  For more information see: https://www.nltk.org/data.html\n\n  Attempted to load \u001b[93mtaggers/averaged_perceptron_tagger/averaged_perceptron_tagger.pickle\u001b[0m\n\n  Searched in:\n    - '/home/muhammad/nltk_data'\n    - '/home/muhammad/anaconda3/nltk_data'\n    - '/home/muhammad/anaconda3/share/nltk_data'\n    - '/home/muhammad/anaconda3/lib/nltk_data'\n    - '/usr/share/nltk_data'\n    - '/usr/local/share/nltk_data'\n    - '/usr/lib/nltk_data'\n    - '/usr/local/lib/nltk_data'\n**********************************************************************\n"
     ]
    }
   ],
   "source": [
    "tagged = nltk.pos_tag(tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
      "[nltk_data]     /home/muhammad/nltk_data...\n",
      "[nltk_data]   Package averaged_perceptron_tagger is already up-to-\n",
      "[nltk_data]       date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# so we need to download the \"averaged_perceptron_tagger\" from NLTK\n",
    "nltk.download('averaged_perceptron_tagger')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "tagged = nltk.pos_tag(tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('At', 'IN'),\n",
       " ('eight', 'CD'),\n",
       " (\"o'clock\", 'NN'),\n",
       " ('on', 'IN'),\n",
       " ('Thursday', 'NNP'),\n",
       " ('morning', 'NN'),\n",
       " ('...', ':'),\n",
       " ('Arthur', 'NNP'),\n",
       " ('did', 'VBD'),\n",
       " (\"n't\", 'RB'),\n",
       " ('feel', 'VB'),\n",
       " ('very', 'RB'),\n",
       " ('good', 'JJ'),\n",
       " ('.', '.')]"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tagged"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "ename": "LookupError",
     "evalue": "\n**********************************************************************\n  Resource \u001b[93mtagsets\u001b[0m not found.\n  Please use the NLTK Downloader to obtain the resource:\n\n  \u001b[31m>>> import nltk\n  >>> nltk.download('tagsets')\n  \u001b[0m\n  For more information see: https://www.nltk.org/data.html\n\n  Attempted to load \u001b[93mhelp/tagsets/PY3/upenn_tagset.pickle\u001b[0m\n\n  Searched in:\n    - '/home/muhammad/nltk_data'\n    - '/home/muhammad/anaconda3/nltk_data'\n    - '/home/muhammad/anaconda3/share/nltk_data'\n    - '/home/muhammad/anaconda3/lib/nltk_data'\n    - '/usr/share/nltk_data'\n    - '/usr/local/share/nltk_data'\n    - '/usr/lib/nltk_data'\n    - '/usr/local/lib/nltk_data'\n    - ''\n**********************************************************************\n",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mLookupError\u001b[0m                               Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-52-8aecbdf48277>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# list all tags\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mnltk\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mhelp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mupenn_tagset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/help.py\u001b[0m in \u001b[0;36mupenn_tagset\u001b[0;34m(tagpattern)\u001b[0m\n\u001b[1;32m     25\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     26\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mupenn_tagset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtagpattern\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 27\u001b[0;31m     \u001b[0m_format_tagset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"upenn_tagset\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtagpattern\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     28\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     29\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/help.py\u001b[0m in \u001b[0;36m_format_tagset\u001b[0;34m(tagset, tagpattern)\u001b[0m\n\u001b[1;32m     44\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     45\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_format_tagset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtagset\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtagpattern\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 46\u001b[0;31m     \u001b[0mtagdict\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"help/tagsets/\"\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mtagset\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m\".pickle\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     47\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mtagpattern\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     48\u001b[0m         \u001b[0m_print_entries\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msorted\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtagdict\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtagdict\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/data.py\u001b[0m in \u001b[0;36mload\u001b[0;34m(resource_url, format, cache, verbose, logic_parser, fstruct_reader, encoding)\u001b[0m\n\u001b[1;32m    750\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    751\u001b[0m     \u001b[0;31m# Load the resource.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 752\u001b[0;31m     \u001b[0mopened_resource\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_open\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresource_url\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    753\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    754\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mformat\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"raw\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/data.py\u001b[0m in \u001b[0;36m_open\u001b[0;34m(resource_url)\u001b[0m\n\u001b[1;32m    875\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    876\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mprotocol\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mprotocol\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlower\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"nltk\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 877\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mfind\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpath_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpath\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m\"\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    878\u001b[0m     \u001b[0;32melif\u001b[0m \u001b[0mprotocol\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlower\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"file\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    879\u001b[0m         \u001b[0;31m# urllib might not use mode='rb', so handle this one ourselves:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/data.py\u001b[0m in \u001b[0;36mfind\u001b[0;34m(resource_name, paths)\u001b[0m\n\u001b[1;32m    583\u001b[0m     \u001b[0msep\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"*\"\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0;36m70\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    584\u001b[0m     \u001b[0mresource_not_found\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"\\n%s\\n%s\\n%s\\n\"\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0msep\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmsg\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msep\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 585\u001b[0;31m     \u001b[0;32mraise\u001b[0m \u001b[0mLookupError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresource_not_found\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    586\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    587\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mLookupError\u001b[0m: \n**********************************************************************\n  Resource \u001b[93mtagsets\u001b[0m not found.\n  Please use the NLTK Downloader to obtain the resource:\n\n  \u001b[31m>>> import nltk\n  >>> nltk.download('tagsets')\n  \u001b[0m\n  For more information see: https://www.nltk.org/data.html\n\n  Attempted to load \u001b[93mhelp/tagsets/PY3/upenn_tagset.pickle\u001b[0m\n\n  Searched in:\n    - '/home/muhammad/nltk_data'\n    - '/home/muhammad/anaconda3/nltk_data'\n    - '/home/muhammad/anaconda3/share/nltk_data'\n    - '/home/muhammad/anaconda3/lib/nltk_data'\n    - '/usr/share/nltk_data'\n    - '/usr/local/share/nltk_data'\n    - '/usr/lib/nltk_data'\n    - '/usr/local/lib/nltk_data'\n    - ''\n**********************************************************************\n"
     ]
    }
   ],
   "source": [
    "# list all tags\n",
    "nltk.help.upenn_tagset()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package tagsets to /home/muhammad/nltk_data...\n",
      "[nltk_data]   Unzipping help/tagsets.zip.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.download('tagsets')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "$: dollar\n",
      "    $ -$ --$ A$ C$ HK$ M$ NZ$ S$ U.S.$ US$\n",
      "'': closing quotation mark\n",
      "    ' ''\n",
      "(: opening parenthesis\n",
      "    ( [ {\n",
      "): closing parenthesis\n",
      "    ) ] }\n",
      ",: comma\n",
      "    ,\n",
      "--: dash\n",
      "    --\n",
      ".: sentence terminator\n",
      "    . ! ?\n",
      ":: colon or ellipsis\n",
      "    : ; ...\n",
      "CC: conjunction, coordinating\n",
      "    & 'n and both but either et for less minus neither nor or plus so\n",
      "    therefore times v. versus vs. whether yet\n",
      "CD: numeral, cardinal\n",
      "    mid-1890 nine-thirty forty-two one-tenth ten million 0.5 one forty-\n",
      "    seven 1987 twenty '79 zero two 78-degrees eighty-four IX '60s .025\n",
      "    fifteen 271,124 dozen quintillion DM2,000 ...\n",
      "DT: determiner\n",
      "    all an another any both del each either every half la many much nary\n",
      "    neither no some such that the them these this those\n",
      "EX: existential there\n",
      "    there\n",
      "FW: foreign word\n",
      "    gemeinschaft hund ich jeux habeas Haementeria Herr K'ang-si vous\n",
      "    lutihaw alai je jour objets salutaris fille quibusdam pas trop Monte\n",
      "    terram fiche oui corporis ...\n",
      "IN: preposition or conjunction, subordinating\n",
      "    astride among uppon whether out inside pro despite on by throughout\n",
      "    below within for towards near behind atop around if like until below\n",
      "    next into if beside ...\n",
      "JJ: adjective or numeral, ordinal\n",
      "    third ill-mannered pre-war regrettable oiled calamitous first separable\n",
      "    ectoplasmic battery-powered participatory fourth still-to-be-named\n",
      "    multilingual multi-disciplinary ...\n",
      "JJR: adjective, comparative\n",
      "    bleaker braver breezier briefer brighter brisker broader bumper busier\n",
      "    calmer cheaper choosier cleaner clearer closer colder commoner costlier\n",
      "    cozier creamier crunchier cuter ...\n",
      "JJS: adjective, superlative\n",
      "    calmest cheapest choicest classiest cleanest clearest closest commonest\n",
      "    corniest costliest crassest creepiest crudest cutest darkest deadliest\n",
      "    dearest deepest densest dinkiest ...\n",
      "LS: list item marker\n",
      "    A A. B B. C C. D E F First G H I J K One SP-44001 SP-44002 SP-44005\n",
      "    SP-44007 Second Third Three Two * a b c d first five four one six three\n",
      "    two\n",
      "MD: modal auxiliary\n",
      "    can cannot could couldn't dare may might must need ought shall should\n",
      "    shouldn't will would\n",
      "NN: noun, common, singular or mass\n",
      "    common-carrier cabbage knuckle-duster Casino afghan shed thermostat\n",
      "    investment slide humour falloff slick wind hyena override subhumanity\n",
      "    machinist ...\n",
      "NNP: noun, proper, singular\n",
      "    Motown Venneboerger Czestochwa Ranzer Conchita Trumplane Christos\n",
      "    Oceanside Escobar Kreisler Sawyer Cougar Yvette Ervin ODI Darryl CTCA\n",
      "    Shannon A.K.C. Meltex Liverpool ...\n",
      "NNPS: noun, proper, plural\n",
      "    Americans Americas Amharas Amityvilles Amusements Anarcho-Syndicalists\n",
      "    Andalusians Andes Andruses Angels Animals Anthony Antilles Antiques\n",
      "    Apache Apaches Apocrypha ...\n",
      "NNS: noun, common, plural\n",
      "    undergraduates scotches bric-a-brac products bodyguards facets coasts\n",
      "    divestitures storehouses designs clubs fragrances averages\n",
      "    subjectivists apprehensions muses factory-jobs ...\n",
      "PDT: pre-determiner\n",
      "    all both half many quite such sure this\n",
      "POS: genitive marker\n",
      "    ' 's\n",
      "PRP: pronoun, personal\n",
      "    hers herself him himself hisself it itself me myself one oneself ours\n",
      "    ourselves ownself self she thee theirs them themselves they thou thy us\n",
      "PRP$: pronoun, possessive\n",
      "    her his mine my our ours their thy your\n",
      "RB: adverb\n",
      "    occasionally unabatingly maddeningly adventurously professedly\n",
      "    stirringly prominently technologically magisterially predominately\n",
      "    swiftly fiscally pitilessly ...\n",
      "RBR: adverb, comparative\n",
      "    further gloomier grander graver greater grimmer harder harsher\n",
      "    healthier heavier higher however larger later leaner lengthier less-\n",
      "    perfectly lesser lonelier longer louder lower more ...\n",
      "RBS: adverb, superlative\n",
      "    best biggest bluntest earliest farthest first furthest hardest\n",
      "    heartiest highest largest least less most nearest second tightest worst\n",
      "RP: particle\n",
      "    aboard about across along apart around aside at away back before behind\n",
      "    by crop down ever fast for forth from go high i.e. in into just later\n",
      "    low more off on open out over per pie raising start teeth that through\n",
      "    under unto up up-pp upon whole with you\n",
      "SYM: symbol\n",
      "    % & ' '' ''. ) ). * + ,. < = > @ A[fj] U.S U.S.S.R * ** ***\n",
      "TO: \"to\" as preposition or infinitive marker\n",
      "    to\n",
      "UH: interjection\n",
      "    Goodbye Goody Gosh Wow Jeepers Jee-sus Hubba Hey Kee-reist Oops amen\n",
      "    huh howdy uh dammit whammo shucks heck anyways whodunnit honey golly\n",
      "    man baby diddle hush sonuvabitch ...\n",
      "VB: verb, base form\n",
      "    ask assemble assess assign assume atone attention avoid bake balkanize\n",
      "    bank begin behold believe bend benefit bevel beware bless boil bomb\n",
      "    boost brace break bring broil brush build ...\n",
      "VBD: verb, past tense\n",
      "    dipped pleaded swiped regummed soaked tidied convened halted registered\n",
      "    cushioned exacted snubbed strode aimed adopted belied figgered\n",
      "    speculated wore appreciated contemplated ...\n",
      "VBG: verb, present participle or gerund\n",
      "    telegraphing stirring focusing angering judging stalling lactating\n",
      "    hankerin' alleging veering capping approaching traveling besieging\n",
      "    encrypting interrupting erasing wincing ...\n",
      "VBN: verb, past participle\n",
      "    multihulled dilapidated aerosolized chaired languished panelized used\n",
      "    experimented flourished imitated reunifed factored condensed sheared\n",
      "    unsettled primed dubbed desired ...\n",
      "VBP: verb, present tense, not 3rd person singular\n",
      "    predominate wrap resort sue twist spill cure lengthen brush terminate\n",
      "    appear tend stray glisten obtain comprise detest tease attract\n",
      "    emphasize mold postpone sever return wag ...\n",
      "VBZ: verb, present tense, 3rd person singular\n",
      "    bases reconstructs marks mixes displeases seals carps weaves snatches\n",
      "    slumps stretches authorizes smolders pictures emerges stockpiles\n",
      "    seduces fizzes uses bolsters slaps speaks pleads ...\n",
      "WDT: WH-determiner\n",
      "    that what whatever which whichever\n",
      "WP: WH-pronoun\n",
      "    that what whatever whatsoever which who whom whosoever\n",
      "WP$: WH-pronoun, possessive\n",
      "    whose\n",
      "WRB: Wh-adverb\n",
      "    how however whence whenever where whereby whereever wherein whereof why\n",
      "``: opening quotation mark\n",
      "    ` ``\n"
     ]
    }
   ],
   "source": [
    "nltk.help.upenn_tagset()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "list"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(tagged)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('At', 'IN'),\n",
       " ('eight', 'CD'),\n",
       " (\"o'clock\", 'NN'),\n",
       " ('on', 'IN'),\n",
       " ('Thursday', 'NNP'),\n",
       " ('morning', 'NN')]"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tagged[0:6]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Identify named entities in NLTK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "ename": "LookupError",
     "evalue": "\n**********************************************************************\n  Resource \u001b[93mmaxent_ne_chunker\u001b[0m not found.\n  Please use the NLTK Downloader to obtain the resource:\n\n  \u001b[31m>>> import nltk\n  >>> nltk.download('maxent_ne_chunker')\n  \u001b[0m\n  For more information see: https://www.nltk.org/data.html\n\n  Attempted to load \u001b[93mchunkers/maxent_ne_chunker/PY3/english_ace_multiclass.pickle\u001b[0m\n\n  Searched in:\n    - '/home/muhammad/nltk_data'\n    - '/home/muhammad/anaconda3/nltk_data'\n    - '/home/muhammad/anaconda3/share/nltk_data'\n    - '/home/muhammad/anaconda3/lib/nltk_data'\n    - '/usr/share/nltk_data'\n    - '/usr/local/share/nltk_data'\n    - '/usr/lib/nltk_data'\n    - '/usr/local/lib/nltk_data'\n    - ''\n**********************************************************************\n",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mLookupError\u001b[0m                               Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-58-fde12775ac24>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mentities\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnltk\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mchunk\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mne_chunk\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtagged\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/chunk/__init__.py\u001b[0m in \u001b[0;36mne_chunk\u001b[0;34m(tagged_tokens, binary)\u001b[0m\n\u001b[1;32m    183\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    184\u001b[0m         \u001b[0mchunker_pickle\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_MULTICLASS_NE_CHUNKER\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 185\u001b[0;31m     \u001b[0mchunker\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mchunker_pickle\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    186\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mchunker\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mparse\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtagged_tokens\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    187\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/data.py\u001b[0m in \u001b[0;36mload\u001b[0;34m(resource_url, format, cache, verbose, logic_parser, fstruct_reader, encoding)\u001b[0m\n\u001b[1;32m    750\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    751\u001b[0m     \u001b[0;31m# Load the resource.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 752\u001b[0;31m     \u001b[0mopened_resource\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_open\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresource_url\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    753\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    754\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mformat\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"raw\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/data.py\u001b[0m in \u001b[0;36m_open\u001b[0;34m(resource_url)\u001b[0m\n\u001b[1;32m    875\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    876\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mprotocol\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mprotocol\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlower\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"nltk\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 877\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mfind\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpath_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpath\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m\"\"\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    878\u001b[0m     \u001b[0;32melif\u001b[0m \u001b[0mprotocol\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlower\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"file\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    879\u001b[0m         \u001b[0;31m# urllib might not use mode='rb', so handle this one ourselves:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/data.py\u001b[0m in \u001b[0;36mfind\u001b[0;34m(resource_name, paths)\u001b[0m\n\u001b[1;32m    583\u001b[0m     \u001b[0msep\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"*\"\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0;36m70\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    584\u001b[0m     \u001b[0mresource_not_found\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"\\n%s\\n%s\\n%s\\n\"\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0msep\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmsg\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msep\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 585\u001b[0;31m     \u001b[0;32mraise\u001b[0m \u001b[0mLookupError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresource_not_found\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    586\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    587\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mLookupError\u001b[0m: \n**********************************************************************\n  Resource \u001b[93mmaxent_ne_chunker\u001b[0m not found.\n  Please use the NLTK Downloader to obtain the resource:\n\n  \u001b[31m>>> import nltk\n  >>> nltk.download('maxent_ne_chunker')\n  \u001b[0m\n  For more information see: https://www.nltk.org/data.html\n\n  Attempted to load \u001b[93mchunkers/maxent_ne_chunker/PY3/english_ace_multiclass.pickle\u001b[0m\n\n  Searched in:\n    - '/home/muhammad/nltk_data'\n    - '/home/muhammad/anaconda3/nltk_data'\n    - '/home/muhammad/anaconda3/share/nltk_data'\n    - '/home/muhammad/anaconda3/lib/nltk_data'\n    - '/usr/share/nltk_data'\n    - '/usr/local/share/nltk_data'\n    - '/usr/lib/nltk_data'\n    - '/usr/local/lib/nltk_data'\n    - ''\n**********************************************************************\n"
     ]
    }
   ],
   "source": [
    "entities = nltk.chunk.ne_chunk(tagged)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package maxent_ne_chunker to\n",
      "[nltk_data]     /home/muhammad/nltk_data...\n",
      "[nltk_data]   Unzipping chunkers/maxent_ne_chunker.zip.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# download maxent_ne_chunker package\n",
    "nltk.download('maxent_ne_chunker')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "ename": "LookupError",
     "evalue": "\n**********************************************************************\n  Resource \u001b[93mwords\u001b[0m not found.\n  Please use the NLTK Downloader to obtain the resource:\n\n  \u001b[31m>>> import nltk\n  >>> nltk.download('words')\n  \u001b[0m\n  For more information see: https://www.nltk.org/data.html\n\n  Attempted to load \u001b[93mcorpora/words\u001b[0m\n\n  Searched in:\n    - '/home/muhammad/nltk_data'\n    - '/home/muhammad/anaconda3/nltk_data'\n    - '/home/muhammad/anaconda3/share/nltk_data'\n    - '/home/muhammad/anaconda3/lib/nltk_data'\n    - '/usr/share/nltk_data'\n    - '/usr/local/share/nltk_data'\n    - '/usr/lib/nltk_data'\n    - '/usr/local/lib/nltk_data'\n**********************************************************************\n",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mLookupError\u001b[0m                               Traceback (most recent call last)",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/corpus/util.py\u001b[0m in \u001b[0;36m__load\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m     82\u001b[0m                 \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 83\u001b[0;31m                     \u001b[0mroot\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnltk\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfind\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"{}/{}\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msubdir\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mzip_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     84\u001b[0m                 \u001b[0;32mexcept\u001b[0m \u001b[0mLookupError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/data.py\u001b[0m in \u001b[0;36mfind\u001b[0;34m(resource_name, paths)\u001b[0m\n\u001b[1;32m    584\u001b[0m     \u001b[0mresource_not_found\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"\\n%s\\n%s\\n%s\\n\"\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0msep\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmsg\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msep\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 585\u001b[0;31m     \u001b[0;32mraise\u001b[0m \u001b[0mLookupError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresource_not_found\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    586\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mLookupError\u001b[0m: \n**********************************************************************\n  Resource \u001b[93mwords\u001b[0m not found.\n  Please use the NLTK Downloader to obtain the resource:\n\n  \u001b[31m>>> import nltk\n  >>> nltk.download('words')\n  \u001b[0m\n  For more information see: https://www.nltk.org/data.html\n\n  Attempted to load \u001b[93mcorpora/words.zip/words/\u001b[0m\n\n  Searched in:\n    - '/home/muhammad/nltk_data'\n    - '/home/muhammad/anaconda3/nltk_data'\n    - '/home/muhammad/anaconda3/share/nltk_data'\n    - '/home/muhammad/anaconda3/lib/nltk_data'\n    - '/usr/share/nltk_data'\n    - '/usr/local/share/nltk_data'\n    - '/usr/lib/nltk_data'\n    - '/usr/local/lib/nltk_data'\n**********************************************************************\n",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mLookupError\u001b[0m                               Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-60-fde12775ac24>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mentities\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnltk\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mchunk\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mne_chunk\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtagged\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/chunk/__init__.py\u001b[0m in \u001b[0;36mne_chunk\u001b[0;34m(tagged_tokens, binary)\u001b[0m\n\u001b[1;32m    184\u001b[0m         \u001b[0mchunker_pickle\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_MULTICLASS_NE_CHUNKER\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    185\u001b[0m     \u001b[0mchunker\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mchunker_pickle\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 186\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0mchunker\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mparse\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtagged_tokens\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    187\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    188\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/chunk/named_entity.py\u001b[0m in \u001b[0;36mparse\u001b[0;34m(self, tokens)\u001b[0m\n\u001b[1;32m    124\u001b[0m         \u001b[0mEach\u001b[0m \u001b[0mtoken\u001b[0m \u001b[0mshould\u001b[0m \u001b[0mbe\u001b[0m \u001b[0ma\u001b[0m \u001b[0mpos\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0mtagged\u001b[0m \u001b[0mword\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    125\u001b[0m         \"\"\"\n\u001b[0;32m--> 126\u001b[0;31m         \u001b[0mtagged\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_tagger\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtag\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtokens\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    127\u001b[0m         \u001b[0mtree\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_tagged_to_parse\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtagged\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    128\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mtree\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/tag/sequential.py\u001b[0m in \u001b[0;36mtag\u001b[0;34m(self, tokens)\u001b[0m\n\u001b[1;32m     61\u001b[0m         \u001b[0mtags\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     62\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtokens\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 63\u001b[0;31m             \u001b[0mtags\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtag_one\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtokens\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtags\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     64\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mzip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtokens\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtags\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     65\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/tag/sequential.py\u001b[0m in \u001b[0;36mtag_one\u001b[0;34m(self, tokens, index, history)\u001b[0m\n\u001b[1;32m     81\u001b[0m         \u001b[0mtag\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     82\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mtagger\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_taggers\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 83\u001b[0;31m             \u001b[0mtag\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtagger\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mchoose_tag\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtokens\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mindex\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhistory\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     84\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mtag\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     85\u001b[0m                 \u001b[0;32mbreak\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/tag/sequential.py\u001b[0m in \u001b[0;36mchoose_tag\u001b[0;34m(self, tokens, index, history)\u001b[0m\n\u001b[1;32m    640\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mchoose_tag\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtokens\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mindex\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhistory\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    641\u001b[0m         \u001b[0;31m# Use our feature detector to get the featureset.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 642\u001b[0;31m         \u001b[0mfeatureset\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfeature_detector\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtokens\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mindex\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhistory\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    643\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    644\u001b[0m         \u001b[0;31m# Use the classifier to pick a tag.  If a cutoff probability\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/tag/sequential.py\u001b[0m in \u001b[0;36mfeature_detector\u001b[0;34m(self, tokens, index, history)\u001b[0m\n\u001b[1;32m    687\u001b[0m         \u001b[0mSee\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0mclassifier\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    688\u001b[0m         \"\"\"\n\u001b[0;32m--> 689\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_feature_detector\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtokens\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mindex\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhistory\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    690\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    691\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mclassifier\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/chunk/named_entity.py\u001b[0m in \u001b[0;36m_feature_detector\u001b[0;34m(self, tokens, index, history)\u001b[0m\n\u001b[1;32m     98\u001b[0m             \u001b[0;34m\"pos\"\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mpos\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     99\u001b[0m             \u001b[0;34m\"word\"\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mword\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 100\u001b[0;31m             \u001b[0;34m\"en-wordlist\"\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mword\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_english_wordlist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    101\u001b[0m             \u001b[0;34m\"prevtag\"\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mprevtag\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    102\u001b[0m             \u001b[0;34m\"prevpos\"\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mprevpos\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/chunk/named_entity.py\u001b[0m in \u001b[0;36m_english_wordlist\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m     49\u001b[0m             \u001b[0;32mfrom\u001b[0m \u001b[0mnltk\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcorpus\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mwords\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     50\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 51\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_en_wordlist\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mwords\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwords\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"en-basic\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     52\u001b[0m             \u001b[0mwl\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_en_wordlist\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     53\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mwl\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/corpus/util.py\u001b[0m in \u001b[0;36m__getattr__\u001b[0;34m(self, attr)\u001b[0m\n\u001b[1;32m    118\u001b[0m             \u001b[0;32mraise\u001b[0m \u001b[0mAttributeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"LazyCorpusLoader object has no attribute '__bases__'\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    119\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 120\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__load\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    121\u001b[0m         \u001b[0;31m# This looks circular, but its not, since __load() changes our\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    122\u001b[0m         \u001b[0;31m# __class__ to something new:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/corpus/util.py\u001b[0m in \u001b[0;36m__load\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m     83\u001b[0m                     \u001b[0mroot\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnltk\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfind\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"{}/{}\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msubdir\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mzip_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     84\u001b[0m                 \u001b[0;32mexcept\u001b[0m \u001b[0mLookupError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 85\u001b[0;31m                     \u001b[0;32mraise\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     86\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     87\u001b[0m         \u001b[0;31m# Load the corpus.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/corpus/util.py\u001b[0m in \u001b[0;36m__load\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m     78\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     79\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 80\u001b[0;31m                 \u001b[0mroot\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnltk\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfind\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"{}/{}\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msubdir\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     81\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mLookupError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     82\u001b[0m                 \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/data.py\u001b[0m in \u001b[0;36mfind\u001b[0;34m(resource_name, paths)\u001b[0m\n\u001b[1;32m    583\u001b[0m     \u001b[0msep\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"*\"\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0;36m70\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    584\u001b[0m     \u001b[0mresource_not_found\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"\\n%s\\n%s\\n%s\\n\"\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0msep\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmsg\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msep\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 585\u001b[0;31m     \u001b[0;32mraise\u001b[0m \u001b[0mLookupError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresource_not_found\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    586\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    587\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mLookupError\u001b[0m: \n**********************************************************************\n  Resource \u001b[93mwords\u001b[0m not found.\n  Please use the NLTK Downloader to obtain the resource:\n\n  \u001b[31m>>> import nltk\n  >>> nltk.download('words')\n  \u001b[0m\n  For more information see: https://www.nltk.org/data.html\n\n  Attempted to load \u001b[93mcorpora/words\u001b[0m\n\n  Searched in:\n    - '/home/muhammad/nltk_data'\n    - '/home/muhammad/anaconda3/nltk_data'\n    - '/home/muhammad/anaconda3/share/nltk_data'\n    - '/home/muhammad/anaconda3/lib/nltk_data'\n    - '/usr/share/nltk_data'\n    - '/usr/local/share/nltk_data'\n    - '/usr/lib/nltk_data'\n    - '/usr/local/lib/nltk_data'\n**********************************************************************\n"
     ]
    }
   ],
   "source": [
    "entities = nltk.chunk.ne_chunk(tagged)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package words to /home/muhammad/nltk_data...\n",
      "[nltk_data]   Unzipping corpora/words.zip.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# download words package\n",
    "nltk.download('words')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "entities = nltk.chunk.ne_chunk(tagged)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABEYAAABTCAIAAAAdjpV5AAAJMmlDQ1BkZWZhdWx0X3JnYi5pY2MAAEiJlZVnUJNZF8fv8zzphUASQodQQ5EqJYCUEFoo0quoQOidUEVsiLgCK4qINEWQRQEXXJUia0UUC4uCAhZ0gywCyrpxFVFBWXDfGZ33HT+8/5l7z2/+c+bec8/5cAEgiINlwct7YlK6wNvJjhkYFMwE3yiMn5bC8fR0A9/VuxEArcR7ut/P+a4IEZFp/OW4uLxy+SmCdACg7GXWzEpPWeGjy0wPj//CZ1dYsFzgMt9Y4eh/eexLzr8s+pLj681dfhUKABwp+hsO/4b/c++KVDiC9NioyGymT3JUelaYIJKZttIJHpfL9BQkR8UmRH5T8P+V/B2lR2anr0RucsomQWx0TDrzfw41MjA0BF9n8cbrS48hRv9/z2dFX73kegDYcwAg+7564ZUAdO4CQPrRV09tua+UfAA67vAzBJn/eqiVDQ0IgALoQAYoAlWgCXSBETADlsAWOAAX4AF8QRDYAPggBiQCAcgCuWAHKABFYB84CKpALWgATaAVnAad4Dy4Aq6D2+AuGAaPgRBMgpdABN6BBQiCsBAZokEykBKkDulARhAbsoYcIDfIGwqCQqFoKAnKgHKhnVARVApVQXVQE/QLdA66At2EBqGH0Dg0A/0NfYQRmATTYQVYA9aH2TAHdoV94fVwNJwK58D58F64Aq6HT8Id8BX4NjwMC+GX8BwCECLCQJQRXYSNcBEPJBiJQgTIVqQQKUfqkVakG+lD7iFCZBb5gMKgaCgmShdliXJG+aH4qFTUVlQxqgp1AtWB6kXdQ42jRKjPaDJaHq2DtkDz0IHoaHQWugBdjm5Et6OvoYfRk+h3GAyGgWFhzDDOmCBMHGYzphhzGNOGuYwZxExg5rBYrAxWB2uF9cCGYdOxBdhK7EnsJewQdhL7HkfEKeGMcI64YFwSLg9XjmvGXcQN4aZwC3hxvDreAu+Bj8BvwpfgG/Dd+Dv4SfwCQYLAIlgRfAlxhB2ECkIr4RphjPCGSCSqEM2JXsRY4nZiBfEU8QZxnPiBRCVpk7ikEFIGaS/pOOky6SHpDZlM1iDbkoPJ6eS95CbyVfJT8nsxmpieGE8sQmybWLVYh9iQ2CsKnqJO4VA2UHIo5ZQzlDuUWXG8uIY4VzxMfKt4tfg58VHxOQmahKGEh0SiRLFEs8RNiWkqlqpBdaBGUPOpx6hXqRM0hKZK49L4tJ20Bto12iQdQ2fRefQ4ehH9Z/oAXSRJlTSW9JfMlqyWvCApZCAMDQaPkcAoYZxmjDA+SilIcaQipfZItUoNSc1Ly0nbSkdKF0q3SQ9Lf5RhyjjIxMvsl+mUeSKLktWW9ZLNkj0ie012Vo4uZynHlyuUOy33SB6W15b3lt8sf0y+X35OQVHBSSFFoVLhqsKsIkPRVjFOsUzxouKMEk3JWilWqUzpktILpiSTw0xgVjB7mSJleWVn5QzlOuUB5QUVloqfSp5Km8oTVYIqWzVKtUy1R1WkpqTmrpar1qL2SB2vzlaPUT+k3qc+r8HSCNDYrdGpMc2SZvFYOawW1pgmWdNGM1WzXvO+FkaLrRWvdVjrrjasbaIdo12tfUcH1jHVidU5rDO4Cr3KfFXSqvpVo7okXY5upm6L7rgeQ89NL0+vU++Vvpp+sP5+/T79zwYmBgkGDQaPDamGLoZ5ht2GfxtpG/GNqo3uryavdly9bXXX6tfGOsaRxkeMH5jQTNxNdpv0mHwyNTMVmLaazpipmYWa1ZiNsulsT3Yx+4Y52tzOfJv5efMPFqYW6RanLf6y1LWMt2y2nF7DWhO5pmHNhJWKVZhVnZXQmmkdan3UWmijbBNmU2/zzFbVNsK20XaKo8WJ45zkvLIzsBPYtdvNcy24W7iX7RF7J/tC+wEHqoOfQ5XDU0cVx2jHFkeRk4nTZqfLzmhnV+f9zqM8BR6f18QTuZi5bHHpdSW5+rhWuT5z03YTuHW7w+4u7gfcx9aqr01a2+kBPHgeBzyeeLI8Uz1/9cJ4eXpVez33NvTO9e7zofls9Gn2eedr51vi+9hP0y/Dr8ef4h/i3+Q/H2AfUBogDNQP3BJ4O0g2KDaoKxgb7B/cGDy3zmHdwXWTISYhBSEj61nrs9ff3CC7IWHDhY2UjWEbz4SiQwNCm0MXwzzC6sPmwnnhNeEiPpd/iP8ywjaiLGIm0iqyNHIqyiqqNGo62ir6QPRMjE1MecxsLDe2KvZ1nHNcbdx8vEf88filhICEtkRcYmjiuSRqUnxSb7JicnbyYIpOSkGKMNUi9WCqSOAqaEyD0tandaXTlz/F/gzNjF0Z45nWmdWZ77P8s85kS2QnZfdv0t60Z9NUjmPOT5tRm/mbe3KVc3fkjm/hbKnbCm0N39qzTXVb/rbJ7U7bT+wg7Ijf8VueQV5p3tudATu78xXyt+dP7HLa1VIgViAoGN1tubv2B9QPsT8M7Fm9p3LP58KIwltFBkXlRYvF/OJbPxr+WPHj0t6ovQMlpiVH9mH2Je0b2W+z/0SpRGlO6cQB9wMdZcyywrK3BzcevFluXF57iHAo45Cwwq2iq1Ktcl/lYlVM1XC1XXVbjXzNnpr5wxGHh47YHmmtVagtqv14NPbogzqnuo56jfryY5hjmceeN/g39P3E/qmpUbaxqPHT8aTjwhPeJ3qbzJqamuWbS1rgloyWmZMhJ+/+bP9zV6tua10bo63oFDiVcerFL6G/jJx2Pd1zhn2m9az62Zp2WnthB9SxqUPUGdMp7ArqGjzncq6n27K7/Ve9X4+fVz5ffUHyQslFwsX8i0uXci7NXU65PHsl+spEz8aex1cDr97v9eoduOZ67cZ1x+tX+zh9l25Y3Th/0+LmuVvsW523TW939Jv0t/9m8lv7gOlAxx2zO113ze92D64ZvDhkM3Tlnv296/d5928Prx0eHPEbeTAaMip8EPFg+mHCw9ePMh8tPN4+hh4rfCL+pPyp/NP637V+bxOaCi+M24/3P/N59niCP/Hyj7Q/Fifzn5Ofl08pTTVNG02fn3Gcufti3YvJlykvF2YL/pT4s+aV5quzf9n+1S8KFE2+Frxe+rv4jcyb42+N3/bMec49fZf4bmG+8L3M+xMf2B/6PgZ8nFrIWsQuVnzS+tT92fXz2FLi0tI/QiyQvpTNDAsAAAAJcEhZcwAADdcAAA3XAUIom3gAAAAddEVYdFNvZnR3YXJlAEdQTCBHaG9zdHNjcmlwdCA5LjUw/rJdRQAAIABJREFUeJzt3U1sG0eaN/CiPihRsi21YkqOdieyqU2wkN59gTWZOQmYGYgEdrDwnEIdvZmDqcsCmcNG5NFHEjl5gB2A3MNm7cMLkAPMYYBc2AESQAb8JuzsicbuZNmS5kOy5Yxasi2JFmVxDw9UU+nqppoUP7rl/+9gUO1ms7q6uquerqpuX71eZwAAAAAAAN7U1+sEAAAAAAAAtA4hDQAAAAAAeBhCGgAAAAAA8LCBXicAAADgDIVCoVQqLS0tKYoSCoV6nRwAAHAX9NIAAICrJZNJwzBSqZSqqtlsttfJAQAA1/HhiWcAAOBm8Xi8UCjQZ1VVo9Fob9MDAABug5AGAABcTdO0bDarKEokEonH471ODgAAuA5CGgAA8AaaUZPJZHqdEAAAcBfMpQEAAFdLJpP0IR6PG4bR28QAAIAL4YlnAADgaqqqUlRjGEYsFut1cgAAwHUw8AwAANzOMAxN0/BgAAAAsISQBgAAAAAAPAxzaQAAAAAAwMMwlwYAAFzK2N//f48effnf//3t06dbhjE2Ojo7OfmTv/3bf/npT3udNAAAcBEMPAMAgPMy9ve19XX+5zcbG/SBFr6sVv9oGK+Oj/nKL6tVxljt5KTO2MnJyUk7aiIf/evz9fl8g/39Pp9vsL//rUuXBvv7GWOXh4dHh4amrlx579q1y8PDjLG3x8beHh+n70bn58+fAAAA6BWENAAAbxB9e1t/9sy00Dg4UMvl3YMD0/L/r+v886vj45OTk6Pj4+fValsiEId8jPl8vpN6fbCvz9/fv1+rded3+3y+Ub/fPzjY39dHfw4NDIyPjFy/enVo4HsDHP7h7/7uBxMTpq8jRgIA6CaENAAAbqStrxv7+6aF+rNnle1t/ueTvb1qrcYY+/bpU1pyWKsdHh3x/+UdI65yeXj4+tWrB0dHA3191VqtWqs9ff6c/+/i3FwoGJydnAzPzETn5xOffvpvX35550c/yn34Ia2gra/nv/pKffyY+oKujY3937/+6/6+Pp/Pt7W7+6fd3W1ha+4xNDBwbWyM/3klEBjo63t7fPzS0BAtuTY2Njw4yBiL3LihjIyYvo4YCQCgAYQ0AADnopbL8kJT7ME90vUXh4emhf/5+993JGXtNur3Dw8OHh0fM8aqx8e1168tV1ucm6MP4ZkZxtjEpUvhmZnfPXlyUq//YWdH29jQ1td5j9D4yEj4+vXwzMzs5GT4+vXw9et8O8b+fuLTT39dKonxjEjf3i6USsVy+fPHj2lT8fffj83PR+fmjP19/dkzOgrGwQF9Xvt+99TQwMBbly4dHR/3+XyHtdqLatVyd/p8vv6+Prud7SGKkSZGR8WF705NXb961bRmTAqHlNFRMasBALwOIQ0AvBFMkz04bWNj5+VL88oHB+LorMOjo8OjoxfV6v9YRSmd4PP5AoOD/T7fq+Pjo2Ya01eGh/v7+xljLw4Pj09OGq88PT4+deUKY+ywVvMPDAwPDOwdHlZrtYDf/19bW5ZfoQiEMaaMjISCQcbY7OQkfQhfv64IzWu1XKa81TY2KOQgN2dmQsFgKBiMzc+HgsHQ5KTlDxn7+9FPPvlmY8MunjGtXPj662K5rD5+TMHSB5EIxTam7dO4u8ahzujQ0HtTUz6fb6CvL+D3jw4NfffixeVAQF6TGwsEhgYGaicngcHBwf7+zd3dM0OgwOAgHamXNqFU910bG3v7tB9psL//ciDATuNSE8RIAOA2CGkAwF3sYo+iTWeIIc0AadD07KhRv390aIgxdnh01NfX19fXN9jXF/D7N/78Z+cbGQsExk8HHdEQsoDfz5yNIrsRDFJ0wRgLBYOvajUalrZ3eDgWCDDGvn36NOD3V2s1Pn3fbgu8IcsHQTUe+EShQrFc1p890zY2xPxfnJsLz8xQX43D0VM8nsn+0z8lfvxjJ1/hKLYpfP01xTY3Z2aWfvjD6Nxc4wa3k1CHwjllZGTE7x8cGHhvauqPOzt0dHg5FDugTP5mcpLiHAojnz5/TmPPmLNuupm33mKMPa9WqVAxxkwD9hrr8/lGTke4varVutPpdG1sLBQMUmo5xEgA0AkIaQCgRXYTzUtra/LKmlUbukH7z6HhwcGA33/8+rXdqKEzzU1PDw0MiFNQxkZGfIyJCTt+/fpPu7vOt/k3k5P0TC3G2M7+PjVhGWPUAA34/YHBQSetWFOIwudXUMcIhSUjfv8fdnZoOWVyg4ju5swMdaSIQ8JYS81HbX1dW1+vbG+bBpJRmsMzM5EbN0LBYAut0vPEM6YU5r/6qlAqUW7cCAbj1HXTzKSUpkKdUDBIWUqZTFOheDcgPwXEPisTmhTEGHt3aqpaqz3Z2+OjyP58OrHKScQ+OjT0g4kJKtJjIyPDAwOvjo93Dw6o7DHGtvb2nuztOcwECrNrr18f1mqMsaH+fno+hLG/XzurJ7C9xDOCQ4wEAAwhDcCbwMlE87+s3JnY4/Lw8EB/v7jk8Oio2tLTq/jYJ3HA2LtTU+vffSeudnxyIl7fvn369OWrV83+BNl5+fLdqSn6TL9CDU0+KX/i0qUG7VSORxTs++0wcTo4Dd/iXVX8MDnpB6BJLPKQsAbjuxyi9GgbG5XtbW19XezkuTkzQw362Py8aexZC7T19fivfrX27Nk54xkRTbnJf/UVJZtPuYm///55ttlaqCOWKz4Li/dAOgl7xKlKuwcHuwcHtM2t3V3/6aPYxN5LJyfv/PQ0/+67U1NPnz+ncIjK+ZO9vRfV6sSlS843yE2Pjw8KJ/7UlSs0uJH+pH6q3z15sn96Q8E5fkmRL27OOYyRePAvOv9pBQBthJAGwEWammhuGXs4aVg3RuNbyPPTrg+aIcAYa2oMlUhszRNlZGR6fFy8VVyt1eiJT9y3T59eCQR4LNTU3pkaK2Ivx/p3310OBN6bmmKnL04Z9vuvXbkiZml7QxTTF3mQabqFb+zvnzkkjO+IwyFhrdG3t7WNjdLamt1AMoqX2vvT2vp69JNPGGOZeLxd8YyIT7n5dalES2jKTfz9988ZiXFtCXXEBFP5FDs/zywq8nwn3iJ/srfHn7omjuRstuTzsIqG4VGIQueUeEIx6TLV1Cn89++8wz+LvZ1EHDtHM39au/qZDv15unyZ1YWOIUYC6AqENABNO89Ec74Fu+aIc3//zjs70u3J4cFBU9dHUwPuTXjDhZOr4YNa7QeKwv/c2t3d3N3lo2VMu9/sLBdTAky/Tm36F9Uq9Za8OzV1fHIiDnvjzSmHGd5yiGJid/fd+ZAwdjpypmujZdRymRri8kCy8MxMKBiM3LgRnpnpXJOLxzPqxx93epeN/X318WN5yk08EunQDrY31DHtiyns4Sddg/LW4DEPvGEtXuhMA0r5meXkjBb7PE3jJ3kxe3tsbNjvN41ZFcOhpq6Zpo7cS8PD9NgDGsunjI7S1DLyR8MQZ/u0fHHmk53oz9amPMkQIwE4hJAGLjI3TDSnpyfxP2snJ/QMX/GOI6/w+HThpkZJiRyOozC9+OKbjQ3xTeqm/DHdZ21q2IlpBBdvP3GmIe/R+XknDSmHzQ4xIjpPiCKSh4TxFqTzIWG8/dGrNgftRbFcNg4OLAeS0Wthzj+QzKFuxjMmarlcLJdNU26WfvjD7iSjc6GO/CvMqtA2uMTJHYONhzKKk+vEvuVmR8HZ3VkQrxWhYNB0r0Tsyj7PnRS5dzcwOMg7k9+7du3y8PCLavV3T54w4VVC8q2r1uoO08uLGGO1168Hvz9kt16v//50+lwLHMZI/FiLECOBmyGkAVdww0RzXpOJPS10e4+PLOeuX73K51FwLb9dRK5jxHuZnDzhlbc4TRkoZ13L9zvl5JnSZqr55MZWF0IUMWfOORSK5yTvc+MNMrthLTxsk4eEdS0kOJO2vq4/e1ZaW9M2NsTGVoPXwnRN7osvlv/jP8ZHRrofz4i09XX18WM+5eZGMBidmzvnlJuWdSfUEcmDIZ1M35J7F50XfnGOX7tGwfFrkdhrYcoW09xCU++66d5WUzWL6VJpihP+z1/91ZYwzvbmzIw8qFgOjVqr2uh53M+r1ePT4Xn0lAhmdQft5atXcnXmnNyZL9+6YjYxknuukHABIKSB5rhhorkcABweHY2PjPCXcBNxGjehSa6MLuX9/ewcnTCmzgfi8EaXXZu7cUXbhRFcf/nfhtWMGD65P0Qx4ZlsGhLmZBKLaUhY29PWXnavheEDyRq/FqZrKJ65OTOT+/BDlzyQSt/epmFpNOVmfGSExzY9b351P9QR2c0BYw1jD7tBlayZM6gLo+DEpwvanRem6Y6NO7SbmtvT4BJtGlQmTojiv2sa8ywPOmit5hUf3sgYe16tviWdAoe1GoVG3M7+fsuzLhliJGgVQpoLyw0TzeULExNqkSd7e3zWB80rpWkY4srXr16V09ZyUCQPyrK8UJra98xxb7s8zq2NFV7jEVzyQGrnbQW74SJiEOUwgup+iCJqMKngzCFhPME9HxLWGj6QjNq4Yni2ODcXCgZpIJnbYjAez6gff+zChgg9TqC0vs6n3CxSbNOxKTct622oIzr/Y9zowzlvHHRzFJyTFJpqh8Yd6U3dsTJVDY170cXLmtxIkIdky/Vvaw2Dxbm554eHx8ITHegZkuJjKvn9R/FJkvThnCPAESO9IRDS9JhLJppbxh5yn0O1Vuvz+fiMC/Lt06dHr19fE2aGyOk8TyKdDMqynBnZ2pXINIJLDgLbOILLlOYGdU9T2hWiiDWlqQLoTohiIs8HcD4k7MxX3XuO3Wth+ECyll8L0zWZzz5LFQqujWdMCl9/XVpb41Nubs7MROfmujblpmXuCXW4tj/GrS2p7dUoOCca10qmCrepW36m23yt9d6bQiPLEeOm0KjlJoHYVnl+eDg8OPj29zusnj5/bnoy3rDf/yfDMG2n7TGS5Vhx+fYoc3ffvtchpHHKDRPNLQc7MZsXjfFzaWt3l4/fpSngllccOc3dH5TVrnpUlfpG2jVUmp01gss03aUt7eZOhyjnrG7bpS1Dwjr6XOPeooFkle1t/dkzsWl1c2YmFAzSQDIPxWmJTz/9ty+/9Eo8I6I3eKqPH/MpN/FIJHLjRk+m3LTMhaGOqHOPcWtv8lhPR8E50dTEoaZ6YBrfoZOf+9J4a3ITS26otOsZDHIThTJfHDkyPT5Od295qsZHRsZPD5DceXWeBp5lkwkxUgsuYEjjqonmIsuOTmY155sxtrW7a+oMsRwz1q5OYeZ4UJac2k6Mz2mqj56173UlTOrw6UQVLtYxFzhEMaE4U74v6/y5xu29KetydB2jgWSWr4Wh3PBoZUbxzJ0f/SgTj3srnhHRGzyL5TJdf+gNnpHr190w5aZlLg915KSyDj/Grb1JZcwVo+Accj5xqNleF/HOYONB1C0/3O8vSzr2DAa51WRqTjzZ23vnrbfEyUgvqlXqXGqcnpaTRN7YGKmXIY07J5ozmwPPbGIPflvUshtH7sNp76AsJ68NsRyU1YUiazq+bXzIJjvr/pDpLO30pAhxT8V7YM1WV94KUURue9W9F9kNJKNakwaSdfS1MF3D45nchx/2Oi3twd/gqT5+TAfuA+q3cd+Um5Z5KNQRdf8xbu1NNnPfKDgnGt+UPE/t3/LEoWbJDdTOPYNBbojavXdBbCu+OzV1eXjY8n79GxsjtSekSebz/HNPJpqLLFvwrPk81be3s198QZ/b2MnocFCWXCx6Xjdw9BoHdr4RXHI+nLPbuqNyX3xBA36aClHEWzh2gwrcc1gdoqPvtlfde5S+vZ3+7DPL18IoIyPeGkjmUPxf//XXpdJFimdMeGzDp9xk4vELXOabCnVi8/OJH/+4Rym1df7HuM1OTvZwvzo3Ci42P9/9ott44lDbH7Hd6WPX5WcwmJbwZlVmaclh8noYI/HUtnxQ2hPS+H7+c/7Z4URzZvNwCfe08Iz9/Yl//mf6LMfQDgdlXbwWCWOs8PXXS7/6FTtrBNeZryvxkOgnn3z++LGTEOXC9zDQ0e/5q+4vBmN/P7Sywl8LEwoGL3Dbl9DdAbv69SLhU25S//iP3ppm0xaWoU4oGFQ//rjXSWtF48e4rfz0px4q0s5HwXlrv9hZE4csx6svzs15okye8xkM9X//944mj2tLjNTyQbmAc2kAAAAAAODN0Xf2KgAAAAAAAG41IC9SVZUxpihKOBzmC3VdVxRFUZTGm9M0zTh9/ncoFAqFQrQkGo3yLdPydu2AE6qqptNp+vUG62Sz2UKhIC50uNekUCiUSqVYLEZZp+u6ruv0X+Fw2OFGOocyoVAoNE6JqqrFYpExlslkmtq+ruvpdHp5eVksNjLL8mAYRs8LiXOmQt6rxBuGoWmaaaFchtsll8tVKpVmS4WJ86PPGOOnj5sLQwucXI4ckvNzf39/dHTUK6eSV4gXc56f4kJ2WmNarkmoglhaWlIUhZbzKoOOlxvqSidMlwK78tz4iiHuIO04VbU899xTjcpVp8OGgVwYxCWmJlYnGIaRTqfpQyaTcZ57TpoBDo/gefeh13p+ENvCSfVtuV+e21mLXhpFUYrFYjqdFq/X2WxWbj/ZSSaT4p+xWIw3s/L5fPevSuFw+MymWDQaNaSXMTnf62QyaRhGKpXSdZ3qJCbkQzKZdJ57nfP555+fmfnRaDSTybSQ2lAopCiKnIcyy/LQ80LSLLGQ9yTxmqbRYaKU5PN5yzLcLolEoi3bcX70eQ53Lk7rCSeXI+dMWXflyhXPnUqeYFkaxYuAeLW3XJMqCLp3Ji7RNC2Xy9Fqnjh2pkuBXXlufMXQdT0vPFWI8oQJuaeqKmWFG6pRU9XZVMOAf8W0O3wHOyeXy8VisUwmk8vlmipLTpoBzo+g1/X2ILaFk+qbSgi1K3hpsVzoZha9NOFwOJvNZjKZQqFAR05VVYrCzwzcKYZTFIU368Ph8OLiYrFYjEaj0Wi0WCx2Ll9UVeXVQCqVog/8foMYX9LtMcMwYrFYsVjkfQu0v3RLg46ik73WNG1iYoIKDS861MSnfIhGo8lksjsBrqZp/ELDM4ExFg6Hb968Ka6p6zq/DDH7HbTcYIPvFgqFfD6/vLzMy4DIsjx0s5BY7pGu68lkMhKJ7OzsnHlDSy7k3Uy8mAy6bUkp4XdzxTKsKArvfqRDNjs7S0U0l8sVi8VUKpXP5/nKYs5EIpF4PM7PqYmJCf7T/Jxip4eefoUOei6XK5VKqVRKvsHs/Ojz/WKM0fJ4PN7ZDJXwnKRYXVEUuizIp4OcmYZhWJYo0+XIruDZXaBM5Kz7yU9+0pPS2F6apqXT6Uwm45I+CtPFnEojX6jrOt27ldeMx+NUbnVdpzMlmUxSfcrPxGQySeFNly+DzWp8KeDl03I1WSKRKBaLdHzp5rfpuspzr2vVqN3JaKo6nTeHLAuDZfHoBLooVSqVYrFIPYF2DSR5oRPOj2BHdzAWiyUSiWQyqes6VUDyHsnXZ03TnFRYrNcHkdnUBZbVkLzQ4clIxCYNzwrLha5Wl1QqlWw2W6/X79y5wxeurKwUi0V5ZUuLi4umPyuVysrKCm3H4UaaxX/C9FlOUqVS4buWTqf5mouLi6VSqV6vl0qldDpdd7zXxWLRcjXxR0150iE7OzsffPAB/ywewXq9ToeVu3Pnzs7ODn22yy67DVp+d2VlJZ/P37lzp1KpNEikZXnoTiGp2++RfPQbMx3c7iTeSUrkvRBXEJO3uLhIf/JDyQ9rqVTK5/PimVIsFuUynM/neclfWVmh78pnnym1Do8+Ne+KxSKVq2aypD2os7per1OBoR20O7/kzGxQohofMrsLlCU563pbGtsin8/TXvQ6IX9hWRrpoMv1XbFYzGazH3zwAR3Zer1eKpXu3LnDv7uysiLuHT+PXHvsGl8K+J9nXjFE2WyW9jqdTvPcoCylXOV5Iv9Qh9ids6aq03lzSC4M8g52jphOywaS81aTJYdHsHN4pUOZXLffI/n67LDCqvf0IFrWBZbVkLywqZORs2zN2jVxXciilyabzVJsp+u6qqqW99qbReFdR3uNdV2nO6P0Z4MROLquL50+lzAej4u9DXSbIRwOiz2qbdG1wUh81+TBvqbOR3GFBl00lhu0+242m43FYmdG85bloQuFhDXMovMc/e4k3omm9oKOHc+EVCqVTCb5/S0xr6LRKN8g3RKjO9PLy8u0cHl5me6sZ7PZVCrV4EedH326D7q0tNSrIbz0u7w8Nz6/TJnJHB8L02oNLlCW5KxzT2lsTadvfLbGsjTSqB522psnrs/vWzPGwuEwrUZjHyYmJsQayjAMXsbceezEMileClpbjcTj8XQ6HY1Gd3Z2xCqDziNd1xOJhKn50YVq1PKcPf+wW7EwNNjBzrFsIDlvNVlq4Qi2F690qBOGNdwj0/XZeYXFdf8gWtYFltWQvLCpkzGTyZgmjNgtdDnrJ55pmka9q3x4yflR0WnX1mTUP5g5xUcnW67Jj25bxkGaLn/yNjVN606fXSgUKpVK/E9xNpRMPNvt1rTboN13eT/vmUm1LA+dLiSsySxqShcSf05n1liFQiGXy2UymVQqlUgkaFod/RefJkhjZgqFQiaTWRJeWcCbYrOzs2e2PBwefTqX3TMlsXOFx/QrzV6gLLOua6XRMAzTKS8vsVvocINuYFcaqb0rxmDRaDSRSMRiMXmCTTweNwwjHo+LVUapVDKFSW67klheClpeja/MGMvlcpFIRP5faj6KW+haNdpecmEg8g52lGUDyXmryVKzR7Dt6Oky9BOUmKbagcxxhdWrg2hZF1hWQ/LCpk7GnZ0d8YsNFrZRJ2qE/rt374p/JxKJra2tiYmJUCj04MGDQqHg8/nC4fDs7Gw2m11dXVVVdW1tza6Rkcvl8vn8w4cPNzc3fT5fKBRSVfX+/fuBQCAcDgcCgUKhcPv2bScpa5aiKGtrazyR33zzzcLCAjsdu0xJopQrilKtVu/du7e6ujo/P7+5uRmLxcR00jTEhYWFSCTiZK8DgcD09PRHH31EoaC4wc3NTVVVy+VyKpUKBAKd2HG7TMjlcqlUanp62m7lra2tfD6vqiqFr7FYjEnZFY1GLTcof1fX9Vwut7CwEI/HE4mEruuRSETeZcvy0LVCYpdFYgJyuZyqqrdu3bI7XqZCrut61xJvomnavXv3KCWRSOThw4dyGVYUJRAI3L9/X1XV1dVVTdNo15LJJH1RVVV+Tb979265XFZV9be//e3PfvYzGpfMMcboanDv3j0q6uVyuVwuLywsUF7Nzs7evn27waXH+dHnp0+D867T0ul0uVy+detWoVCYnp4ul8u7u7sTExPy6SBnpl2JMp1fhmHIq01PT8sXKMsUylk3PT3d/dL4y1/+8he/+AWl3G6J3UKHG+wty9IoXuHpmH700Ud8IXVW84qDnhxDk4ljsdjCwsLa2hpdQguFwvLysukq1OUryZkoeaZLwfz8vKk837p1y3I1u81OTEzcvXv33r179KeYpYVCIRAITExMdK0adV4LOGwOyYXhs88+e/TokbiDSx17kSXtwubm5urqKl235QaS81aT3a+ceQQ7t4MkEAjE4/EHDx7QkbLcI8vKjjmosFivD6JlY9WyDSMvDIfDzk9GvpphGPyyY7mwjTpSIzgfo7azs+P+4XTNJrJSqTSeONHUBovFIh+p2UNtT7PdBl2yvy3wRGFuo0ql4nBmgpwzlUqFTwngLA89n4Z3sXWz8Jx5gXIJuYTIS+wWOtyg18nFxnNXIctLQcureZrnjh2xTLZH96UBh3vkrQrLVBc4PJTOT0bLRoLzlkNr2l4j+Or1ettiLk+hDkTqW3DVg2UAPEdV1WKxqGlae59Q/CbDBQoAoBO8VWGhLnDuzQ1pAAAAAADgArB+PAAAAAAAAIAnIKQBAADoCN/Pf55s9ysBAABAhpAGAAAAAAA8DCENAAAAAAB4GEIaAAAAAADwMIQ0AAAAAADgYQhpAAAAAADAwxDSAAAAAACAhyGkAQAAAAAAD0NIAwAAAAAAHoaQBgAAAAAAPAwhDQAAAAAAeBhCGgAAAAAA8DCENAAAAAAA4GEIaQAAAAAAwMMQ0gAAAAAAgIchpAEAAAAAAA9DSAMAAAAAAB6GkAYAAAAAADwMIQ0AAEBHLM7NzU5O9joVAAAXn69er/c6DQAAAAAAAC1CLw0AAAAAAHjYQK8TAAAA0BuqqjLGFEUJh8MNVtN1XVEURVHEL2az2UKh0K6UaJpmGEY0GuWpCoVChmHICyk99K1QKERLAADecOilAQCAN5SiKMViMZ1O8yDBUjab1TRNXBKNRg3DaG9iYrEYj5Hy+TxFUJYLk8kkT1gbwyoAAO9CSAMAAG+ocDhsGEYmkxEDg1wuF4/HNU1LJpOJROI3v/mNpmn5fD6ZTPJYgtCSRCJB4Y2qqvF4nDGm63oymczlcpYbtIyFwuHw4uJisVikbhnqFLJcGAqFFEWJRqPRaDSTyWSz2Q5mEACAR2DgGQAAvKF0XY9EIqFQqFKp8IWJRCKfz+fz+UwmYxiGoiiPHj2KxWI0AEy0tLQUDoc1TcvlcslkMhqNptNpxlgoFMpkMjz+kTdol55UKpVOpzOZzJkLaRxasVhcXl4+Xx4AAFwECGkAAOANlc1mVVXN5/O6rquqKgYtFEI0CD8YYzQDJxwO5/P5M3/LyQZpYoxpkJvlwmKxyE5jqjN/GgDgwkNIAwAAby4KFQzDSKfTcj9My1qeaZPJZBKJhCnykReaOm0AAN5w/Xfv3u11GgAAALotkUhsbW1NTEyEQqEHDx4UCgWfzxcOh5PJ5MOHDzc3N1VVnZ2dVRRldnY2m82urq6qqrq2thYOh1VVvX//fiAQoPU1TVtYWFAUJRAI3L9/X1XV1dVVTdNu3boVCATkDcqJETcYCAQKhcLt27cbLNzc3KSUdD/fAABcCK/aBAAAOINhGJqrPZEHAAAAS0lEQVSmOenGoYen4dnKAADdhJAGAAAAAAA8DA9xBgAAAAAAD0NIAwAAAAAAHoaQBgAAAAAAPAwhDQAAAAAAeBhCGgAAAAAA8LD/BThir4Hy+x2dAAAAAElFTkSuQmCC",
      "text/plain": [
       "Tree('S', [('At', 'IN'), ('eight', 'CD'), (\"o'clock\", 'NN'), ('on', 'IN'), ('Thursday', 'NNP'), ('morning', 'NN'), ('...', ':'), Tree('PERSON', [('Arthur', 'NNP')]), ('did', 'VBD'), (\"n't\", 'RB'), ('feel', 'VB'), ('very', 'RB'), ('good', 'JJ'), ('.', '.')])"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "entities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(S\n",
      "  At/IN\n",
      "  eight/CD\n",
      "  o'clock/NN\n",
      "  on/IN\n",
      "  Thursday/NNP\n",
      "  morning/NN\n",
      "  .../:\n",
      "  (PERSON Arthur/NNP)\n",
      "  did/VBD\n",
      "  n't/RB\n",
      "  feel/VB\n",
      "  very/RB\n",
      "  good/JJ\n",
      "  ./.)\n"
     ]
    }
   ],
   "source": [
    "print(entities)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Display a parse tree in NLTK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "ename": "LookupError",
     "evalue": "\n**********************************************************************\n  Resource \u001b[93mtreebank\u001b[0m not found.\n  Please use the NLTK Downloader to obtain the resource:\n\n  \u001b[31m>>> import nltk\n  >>> nltk.download('treebank')\n  \u001b[0m\n  For more information see: https://www.nltk.org/data.html\n\n  Attempted to load \u001b[93mcorpora/treebank/combined\u001b[0m\n\n  Searched in:\n    - '/home/muhammad/nltk_data'\n    - '/home/muhammad/anaconda3/nltk_data'\n    - '/home/muhammad/anaconda3/share/nltk_data'\n    - '/home/muhammad/anaconda3/lib/nltk_data'\n    - '/usr/share/nltk_data'\n    - '/usr/local/share/nltk_data'\n    - '/usr/lib/nltk_data'\n    - '/usr/local/lib/nltk_data'\n**********************************************************************\n",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mLookupError\u001b[0m                               Traceback (most recent call last)",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/corpus/util.py\u001b[0m in \u001b[0;36m__load\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m     82\u001b[0m                 \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 83\u001b[0;31m                     \u001b[0mroot\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnltk\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfind\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"{}/{}\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msubdir\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mzip_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     84\u001b[0m                 \u001b[0;32mexcept\u001b[0m \u001b[0mLookupError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/data.py\u001b[0m in \u001b[0;36mfind\u001b[0;34m(resource_name, paths)\u001b[0m\n\u001b[1;32m    584\u001b[0m     \u001b[0mresource_not_found\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"\\n%s\\n%s\\n%s\\n\"\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0msep\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmsg\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msep\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 585\u001b[0;31m     \u001b[0;32mraise\u001b[0m \u001b[0mLookupError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresource_not_found\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    586\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mLookupError\u001b[0m: \n**********************************************************************\n  Resource \u001b[93mtreebank\u001b[0m not found.\n  Please use the NLTK Downloader to obtain the resource:\n\n  \u001b[31m>>> import nltk\n  >>> nltk.download('treebank')\n  \u001b[0m\n  For more information see: https://www.nltk.org/data.html\n\n  Attempted to load \u001b[93mcorpora/treebank.zip/treebank/combined/\u001b[0m\n\n  Searched in:\n    - '/home/muhammad/nltk_data'\n    - '/home/muhammad/anaconda3/nltk_data'\n    - '/home/muhammad/anaconda3/share/nltk_data'\n    - '/home/muhammad/anaconda3/lib/nltk_data'\n    - '/usr/share/nltk_data'\n    - '/usr/local/share/nltk_data'\n    - '/usr/lib/nltk_data'\n    - '/usr/local/lib/nltk_data'\n**********************************************************************\n",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mLookupError\u001b[0m                               Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-65-b0734761784d>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mnltk\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcorpus\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mtreebank\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mt\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtreebank\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mparsed_sents\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'wsj_0001.mrg'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdraw\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/corpus/util.py\u001b[0m in \u001b[0;36m__getattr__\u001b[0;34m(self, attr)\u001b[0m\n\u001b[1;32m    118\u001b[0m             \u001b[0;32mraise\u001b[0m \u001b[0mAttributeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"LazyCorpusLoader object has no attribute '__bases__'\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    119\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 120\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__load\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    121\u001b[0m         \u001b[0;31m# This looks circular, but its not, since __load() changes our\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    122\u001b[0m         \u001b[0;31m# __class__ to something new:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/corpus/util.py\u001b[0m in \u001b[0;36m__load\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m     83\u001b[0m                     \u001b[0mroot\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnltk\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfind\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"{}/{}\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msubdir\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mzip_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     84\u001b[0m                 \u001b[0;32mexcept\u001b[0m \u001b[0mLookupError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 85\u001b[0;31m                     \u001b[0;32mraise\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     86\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     87\u001b[0m         \u001b[0;31m# Load the corpus.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/corpus/util.py\u001b[0m in \u001b[0;36m__load\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m     78\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     79\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 80\u001b[0;31m                 \u001b[0mroot\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnltk\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfind\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"{}/{}\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msubdir\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     81\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mLookupError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     82\u001b[0m                 \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/nltk/data.py\u001b[0m in \u001b[0;36mfind\u001b[0;34m(resource_name, paths)\u001b[0m\n\u001b[1;32m    583\u001b[0m     \u001b[0msep\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"*\"\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0;36m70\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    584\u001b[0m     \u001b[0mresource_not_found\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"\\n%s\\n%s\\n%s\\n\"\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0msep\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmsg\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msep\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 585\u001b[0;31m     \u001b[0;32mraise\u001b[0m \u001b[0mLookupError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresource_not_found\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    586\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    587\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mLookupError\u001b[0m: \n**********************************************************************\n  Resource \u001b[93mtreebank\u001b[0m not found.\n  Please use the NLTK Downloader to obtain the resource:\n\n  \u001b[31m>>> import nltk\n  >>> nltk.download('treebank')\n  \u001b[0m\n  For more information see: https://www.nltk.org/data.html\n\n  Attempted to load \u001b[93mcorpora/treebank/combined\u001b[0m\n\n  Searched in:\n    - '/home/muhammad/nltk_data'\n    - '/home/muhammad/anaconda3/nltk_data'\n    - '/home/muhammad/anaconda3/share/nltk_data'\n    - '/home/muhammad/anaconda3/lib/nltk_data'\n    - '/usr/share/nltk_data'\n    - '/usr/local/share/nltk_data'\n    - '/usr/lib/nltk_data'\n    - '/usr/local/lib/nltk_data'\n**********************************************************************\n"
     ]
    }
   ],
   "source": [
    "from nltk.corpus import treebank\n",
    "t = treebank.parsed_sents('wsj_0001.mrg')[0]\n",
    "t.draw()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package treebank to\n",
      "[nltk_data]     /home/muhammad/nltk_data...\n",
      "[nltk_data]   Package treebank is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# so we need to download treebank package\n",
    "nltk.download('treebank')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import treebank\n",
    "t = treebank.parsed_sents('wsj_0001.mrg')[0]\n",
    "t.draw()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. TextBlob (http://textblob.readthedocs.io/en/dev/index.html)\n",
    "It provides a simple API for diving deep into common NLP tasks, such as part-of-speech tagging, noun\n",
    "phrase extraction, sentiment analysis, classification, and much more.\n",
    "\n",
    "TextBlob stands on the giant shoulders of NLTK and pattern, and plays nicely with both."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/bin/bash: pip: command not found\n",
      "Collecting textblob\n",
      "  Using cached textblob-0.15.3-py2.py3-none-any.whl (636 kB)\n",
      "Collecting nltk>=3.1\n",
      "  Using cached nltk-3.6.2-py3-none-any.whl (1.5 MB)\n",
      "Collecting joblib\n",
      "  Using cached joblib-1.0.1-py3-none-any.whl (303 kB)\n",
      "Requirement already satisfied, skipping upgrade: click in /usr/lib/python3/dist-packages (from nltk>=3.1->textblob) (7.0)\n",
      "Collecting tqdm\n",
      "  Using cached tqdm-4.60.0-py2.py3-none-any.whl (75 kB)\n",
      "Collecting regex\n",
      "  Using cached regex-2021.4.4-cp38-cp38-manylinux2014_x86_64.whl (733 kB)\n",
      "Installing collected packages: joblib, tqdm, regex, nltk, textblob\n",
      "\u001b[33m  WARNING: The script tqdm is installed in '/home/muhammad/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\n",
      "\u001b[33m  WARNING: The script nltk is installed in '/home/muhammad/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\n",
      "Successfully installed joblib-1.0.1 nltk-3.6.2 regex-2021.4.4 textblob-0.15.3 tqdm-4.60.0\n"
     ]
    }
   ],
   "source": [
    "# install TextBlob\n",
    "!pip install -U textblob\n",
    "# or Install it with python3\n",
    "!python3 -m pip install -U textblob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### POS Tagging in TextBlob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('The', 'DT'),\n",
       " ('titular', 'JJ'),\n",
       " ('threat', 'NN'),\n",
       " ('of', 'IN'),\n",
       " ('The', 'DT'),\n",
       " ('Blob', 'NNP'),\n",
       " ('has', 'VBZ'),\n",
       " ('always', 'RB'),\n",
       " ('struck', 'VBN'),\n",
       " ('me', 'PRP'),\n",
       " ('as', 'IN'),\n",
       " ('the', 'DT'),\n",
       " ('ultimate', 'JJ'),\n",
       " ('movie', 'NN'),\n",
       " ('monster', 'NN'),\n",
       " ('an', 'DT'),\n",
       " ('insatiably', 'RB'),\n",
       " ('hungry', 'JJ'),\n",
       " ('amoeba-like', 'JJ'),\n",
       " ('mass', 'NN'),\n",
       " ('able', 'JJ'),\n",
       " ('to', 'TO'),\n",
       " ('penetrate', 'VB'),\n",
       " ('virtually', 'RB'),\n",
       " ('any', 'DT'),\n",
       " ('safeguard', 'NN'),\n",
       " ('capable', 'JJ'),\n",
       " ('of', 'IN'),\n",
       " ('as', 'IN'),\n",
       " ('a', 'DT'),\n",
       " ('doomed', 'JJ'),\n",
       " ('doctor', 'NN'),\n",
       " ('chillingly', 'RB'),\n",
       " ('describes', 'VBZ'),\n",
       " ('it', 'PRP'),\n",
       " ('assimilating', 'VBG'),\n",
       " ('flesh', 'NN'),\n",
       " ('on', 'IN'),\n",
       " ('contact', 'NN'),\n",
       " ('Snide', 'JJ'),\n",
       " ('comparisons', 'NNS'),\n",
       " ('to', 'TO'),\n",
       " ('gelatin', 'VB'),\n",
       " ('be', 'VB'),\n",
       " ('damned', 'VBN'),\n",
       " ('it', 'PRP'),\n",
       " (\"'s\", 'VBZ'),\n",
       " ('a', 'DT'),\n",
       " ('concept', 'NN'),\n",
       " ('with', 'IN'),\n",
       " ('the', 'DT'),\n",
       " ('most', 'RBS'),\n",
       " ('devastating', 'JJ'),\n",
       " ('of', 'IN'),\n",
       " ('potential', 'JJ'),\n",
       " ('consequences', 'NNS'),\n",
       " ('not', 'RB'),\n",
       " ('unlike', 'IN'),\n",
       " ('the', 'DT'),\n",
       " ('grey', 'NN'),\n",
       " ('goo', 'NN'),\n",
       " ('scenario', 'NN'),\n",
       " ('proposed', 'VBN'),\n",
       " ('by', 'IN'),\n",
       " ('technological', 'JJ'),\n",
       " ('theorists', 'NNS'),\n",
       " ('fearful', 'NN'),\n",
       " ('of', 'IN'),\n",
       " ('artificial', 'JJ'),\n",
       " ('intelligence', 'NN'),\n",
       " ('run', 'NN'),\n",
       " ('rampant', 'NN')]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from textblob import TextBlob\n",
    "\n",
    "text = '''\n",
    "The titular threat of The Blob has always struck me as the ultimate movie\n",
    "monster: an insatiably hungry, amoeba-like mass able to penetrate\n",
    "virtually any safeguard, capable of--as a doomed doctor chillingly\n",
    "describes it--\"assimilating flesh on contact.\n",
    "Snide comparisons to gelatin be damned, it's a concept with the most\n",
    "devastating of potential consequences, not unlike the grey goo scenario\n",
    "proposed by technological theorists fearful of\n",
    "artificial intelligence run rampant.\n",
    "'''\n",
    "\n",
    "blob = TextBlob(text)\n",
    "blob.tags"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Noun phrase extraction in TextBlob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "**********************************************************************\n",
      "  Resource \u001b[93mbrown\u001b[0m not found.\n",
      "  Please use the NLTK Downloader to obtain the resource:\n",
      "\n",
      "  \u001b[31m>>> import nltk\n",
      "  >>> nltk.download('brown')\n",
      "  \u001b[0m\n",
      "  For more information see: https://www.nltk.org/data.html\n",
      "\n",
      "  Attempted to load \u001b[93mcorpora/brown\u001b[0m\n",
      "\n",
      "  Searched in:\n",
      "    - '/home/muhammad/nltk_data'\n",
      "    - '/home/muhammad/anaconda3/nltk_data'\n",
      "    - '/home/muhammad/anaconda3/share/nltk_data'\n",
      "    - '/home/muhammad/anaconda3/lib/nltk_data'\n",
      "    - '/usr/share/nltk_data'\n",
      "    - '/usr/local/share/nltk_data'\n",
      "    - '/usr/lib/nltk_data'\n",
      "    - '/usr/local/lib/nltk_data'\n",
      "**********************************************************************\n",
      "\n"
     ]
    },
    {
     "ename": "MissingCorpusError",
     "evalue": "\nLooks like you are missing some required data for this feature.\n\nTo download the necessary data, simply run\n\n    python -m textblob.download_corpora\n\nor use the NLTK downloader to download the missing data: http://nltk.org/data.html\nIf this doesn't fix the problem, file an issue at https://github.com/sloria/TextBlob/issues.\n",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mLookupError\u001b[0m                               Traceback (most recent call last)",
      "\u001b[0;32m~/.local/lib/python3.8/site-packages/nltk/corpus/util.py\u001b[0m in \u001b[0;36m__load\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m     82\u001b[0m                 \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 83\u001b[0;31m                     \u001b[0mroot\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnltk\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfind\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"{}/{}\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msubdir\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mzip_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     84\u001b[0m                 \u001b[0;32mexcept\u001b[0m \u001b[0mLookupError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.local/lib/python3.8/site-packages/nltk/data.py\u001b[0m in \u001b[0;36mfind\u001b[0;34m(resource_name, paths)\u001b[0m\n\u001b[1;32m    582\u001b[0m     \u001b[0mresource_not_found\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"\\n%s\\n%s\\n%s\\n\"\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0msep\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmsg\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msep\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 583\u001b[0;31m     \u001b[0;32mraise\u001b[0m \u001b[0mLookupError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresource_not_found\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    584\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mLookupError\u001b[0m: \n**********************************************************************\n  Resource \u001b[93mbrown\u001b[0m not found.\n  Please use the NLTK Downloader to obtain the resource:\n\n  \u001b[31m>>> import nltk\n  >>> nltk.download('brown')\n  \u001b[0m\n  For more information see: https://www.nltk.org/data.html\n\n  Attempted to load \u001b[93mcorpora/brown.zip/brown/\u001b[0m\n\n  Searched in:\n    - '/home/muhammad/nltk_data'\n    - '/home/muhammad/anaconda3/nltk_data'\n    - '/home/muhammad/anaconda3/share/nltk_data'\n    - '/home/muhammad/anaconda3/lib/nltk_data'\n    - '/usr/share/nltk_data'\n    - '/usr/local/share/nltk_data'\n    - '/usr/lib/nltk_data'\n    - '/usr/local/lib/nltk_data'\n**********************************************************************\n",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mLookupError\u001b[0m                               Traceback (most recent call last)",
      "\u001b[0;32m~/.local/lib/python3.8/site-packages/textblob/decorators.py\u001b[0m in \u001b[0;36mdecorated\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m     34\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 35\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     36\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mLookupError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0merr\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.local/lib/python3.8/site-packages/textblob/en/np_extractors.py\u001b[0m in \u001b[0;36mtrain\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    108\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 109\u001b[0;31m         \u001b[0mtrain_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnltk\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcorpus\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbrown\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtagged_sents\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcategories\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'news'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    110\u001b[0m         regexp_tagger = nltk.RegexpTagger([\n",
      "\u001b[0;32m~/.local/lib/python3.8/site-packages/nltk/corpus/util.py\u001b[0m in \u001b[0;36m__getattr__\u001b[0;34m(self, attr)\u001b[0m\n\u001b[1;32m    119\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 120\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__load\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    121\u001b[0m         \u001b[0;31m# This looks circular, but its not, since __load() changes our\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.local/lib/python3.8/site-packages/nltk/corpus/util.py\u001b[0m in \u001b[0;36m__load\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m     84\u001b[0m                 \u001b[0;32mexcept\u001b[0m \u001b[0mLookupError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 85\u001b[0;31m                     \u001b[0;32mraise\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     86\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.local/lib/python3.8/site-packages/nltk/corpus/util.py\u001b[0m in \u001b[0;36m__load\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m     79\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 80\u001b[0;31m                 \u001b[0mroot\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnltk\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfind\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"{}/{}\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msubdir\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     81\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mLookupError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.local/lib/python3.8/site-packages/nltk/data.py\u001b[0m in \u001b[0;36mfind\u001b[0;34m(resource_name, paths)\u001b[0m\n\u001b[1;32m    582\u001b[0m     \u001b[0mresource_not_found\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"\\n%s\\n%s\\n%s\\n\"\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0msep\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmsg\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msep\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 583\u001b[0;31m     \u001b[0;32mraise\u001b[0m \u001b[0mLookupError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresource_not_found\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    584\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mLookupError\u001b[0m: \n**********************************************************************\n  Resource \u001b[93mbrown\u001b[0m not found.\n  Please use the NLTK Downloader to obtain the resource:\n\n  \u001b[31m>>> import nltk\n  >>> nltk.download('brown')\n  \u001b[0m\n  For more information see: https://www.nltk.org/data.html\n\n  Attempted to load \u001b[93mcorpora/brown\u001b[0m\n\n  Searched in:\n    - '/home/muhammad/nltk_data'\n    - '/home/muhammad/anaconda3/nltk_data'\n    - '/home/muhammad/anaconda3/share/nltk_data'\n    - '/home/muhammad/anaconda3/lib/nltk_data'\n    - '/usr/share/nltk_data'\n    - '/usr/local/share/nltk_data'\n    - '/usr/lib/nltk_data'\n    - '/usr/local/lib/nltk_data'\n**********************************************************************\n",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mMissingCorpusError\u001b[0m                        Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-7-5acd980a1142>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mblob\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnoun_phrases\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/.local/lib/python3.8/site-packages/textblob/decorators.py\u001b[0m in \u001b[0;36m__get__\u001b[0;34m(self, obj, cls)\u001b[0m\n\u001b[1;32m     22\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mobj\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     23\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 24\u001b[0;31m         \u001b[0mvalue\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mobj\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__dict__\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfunc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     25\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     26\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.local/lib/python3.8/site-packages/textblob/blob.py\u001b[0m in \u001b[0;36mnoun_phrases\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    466\u001b[0m         \u001b[0;34m\"\"\"Returns a list of noun phrases for this blob.\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    467\u001b[0m         return WordList([phrase.strip().lower()\n\u001b[0;32m--> 468\u001b[0;31m                         \u001b[0;32mfor\u001b[0m \u001b[0mphrase\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnp_extractor\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mextract\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mraw\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    469\u001b[0m                         if len(phrase) > 1])\n\u001b[1;32m    470\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.local/lib/python3.8/site-packages/textblob/en/np_extractors.py\u001b[0m in \u001b[0;36mextract\u001b[0;34m(self, sentence)\u001b[0m\n\u001b[1;32m    136\u001b[0m         \u001b[0;34m'''Return a list of noun phrases (strings) for body of text.'''\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    137\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_trained\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 138\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtrain\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    139\u001b[0m         \u001b[0mtokens\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_tokenize_sentence\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msentence\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    140\u001b[0m         \u001b[0mtagged\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtagger\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtag\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtokens\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.local/lib/python3.8/site-packages/textblob/decorators.py\u001b[0m in \u001b[0;36mdecorated\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m     36\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mLookupError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0merr\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     37\u001b[0m             \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0merr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 38\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0mMissingCorpusError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     39\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mdecorated\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mMissingCorpusError\u001b[0m: \nLooks like you are missing some required data for this feature.\n\nTo download the necessary data, simply run\n\n    python -m textblob.download_corpora\n\nor use the NLTK downloader to download the missing data: http://nltk.org/data.html\nIf this doesn't fix the problem, file an issue at https://github.com/sloria/TextBlob/issues.\n"
     ]
    }
   ],
   "source": [
    "blob.noun_phrases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package brown to /home/muhammad/nltk_data...\n",
      "[nltk_data]   Unzipping corpora/brown.zip.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# need to install brown package\n",
    "import nltk\n",
    "nltk.download('brown')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "WordList(['titular threat', 'blob', 'ultimate movie monster', 'amoeba-like mass', 'snide', 'potential consequences', 'grey goo scenario', 'technological theorists fearful', 'artificial intelligence run rampant'])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "blob.noun_phrases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "WordList(['titular threat', 'blob', 'ultimate movie monster'])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "blob.noun_phrases[0:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Sentence(\"\n",
       " The titular threat of The Blob has always struck me as the ultimate movie\n",
       " monster: an insatiably hungry, amoeba-like mass able to penetrate\n",
       " virtually any safeguard, capable of--as a doomed doctor chillingly\n",
       " describes it--\"assimilating flesh on contact.\"),\n",
       " Sentence(\"Snide comparisons to gelatin be damned, it's a concept with the most\n",
       " devastating of potential consequences, not unlike the grey goo scenario\n",
       " proposed by technological theorists fearful of\n",
       " artificial intelligence run rampant.\")]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "blob.sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\nThe titular threat of The Blob has always struck me as the ultimate movie\\nmonster: an insatiably hungry, amoeba-like mass able to penetrate\\nvirtually any safeguard, capable of--as a doomed doctor chillingly\\ndescribes it--\"assimilating flesh on contact.'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get first sentence\n",
    "blob.sentences[0].string"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sentiment analysis with TextBlob\n",
    "\n",
    "Polarity defines negativity or positivity in the sentence, whereas\n",
    "subjectivity implies whether the sentence discusses something vaguely or\n",
    "with complete surety."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Sentiment(polarity=0.06000000000000001, subjectivity=0.605)\n",
      "\n",
      " polarity: 0.06000000000000001\n",
      "\n",
      " subjectivity: 0.605\n",
      "\n",
      " Sentiment(polarity=-0.34166666666666673, subjectivity=0.7666666666666666)\n",
      "\n",
      " polarity: -0.34166666666666673\n",
      "\n",
      " subjectivity: 0.7666666666666666\n"
     ]
    }
   ],
   "source": [
    "for sentence in blob.sentences:\n",
    "    print(\"\\n\",sentence.sentiment)\n",
    "    print(\"\\n polarity:\",sentence.sentiment.polarity)\n",
    "    print(\"\\n subjectivity:\",sentence.sentiment.subjectivity)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Correct spelling errors with TextBlob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "sent = TextBlob(\"I thinkk tha model needs to be trainned more!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TextBlob(\"I think the model needs to be trained more!\")"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sent.correct()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Language translation with TextBlob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TextBlob(\"Do you want to learn French?\")"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sent_fr = TextBlob(u\"Voulez-vous apprendre le français?\")\n",
    "sent_fr.translate(from_lang='fr', to='en')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TextBlob(\"Do you want to learn Arabic?\")"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sent_ar = TextBlob(u\"هل تريد تعلم العربية?\")\n",
    "sent_ar.translate(from_lang='ar', to='en')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text Classification using TextBlob (https://textblob.readthedocs.io/en/dev/api_reference.html#module-textblob.classifiers)\n",
    "Text can be classified into various classes, such as positive and negative."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'pos'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from textblob import TextBlob\n",
    "from textblob.classifiers import NaiveBayesClassifier\n",
    "data = [\n",
    "('I love my country.', 'pos'),\n",
    "('This is an amazing place!', 'pos'),\n",
    "('I do not like the smell of this place.', 'neg'),\n",
    "('I do not like this restaurant', 'neg'),\n",
    "('I am tired of hearing your nonsense.', 'neg'),\n",
    "(\"I always aspire to be like him\", 'pos'),\n",
    "(\"It's a horrible performance.\", \"neg\"),\n",
    "(\"My boss is horrible.\", \"neg\")\n",
    "]\n",
    "model = NaiveBayesClassifier(data)\n",
    "model.classify(\"It's an awesome place!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The game is good.\n",
      "neg\n",
      "But the hangover is horrible.\n",
      "neg\n"
     ]
    }
   ],
   "source": [
    "blob = TextBlob(\"The game is good. But the hangover is horrible.\", classifier=model)\n",
    "for s in blob.sentences:\n",
    "    print(s)\n",
    "    print(s.classify())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. SpaCy (https://spacy.io/)\n",
    "provides very fast and accurate syntactic\n",
    "analysis (the fastest of any library released) and also offers named entity\n",
    "recognition and ready access to word vectors. It is written in Cython\n",
    "language and contains a wide variety of trained models on language\n",
    "vocabularies, syntaxes, word-to-vector transformations, and entities\n",
    "recognition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/bin/bash: pip: command not found\n",
      "Collecting spacy\n",
      "  Downloading spacy-3.0.6-cp38-cp38-manylinux2014_x86_64.whl (13.0 MB)\n",
      "\u001b[K     |████████████████████████████████| 13.0 MB 8.1 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting srsly<3.0.0,>=2.4.1\n",
      "  Downloading srsly-2.4.1-cp38-cp38-manylinux2014_x86_64.whl (458 kB)\n",
      "\u001b[K     |████████████████████████████████| 458 kB 8.6 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting murmurhash<1.1.0,>=0.28.0\n",
      "  Downloading murmurhash-1.0.5-cp38-cp38-manylinux2014_x86_64.whl (20 kB)\n",
      "Collecting cymem<2.1.0,>=2.0.2\n",
      "  Downloading cymem-2.0.5-cp38-cp38-manylinux2014_x86_64.whl (35 kB)\n",
      "Collecting pydantic<1.8.0,>=1.7.1\n",
      "  Downloading pydantic-1.7.3-cp38-cp38-manylinux2014_x86_64.whl (12.2 MB)\n",
      "\u001b[K     |████████████████████████████████| 12.2 MB 9.2 MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied, skipping upgrade: setuptools in /usr/lib/python3/dist-packages (from spacy) (45.2.0)\n",
      "Collecting preshed<3.1.0,>=3.0.2\n",
      "  Downloading preshed-3.0.5-cp38-cp38-manylinux2014_x86_64.whl (130 kB)\n",
      "\u001b[K     |████████████████████████████████| 130 kB 9.1 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting thinc<8.1.0,>=8.0.3\n",
      "  Downloading thinc-8.0.3-cp38-cp38-manylinux2014_x86_64.whl (1.1 MB)\n",
      "\u001b[K     |████████████████████████████████| 1.1 MB 8.0 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting spacy-legacy<3.1.0,>=3.0.4\n",
      "  Downloading spacy_legacy-3.0.4-py2.py3-none-any.whl (12 kB)\n",
      "Collecting wasabi<1.1.0,>=0.8.1\n",
      "  Downloading wasabi-0.8.2-py3-none-any.whl (23 kB)\n",
      "Collecting catalogue<2.1.0,>=2.0.3\n",
      "  Downloading catalogue-2.0.3-py3-none-any.whl (16 kB)\n",
      "Collecting pathy>=0.3.5\n",
      "  Downloading pathy-0.5.2-py3-none-any.whl (42 kB)\n",
      "\u001b[K     |████████████████████████████████| 42 kB 235 kB/s  eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied, skipping upgrade: requests<3.0.0,>=2.13.0 in /usr/lib/python3/dist-packages (from spacy) (2.22.0)\n",
      "Collecting jinja2\n",
      "  Downloading Jinja2-2.11.3-py2.py3-none-any.whl (125 kB)\n",
      "\u001b[K     |████████████████████████████████| 125 kB 8.7 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting numpy>=1.15.0\n",
      "  Downloading numpy-1.20.2-cp38-cp38-manylinux2010_x86_64.whl (15.4 MB)\n",
      "\u001b[K     |████████████████████████████████| 15.4 MB 18.6 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting typer<0.4.0,>=0.3.0\n",
      "  Downloading typer-0.3.2-py3-none-any.whl (21 kB)\n",
      "Requirement already satisfied, skipping upgrade: packaging>=20.0 in /usr/lib/python3/dist-packages (from spacy) (20.3)\n",
      "Requirement already satisfied, skipping upgrade: tqdm<5.0.0,>=4.38.0 in /home/muhammad/.local/lib/python3.8/site-packages (from spacy) (4.60.0)\n",
      "Collecting blis<0.8.0,>=0.4.0\n",
      "  Downloading blis-0.7.4-cp38-cp38-manylinux2014_x86_64.whl (9.8 MB)\n",
      "\u001b[K     |████████████████████████████████| 9.8 MB 8.1 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting smart-open<4.0.0,>=2.2.0\n",
      "  Downloading smart_open-3.0.0.tar.gz (113 kB)\n",
      "\u001b[K     |████████████████████████████████| 113 kB 8.7 MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied, skipping upgrade: MarkupSafe>=0.23 in /usr/lib/python3/dist-packages (from jinja2->spacy) (1.1.0)\n",
      "Collecting click<7.2.0,>=7.1.1\n",
      "  Using cached click-7.1.2-py2.py3-none-any.whl (82 kB)\n",
      "Building wheels for collected packages: smart-open\n",
      "  Building wheel for smart-open (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for smart-open: filename=smart_open-3.0.0-py3-none-any.whl size=107092 sha256=10cc323a0ab90370e140e462bb831ca84361083d8257bdd2c7ed36c0cb652b42\n",
      "  Stored in directory: /home/muhammad/.cache/pip/wheels/11/73/9a/f91ac1f1816436b16423617c5be5db048697ff152a9c4346f2\n",
      "Successfully built smart-open\n",
      "Installing collected packages: catalogue, srsly, murmurhash, cymem, pydantic, preshed, wasabi, numpy, blis, thinc, spacy-legacy, smart-open, click, typer, pathy, jinja2, spacy\n",
      "\u001b[33m  WARNING: The scripts f2py, f2py3 and f2py3.8 are installed in '/home/muhammad/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\n",
      "\u001b[33m  WARNING: The script pathy is installed in '/home/muhammad/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\n",
      "\u001b[33m  WARNING: The script spacy is installed in '/home/muhammad/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\n",
      "Successfully installed blis-0.7.4 catalogue-2.0.3 click-7.1.2 cymem-2.0.5 jinja2-2.11.3 murmurhash-1.0.5 numpy-1.20.2 pathy-0.5.2 preshed-3.0.5 pydantic-1.7.3 smart-open-3.0.0 spacy-3.0.6 spacy-legacy-3.0.4 srsly-2.4.1 thinc-8.0.3 typer-0.3.2 wasabi-0.8.2\n"
     ]
    }
   ],
   "source": [
    "# install Spacy\n",
    "!pip install -U spacy\n",
    "# or Install it with python3\n",
    "!python3 -m pip install -U spacy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "ename": "OSError",
     "evalue": "[E941] Can't find model 'en'. It looks like you're trying to load a model from a shortcut, which is obsolete as of spaCy v3.0. To load the model, use its full name instead:\n\nnlp = spacy.load(\"en_core_web_sm\")\n\nFor more details on the available models, see the models directory: https://spacy.io/models. If you want to create a blank model, use spacy.blank: nlp = spacy.blank(\"en\")",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mOSError\u001b[0m                                   Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-2-f405c02e94a9>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mspacy\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mnlp\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mspacy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"en\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/.local/lib/python3.8/site-packages/spacy/__init__.py\u001b[0m in \u001b[0;36mload\u001b[0;34m(name, vocab, disable, exclude, config)\u001b[0m\n\u001b[1;32m     48\u001b[0m     \u001b[0mRETURNS\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mLanguage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mThe\u001b[0m \u001b[0mloaded\u001b[0m \u001b[0mnlp\u001b[0m \u001b[0mobject\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     49\u001b[0m     \"\"\"\n\u001b[0;32m---> 50\u001b[0;31m     return util.load_model(\n\u001b[0m\u001b[1;32m     51\u001b[0m         \u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvocab\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mvocab\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdisable\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdisable\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mexclude\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mexclude\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mconfig\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mconfig\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     52\u001b[0m     )\n",
      "\u001b[0;32m~/.local/lib/python3.8/site-packages/spacy/util.py\u001b[0m in \u001b[0;36mload_model\u001b[0;34m(name, vocab, disable, exclude, config)\u001b[0m\n\u001b[1;32m    328\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mload_model_from_path\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    329\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mname\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mOLD_MODEL_SHORTCUTS\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 330\u001b[0;31m         \u001b[0;32mraise\u001b[0m \u001b[0mIOError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mErrors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mE941\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfull\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mOLD_MODEL_SHORTCUTS\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    331\u001b[0m     \u001b[0;32mraise\u001b[0m \u001b[0mIOError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mErrors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mE050\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    332\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mOSError\u001b[0m: [E941] Can't find model 'en'. It looks like you're trying to load a model from a shortcut, which is obsolete as of spaCy v3.0. To load the model, use its full name instead:\n\nnlp = spacy.load(\"en_core_web_sm\")\n\nFor more details on the available models, see the models directory: https://spacy.io/models. If you want to create a blank model, use spacy.blank: nlp = spacy.blank(\"en\")"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "nlp = spacy.load(\"en\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[38;5;3m⚠ As of spaCy v3.0, shortcuts like 'en' are deprecated. Please use the\n",
      "full pipeline package name 'en_core_web_sm' instead.\u001b[0m\n",
      "Requirement already satisfied: en-core-web-sm==3.0.0 from https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.0.0/en_core_web_sm-3.0.0-py3-none-any.whl#egg=en_core_web_sm==3.0.0 in /home/muhammad/.local/lib/python3.8/site-packages (3.0.0)\n",
      "Requirement already satisfied: spacy<3.1.0,>=3.0.0 in /home/muhammad/.local/lib/python3.8/site-packages (from en-core-web-sm==3.0.0) (3.0.6)\n",
      "Requirement already satisfied: typer<0.4.0,>=0.3.0 in /home/muhammad/.local/lib/python3.8/site-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (0.3.2)\n",
      "Requirement already satisfied: blis<0.8.0,>=0.4.0 in /home/muhammad/.local/lib/python3.8/site-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (0.7.4)\n",
      "Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (45.2.0)\n",
      "Requirement already satisfied: numpy>=1.15.0 in /home/muhammad/.local/lib/python3.8/site-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (1.20.2)\n",
      "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /home/muhammad/.local/lib/python3.8/site-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (1.0.5)\n",
      "Requirement already satisfied: catalogue<2.1.0,>=2.0.3 in /home/muhammad/.local/lib/python3.8/site-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (2.0.3)\n",
      "Requirement already satisfied: packaging>=20.0 in /usr/lib/python3/dist-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (20.3)\n",
      "Requirement already satisfied: srsly<3.0.0,>=2.4.1 in /home/muhammad/.local/lib/python3.8/site-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (2.4.1)\n",
      "Requirement already satisfied: wasabi<1.1.0,>=0.8.1 in /home/muhammad/.local/lib/python3.8/site-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (0.8.2)\n",
      "Requirement already satisfied: pydantic<1.8.0,>=1.7.1 in /home/muhammad/.local/lib/python3.8/site-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (1.7.3)\n",
      "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.4 in /home/muhammad/.local/lib/python3.8/site-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (3.0.4)\n",
      "Requirement already satisfied: pathy>=0.3.5 in /home/muhammad/.local/lib/python3.8/site-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (0.5.2)\n",
      "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /home/muhammad/.local/lib/python3.8/site-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (3.0.5)\n",
      "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /home/muhammad/.local/lib/python3.8/site-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (2.0.5)\n",
      "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/lib/python3/dist-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (2.22.0)\n",
      "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /home/muhammad/.local/lib/python3.8/site-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (4.60.0)\n",
      "Requirement already satisfied: jinja2 in /home/muhammad/.local/lib/python3.8/site-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (2.11.3)\n",
      "Requirement already satisfied: thinc<8.1.0,>=8.0.3 in /home/muhammad/.local/lib/python3.8/site-packages (from spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (8.0.3)\n",
      "Requirement already satisfied: click<7.2.0,>=7.1.1 in /home/muhammad/.local/lib/python3.8/site-packages (from typer<0.4.0,>=0.3.0->spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (7.1.2)\n",
      "Requirement already satisfied: smart-open<4.0.0,>=2.2.0 in /home/muhammad/.local/lib/python3.8/site-packages (from pathy>=0.3.5->spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (3.0.0)\n",
      "Requirement already satisfied: MarkupSafe>=0.23 in /usr/lib/python3/dist-packages (from jinja2->spacy<3.1.0,>=3.0.0->en-core-web-sm==3.0.0) (1.1.0)\n",
      "\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
      "You can now load the package via spacy.load('en_core_web_sm')\n"
     ]
    }
   ],
   "source": [
    "# To fix that error run the command\n",
    "!python3 -m spacy download en"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "nlp = spacy.load(\"en_core_web_sm\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "william_wikidef_str = \"\"\"William was the son of King William II and Anna Pavlovna of Russia.\n",
    "On the abdication of his grandfather William I in 1840, he became the Prince of Orange.\n",
    "On the death of his father in 1849, he succeeded as king of the\n",
    "Netherlands. William married his cousin Sophie of Württemberg\n",
    "in 1839 and they had three sons, William, Maurice, and\n",
    "Alexander, all of whom predeceased him. \"\"\"\n",
    "\n",
    "william = nlp(william_wikidef_str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(William,\n",
       " Anna Pavlovna,\n",
       " Russia,\n",
       " William I,\n",
       " 1840,\n",
       " the Prince of Orange,\n",
       " 1849,\n",
       " the\n",
       " Netherlands,\n",
       " William,\n",
       " Sophie of Württemberg,\n",
       " 1839,\n",
       " three,\n",
       " William,\n",
       " Maurice)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "william.ents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(William, 'PERSON', 380),\n",
       " (Anna Pavlovna, 'PERSON', 380),\n",
       " (Russia, 'GPE', 384),\n",
       " (William I, 'PERSON', 380),\n",
       " (1840, 'DATE', 391),\n",
       " (the Prince of Orange, 'ORG', 383),\n",
       " (1849, 'DATE', 391),\n",
       " (the\n",
       "  Netherlands,\n",
       "  'GPE',\n",
       "  384),\n",
       " (William, 'PERSON', 380),\n",
       " (Sophie of Württemberg, 'ORG', 383),\n",
       " (1839, 'DATE', 391),\n",
       " (three, 'CARDINAL', 397),\n",
       " (William, 'PERSON', 380),\n",
       " (Maurice, 'GPE', 384)]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[ (i, i.label_, i.label) for i in william.ents]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">\n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    William\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       " was the son of King \n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    William II\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       " and \n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Anna Pavlovna\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       " of \n",
       "<mark class=\"entity\" style=\"background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Russia\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">GPE</span>\n",
       "</mark>\n",
       ".</br>On the abdication of his grandfather \n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    William I\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       " in \n",
       "<mark class=\"entity\" style=\"background: #bfe1d9; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    1840\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">DATE</span>\n",
       "</mark>\n",
       ", he became \n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    the Prince of Orange\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       ".</br>On the death of his father in \n",
       "<mark class=\"entity\" style=\"background: #bfe1d9; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    1849\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">DATE</span>\n",
       "</mark>\n",
       ", he succeeded as king of \n",
       "<mark class=\"entity\" style=\"background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    the\n",
       "Netherlands\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">GPE</span>\n",
       "</mark>\n",
       ". \n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    William\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       " married his cousin \n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Sophie of Württemberg\n",
       "\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       "in \n",
       "<mark class=\"entity\" style=\"background: #bfe1d9; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    1839\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">DATE</span>\n",
       "</mark>\n",
       " and they had \n",
       "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    three\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">CARDINAL</span>\n",
       "</mark>\n",
       " sons, \n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    William\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       ", \n",
       "<mark class=\"entity\" style=\"background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Maurice\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">GPE</span>\n",
       "</mark>\n",
       ", and</br>Alexander, all of whom predeceased him. </div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# let's visualize the recognized entities\n",
    "import spacy\n",
    "from spacy import displacy\n",
    "# if you want to run a web server to serve the visualization\n",
    "# displacy.serve(william, style=\"ent\") \n",
    "# or if you are running jupyter notebook use render() instead\n",
    "displacy.render(william, style=\"ent\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Noun Phrase extraction with Spacy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " <class 'spacy.tokens.span.Span'>\n",
      "The book\n",
      "---\n",
      "nsubj\n",
      "---\n",
      "head: deals\n",
      "\n",
      " <class 'spacy.tokens.span.Span'>\n",
      "NLP\n",
      "---\n",
      "pobj\n",
      "---\n",
      "head: with\n"
     ]
    }
   ],
   "source": [
    "sent = nlp('The book deals with NLP')\n",
    "for noun_ in sent.noun_chunks:\n",
    "    print(\"\\n\",type(noun_))\n",
    "    print(noun_.text)\n",
    "    print('---')\n",
    "    print(noun_.root.dep_)\n",
    "    print('---')\n",
    "    print(\"head:\",noun_.root.head.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Gensim (https://pypi.python.org/pypi/gensim or https://radimrehurek.com/gensim/) \n",
    "It is used primarily for topic modeling and document similarity.\n",
    "\n",
    "Gensim offers LDA (latent dirichlet allocation—a generative statistical\n",
    "model that allows sets of observations to be explained by unobserved\n",
    "groups that explain why some parts of the data are similar).\n",
    "\n",
    "For example, if observations are words collected into documents, it posits that each document is a mixture of a small number of topics and that each word's presence is attributable to one of the document's topics. LDA is an example of a topic model and belongs to the machine learning toolbox and in wider sense to the artificial intelligence toolbox.\n",
    "\n",
    "#### Read more! [about LDA topic model](https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/bin/bash: pip: command not found\n",
      "Collecting gensim\n",
      "  Downloading gensim-4.0.1-cp38-cp38-manylinux1_x86_64.whl (23.9 MB)\n",
      "\u001b[K     |████████████████████████████████| 23.9 MB 9.3 MB/s eta 0:00:01    |██████▍                         | 4.8 MB 1.2 MB/s eta 0:00:16\n",
      "\u001b[?25hRequirement already satisfied, skipping upgrade: numpy>=1.11.3 in /home/muhammad/.local/lib/python3.8/site-packages (from gensim) (1.20.2)\n",
      "Collecting scipy>=0.18.1\n",
      "  Downloading scipy-1.6.2-cp38-cp38-manylinux1_x86_64.whl (27.2 MB)\n",
      "\u001b[K     |████████████████████████████████| 27.2 MB 8.8 MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied, skipping upgrade: smart-open>=1.8.1 in /home/muhammad/.local/lib/python3.8/site-packages (from gensim) (3.0.0)\n",
      "Requirement already satisfied, skipping upgrade: requests in /usr/lib/python3/dist-packages (from smart-open>=1.8.1->gensim) (2.22.0)\n",
      "Installing collected packages: scipy, gensim\n",
      "Successfully installed gensim-4.0.1 scipy-1.6.2\n"
     ]
    }
   ],
   "source": [
    "# install Gensim\n",
    "!pip install --upgrade gensim\n",
    "# or\n",
    "!python3 -m pip install -U gensim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['bitcoin', 'is', 'an', 'innovative', 'payment', 'network', 'and', 'a', 'new', 'kind', 'of', 'money.']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/muhammad/.local/lib/python3.8/site-packages/gensim/similarities/__init__.py:15: UserWarning: The gensim.similarities.levenshtein submodule is disabled, because the optional Levenshtein package <https://pypi.org/project/python-Levenshtein/> is unavailable. Install Levenhstein (e.g. `pip install python-Levenshtein`) to suppress this warning.\n",
      "  warnings.warn(msg)\n"
     ]
    }
   ],
   "source": [
    "from gensim.models import Word2Vec\n",
    "min_count = 0\n",
    "size = 50\n",
    "window = 2\n",
    "sentence= \"bitcoin is an innovative payment network and a new kind of money.\"\n",
    "tokens=sentence.split()\n",
    "print(tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting a word vector using Gensim."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Word2Vec(tokens, min_count=min_count, vector_size=size,window=window)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<gensim.models.word2vec.Word2Vec at 0x7f9eedc25520>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 1.56321377e-02, -1.90189127e-02, -4.17724892e-04,  6.93716388e-03,\n",
       "       -1.87265233e-03,  1.67726874e-02,  1.80236828e-02,  1.30769610e-02,\n",
       "       -1.42979168e-03,  1.54274460e-02, -1.70741510e-02,  6.41523115e-03,\n",
       "       -9.27207991e-03, -1.01839453e-02,  7.18633179e-03,  1.07350927e-02,\n",
       "        1.55413030e-02, -1.15356091e-02,  1.48623688e-02,  1.32507896e-02,\n",
       "       -7.42820464e-03, -1.74971707e-02,  1.08766230e-02,  1.30173545e-02,\n",
       "       -1.57562713e-03, -1.34256398e-02, -1.41616967e-02, -4.99728601e-03,\n",
       "        1.02840560e-02, -7.32584484e-03, -1.87345482e-02,  7.64697697e-03,\n",
       "        9.75919515e-03, -1.28610805e-02,  2.41353363e-03, -4.14449396e-03,\n",
       "        4.60930496e-05, -1.97694749e-02,  5.38536347e-03, -9.49801225e-03,\n",
       "        2.17481446e-03, -3.14943912e-03,  4.39422950e-03, -1.57589372e-02,\n",
       "       -5.43286046e-03,  5.33067901e-03,  1.06922844e-02, -4.78387717e-03,\n",
       "       -1.90264061e-02,  9.01062042e-03], dtype=float32)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# vector of 'a'\n",
    "model.wv['a']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Pattern (https://pypi.python.org/pypi/Pattern)\n",
    "Is useful for a variety of NLP tasks, such as part-of-speech taggers, n-gram searches, sentiment\n",
    "analysis, and WordNet and machine learning, such as vector space\n",
    "modeling, k-means clustering, Naive Bayes, K-NN, and SVM classifiers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Stanford CoreNLP (https://stanfordnlp.github.io/CoreNLP/)\n",
    "provides the base forms of words; their parts of speech; whether they are names of\n",
    "companies, people, etc.; normalizes dates, times, and numeric quantities;\n",
    "marks up the structure of sentences in terms of phrases and syntactic\n",
    "dependencies; indicates which noun phrases refer to the same entities;\n",
    "indicates sentiment; extracts particular or open-class relations between\n",
    "entity mentions; gets the quotes people said; etc.\n",
    "\n",
    "There is a live online demo of CoreNLP available at (https://corenlp.run/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}