Sentiment Classification - Mini Project 1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sentiment Classification & How To \"Frame Problems\" for a Neural Network\n",
    "\n",
    "by Andrew Trask\n",
    "\n",
    "- **Twitter**: @iamtrask\n",
    "- **Blog**: http://iamtrask.github.io"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What You Should Already Know\n",
    "\n",
    "- neural networks, forward and back-propagation\n",
    "- stochastic gradient descent\n",
    "- mean squared error\n",
    "- and train/test splits\n",
    "\n",
    "### Where to Get Help if You Need it\n",
    "- Re-watch previous Udacity Lectures\n",
    "- Leverage the recommended Course Reading Material - [Grokking Deep Learning](https://www.manning.com/books/grokking-deep-learning) (40% Off: **traskud17**)\n",
    "- Shoot me a tweet @iamtrask\n",
    "\n",
    "\n",
    "### Tutorial Outline:\n",
    "\n",
    "- Intro: The Importance of \"Framing a Problem\"\n",
    "\n",
    "\n",
    "- Curate a Dataset\n",
    "- Developing a \"Predictive Theory\"\n",
    "- **PROJECT 1**: Quick Theory Validation\n",
    "\n",
    "\n",
    "- Transforming Text to Numbers\n",
    "- **PROJECT 2**: Creating the Input/Output Data\n",
    "\n",
    "\n",
    "- Putting it all together in a Neural Network\n",
    "- **PROJECT 3**: Building our Neural Network\n",
    "\n",
    "\n",
    "- Understanding Neural Noise\n",
    "- **PROJECT 4**: Making Learning Faster by Reducing Noise\n",
    "\n",
    "\n",
    "- Analyzing Inefficiencies in our Network\n",
    "- **PROJECT 5**: Making our Network Train and Run Faster\n",
    "\n",
    "\n",
    "- Further Noise Reduction\n",
    "- **PROJECT 6**: Reducing Noise by Strategically Reducing the Vocabulary\n",
    "\n",
    "\n",
    "- Analysis: What's going on in the weights?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "56bb3cba-260c-4ebe-9ed6-b995b4c72aa3"
    }
   },
   "source": [
    "# Lesson: Curate a Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "nbpresent": {
     "id": "eba2b193-0419-431e-8db9-60f34dd3fe83"
    }
   },
   "outputs": [],
   "source": [
    "def pretty_print_review_and_label(i):\n",
    "    print(labels[i] + \"\\t:\\t\" + reviews[i][:80] + \"...\")\n",
    "\n",
    "g = open('reviews.txt','r') # What we know!\n",
    "reviews = list(map(lambda x:x[:-1],g.readlines()))\n",
    "g.close()\n",
    "\n",
    "g = open('labels.txt','r') # What we WANT to know!\n",
    "labels = list(map(lambda x:x[:-1].upper(),g.readlines()))\n",
    "g.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25000"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(reviews)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "nbpresent": {
     "id": "bb95574b-21a0-4213-ae50-34363cf4f87f"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'bromwell high is a cartoon comedy . it ran at the same time as some other programs about school life  such as  teachers  . my   years in the teaching profession lead me to believe that bromwell high  s satire is much closer to reality than is  teachers  . the scramble to survive financially  the insightful students who can see right through their pathetic teachers  pomp  the pettiness of the whole situation  all remind me of the schools i knew and their students . when i saw the episode in which a student repeatedly tried to burn down the school  i immediately recalled . . . . . . . . . at . . . . . . . . . . high . a classic line inspector i  m here to sack one of your teachers . student welcome to bromwell high . i expect that many adults of my age think that bromwell high is far fetched . what a pity that it isn  t   '"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "nbpresent": {
     "id": "e0408810-c424-4ed4-afb9-1735e9ddbd0a"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'POSITIVE'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lesson: Develop a Predictive Theory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "nbpresent": {
     "id": "e67a709f-234f-4493-bae6-4fb192141ee0"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "labels.txt \t : \t reviews.txt\n",
      "\n",
      "NEGATIVE\t:\tthis movie is terrible but it has some good effects .  ...\n",
      "POSITIVE\t:\tadrian pasdar is excellent is this film . he makes a fascinating woman .  ...\n",
      "NEGATIVE\t:\tcomment this movie is impossible . is terrible  very improbable  bad interpretat...\n",
      "POSITIVE\t:\texcellent episode movie ala pulp fiction .  days   suicides . it doesnt get more...\n",
      "NEGATIVE\t:\tif you haven  t seen this  it  s terrible . it is pure trash . i saw this about ...\n",
      "POSITIVE\t:\tthis schiffer guy is a real genius  the movie is of excellent quality and both e...\n"
     ]
    }
   ],
   "source": [
    "print(\"labels.txt \\t : \\t reviews.txt\\n\")\n",
    "pretty_print_review_and_label(2137)\n",
    "pretty_print_review_and_label(12816)\n",
    "pretty_print_review_and_label(6267)\n",
    "pretty_print_review_and_label(21934)\n",
    "pretty_print_review_and_label(5297)\n",
    "pretty_print_review_and_label(4998)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create lists of positive and negative word counts using Counter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "\n",
    "word_count_positive = Counter([word for i, review in enumerate(reviews) for word in review.split() if labels[i] == 'POSITIVE'])\n",
    "word_count_negative = Counter([word for i, review in enumerate(reviews) for word in review.split() if labels[i] == 'NEGATIVE'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### It is possible to use the Counter object to sum counts for the same word to get the total count per word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "19074  +  24965  =  44039\n"
     ]
    }
   ],
   "source": [
    "# FYI, total counts computed by summing counter object\n",
    "word_count_total = word_count_positive + word_count_negative\n",
    "print(word_count_positive['movie'], ' + ' , word_count_negative['movie'], ' = ', word_count_total['movie'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a ratio of positive to negative word counts for each word and show most positive words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('edie', 4.691347882229153),\n",
       " ('paulie', 4.077537443905737),\n",
       " ('felix', 3.1527360223636984),\n",
       " ('polanski', 2.823361047613264),\n",
       " ('matthau', 2.8067217286093),\n",
       " ('victoria', 2.6810215287143593),\n",
       " ('mildred', 2.602689685444458),\n",
       " ('gandhi', 2.538973871058355),\n",
       " ('flawless', 2.451005098112405),\n",
       " ('superbly', 2.260025478575354),\n",
       " ('perfection', 2.1594842493534876),\n",
       " ('astaire', 2.1400661634963885),\n",
       " ('captures', 2.038619547159711),\n",
       " ('voight', 2.0301704926731845),\n",
       " ('wonderfully', 2.0218960560333676),\n",
       " ('powell', 1.9783454248086054),\n",
       " ('brosnan', 1.954799096472701),\n",
       " ('lily', 1.920376847050295),\n",
       " ('bakshi', 1.9029851043384285),\n",
       " ('lincoln', 1.9014583864846288),\n",
       " ('refreshing', 1.8551812956657077),\n",
       " ('breathtaking', 1.8481124057793443),\n",
       " ('bourne', 1.847848935879256),\n",
       " ('lemmon', 1.8458266904984886),\n",
       " ('delightful', 1.800270158896129),\n",
       " ('flynn', 1.7996646487353336),\n",
       " ('andrews', 1.7764919970974358),\n",
       " ('homer', 1.7692866133761669),\n",
       " ('beautifully', 1.7626953362843152),\n",
       " ('soccer', 1.757857917552546),\n",
       " ('elvira', 1.7397031072721776),\n",
       " ('underrated', 1.7197859696031448),\n",
       " ('gripping', 1.716536047990647),\n",
       " ('superb', 1.7091514458968762),\n",
       " ('delight', 1.671473303353741),\n",
       " ('welles', 1.6677068205582648),\n",
       " ('sadness', 1.6635051337045657),\n",
       " ('sinatra', 1.638996714675839),\n",
       " ('touching', 1.6372174765413705),\n",
       " ('timeless', 1.6292405397304761)]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import math\n",
    "\n",
    "pos_neg_ratios = Counter()\n",
    "\n",
    "[pos_neg_ratios.update({term: math.log(word_count_positive[term] / (word_count_negative[term] + 1) + .000000000001) }) \n",
    " for term, count in word_count_total.most_common() if count > 100]  # Higher count reveals words \n",
    "\n",
    "pos_neg_ratios.most_common(40) # positive sentiments\n",
    "\n",
    "\n",
    "# for term, cnt in word_count_total.most_common():\n",
    "#     if cnt > 100:    # Adjusting this higher from 10 yields more sentiment-type words\n",
    "#         pos_neg_ratios[term] = word_count_positive[term] / (word_count_negative[term] + 1)        \n",
    "\n",
    "# for word, ratio in pos_neg_ratios.most_common():\n",
    "# #     if ratio > 1:\n",
    "#     pos_neg_ratios[word] = math.log(ratio + .000000000001) \n",
    "# #     else:\n",
    "# #         pos_neg_ratios[word] = -math.log(1/(ratio + .000000000001))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Show the words more likely to be negative"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('boll', -4.969813299432),\n",
       " ('uwe', -4.624972813182271),\n",
       " ('seagal', -3.6441435602342946),\n",
       " ('unwatchable', -3.258096537995482),\n",
       " ('stinker', -3.2088254889899495),\n",
       " ('mst', -2.9502698994581227),\n",
       " ('incoherent', -2.9368917735122),\n",
       " ('unfunny', -2.692239595060803),\n",
       " ('waste', -2.6193845640028264),\n",
       " ('blah', -2.5704288232130907),\n",
       " ('horrid', -2.484906649776),\n",
       " ('pointless', -2.4553061800000595),\n",
       " ('atrocious', -2.425908309014732),\n",
       " ('redeeming', -2.368239063204804),\n",
       " ('prom', -2.3608540011074215),\n",
       " ('drivel', -2.3470368555544248),\n",
       " ('lousy', -2.3075726344950347),\n",
       " ('worst', -2.2869878961705323),\n",
       " ('laughable', -2.2643638801642227),\n",
       " ('awful', -2.227194247018161),\n",
       " ('poorly', -2.220755074737199),\n",
       " ('wasting', -2.2046046846247753),\n",
       " ('remotely', -2.197224577327219),\n",
       " ('existent', -2.079441541671836),\n",
       " ('boredom', -1.995100393238732),\n",
       " ('miserably', -1.9924301646828728),\n",
       " ('sucks', -1.9870682215415267),\n",
       " ('uninspired', -1.983297681119667),\n",
       " ('lame', -1.9817674589389105),\n",
       " ('insult', -1.9783454248012362)]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(reversed(pos_neg_ratios.most_common()[-30:]))"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}