New Twitter.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# we are using Tweepy library for twitter sentiment analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Importing tweepy\n",
    "\n",
    "import tweepy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Variables that contains the credentials to access Twitter API\n",
    "ACCESS_TOKEN = '1168456107657854976-JOJpUbKFexmILeI7P5vndarldz4nWo'\n",
    "ACCESS_SECRET = 'tKte6BpxPxQUhUxJIH57OaGm9XW9SCnKRGMuKS5fCYPHF'\n",
    "CONSUMER_KEY = 'xg59Dpojes0xFOQyCc0xFPfeN'\n",
    "CONSUMER_SECRET = 'Y46PIy0VhtL2FyBdmMzFt0RBaHDKGTPHqYH0hgwi6MjKUgaRDY'\n",
    "\n",
    "\n",
    "# Setup access to API\n",
    "def connect_to_twitter_OAuth():\n",
    "    auth = tweepy.OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)\n",
    "    auth.set_access_token(ACCESS_TOKEN, ACCESS_SECRET)\n",
    "\n",
    "    api = tweepy.API(auth)\n",
    "    return api\n",
    "\n",
    "\n",
    "# Create API object\n",
    "api = connect_to_twitter_OAuth()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Not many of us get to live to see our own legacy play out in such a meaningful, remarkable way. John Lewis did:\n",
      "\n",
      "https://t.co/KbVfYt5CeQ\n",
      "Some thoughts on the Reverend C.T. Vivian, a pioneer who pulled America closer to our founding ideals and a friend… https://t.co/Pw8vNppa47\n",
      "Here’s a quick look at how our response stacks up with countries around the world:\n",
      "\n",
      "https://t.co/AEIlpub72e\n",
      "The latest data offers a tragic reminder that the virus doesn’t care about spin or ideology––and that the best thin… https://t.co/MFinvcuNu5\n",
      "Get registered and find out how you can vote by mail at https://t.co/JQJjdF7JxM.\n",
      "Voting by mail shouldn't be a partisan issue––especially during a pandemic. Everybody should be able to request an… https://t.co/GTfpiOKIkC\n",
      "It’s clear our communities are crying out for change—and what that change looks like is up to us. That’s why it’s i… https://t.co/nP4zjrNUUt\n",
      "Finally, here’s a look at how Germany reshaped police training after the horrors of World War II and how this appro… https://t.co/pyxJeIvRMc\n",
      "And here’s a counterbalance that argues for substantial reforms to policing and oversight but not fewer police.\n",
      "https://t.co/DwQoyobGRX\n",
      "Here’s a piece that outlines the need for a more aggressive overhaul of policing.\n",
      "https://t.co/yJlSb0MmyK\n",
      "As communities around the country try to reimagine how law enforcement can operate in a just, effective and non-dis… https://t.co/omdC67UnSY\n",
      "For too long, gerrymandering has contributed to stalled progress and warped our representative government. Redistri… https://t.co/UxR0zIcHQc\n",
      "Our founding promise has never been guaranteed. Each generation has been called to bring us a little closer to our… https://t.co/GVmt02uZXG\n",
      "This holiday weekend, let’s be safe and smart. It’s going to take all of us to beat this virus. So wear a mask. Was… https://t.co/0dVvsNXKpO\n",
      "Today I’m tipping my hat to all the giants in the Negro Leagues, from Satchel Paige to Toni Stone and so many other… https://t.co/iko4cg7avO\n",
      "RT @MichelleObama: Thinking back to that day reminds me of how much strength we can find in one another, in good times and in bad. We’ve se…\n",
      "RT @ObamaFoundation: In the afternoon of June 26, 2015, @BarackObama traveled to Charleston to eulogize Reverend Pinckney after a racist sh…\n",
      "Jim’s courage and persistence helped change America for the better. And so did every American who marched, organize… https://t.co/o0f9SogkU5\n",
      "Michelle and I want every kid in Chicago to grow up knowing even better opportunities than we had. We’re happy to h… https://t.co/7UFpMs1tYt\n",
      "RT @UniNoticias: 👉 En exclusiva el expresidente @BarackObama envió un mensaje especial a los hispanos sobre la importancia de salir a votar…\n"
     ]
    }
   ],
   "source": [
    "# tweets from a specific user\n",
    "bo_tweets = api.user_timeline('@BarackObama')\n",
    "for tweet in bo_tweets:\n",
    "    print(tweet.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Now importing some important Libraries \n",
    "\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np \n",
    "import nltk\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining a function to extract data from tweet object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_tweet_attributes(messy_tweet):\n",
    "    # creating an empty list\n",
    "    \n",
    "    tweet_list = []\n",
    "    # Now looping through messy tweets\n",
    "    for tweet in messy_tweet:\n",
    "        tweet_id = tweet.id # unique integer idetifier for tweet\n",
    "        text = tweet.text # get text from every tweet\n",
    "        favorite_count = tweet.favorite_count\n",
    "        retweet_count = tweet.retweet_count\n",
    "        created_at = tweet.created_at # utc time tweet created\n",
    "        source = tweet.source # utility used to post tweet\n",
    "        reply_to_status = tweet.in_reply_to_status_id # if reply int of orginal tweet id\n",
    "        reply_to_user = tweet.in_reply_to_screen_name # if reply original tweetes screenname\n",
    "        retweets = tweet.retweet_count # number of times this tweet retweeted\n",
    "        favorites = tweet.favorite_count # number of time this tweet liked\n",
    "        # append attributes to list\n",
    "        \n",
    "        tweet_list.append({'tweet_id':tweet_id, \n",
    "                          'text':text, \n",
    "                          'favorite_count':favorite_count,\n",
    "                          'retweet_count':retweet_count,\n",
    "                          'created_at':created_at, \n",
    "                          'source':source, \n",
    "                          'reply_to_status':reply_to_status, \n",
    "                          'reply_to_user':reply_to_user,\n",
    "                          'retweets':retweets,\n",
    "                          'favorites':favorites})\n",
    "        df = pd.DataFrame(tweet_list, columns=['tweet_id',\n",
    "                                           'text',\n",
    "                                           'favorite_count',\n",
    "                                           'retweet_count',\n",
    "                                           'created_at',\n",
    "                                           'source',\n",
    "                                           'reply_to_status',\n",
    "                                           'reply_to_user',\n",
    "                                           'retweets',\n",
    "                                           'favorites'])\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tweet_id</th>\n",
       "      <th>text</th>\n",
       "      <th>favorite_count</th>\n",
       "      <th>retweet_count</th>\n",
       "      <th>created_at</th>\n",
       "      <th>source</th>\n",
       "      <th>reply_to_status</th>\n",
       "      <th>reply_to_user</th>\n",
       "      <th>retweets</th>\n",
       "      <th>favorites</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1284358592083169280</td>\n",
       "      <td>Not many of us get to live to see our own lega...</td>\n",
       "      <td>454823</td>\n",
       "      <td>96294</td>\n",
       "      <td>2020-07-18 05:25:25</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>96294</td>\n",
       "      <td>454823</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1284140415965429766</td>\n",
       "      <td>Some thoughts on the Reverend C.T. Vivian, a p...</td>\n",
       "      <td>66851</td>\n",
       "      <td>10488</td>\n",
       "      <td>2020-07-17 14:58:27</td>\n",
       "      <td>Twitter for iPhone</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>10488</td>\n",
       "      <td>66851</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1283410552853757954</td>\n",
       "      <td>Here’s a quick look at how our response stacks...</td>\n",
       "      <td>25749</td>\n",
       "      <td>6178</td>\n",
       "      <td>2020-07-15 14:38:15</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>1.283411e+18</td>\n",
       "      <td>BarackObama</td>\n",
       "      <td>6178</td>\n",
       "      <td>25749</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1283410551532597254</td>\n",
       "      <td>The latest data offers a tragic reminder that ...</td>\n",
       "      <td>73873</td>\n",
       "      <td>17408</td>\n",
       "      <td>2020-07-15 14:38:14</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>17408</td>\n",
       "      <td>73873</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1282745078163939330</td>\n",
       "      <td>Get registered and find out how you can vote b...</td>\n",
       "      <td>22953</td>\n",
       "      <td>7104</td>\n",
       "      <td>2020-07-13 18:33:53</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>1.282745e+18</td>\n",
       "      <td>BarackObama</td>\n",
       "      <td>7104</td>\n",
       "      <td>22953</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1282745076909735937</td>\n",
       "      <td>Voting by mail shouldn't be a partisan issue––...</td>\n",
       "      <td>114431</td>\n",
       "      <td>24289</td>\n",
       "      <td>2020-07-13 18:33:53</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>24289</td>\n",
       "      <td>114431</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1281290986522644483</td>\n",
       "      <td>It’s clear our communities are crying out for ...</td>\n",
       "      <td>29142</td>\n",
       "      <td>3928</td>\n",
       "      <td>2020-07-09 18:15:51</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>1.281291e+18</td>\n",
       "      <td>BarackObama</td>\n",
       "      <td>3928</td>\n",
       "      <td>29142</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1281290985012682752</td>\n",
       "      <td>Finally, here’s a look at how Germany reshaped...</td>\n",
       "      <td>26216</td>\n",
       "      <td>4378</td>\n",
       "      <td>2020-07-09 18:15:50</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>1.281291e+18</td>\n",
       "      <td>BarackObama</td>\n",
       "      <td>4378</td>\n",
       "      <td>26216</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1281290983519518720</td>\n",
       "      <td>And here’s a counterbalance that argues for su...</td>\n",
       "      <td>5631</td>\n",
       "      <td>1151</td>\n",
       "      <td>2020-07-09 18:15:50</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>1.281291e+18</td>\n",
       "      <td>BarackObama</td>\n",
       "      <td>1151</td>\n",
       "      <td>5631</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1281290982198255620</td>\n",
       "      <td>Here’s a piece that outlines the need for a mo...</td>\n",
       "      <td>6988</td>\n",
       "      <td>1525</td>\n",
       "      <td>2020-07-09 18:15:50</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>1.281291e+18</td>\n",
       "      <td>BarackObama</td>\n",
       "      <td>1525</td>\n",
       "      <td>6988</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>1281290980264751111</td>\n",
       "      <td>As communities around the country try to reima...</td>\n",
       "      <td>51497</td>\n",
       "      <td>9510</td>\n",
       "      <td>2020-07-09 18:15:49</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>9510</td>\n",
       "      <td>51497</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>1280909148855635969</td>\n",
       "      <td>For too long, gerrymandering has contributed t...</td>\n",
       "      <td>42646</td>\n",
       "      <td>10431</td>\n",
       "      <td>2020-07-08 16:58:33</td>\n",
       "      <td>Twitter for iPhone</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>10431</td>\n",
       "      <td>42646</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>1279423478542852098</td>\n",
       "      <td>Our founding promise has never been guaranteed...</td>\n",
       "      <td>197971</td>\n",
       "      <td>42430</td>\n",
       "      <td>2020-07-04 14:35:02</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>42430</td>\n",
       "      <td>197971</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>1279082492193050625</td>\n",
       "      <td>This holiday weekend, let’s be safe and smart....</td>\n",
       "      <td>266377</td>\n",
       "      <td>50339</td>\n",
       "      <td>2020-07-03 16:00:04</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>50339</td>\n",
       "      <td>266377</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>1277602792715300864</td>\n",
       "      <td>Today I’m tipping my hat to all the giants in ...</td>\n",
       "      <td>250271</td>\n",
       "      <td>32260</td>\n",
       "      <td>2020-06-29 14:00:17</td>\n",
       "      <td>Twitter Media Studio</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>32260</td>\n",
       "      <td>250271</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>1276654351109500932</td>\n",
       "      <td>RT @MichelleObama: Thinking back to that day r...</td>\n",
       "      <td>0</td>\n",
       "      <td>14500</td>\n",
       "      <td>2020-06-26 23:11:31</td>\n",
       "      <td>Twitter for iPhone</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>14500</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>1276568419979141121</td>\n",
       "      <td>RT @ObamaFoundation: In the afternoon of June ...</td>\n",
       "      <td>0</td>\n",
       "      <td>5353</td>\n",
       "      <td>2020-06-26 17:30:03</td>\n",
       "      <td>Twitter for iPhone</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>5353</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>1276529890867793920</td>\n",
       "      <td>Jim’s courage and persistence helped change Am...</td>\n",
       "      <td>73278</td>\n",
       "      <td>11219</td>\n",
       "      <td>2020-06-26 14:56:57</td>\n",
       "      <td>Twitter for iPhone</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>11219</td>\n",
       "      <td>73278</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>1276240519182647297</td>\n",
       "      <td>Michelle and I want every kid in Chicago to gr...</td>\n",
       "      <td>98791</td>\n",
       "      <td>11203</td>\n",
       "      <td>2020-06-25 19:47:05</td>\n",
       "      <td>Twitter for iPhone</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>11203</td>\n",
       "      <td>98791</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>1275912896791433217</td>\n",
       "      <td>RT @UniNoticias: 👉 En exclusiva el expresident...</td>\n",
       "      <td>0</td>\n",
       "      <td>1381</td>\n",
       "      <td>2020-06-24 22:05:14</td>\n",
       "      <td>Twitter for iPhone</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>1381</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               tweet_id                                               text  \\\n",
       "0   1284358592083169280  Not many of us get to live to see our own lega...   \n",
       "1   1284140415965429766  Some thoughts on the Reverend C.T. Vivian, a p...   \n",
       "2   1283410552853757954  Here’s a quick look at how our response stacks...   \n",
       "3   1283410551532597254  The latest data offers a tragic reminder that ...   \n",
       "4   1282745078163939330  Get registered and find out how you can vote b...   \n",
       "5   1282745076909735937  Voting by mail shouldn't be a partisan issue––...   \n",
       "6   1281290986522644483  It’s clear our communities are crying out for ...   \n",
       "7   1281290985012682752  Finally, here’s a look at how Germany reshaped...   \n",
       "8   1281290983519518720  And here’s a counterbalance that argues for su...   \n",
       "9   1281290982198255620  Here’s a piece that outlines the need for a mo...   \n",
       "10  1281290980264751111  As communities around the country try to reima...   \n",
       "11  1280909148855635969  For too long, gerrymandering has contributed t...   \n",
       "12  1279423478542852098  Our founding promise has never been guaranteed...   \n",
       "13  1279082492193050625  This holiday weekend, let’s be safe and smart....   \n",
       "14  1277602792715300864  Today I’m tipping my hat to all the giants in ...   \n",
       "15  1276654351109500932  RT @MichelleObama: Thinking back to that day r...   \n",
       "16  1276568419979141121  RT @ObamaFoundation: In the afternoon of June ...   \n",
       "17  1276529890867793920  Jim’s courage and persistence helped change Am...   \n",
       "18  1276240519182647297  Michelle and I want every kid in Chicago to gr...   \n",
       "19  1275912896791433217  RT @UniNoticias: 👉 En exclusiva el expresident...   \n",
       "\n",
       "    favorite_count  retweet_count          created_at                source  \\\n",
       "0           454823          96294 2020-07-18 05:25:25       Twitter Web App   \n",
       "1            66851          10488 2020-07-17 14:58:27    Twitter for iPhone   \n",
       "2            25749           6178 2020-07-15 14:38:15       Twitter Web App   \n",
       "3            73873          17408 2020-07-15 14:38:14       Twitter Web App   \n",
       "4            22953           7104 2020-07-13 18:33:53       Twitter Web App   \n",
       "5           114431          24289 2020-07-13 18:33:53       Twitter Web App   \n",
       "6            29142           3928 2020-07-09 18:15:51       Twitter Web App   \n",
       "7            26216           4378 2020-07-09 18:15:50       Twitter Web App   \n",
       "8             5631           1151 2020-07-09 18:15:50       Twitter Web App   \n",
       "9             6988           1525 2020-07-09 18:15:50       Twitter Web App   \n",
       "10           51497           9510 2020-07-09 18:15:49       Twitter Web App   \n",
       "11           42646          10431 2020-07-08 16:58:33    Twitter for iPhone   \n",
       "12          197971          42430 2020-07-04 14:35:02       Twitter Web App   \n",
       "13          266377          50339 2020-07-03 16:00:04       Twitter Web App   \n",
       "14          250271          32260 2020-06-29 14:00:17  Twitter Media Studio   \n",
       "15               0          14500 2020-06-26 23:11:31    Twitter for iPhone   \n",
       "16               0           5353 2020-06-26 17:30:03    Twitter for iPhone   \n",
       "17           73278          11219 2020-06-26 14:56:57    Twitter for iPhone   \n",
       "18           98791          11203 2020-06-25 19:47:05    Twitter for iPhone   \n",
       "19               0           1381 2020-06-24 22:05:14    Twitter for iPhone   \n",
       "\n",
       "    reply_to_status reply_to_user  retweets  favorites  \n",
       "0               NaN          None     96294     454823  \n",
       "1               NaN          None     10488      66851  \n",
       "2      1.283411e+18   BarackObama      6178      25749  \n",
       "3               NaN          None     17408      73873  \n",
       "4      1.282745e+18   BarackObama      7104      22953  \n",
       "5               NaN          None     24289     114431  \n",
       "6      1.281291e+18   BarackObama      3928      29142  \n",
       "7      1.281291e+18   BarackObama      4378      26216  \n",
       "8      1.281291e+18   BarackObama      1151       5631  \n",
       "9      1.281291e+18   BarackObama      1525       6988  \n",
       "10              NaN          None      9510      51497  \n",
       "11              NaN          None     10431      42646  \n",
       "12              NaN          None     42430     197971  \n",
       "13              NaN          None     50339     266377  \n",
       "14              NaN          None     32260     250271  \n",
       "15              NaN          None     14500          0  \n",
       "16              NaN          None      5353          0  \n",
       "17              NaN          None     11219      73278  \n",
       "18              NaN          None     11203      98791  \n",
       "19              NaN          None      1381          0  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = extract_tweet_attributes(bo_tweets)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tweet_id</th>\n",
       "      <th>text</th>\n",
       "      <th>favorite_count</th>\n",
       "      <th>retweet_count</th>\n",
       "      <th>created_at</th>\n",
       "      <th>source</th>\n",
       "      <th>reply_to_status</th>\n",
       "      <th>reply_to_user</th>\n",
       "      <th>retweets</th>\n",
       "      <th>favorites</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1284358592083169280</td>\n",
       "      <td>Not many of us get to live to see our own lega...</td>\n",
       "      <td>454823</td>\n",
       "      <td>96294</td>\n",
       "      <td>2020-07-18 05:25:25</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>96294</td>\n",
       "      <td>454823</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1284140415965429766</td>\n",
       "      <td>Some thoughts on the Reverend C.T. Vivian, a p...</td>\n",
       "      <td>66851</td>\n",
       "      <td>10488</td>\n",
       "      <td>2020-07-17 14:58:27</td>\n",
       "      <td>Twitter for iPhone</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>10488</td>\n",
       "      <td>66851</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1283410552853757954</td>\n",
       "      <td>Here’s a quick look at how our response stacks...</td>\n",
       "      <td>25749</td>\n",
       "      <td>6178</td>\n",
       "      <td>2020-07-15 14:38:15</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>1.283411e+18</td>\n",
       "      <td>BarackObama</td>\n",
       "      <td>6178</td>\n",
       "      <td>25749</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1283410551532597254</td>\n",
       "      <td>The latest data offers a tragic reminder that ...</td>\n",
       "      <td>73873</td>\n",
       "      <td>17408</td>\n",
       "      <td>2020-07-15 14:38:14</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>17408</td>\n",
       "      <td>73873</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1282745078163939330</td>\n",
       "      <td>Get registered and find out how you can vote b...</td>\n",
       "      <td>22953</td>\n",
       "      <td>7104</td>\n",
       "      <td>2020-07-13 18:33:53</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>1.282745e+18</td>\n",
       "      <td>BarackObama</td>\n",
       "      <td>7104</td>\n",
       "      <td>22953</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              tweet_id                                               text  \\\n",
       "0  1284358592083169280  Not many of us get to live to see our own lega...   \n",
       "1  1284140415965429766  Some thoughts on the Reverend C.T. Vivian, a p...   \n",
       "2  1283410552853757954  Here’s a quick look at how our response stacks...   \n",
       "3  1283410551532597254  The latest data offers a tragic reminder that ...   \n",
       "4  1282745078163939330  Get registered and find out how you can vote b...   \n",
       "\n",
       "   favorite_count  retweet_count          created_at              source  \\\n",
       "0          454823          96294 2020-07-18 05:25:25     Twitter Web App   \n",
       "1           66851          10488 2020-07-17 14:58:27  Twitter for iPhone   \n",
       "2           25749           6178 2020-07-15 14:38:15     Twitter Web App   \n",
       "3           73873          17408 2020-07-15 14:38:14     Twitter Web App   \n",
       "4           22953           7104 2020-07-13 18:33:53     Twitter Web App   \n",
       "\n",
       "   reply_to_status reply_to_user  retweets  favorites  \n",
       "0              NaN          None     96294     454823  \n",
       "1              NaN          None     10488      66851  \n",
       "2     1.283411e+18   BarackObama      6178      25749  \n",
       "3              NaN          None     17408      73873  \n",
       "4     1.282745e+18   BarackObama      7104      22953  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     Not many of us get to live to see our own lega...\n",
       "1     Some thoughts on the Reverend C.T. Vivian, a p...\n",
       "2     Here’s a quick look at how our response stacks...\n",
       "3     The latest data offers a tragic reminder that ...\n",
       "4     Get registered and find out how you can vote b...\n",
       "5     Voting by mail shouldn't be a partisan issue––...\n",
       "6     It’s clear our communities are crying out for ...\n",
       "7     Finally, here’s a look at how Germany reshaped...\n",
       "8     And here’s a counterbalance that argues for su...\n",
       "9     Here’s a piece that outlines the need for a mo...\n",
       "10    As communities around the country try to reima...\n",
       "11    For too long, gerrymandering has contributed t...\n",
       "12    Our founding promise has never been guaranteed...\n",
       "13    This holiday weekend, let’s be safe and smart....\n",
       "14    Today I’m tipping my hat to all the giants in ...\n",
       "15    RT @MichelleObama: Thinking back to that day r...\n",
       "16    RT @ObamaFoundation: In the afternoon of June ...\n",
       "17    Jim’s courage and persistence helped change Am...\n",
       "18    Michelle and I want every kid in Chicago to gr...\n",
       "19    RT @UniNoticias: 👉 En exclusiva el expresident...\n",
       "Name: text, dtype: object"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['text']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# removing user handles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tweet_id</th>\n",
       "      <th>text</th>\n",
       "      <th>favorite_count</th>\n",
       "      <th>retweet_count</th>\n",
       "      <th>created_at</th>\n",
       "      <th>source</th>\n",
       "      <th>reply_to_status</th>\n",
       "      <th>reply_to_user</th>\n",
       "      <th>retweets</th>\n",
       "      <th>favorites</th>\n",
       "      <th>text_without_handle</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1284358592083169280</td>\n",
       "      <td>Not many of us get to live to see our own lega...</td>\n",
       "      <td>454823</td>\n",
       "      <td>96294</td>\n",
       "      <td>2020-07-18 05:25:25</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>96294</td>\n",
       "      <td>454823</td>\n",
       "      <td>Not many of us get to live to see our own lega...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1284140415965429766</td>\n",
       "      <td>Some thoughts on the Reverend C.T. Vivian, a p...</td>\n",
       "      <td>66851</td>\n",
       "      <td>10488</td>\n",
       "      <td>2020-07-17 14:58:27</td>\n",
       "      <td>Twitter for iPhone</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>10488</td>\n",
       "      <td>66851</td>\n",
       "      <td>Some thoughts on the Reverend C.T. Vivian, a p...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1283410552853757954</td>\n",
       "      <td>Here’s a quick look at how our response stacks...</td>\n",
       "      <td>25749</td>\n",
       "      <td>6178</td>\n",
       "      <td>2020-07-15 14:38:15</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>1.283411e+18</td>\n",
       "      <td>BarackObama</td>\n",
       "      <td>6178</td>\n",
       "      <td>25749</td>\n",
       "      <td>Here’s a quick look at how our response stacks...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1283410551532597254</td>\n",
       "      <td>The latest data offers a tragic reminder that ...</td>\n",
       "      <td>73873</td>\n",
       "      <td>17408</td>\n",
       "      <td>2020-07-15 14:38:14</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>17408</td>\n",
       "      <td>73873</td>\n",
       "      <td>The latest data offers a tragic reminder that ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1282745078163939330</td>\n",
       "      <td>Get registered and find out how you can vote b...</td>\n",
       "      <td>22953</td>\n",
       "      <td>7104</td>\n",
       "      <td>2020-07-13 18:33:53</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>1.282745e+18</td>\n",
       "      <td>BarackObama</td>\n",
       "      <td>7104</td>\n",
       "      <td>22953</td>\n",
       "      <td>Get registered and find out how you can vote b...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1282745076909735937</td>\n",
       "      <td>Voting by mail shouldn't be a partisan issue––...</td>\n",
       "      <td>114431</td>\n",
       "      <td>24289</td>\n",
       "      <td>2020-07-13 18:33:53</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>24289</td>\n",
       "      <td>114431</td>\n",
       "      <td>Voting by mail shouldn't be a partisan issue––...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1281290986522644483</td>\n",
       "      <td>It’s clear our communities are crying out for ...</td>\n",
       "      <td>29142</td>\n",
       "      <td>3928</td>\n",
       "      <td>2020-07-09 18:15:51</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>1.281291e+18</td>\n",
       "      <td>BarackObama</td>\n",
       "      <td>3928</td>\n",
       "      <td>29142</td>\n",
       "      <td>It’s clear our communities are crying out for ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1281290985012682752</td>\n",
       "      <td>Finally, here’s a look at how Germany reshaped...</td>\n",
       "      <td>26216</td>\n",
       "      <td>4378</td>\n",
       "      <td>2020-07-09 18:15:50</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>1.281291e+18</td>\n",
       "      <td>BarackObama</td>\n",
       "      <td>4378</td>\n",
       "      <td>26216</td>\n",
       "      <td>Finally, here’s a look at how Germany reshaped...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1281290983519518720</td>\n",
       "      <td>And here’s a counterbalance that argues for su...</td>\n",
       "      <td>5631</td>\n",
       "      <td>1151</td>\n",
       "      <td>2020-07-09 18:15:50</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>1.281291e+18</td>\n",
       "      <td>BarackObama</td>\n",
       "      <td>1151</td>\n",
       "      <td>5631</td>\n",
       "      <td>And here’s a counterbalance that argues for su...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1281290982198255620</td>\n",
       "      <td>Here’s a piece that outlines the need for a mo...</td>\n",
       "      <td>6988</td>\n",
       "      <td>1525</td>\n",
       "      <td>2020-07-09 18:15:50</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>1.281291e+18</td>\n",
       "      <td>BarackObama</td>\n",
       "      <td>1525</td>\n",
       "      <td>6988</td>\n",
       "      <td>Here’s a piece that outlines the need for a mo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>1281290980264751111</td>\n",
       "      <td>As communities around the country try to reima...</td>\n",
       "      <td>51497</td>\n",
       "      <td>9510</td>\n",
       "      <td>2020-07-09 18:15:49</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>9510</td>\n",
       "      <td>51497</td>\n",
       "      <td>As communities around the country try to reima...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>1280909148855635969</td>\n",
       "      <td>For too long, gerrymandering has contributed t...</td>\n",
       "      <td>42646</td>\n",
       "      <td>10431</td>\n",
       "      <td>2020-07-08 16:58:33</td>\n",
       "      <td>Twitter for iPhone</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>10431</td>\n",
       "      <td>42646</td>\n",
       "      <td>For too long, gerrymandering has contributed t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>1279423478542852098</td>\n",
       "      <td>Our founding promise has never been guaranteed...</td>\n",
       "      <td>197971</td>\n",
       "      <td>42430</td>\n",
       "      <td>2020-07-04 14:35:02</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>42430</td>\n",
       "      <td>197971</td>\n",
       "      <td>Our founding promise has never been guaranteed...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>1279082492193050625</td>\n",
       "      <td>This holiday weekend, let’s be safe and smart....</td>\n",
       "      <td>266377</td>\n",
       "      <td>50339</td>\n",
       "      <td>2020-07-03 16:00:04</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>50339</td>\n",
       "      <td>266377</td>\n",
       "      <td>This holiday weekend, let’s be safe and smart....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>1277602792715300864</td>\n",
       "      <td>Today I’m tipping my hat to all the giants in ...</td>\n",
       "      <td>250271</td>\n",
       "      <td>32260</td>\n",
       "      <td>2020-06-29 14:00:17</td>\n",
       "      <td>Twitter Media Studio</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>32260</td>\n",
       "      <td>250271</td>\n",
       "      <td>Today I’m tipping my hat to all the giants in ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>1276654351109500932</td>\n",
       "      <td>RT @MichelleObama: Thinking back to that day r...</td>\n",
       "      <td>0</td>\n",
       "      <td>14500</td>\n",
       "      <td>2020-06-26 23:11:31</td>\n",
       "      <td>Twitter for iPhone</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>14500</td>\n",
       "      <td>0</td>\n",
       "      <td>: Thinking back to that day reminds me of how...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>1276568419979141121</td>\n",
       "      <td>RT @ObamaFoundation: In the afternoon of June ...</td>\n",
       "      <td>0</td>\n",
       "      <td>5353</td>\n",
       "      <td>2020-06-26 17:30:03</td>\n",
       "      <td>Twitter for iPhone</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>5353</td>\n",
       "      <td>0</td>\n",
       "      <td>: In the afternoon of June 26, 2015,  travele...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>1276529890867793920</td>\n",
       "      <td>Jim’s courage and persistence helped change Am...</td>\n",
       "      <td>73278</td>\n",
       "      <td>11219</td>\n",
       "      <td>2020-06-26 14:56:57</td>\n",
       "      <td>Twitter for iPhone</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>11219</td>\n",
       "      <td>73278</td>\n",
       "      <td>Jim’s courage and persistence helped change Am...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>1276240519182647297</td>\n",
       "      <td>Michelle and I want every kid in Chicago to gr...</td>\n",
       "      <td>98791</td>\n",
       "      <td>11203</td>\n",
       "      <td>2020-06-25 19:47:05</td>\n",
       "      <td>Twitter for iPhone</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>11203</td>\n",
       "      <td>98791</td>\n",
       "      <td>Michelle and I want every kid in Chicago to gr...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>1275912896791433217</td>\n",
       "      <td>RT @UniNoticias: 👉 En exclusiva el expresident...</td>\n",
       "      <td>0</td>\n",
       "      <td>1381</td>\n",
       "      <td>2020-06-24 22:05:14</td>\n",
       "      <td>Twitter for iPhone</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>1381</td>\n",
       "      <td>0</td>\n",
       "      <td>: 👉 En exclusiva el expresidente  envió un me...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               tweet_id                                               text  \\\n",
       "0   1284358592083169280  Not many of us get to live to see our own lega...   \n",
       "1   1284140415965429766  Some thoughts on the Reverend C.T. Vivian, a p...   \n",
       "2   1283410552853757954  Here’s a quick look at how our response stacks...   \n",
       "3   1283410551532597254  The latest data offers a tragic reminder that ...   \n",
       "4   1282745078163939330  Get registered and find out how you can vote b...   \n",
       "5   1282745076909735937  Voting by mail shouldn't be a partisan issue––...   \n",
       "6   1281290986522644483  It’s clear our communities are crying out for ...   \n",
       "7   1281290985012682752  Finally, here’s a look at how Germany reshaped...   \n",
       "8   1281290983519518720  And here’s a counterbalance that argues for su...   \n",
       "9   1281290982198255620  Here’s a piece that outlines the need for a mo...   \n",
       "10  1281290980264751111  As communities around the country try to reima...   \n",
       "11  1280909148855635969  For too long, gerrymandering has contributed t...   \n",
       "12  1279423478542852098  Our founding promise has never been guaranteed...   \n",
       "13  1279082492193050625  This holiday weekend, let’s be safe and smart....   \n",
       "14  1277602792715300864  Today I’m tipping my hat to all the giants in ...   \n",
       "15  1276654351109500932  RT @MichelleObama: Thinking back to that day r...   \n",
       "16  1276568419979141121  RT @ObamaFoundation: In the afternoon of June ...   \n",
       "17  1276529890867793920  Jim’s courage and persistence helped change Am...   \n",
       "18  1276240519182647297  Michelle and I want every kid in Chicago to gr...   \n",
       "19  1275912896791433217  RT @UniNoticias: 👉 En exclusiva el expresident...   \n",
       "\n",
       "    favorite_count  retweet_count          created_at                source  \\\n",
       "0           454823          96294 2020-07-18 05:25:25       Twitter Web App   \n",
       "1            66851          10488 2020-07-17 14:58:27    Twitter for iPhone   \n",
       "2            25749           6178 2020-07-15 14:38:15       Twitter Web App   \n",
       "3            73873          17408 2020-07-15 14:38:14       Twitter Web App   \n",
       "4            22953           7104 2020-07-13 18:33:53       Twitter Web App   \n",
       "5           114431          24289 2020-07-13 18:33:53       Twitter Web App   \n",
       "6            29142           3928 2020-07-09 18:15:51       Twitter Web App   \n",
       "7            26216           4378 2020-07-09 18:15:50       Twitter Web App   \n",
       "8             5631           1151 2020-07-09 18:15:50       Twitter Web App   \n",
       "9             6988           1525 2020-07-09 18:15:50       Twitter Web App   \n",
       "10           51497           9510 2020-07-09 18:15:49       Twitter Web App   \n",
       "11           42646          10431 2020-07-08 16:58:33    Twitter for iPhone   \n",
       "12          197971          42430 2020-07-04 14:35:02       Twitter Web App   \n",
       "13          266377          50339 2020-07-03 16:00:04       Twitter Web App   \n",
       "14          250271          32260 2020-06-29 14:00:17  Twitter Media Studio   \n",
       "15               0          14500 2020-06-26 23:11:31    Twitter for iPhone   \n",
       "16               0           5353 2020-06-26 17:30:03    Twitter for iPhone   \n",
       "17           73278          11219 2020-06-26 14:56:57    Twitter for iPhone   \n",
       "18           98791          11203 2020-06-25 19:47:05    Twitter for iPhone   \n",
       "19               0           1381 2020-06-24 22:05:14    Twitter for iPhone   \n",
       "\n",
       "    reply_to_status reply_to_user  retweets  favorites  \\\n",
       "0               NaN          None     96294     454823   \n",
       "1               NaN          None     10488      66851   \n",
       "2      1.283411e+18   BarackObama      6178      25749   \n",
       "3               NaN          None     17408      73873   \n",
       "4      1.282745e+18   BarackObama      7104      22953   \n",
       "5               NaN          None     24289     114431   \n",
       "6      1.281291e+18   BarackObama      3928      29142   \n",
       "7      1.281291e+18   BarackObama      4378      26216   \n",
       "8      1.281291e+18   BarackObama      1151       5631   \n",
       "9      1.281291e+18   BarackObama      1525       6988   \n",
       "10              NaN          None      9510      51497   \n",
       "11              NaN          None     10431      42646   \n",
       "12              NaN          None     42430     197971   \n",
       "13              NaN          None     50339     266377   \n",
       "14              NaN          None     32260     250271   \n",
       "15              NaN          None     14500          0   \n",
       "16              NaN          None      5353          0   \n",
       "17              NaN          None     11219      73278   \n",
       "18              NaN          None     11203      98791   \n",
       "19              NaN          None      1381          0   \n",
       "\n",
       "                                  text_without_handle  \n",
       "0   Not many of us get to live to see our own lega...  \n",
       "1   Some thoughts on the Reverend C.T. Vivian, a p...  \n",
       "2   Here’s a quick look at how our response stacks...  \n",
       "3   The latest data offers a tragic reminder that ...  \n",
       "4   Get registered and find out how you can vote b...  \n",
       "5   Voting by mail shouldn't be a partisan issue––...  \n",
       "6   It’s clear our communities are crying out for ...  \n",
       "7   Finally, here’s a look at how Germany reshaped...  \n",
       "8   And here’s a counterbalance that argues for su...  \n",
       "9   Here’s a piece that outlines the need for a mo...  \n",
       "10  As communities around the country try to reima...  \n",
       "11  For too long, gerrymandering has contributed t...  \n",
       "12  Our founding promise has never been guaranteed...  \n",
       "13  This holiday weekend, let’s be safe and smart....  \n",
       "14  Today I’m tipping my hat to all the giants in ...  \n",
       "15   : Thinking back to that day reminds me of how...  \n",
       "16   : In the afternoon of June 26, 2015,  travele...  \n",
       "17  Jim’s courage and persistence helped change Am...  \n",
       "18  Michelle and I want every kid in Chicago to gr...  \n",
       "19   : 👉 En exclusiva el expresidente  envió un me...  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['text_without_handle'] = df['text'].replace(to_replace = '(@[\\w]+)',value ='',regex = True)\n",
    "df['text_without_handle'] = df['text_without_handle'].replace(to_replace =('RT'), value ='',regex = True)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# removing any kind of links from tweet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tweet_id</th>\n",
       "      <th>text</th>\n",
       "      <th>favorite_count</th>\n",
       "      <th>retweet_count</th>\n",
       "      <th>created_at</th>\n",
       "      <th>source</th>\n",
       "      <th>reply_to_status</th>\n",
       "      <th>reply_to_user</th>\n",
       "      <th>retweets</th>\n",
       "      <th>favorites</th>\n",
       "      <th>text_without_handle</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1284358592083169280</td>\n",
       "      <td>Not many of us get to live to see our own lega...</td>\n",
       "      <td>454823</td>\n",
       "      <td>96294</td>\n",
       "      <td>2020-07-18 05:25:25</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>96294</td>\n",
       "      <td>454823</td>\n",
       "      <td>Not many of us get to live to see our own lega...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1284140415965429766</td>\n",
       "      <td>Some thoughts on the Reverend C.T. Vivian, a p...</td>\n",
       "      <td>66851</td>\n",
       "      <td>10488</td>\n",
       "      <td>2020-07-17 14:58:27</td>\n",
       "      <td>Twitter for iPhone</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>10488</td>\n",
       "      <td>66851</td>\n",
       "      <td>Some thoughts on the Reverend C.T. Vivian, a p...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1283410552853757954</td>\n",
       "      <td>Here’s a quick look at how our response stacks...</td>\n",
       "      <td>25749</td>\n",
       "      <td>6178</td>\n",
       "      <td>2020-07-15 14:38:15</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>1.283411e+18</td>\n",
       "      <td>BarackObama</td>\n",
       "      <td>6178</td>\n",
       "      <td>25749</td>\n",
       "      <td>Here’s a quick look at how our response stacks...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1283410551532597254</td>\n",
       "      <td>The latest data offers a tragic reminder that ...</td>\n",
       "      <td>73873</td>\n",
       "      <td>17408</td>\n",
       "      <td>2020-07-15 14:38:14</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>17408</td>\n",
       "      <td>73873</td>\n",
       "      <td>The latest data offers a tragic reminder that ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1282745078163939330</td>\n",
       "      <td>Get registered and find out how you can vote b...</td>\n",
       "      <td>22953</td>\n",
       "      <td>7104</td>\n",
       "      <td>2020-07-13 18:33:53</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>1.282745e+18</td>\n",
       "      <td>BarackObama</td>\n",
       "      <td>7104</td>\n",
       "      <td>22953</td>\n",
       "      <td>Get registered and find out how you can vote b...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1282745076909735937</td>\n",
       "      <td>Voting by mail shouldn't be a partisan issue––...</td>\n",
       "      <td>114431</td>\n",
       "      <td>24289</td>\n",
       "      <td>2020-07-13 18:33:53</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>24289</td>\n",
       "      <td>114431</td>\n",
       "      <td>Voting by mail shouldn't be a partisan issue––...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1281290986522644483</td>\n",
       "      <td>It’s clear our communities are crying out for ...</td>\n",
       "      <td>29142</td>\n",
       "      <td>3928</td>\n",
       "      <td>2020-07-09 18:15:51</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>1.281291e+18</td>\n",
       "      <td>BarackObama</td>\n",
       "      <td>3928</td>\n",
       "      <td>29142</td>\n",
       "      <td>It’s clear our communities are crying out for ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1281290985012682752</td>\n",
       "      <td>Finally, here’s a look at how Germany reshaped...</td>\n",
       "      <td>26216</td>\n",
       "      <td>4378</td>\n",
       "      <td>2020-07-09 18:15:50</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>1.281291e+18</td>\n",
       "      <td>BarackObama</td>\n",
       "      <td>4378</td>\n",
       "      <td>26216</td>\n",
       "      <td>Finally, here’s a look at how Germany reshaped...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1281290983519518720</td>\n",
       "      <td>And here’s a counterbalance that argues for su...</td>\n",
       "      <td>5631</td>\n",
       "      <td>1151</td>\n",
       "      <td>2020-07-09 18:15:50</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>1.281291e+18</td>\n",
       "      <td>BarackObama</td>\n",
       "      <td>1151</td>\n",
       "      <td>5631</td>\n",
       "      <td>And here’s a counterbalance that argues for su...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1281290982198255620</td>\n",
       "      <td>Here’s a piece that outlines the need for a mo...</td>\n",
       "      <td>6988</td>\n",
       "      <td>1525</td>\n",
       "      <td>2020-07-09 18:15:50</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>1.281291e+18</td>\n",
       "      <td>BarackObama</td>\n",
       "      <td>1525</td>\n",
       "      <td>6988</td>\n",
       "      <td>Here’s a piece that outlines the need for a mo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>1281290980264751111</td>\n",
       "      <td>As communities around the country try to reima...</td>\n",
       "      <td>51497</td>\n",
       "      <td>9510</td>\n",
       "      <td>2020-07-09 18:15:49</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>9510</td>\n",
       "      <td>51497</td>\n",
       "      <td>As communities around the country try to reima...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>1280909148855635969</td>\n",
       "      <td>For too long, gerrymandering has contributed t...</td>\n",
       "      <td>42646</td>\n",
       "      <td>10431</td>\n",
       "      <td>2020-07-08 16:58:33</td>\n",
       "      <td>Twitter for iPhone</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>10431</td>\n",
       "      <td>42646</td>\n",
       "      <td>For too long, gerrymandering has contributed t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>1279423478542852098</td>\n",
       "      <td>Our founding promise has never been guaranteed...</td>\n",
       "      <td>197971</td>\n",
       "      <td>42430</td>\n",
       "      <td>2020-07-04 14:35:02</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>42430</td>\n",
       "      <td>197971</td>\n",
       "      <td>Our founding promise has never been guaranteed...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>1279082492193050625</td>\n",
       "      <td>This holiday weekend, let’s be safe and smart....</td>\n",
       "      <td>266377</td>\n",
       "      <td>50339</td>\n",
       "      <td>2020-07-03 16:00:04</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>50339</td>\n",
       "      <td>266377</td>\n",
       "      <td>This holiday weekend, let’s be safe and smart....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>1277602792715300864</td>\n",
       "      <td>Today I’m tipping my hat to all the giants in ...</td>\n",
       "      <td>250271</td>\n",
       "      <td>32260</td>\n",
       "      <td>2020-06-29 14:00:17</td>\n",
       "      <td>Twitter Media Studio</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>32260</td>\n",
       "      <td>250271</td>\n",
       "      <td>Today I’m tipping my hat to all the giants in ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>1276654351109500932</td>\n",
       "      <td>RT @MichelleObama: Thinking back to that day r...</td>\n",
       "      <td>0</td>\n",
       "      <td>14500</td>\n",
       "      <td>2020-06-26 23:11:31</td>\n",
       "      <td>Twitter for iPhone</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>14500</td>\n",
       "      <td>0</td>\n",
       "      <td>: Thinking back to that day reminds me of how...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>1276568419979141121</td>\n",
       "      <td>RT @ObamaFoundation: In the afternoon of June ...</td>\n",
       "      <td>0</td>\n",
       "      <td>5353</td>\n",
       "      <td>2020-06-26 17:30:03</td>\n",
       "      <td>Twitter for iPhone</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>5353</td>\n",
       "      <td>0</td>\n",
       "      <td>: In the afternoon of June 26, 2015,  travele...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>1276529890867793920</td>\n",
       "      <td>Jim’s courage and persistence helped change Am...</td>\n",
       "      <td>73278</td>\n",
       "      <td>11219</td>\n",
       "      <td>2020-06-26 14:56:57</td>\n",
       "      <td>Twitter for iPhone</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>11219</td>\n",
       "      <td>73278</td>\n",
       "      <td>Jim’s courage and persistence helped change Am...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>1276240519182647297</td>\n",
       "      <td>Michelle and I want every kid in Chicago to gr...</td>\n",
       "      <td>98791</td>\n",
       "      <td>11203</td>\n",
       "      <td>2020-06-25 19:47:05</td>\n",
       "      <td>Twitter for iPhone</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>11203</td>\n",
       "      <td>98791</td>\n",
       "      <td>Michelle and I want every kid in Chicago to gr...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>1275912896791433217</td>\n",
       "      <td>RT @UniNoticias: 👉 En exclusiva el expresident...</td>\n",
       "      <td>0</td>\n",
       "      <td>1381</td>\n",
       "      <td>2020-06-24 22:05:14</td>\n",
       "      <td>Twitter for iPhone</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>1381</td>\n",
       "      <td>0</td>\n",
       "      <td>: 👉 En exclusiva el expresidente  envió un me...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               tweet_id                                               text  \\\n",
       "0   1284358592083169280  Not many of us get to live to see our own lega...   \n",
       "1   1284140415965429766  Some thoughts on the Reverend C.T. Vivian, a p...   \n",
       "2   1283410552853757954  Here’s a quick look at how our response stacks...   \n",
       "3   1283410551532597254  The latest data offers a tragic reminder that ...   \n",
       "4   1282745078163939330  Get registered and find out how you can vote b...   \n",
       "5   1282745076909735937  Voting by mail shouldn't be a partisan issue––...   \n",
       "6   1281290986522644483  It’s clear our communities are crying out for ...   \n",
       "7   1281290985012682752  Finally, here’s a look at how Germany reshaped...   \n",
       "8   1281290983519518720  And here’s a counterbalance that argues for su...   \n",
       "9   1281290982198255620  Here’s a piece that outlines the need for a mo...   \n",
       "10  1281290980264751111  As communities around the country try to reima...   \n",
       "11  1280909148855635969  For too long, gerrymandering has contributed t...   \n",
       "12  1279423478542852098  Our founding promise has never been guaranteed...   \n",
       "13  1279082492193050625  This holiday weekend, let’s be safe and smart....   \n",
       "14  1277602792715300864  Today I’m tipping my hat to all the giants in ...   \n",
       "15  1276654351109500932  RT @MichelleObama: Thinking back to that day r...   \n",
       "16  1276568419979141121  RT @ObamaFoundation: In the afternoon of June ...   \n",
       "17  1276529890867793920  Jim’s courage and persistence helped change Am...   \n",
       "18  1276240519182647297  Michelle and I want every kid in Chicago to gr...   \n",
       "19  1275912896791433217  RT @UniNoticias: 👉 En exclusiva el expresident...   \n",
       "\n",
       "    favorite_count  retweet_count          created_at                source  \\\n",
       "0           454823          96294 2020-07-18 05:25:25       Twitter Web App   \n",
       "1            66851          10488 2020-07-17 14:58:27    Twitter for iPhone   \n",
       "2            25749           6178 2020-07-15 14:38:15       Twitter Web App   \n",
       "3            73873          17408 2020-07-15 14:38:14       Twitter Web App   \n",
       "4            22953           7104 2020-07-13 18:33:53       Twitter Web App   \n",
       "5           114431          24289 2020-07-13 18:33:53       Twitter Web App   \n",
       "6            29142           3928 2020-07-09 18:15:51       Twitter Web App   \n",
       "7            26216           4378 2020-07-09 18:15:50       Twitter Web App   \n",
       "8             5631           1151 2020-07-09 18:15:50       Twitter Web App   \n",
       "9             6988           1525 2020-07-09 18:15:50       Twitter Web App   \n",
       "10           51497           9510 2020-07-09 18:15:49       Twitter Web App   \n",
       "11           42646          10431 2020-07-08 16:58:33    Twitter for iPhone   \n",
       "12          197971          42430 2020-07-04 14:35:02       Twitter Web App   \n",
       "13          266377          50339 2020-07-03 16:00:04       Twitter Web App   \n",
       "14          250271          32260 2020-06-29 14:00:17  Twitter Media Studio   \n",
       "15               0          14500 2020-06-26 23:11:31    Twitter for iPhone   \n",
       "16               0           5353 2020-06-26 17:30:03    Twitter for iPhone   \n",
       "17           73278          11219 2020-06-26 14:56:57    Twitter for iPhone   \n",
       "18           98791          11203 2020-06-25 19:47:05    Twitter for iPhone   \n",
       "19               0           1381 2020-06-24 22:05:14    Twitter for iPhone   \n",
       "\n",
       "    reply_to_status reply_to_user  retweets  favorites  \\\n",
       "0               NaN          None     96294     454823   \n",
       "1               NaN          None     10488      66851   \n",
       "2      1.283411e+18   BarackObama      6178      25749   \n",
       "3               NaN          None     17408      73873   \n",
       "4      1.282745e+18   BarackObama      7104      22953   \n",
       "5               NaN          None     24289     114431   \n",
       "6      1.281291e+18   BarackObama      3928      29142   \n",
       "7      1.281291e+18   BarackObama      4378      26216   \n",
       "8      1.281291e+18   BarackObama      1151       5631   \n",
       "9      1.281291e+18   BarackObama      1525       6988   \n",
       "10              NaN          None      9510      51497   \n",
       "11              NaN          None     10431      42646   \n",
       "12              NaN          None     42430     197971   \n",
       "13              NaN          None     50339     266377   \n",
       "14              NaN          None     32260     250271   \n",
       "15              NaN          None     14500          0   \n",
       "16              NaN          None      5353          0   \n",
       "17              NaN          None     11219      73278   \n",
       "18              NaN          None     11203      98791   \n",
       "19              NaN          None      1381          0   \n",
       "\n",
       "                                  text_without_handle  \n",
       "0   Not many of us get to live to see our own lega...  \n",
       "1   Some thoughts on the Reverend C.T. Vivian, a p...  \n",
       "2   Here’s a quick look at how our response stacks...  \n",
       "3   The latest data offers a tragic reminder that ...  \n",
       "4   Get registered and find out how you can vote b...  \n",
       "5   Voting by mail shouldn't be a partisan issue––...  \n",
       "6   It’s clear our communities are crying out for ...  \n",
       "7   Finally, here’s a look at how Germany reshaped...  \n",
       "8   And here’s a counterbalance that argues for su...  \n",
       "9   Here’s a piece that outlines the need for a mo...  \n",
       "10  As communities around the country try to reima...  \n",
       "11  For too long, gerrymandering has contributed t...  \n",
       "12  Our founding promise has never been guaranteed...  \n",
       "13  This holiday weekend, let’s be safe and smart....  \n",
       "14  Today I’m tipping my hat to all the giants in ...  \n",
       "15   : Thinking back to that day reminds me of how...  \n",
       "16   : In the afternoon of June 26, 2015,  travele...  \n",
       "17  Jim’s courage and persistence helped change Am...  \n",
       "18  Michelle and I want every kid in Chicago to gr...  \n",
       "19   : 👉 En exclusiva el expresidente  envió un me...  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['text_without_handle'] = df['text_without_handle'].str.replace('((www\\.[\\s]+)|(https?://[^\\s]+))','\\0',regex=True)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Removing special char,numbers and punctuation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tweet_id</th>\n",
       "      <th>text</th>\n",
       "      <th>favorite_count</th>\n",
       "      <th>retweet_count</th>\n",
       "      <th>created_at</th>\n",
       "      <th>source</th>\n",
       "      <th>reply_to_status</th>\n",
       "      <th>reply_to_user</th>\n",
       "      <th>retweets</th>\n",
       "      <th>favorites</th>\n",
       "      <th>text_without_handle</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1284358592083169280</td>\n",
       "      <td>Not many of us get to live to see our own lega...</td>\n",
       "      <td>454823</td>\n",
       "      <td>96294</td>\n",
       "      <td>2020-07-18 05:25:25</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>96294</td>\n",
       "      <td>454823</td>\n",
       "      <td>Not many of us get to live to see our own lega...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1284140415965429766</td>\n",
       "      <td>Some thoughts on the Reverend C.T. Vivian, a p...</td>\n",
       "      <td>66851</td>\n",
       "      <td>10488</td>\n",
       "      <td>2020-07-17 14:58:27</td>\n",
       "      <td>Twitter for iPhone</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>10488</td>\n",
       "      <td>66851</td>\n",
       "      <td>Some thoughts on the Reverend C T Vivian a pio...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1283410552853757954</td>\n",
       "      <td>Here’s a quick look at how our response stacks...</td>\n",
       "      <td>25749</td>\n",
       "      <td>6178</td>\n",
       "      <td>2020-07-15 14:38:15</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>1.283411e+18</td>\n",
       "      <td>BarackObama</td>\n",
       "      <td>6178</td>\n",
       "      <td>25749</td>\n",
       "      <td>Here s a quick look at how our response stacks...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1283410551532597254</td>\n",
       "      <td>The latest data offers a tragic reminder that ...</td>\n",
       "      <td>73873</td>\n",
       "      <td>17408</td>\n",
       "      <td>2020-07-15 14:38:14</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>17408</td>\n",
       "      <td>73873</td>\n",
       "      <td>The latest data offers a tragic reminder that ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1282745078163939330</td>\n",
       "      <td>Get registered and find out how you can vote b...</td>\n",
       "      <td>22953</td>\n",
       "      <td>7104</td>\n",
       "      <td>2020-07-13 18:33:53</td>\n",
       "      <td>Twitter Web App</td>\n",
       "      <td>1.282745e+18</td>\n",
       "      <td>BarackObama</td>\n",
       "      <td>7104</td>\n",
       "      <td>22953</td>\n",
       "      <td>Get registered and find out how you can vote b...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              tweet_id                                               text  \\\n",
       "0  1284358592083169280  Not many of us get to live to see our own lega...   \n",
       "1  1284140415965429766  Some thoughts on the Reverend C.T. Vivian, a p...   \n",
       "2  1283410552853757954  Here’s a quick look at how our response stacks...   \n",
       "3  1283410551532597254  The latest data offers a tragic reminder that ...   \n",
       "4  1282745078163939330  Get registered and find out how you can vote b...   \n",
       "\n",
       "   favorite_count  retweet_count          created_at              source  \\\n",
       "0          454823          96294 2020-07-18 05:25:25     Twitter Web App   \n",
       "1           66851          10488 2020-07-17 14:58:27  Twitter for iPhone   \n",
       "2           25749           6178 2020-07-15 14:38:15     Twitter Web App   \n",
       "3           73873          17408 2020-07-15 14:38:14     Twitter Web App   \n",
       "4           22953           7104 2020-07-13 18:33:53     Twitter Web App   \n",
       "\n",
       "   reply_to_status reply_to_user  retweets  favorites  \\\n",
       "0              NaN          None     96294     454823   \n",
       "1              NaN          None     10488      66851   \n",
       "2     1.283411e+18   BarackObama      6178      25749   \n",
       "3              NaN          None     17408      73873   \n",
       "4     1.282745e+18   BarackObama      7104      22953   \n",
       "\n",
       "                                 text_without_handle  \n",
       "0  Not many of us get to live to see our own lega...  \n",
       "1  Some thoughts on the Reverend C T Vivian a pio...  \n",
       "2  Here s a quick look at how our response stacks...  \n",
       "3  The latest data offers a tragic reminder that ...  \n",
       "4  Get registered and find out how you can vote b...  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['text_without_handle'] = df['text_without_handle'].str.replace(\"[^a-zA-Z]+\", \" \")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     Not many of us get to live to see our own lega...\n",
       "1     Some thoughts on the Reverend C T Vivian a pio...\n",
       "2     Here s a quick look at how our response stacks...\n",
       "3     The latest data offers a tragic reminder that ...\n",
       "4     Get registered and find out how you can vote b...\n",
       "5     Voting by mail shouldn t be a partisan issue e...\n",
       "6     It s clear our communities are crying out for ...\n",
       "7     Finally here s a look at how Germany reshaped ...\n",
       "8     And here s a counterbalance that argues for su...\n",
       "9     Here s a piece that outlines the need for a mo...\n",
       "10    As communities around the country try to reima...\n",
       "11    For too long gerrymandering has contributed to...\n",
       "12    Our founding promise has never been guaranteed...\n",
       "13    This holiday weekend let s be safe and smart I...\n",
       "14    Today I m tipping my hat to all the giants in ...\n",
       "15     Thinking back to that day reminds me of how m...\n",
       "16     In the afternoon of June traveled to Charlest...\n",
       "17    Jim s courage and persistence helped change Am...\n",
       "18    Michelle and I want every kid in Chicago to gr...\n",
       "19     En exclusiva el expresidente envi un mensaje ...\n",
       "Name: text_without_handle, dtype: object"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['text_without_handle']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.tokenize import word_tokenize\n",
    "from nltk.probability import FreqDist\n",
    "import re\n",
    "from nltk.corpus import stopwords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['text_without_handle'] = df['text_without_handle'].str.lower()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['text_without_handle'] = df['text_without_handle'].apply(lambda x: [item for item in word_tokenize(x)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     [not, many, of, us, get, to, live, to, see, ou...\n",
       "1     [some, thoughts, on, the, reverend, c, t, vivi...\n",
       "2     [here, s, a, quick, look, at, how, our, respon...\n",
       "3     [the, latest, data, offers, a, tragic, reminde...\n",
       "4     [get, registered, and, find, out, how, you, ca...\n",
       "5     [voting, by, mail, shouldn, t, be, a, partisan...\n",
       "6     [it, s, clear, our, communities, are, crying, ...\n",
       "7     [finally, here, s, a, look, at, how, germany, ...\n",
       "8     [and, here, s, a, counterbalance, that, argues...\n",
       "9     [here, s, a, piece, that, outlines, the, need,...\n",
       "10    [as, communities, around, the, country, try, t...\n",
       "11    [for, too, long, gerrymandering, has, contribu...\n",
       "12    [our, founding, promise, has, never, been, gua...\n",
       "13    [this, holiday, weekend, let, s, be, safe, and...\n",
       "14    [today, i, m, tipping, my, hat, to, all, the, ...\n",
       "15    [thinking, back, to, that, day, reminds, me, o...\n",
       "16    [in, the, afternoon, of, june, traveled, to, c...\n",
       "17    [jim, s, courage, and, persistence, helped, ch...\n",
       "18    [michelle, and, i, want, every, kid, in, chica...\n",
       "19    [en, exclusiva, el, expresidente, envi, un, me...\n",
       "Name: text_without_handle, dtype: object"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['text_without_handle']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "stop = stopwords.words('english')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['text_without_handle'] = df['text_without_handle'].apply(lambda x: [item for item in x if item not in stop])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     [many, us, get, live, see, legacy, play, meani...\n",
       "1     [thoughts, reverend, c, vivian, pioneer, pulle...\n",
       "2     [quick, look, response, stacks, countries, aro...\n",
       "3     [latest, data, offers, tragic, reminder, virus...\n",
       "4                   [get, registered, find, vote, mail]\n",
       "5     [voting, mail, partisan, issue, especially, pa...\n",
       "6     [clear, communities, crying, change, change, l...\n",
       "7     [finally, look, germany, reshaped, police, tra...\n",
       "8     [counterbalance, argues, substantial, reforms,...\n",
       "9     [piece, outlines, need, aggressive, overhaul, ...\n",
       "10    [communities, around, country, try, reimagine,...\n",
       "11    [long, gerrymandering, contributed, stalled, p...\n",
       "12    [founding, promise, never, guaranteed, generat...\n",
       "13    [holiday, weekend, let, safe, smart, going, ta...\n",
       "14    [today, tipping, hat, giants, negro, leagues, ...\n",
       "15    [thinking, back, day, reminds, much, strength,...\n",
       "16    [afternoon, june, traveled, charleston, eulogi...\n",
       "17    [jim, courage, persistence, helped, change, am...\n",
       "18    [michelle, want, every, kid, chicago, grow, kn...\n",
       "19    [en, exclusiva, el, expresidente, envi, un, me...\n",
       "Name: text_without_handle, dtype: object"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['text_without_handle']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "def rejoin_words(row):\n",
    "    my_list = row['text_without_handle']\n",
    "    joined_words = ( \" \".join(my_list))\n",
    "    return joined_words\n",
    "\n",
    "df['text_without_handle'] = df.apply(rejoin_words, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     many us get live see legacy play meaningful re...\n",
       "1     thoughts reverend c vivian pioneer pulled amer...\n",
       "2     quick look response stacks countries around world\n",
       "3     latest data offers tragic reminder virus care ...\n",
       "4                         get registered find vote mail\n",
       "5     voting mail partisan issue especially pandemic...\n",
       "6     clear communities crying change change looks l...\n",
       "7     finally look germany reshaped police training ...\n",
       "8     counterbalance argues substantial reforms poli...\n",
       "9      piece outlines need aggressive overhaul policing\n",
       "10    communities around country try reimagine law e...\n",
       "11    long gerrymandering contributed stalled progre...\n",
       "12    founding promise never guaranteed generation c...\n",
       "13    holiday weekend let safe smart going take us b...\n",
       "14    today tipping hat giants negro leagues satchel...\n",
       "15    thinking back day reminds much strength find o...\n",
       "16    afternoon june traveled charleston eulogize re...\n",
       "17    jim courage persistence helped change america ...\n",
       "18    michelle want every kid chicago grow knowing e...\n",
       "19    en exclusiva el expresidente envi un mensaje e...\n",
       "Name: text_without_handle, dtype: object"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['text_without_handle']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# applying lemmatizatio\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "w_tokenizer = nltk.tokenize.WhitespaceTokenizer()\n",
    "lemmatizer = nltk.stem.WordNetLemmatizer()\n",
    "\n",
    "def lemmatized_text(text):\n",
    "    return [lemmatizer.lemmatize(w) for w in w_tokenizer.tokenize(text)]\n",
    "\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['text_without_handle'] = df['text_without_handle'].apply(lemmatized_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     [many, u, get, live, see, legacy, play, meanin...\n",
       "1     [thought, reverend, c, vivian, pioneer, pulled...\n",
       "2     [quick, look, response, stack, country, around...\n",
       "3     [latest, data, offer, tragic, reminder, virus,...\n",
       "4                   [get, registered, find, vote, mail]\n",
       "5     [voting, mail, partisan, issue, especially, pa...\n",
       "6     [clear, community, cry, change, change, look, ...\n",
       "7     [finally, look, germany, reshaped, police, tra...\n",
       "8     [counterbalance, argues, substantial, reform, ...\n",
       "9     [piece, outline, need, aggressive, overhaul, p...\n",
       "10    [community, around, country, try, reimagine, l...\n",
       "11    [long, gerrymandering, contributed, stalled, p...\n",
       "12    [founding, promise, never, guaranteed, generat...\n",
       "13    [holiday, weekend, let, safe, smart, going, ta...\n",
       "14    [today, tipping, hat, giant, negro, league, sa...\n",
       "15    [thinking, back, day, reminds, much, strength,...\n",
       "16    [afternoon, june, traveled, charleston, eulogi...\n",
       "17    [jim, courage, persistence, helped, change, am...\n",
       "18    [michelle, want, every, kid, chicago, grow, kn...\n",
       "19    [en, exclusiva, el, expresidente, envi, un, me...\n",
       "Name: text_without_handle, dtype: object"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['text_without_handle']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "def rejoin_words(row):\n",
    "    my_list = row['text_without_handle']\n",
    "    joined_words = ( \" \".join(my_list))\n",
    "    return joined_words\n",
    "\n",
    "df['text_without_handle'] = df.apply(rejoin_words, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     many u get live see legacy play meaningful rem...\n",
       "1     thought reverend c vivian pioneer pulled ameri...\n",
       "2        quick look response stack country around world\n",
       "3     latest data offer tragic reminder virus care s...\n",
       "4                         get registered find vote mail\n",
       "5     voting mail partisan issue especially pandemic...\n",
       "6         clear community cry change change look like u\n",
       "7     finally look germany reshaped police training ...\n",
       "8     counterbalance argues substantial reform polic...\n",
       "9       piece outline need aggressive overhaul policing\n",
       "10    community around country try reimagine law enf...\n",
       "11    long gerrymandering contributed stalled progre...\n",
       "12    founding promise never guaranteed generation c...\n",
       "13    holiday weekend let safe smart going take u be...\n",
       "14    today tipping hat giant negro league satchel p...\n",
       "15    thinking back day reminds much strength find o...\n",
       "16    afternoon june traveled charleston eulogize re...\n",
       "17    jim courage persistence helped change america ...\n",
       "18    michelle want every kid chicago grow knowing e...\n",
       "19    en exclusiva el expresidente envi un mensaje e...\n",
       "Name: text_without_handle, dtype: object"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['text_without_handle']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tweet_id                0\n",
       "text                    0\n",
       "favorite_count          0\n",
       "retweet_count           0\n",
       "created_at              0\n",
       "source                  0\n",
       "reply_to_status        14\n",
       "reply_to_user          14\n",
       "retweets                0\n",
       "favorites               0\n",
       "text_without_handle     0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Check the Sentiment using TextBlob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting textblobNote: you may need to restart the kernel to use updated packages.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ERROR: Error checking for conflicts.\n",
      "Traceback (most recent call last):\n",
      "  File \"C:\\Users\\ANIMESH\\Anaconda3\\lib\\site-packages\\pip\\_vendor\\pkg_resources\\__init__.py\", line 3021, in _dep_map\n",
      "    return self.__dep_map\n",
      "  File \"C:\\Users\\ANIMESH\\Anaconda3\\lib\\site-packages\\pip\\_vendor\\pkg_resources\\__init__.py\", line 2815, in __getattr__\n",
      "    raise AttributeError(attr)\n",
      "AttributeError: _DistInfoDistribution__dep_map\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"C:\\Users\\ANIMESH\\Anaconda3\\lib\\site-packages\\pip\\_vendor\\pkg_resources\\__init__.py\", line 3012, in _parsed_pkg_info\n",
      "    return self._pkg_info\n",
      "  File \"C:\\Users\\ANIMESH\\Anaconda3\\lib\\site-packages\\pip\\_vendor\\pkg_resources\\__init__.py\", line 2815, in __getattr__\n",
      "    raise AttributeError(attr)\n",
      "AttributeError: _pkg_info\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"C:\\Users\\ANIMESH\\Anaconda3\\lib\\site-packages\\pip\\_internal\\commands\\install.py\", line 517, in _warn_about_conflicts\n",
      "    package_set, _dep_info = check_install_conflicts(to_install)\n",
      "  File \"C:\\Users\\ANIMESH\\Anaconda3\\lib\\site-packages\\pip\\_internal\\operations\\check.py\", line 114, in check_install_conflicts\n",
      "    package_set, _ = create_package_set_from_installed()\n",
      "  File \"C:\\Users\\ANIMESH\\Anaconda3\\lib\\site-packages\\pip\\_internal\\operations\\check.py\", line 53, in create_package_set_from_installed\n",
      "    package_set[name] = PackageDetails(dist.version, dist.requires())\n",
      "  File \"C:\\Users\\ANIMESH\\Anaconda3\\lib\\site-packages\\pip\\_vendor\\pkg_resources\\__init__.py\", line 2736, in requires\n",
      "    dm = self._dep_map\n",
      "  File \"C:\\Users\\ANIMESH\\Anaconda3\\lib\\site-packages\\pip\\_vendor\\pkg_resources\\__init__.py\", line 3023, in _dep_map\n",
      "    self.__dep_map = self._compute_dependencies()\n",
      "  File \"C:\\Users\\ANIMESH\\Anaconda3\\lib\\site-packages\\pip\\_vendor\\pkg_resources\\__init__.py\", line 3032, in _compute_dependencies\n",
      "    for req in self._parsed_pkg_info.get_all('Requires-Dist') or []:\n",
      "  File \"C:\\Users\\ANIMESH\\Anaconda3\\lib\\site-packages\\pip\\_vendor\\pkg_resources\\__init__.py\", line 3014, in _parsed_pkg_info\n",
      "    metadata = self.get_metadata(self.PKG_INFO)\n",
      "  File \"C:\\Users\\ANIMESH\\Anaconda3\\lib\\site-packages\\pip\\_vendor\\pkg_resources\\__init__.py\", line 1420, in get_metadata\n",
      "    value = self._get(path)\n",
      "  File \"C:\\Users\\ANIMESH\\Anaconda3\\lib\\site-packages\\pip\\_vendor\\pkg_resources\\__init__.py\", line 1616, in _get\n",
      "    with open(path, 'rb') as stream:\n",
      "FileNotFoundError: [Errno 2] No such file or directory: 'c:\\\\users\\\\animesh\\\\anaconda3\\\\lib\\\\site-packages\\\\pyodbc-4.0.27.dist-info\\\\METADATA'\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "  Downloading textblob-0.15.3-py2.py3-none-any.whl (636 kB)\n",
      "Requirement already satisfied: nltk>=3.1 in c:\\users\\animesh\\anaconda3\\lib\\site-packages (from textblob) (3.4.5)\n",
      "Requirement already satisfied: six in c:\\users\\animesh\\anaconda3\\lib\\site-packages (from nltk>=3.1->textblob) (1.14.0)\n",
      "Installing collected packages: textblob\n",
      "Successfully installed textblob-0.15.3\n"
     ]
    }
   ],
   "source": [
    "pip install textblob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "from textblob import TextBlob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>polarity</th>\n",
       "      <th>subjectivity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Not many of us get to live to see our own lega...</td>\n",
       "      <td>0.471591</td>\n",
       "      <td>0.562500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Some thoughts on the Reverend C.T. Vivian, a p...</td>\n",
       "      <td>0.900000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Here’s a quick look at how our response stacks...</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>The latest data offers a tragic reminder that ...</td>\n",
       "      <td>0.087500</td>\n",
       "      <td>0.700000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Get registered and find out how you can vote b...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Voting by mail shouldn't be a partisan issue––...</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>0.812500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>It’s clear our communities are crying out for ...</td>\n",
       "      <td>0.100000</td>\n",
       "      <td>0.383333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Finally, here’s a look at how Germany reshaped...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>And here’s a counterbalance that argues for su...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Here’s a piece that outlines the need for a mo...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>As communities around the country try to reima...</td>\n",
       "      <td>0.600000</td>\n",
       "      <td>0.800000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>For too long, gerrymandering has contributed t...</td>\n",
       "      <td>-0.050000</td>\n",
       "      <td>0.400000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Our founding promise has never been guaranteed...</td>\n",
       "      <td>-0.187500</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>This holiday weekend, let’s be safe and smart....</td>\n",
       "      <td>0.357143</td>\n",
       "      <td>0.571429</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Today I’m tipping my hat to all the giants in ...</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>0.750000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>RT @MichelleObama: Thinking back to that day r...</td>\n",
       "      <td>0.050000</td>\n",
       "      <td>0.366667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>RT @ObamaFoundation: In the afternoon of June ...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Jim’s courage and persistence helped change Am...</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>0.250000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Michelle and I want every kid in Chicago to gr...</td>\n",
       "      <td>0.650000</td>\n",
       "      <td>0.750000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>RT @UniNoticias: 👉 En exclusiva el expresident...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 text  polarity  subjectivity\n",
       "0   Not many of us get to live to see our own lega...  0.471591      0.562500\n",
       "1   Some thoughts on the Reverend C.T. Vivian, a p...  0.900000      1.000000\n",
       "2   Here’s a quick look at how our response stacks...  0.333333      0.500000\n",
       "3   The latest data offers a tragic reminder that ...  0.087500      0.700000\n",
       "4   Get registered and find out how you can vote b...  0.000000      0.000000\n",
       "5   Voting by mail shouldn't be a partisan issue––...  0.250000      0.812500\n",
       "6   It’s clear our communities are crying out for ...  0.100000      0.383333\n",
       "7   Finally, here’s a look at how Germany reshaped...  0.000000      1.000000\n",
       "8   And here’s a counterbalance that argues for su...  0.000000      0.000000\n",
       "9   Here’s a piece that outlines the need for a mo...  0.000000      0.000000\n",
       "10  As communities around the country try to reima...  0.600000      0.800000\n",
       "11  For too long, gerrymandering has contributed t... -0.050000      0.400000\n",
       "12  Our founding promise has never been guaranteed... -0.187500      0.500000\n",
       "13  This holiday weekend, let’s be safe and smart....  0.357143      0.571429\n",
       "14  Today I’m tipping my hat to all the giants in ...  0.250000      0.750000\n",
       "15  RT @MichelleObama: Thinking back to that day r...  0.050000      0.366667\n",
       "16  RT @ObamaFoundation: In the afternoon of June ...  0.000000      0.000000\n",
       "17  Jim’s courage and persistence helped change Am...  0.250000      0.250000\n",
       "18  Michelle and I want every kid in Chicago to gr...  0.650000      0.750000\n",
       "19  RT @UniNoticias: 👉 En exclusiva el expresident...  0.000000      0.000000"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[['polarity', 'subjectivity']] = df['text_without_handle'].apply(lambda Text: pd.Series(TextBlob(Text).sentiment))\n",
    "df[['text','polarity', 'subjectivity']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text_without_handle</th>\n",
       "      <th>polarity</th>\n",
       "      <th>subjectivity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>many u get live see legacy play meaningful rem...</td>\n",
       "      <td>0.471591</td>\n",
       "      <td>0.562500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>thought reverend c vivian pioneer pulled ameri...</td>\n",
       "      <td>0.900000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>quick look response stack country around world</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>latest data offer tragic reminder virus care s...</td>\n",
       "      <td>0.087500</td>\n",
       "      <td>0.700000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>get registered find vote mail</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>voting mail partisan issue especially pandemic...</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>0.812500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>clear community cry change change look like u</td>\n",
       "      <td>0.100000</td>\n",
       "      <td>0.383333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>finally look germany reshaped police training ...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>counterbalance argues substantial reform polic...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>piece outline need aggressive overhaul policing</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>community around country try reimagine law enf...</td>\n",
       "      <td>0.600000</td>\n",
       "      <td>0.800000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>long gerrymandering contributed stalled progre...</td>\n",
       "      <td>-0.050000</td>\n",
       "      <td>0.400000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>founding promise never guaranteed generation c...</td>\n",
       "      <td>-0.187500</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>holiday weekend let safe smart going take u be...</td>\n",
       "      <td>0.357143</td>\n",
       "      <td>0.571429</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>today tipping hat giant negro league satchel p...</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>0.750000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>thinking back day reminds much strength find o...</td>\n",
       "      <td>0.050000</td>\n",
       "      <td>0.366667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>afternoon june traveled charleston eulogize re...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>jim courage persistence helped change america ...</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>0.250000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>michelle want every kid chicago grow knowing e...</td>\n",
       "      <td>0.650000</td>\n",
       "      <td>0.750000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>en exclusiva el expresidente envi un mensaje e...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                  text_without_handle  polarity  subjectivity\n",
       "0   many u get live see legacy play meaningful rem...  0.471591      0.562500\n",
       "1   thought reverend c vivian pioneer pulled ameri...  0.900000      1.000000\n",
       "2      quick look response stack country around world  0.333333      0.500000\n",
       "3   latest data offer tragic reminder virus care s...  0.087500      0.700000\n",
       "4                       get registered find vote mail  0.000000      0.000000\n",
       "5   voting mail partisan issue especially pandemic...  0.250000      0.812500\n",
       "6       clear community cry change change look like u  0.100000      0.383333\n",
       "7   finally look germany reshaped police training ...  0.000000      1.000000\n",
       "8   counterbalance argues substantial reform polic...  0.000000      0.000000\n",
       "9     piece outline need aggressive overhaul policing  0.000000      0.000000\n",
       "10  community around country try reimagine law enf...  0.600000      0.800000\n",
       "11  long gerrymandering contributed stalled progre... -0.050000      0.400000\n",
       "12  founding promise never guaranteed generation c... -0.187500      0.500000\n",
       "13  holiday weekend let safe smart going take u be...  0.357143      0.571429\n",
       "14  today tipping hat giant negro league satchel p...  0.250000      0.750000\n",
       "15  thinking back day reminds much strength find o...  0.050000      0.366667\n",
       "16  afternoon june traveled charleston eulogize re...  0.000000      0.000000\n",
       "17  jim courage persistence helped change america ...  0.250000      0.250000\n",
       "18  michelle want every kid chicago grow knowing e...  0.650000      0.750000\n",
       "19  en exclusiva el expresidente envi un mensaje e...  0.000000      0.000000"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[['text_without_handle','polarity', 'subjectivity']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Jim’s courage and persistence helped change America for the better. And so did every American who marched, organize… https://t.co/o0f9SogkU5'"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['text'][17]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'thought reverend c vivian pioneer pulled america closer founding ideal friend'"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['text_without_handle'][1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "many u get live see legacy play meaningful remarkable way john lewis\n",
      "Sentiment(classification='pos', p_pos=0.7956349851023108, p_neg=0.2043650148976917)\n",
      "thought reverend c vivian pioneer pulled america closer founding ideal friend\n",
      "Sentiment(classification='pos', p_pos=0.9782732350031985, p_neg=0.02172676499680046)\n",
      "quick look response stack country around world\n",
      "Sentiment(classification='neg', p_pos=0.489585305042194, p_neg=0.5104146949578059)\n",
      "latest data offer tragic reminder virus care spin ideology best thin\n",
      "Sentiment(classification='pos', p_pos=0.9891390386185123, p_neg=0.010860961381486553)\n",
      "get registered find vote mail\n",
      "Sentiment(classification='pos', p_pos=0.5216540968655804, p_neg=0.47834590313441855)\n",
      "voting mail partisan issue especially pandemic everybody able request\n",
      "Sentiment(classification='pos', p_pos=0.7538092942392617, p_neg=0.24619070576073912)\n",
      "clear community cry change change look like u\n",
      "Sentiment(classification='pos', p_pos=0.8089998972536779, p_neg=0.19100010274632315)\n",
      "finally look germany reshaped police training horror world war ii appro\n",
      "Sentiment(classification='pos', p_pos=0.915821924405719, p_neg=0.08417807559428199)\n",
      "counterbalance argues substantial reform policing oversight fewer police\n",
      "Sentiment(classification='pos', p_pos=0.9735576923076925, p_neg=0.026442307692307702)\n",
      "piece outline need aggressive overhaul policing\n",
      "Sentiment(classification='neg', p_pos=0.29219262075105995, p_neg=0.7078073792489407)\n",
      "community around country try reimagine law enforcement operate effective non dis\n",
      "Sentiment(classification='pos', p_pos=0.8871657197636821, p_neg=0.11283428023631797)\n",
      "long gerrymandering contributed stalled progress warped representative government redistri\n",
      "Sentiment(classification='pos', p_pos=0.7266970313093216, p_neg=0.2733029686906809)\n",
      "founding promise never guaranteed generation called bring u little closer\n",
      "Sentiment(classification='neg', p_pos=0.09469807366882242, p_neg=0.9053019263311782)\n",
      "holiday weekend let safe smart going take u beat virus wear mask\n",
      "Sentiment(classification='neg', p_pos=0.2588838714150783, p_neg=0.7411161285849233)\n",
      "today tipping hat giant negro league satchel paige toni stone many\n",
      "Sentiment(classification='pos', p_pos=0.9981363611757509, p_neg=0.0018636388242496288)\n",
      "thinking back day reminds much strength find one another good time bad se\n",
      "Sentiment(classification='pos', p_pos=0.60488051333199, p_neg=0.39511948666800994)\n",
      "afternoon june traveled charleston eulogize reverend pinckney racist sh\n",
      "Sentiment(classification='pos', p_pos=0.9857677293428747, p_neg=0.01423227065712425)\n",
      "jim courage persistence helped change america better every american marched organize\n",
      "Sentiment(classification='pos', p_pos=0.9965706111915441, p_neg=0.0034293888084556603)\n",
      "michelle want every kid chicago grow knowing even better opportunity happy h\n",
      "Sentiment(classification='pos', p_pos=0.7471483501787803, p_neg=0.2528516498212175)\n",
      "en exclusiva el expresidente envi un mensaje especial los hispanos sobre la importancia de salir votar\n",
      "Sentiment(classification='pos', p_pos=0.5374999999999999, p_neg=0.4625)\n"
     ]
    }
   ],
   "source": [
    "from textblob.sentiments import NaiveBayesAnalyzer\n",
    "for tweet in df['text_without_handle']:\n",
    "    print(tweet)\n",
    "    analysis = TextBlob(tweet,analyzer=NaiveBayesAnalyzer())\n",
    "    print(analysis.sentiment)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}