Big_Data_Project_Twitter_Sentiment_Analysis.ipynb

{"cells":[{"cell_type":"markdown","source":["# Big Data Project - Twitter Sentiment Analysis\n\n---\n## Introduction\n\nIn today's digital age, social media platforms like Twitter have become a crucial source of information for businesses, researchers, and policymakers. Black Friday is a significant shopping event that takes place every year, and Twitter is a platform where people express their views and share their experiences about this event. Analyzing the sentiment of tweets during Black Friday can provide valuable insights into consumer behavior and preferences.\n \nThis project will go through the following steps to build a Machine Learning model for Twitter Sentiment Analysis:\n1. Mount the data on tweets from weclouddata\n2. Create spark session and the spark dataframe for the chosen topic (Black Friday)\n3. Create a sentiment column and clean the text\n4. Feature transfermation\n5. Model training and evaluation\n6. Save the data and predictions to my bucket\n\nOverall, this project aims to demonstrate how machine learning techniques can be used to analyze social media data and provide insights into consumer behavior during significant events like Black Friday. With the increasing popularity of social media platforms, there is a vast amount of unstructured data available that can be used to improve business operations and decision-making."],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"45335157-ca10-412a-b52b-59f68ca406db","inputWidgets":{},"title":""}}},{"cell_type":"markdown","source":["## Import Libraries"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"ca3ad82c-a9fc-4caa-aaf9-a64ab8bebee2","inputWidgets":{},"title":""}}},{"cell_type":"code","source":["# Pyspark SQL\nfrom pyspark.sql import SparkSession\nfrom pyspark.sql.types import StructType, StructField, StringType, FloatType, IntegerType\nfrom pyspark.sql.functions import col, sum, udf\nimport pyspark.sql.functions as F\n\n# Sentiment Analyzer\n# !pip install vaderSentiment\nfrom vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer\n\n# Pyspark Machine Learning\nfrom pyspark.ml.feature import Tokenizer, StopWordsRemover, CountVectorizer, HashingTF, IDF\nfrom pyspark.ml.classification import LogisticRegression\nfrom pyspark.ml.evaluation import BinaryClassificationEvaluator\n"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"8bdcd106-4177-4151-bc81-cad223ff610d","inputWidgets":{},"title":""}},"outputs":[{"output_type":"stream","output_type":"stream","name":"stdout","text":["Collecting vaderSentiment\r\n  Using cached vaderSentiment-3.3.2-py2.py3-none-any.whl (125 kB)\r\nRequirement already satisfied: requests in /databricks/python3/lib/python3.9/site-packages (from vaderSentiment) (2.26.0)\r\nRequirement already satisfied: idna<4,>=2.5 in /databricks/python3/lib/python3.9/site-packages (from requests->vaderSentiment) (3.2)\r\nRequirement already satisfied: charset-normalizer~=2.0.0 in /databricks/python3/lib/python3.9/site-packages (from requests->vaderSentiment) (2.0.4)\r\nRequirement already satisfied: urllib3<1.27,>=1.21.1 in /databricks/python3/lib/python3.9/site-packages (from requests->vaderSentiment) (1.26.7)\r\nRequirement already satisfied: certifi>=2017.4.17 in /databricks/python3/lib/python3.9/site-packages (from requests->vaderSentiment) (2021.10.8)\r\nInstalling collected packages: vaderSentiment\r\nSuccessfully installed vaderSentiment-3.3.2\r\n\u001B[33mWARNING: You are using pip version 21.2.4; however, version 23.1.2 is available.\r\nYou should consider upgrading via the '/local_disk0/.ephemeral_nfs/envs/pythonEnv-00d252a4-0060-4ba9-9749-c3bfd3dffd13/bin/python -m pip install --upgrade pip' command.\u001B[0m\r\n"]}],"execution_count":0},{"cell_type":"markdown","source":["## Load Data"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"c5da603b-a340-43bf-bb74-c30a8732b926","inputWidgets":{},"title":""}}},{"cell_type":"markdown","source":["#### Mount to weclouddata public dataset bucket"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"0df6c3a4-bee9-4499-9121-808533cc0a64","inputWidgets":{},"title":""}}},{"cell_type":"code","source":["# Define a function to mount\n\ndef mount_s3_bucket(access_key, secret_key, bucket_name, mount_folder):\n  ACCESS_KEY_ID = access_key\n  SECRET_ACCESS_KEY = secret_key\n  ENCODED_SECRET_KEY = SECRET_ACCESS_KEY.replace(\"/\", \"%2F\")\n\n  print (\"Mounting\", bucket_name)\n\n  try:\n    # Unmount the data in case it was already mounted.\n    dbutils.fs.unmount(\"/mnt/%s\" % mount_folder)\n    \n  except:\n    # If it fails to unmount it most likely wasn't mounted in the first place\n    print (\"Directory not unmounted: \", mount_folder)\n    \n  finally:\n    # Lastly, mount our bucket.\n    dbutils.fs.mount(\"s3a://%s:%s@%s\" % (ACCESS_KEY_ID, ENCODED_SECRET_KEY, bucket_name), \"/mnt/%s\" % mount_folder)\n    #dbutils.fs.mount(\"s3a://\"+ ACCESS_KEY_ID + \":\" + ENCODED_SECRET_KEY + \"@\" + bucket_name, mount_folder)\n    print (\"The bucket\", bucket_name, \"was mounted to\", mount_folder, \"\\n\")"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"cfd4cea6-239b-436c-9bac-e355d8ad5600","inputWidgets":{},"title":""}},"outputs":[],"execution_count":0},{"cell_type":"code","source":["# Set my access key and secret access key\nACCESS_KEY = my access key\nSECRET_ACCESS_KEY = my secret access key"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"22bb0aed-a612-4407-847b-3bae9e57269c","inputWidgets":{},"title":""}},"outputs":[],"execution_count":0},{"cell_type":"code","source":["# Mount the dataset\nmount_s3_bucket(ACCESS_KEY, SECRET_ACCESS_KEY, \"weclouddata/twitter/\", \"twitter_folder\")"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"0c1cc8a1-1f48-4b0a-90ef-d72ddba62e1f","inputWidgets":{},"title":""}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"data":"","errorSummary":"Cancelled","metadata":{},"errorTraceType":"html","type":"ipynbError","arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n","  .ansiout {\n","    display: block;\n","    unicode-bidi: embed;\n","    white-space: pre-wrap;\n","    word-wrap: break-word;\n","    word-break: break-all;\n","    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n","    font-size: 13px;\n","    color: #555;\n","    margin-left: 4px;\n","    line-height: 19px;\n","  }\n","</style>"]}}],"execution_count":0},{"cell_type":"code","source":["# Explore the mounted folder\n%fs ls /mnt/twitter_folder/"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"69cff712-6713-43f1-9052-a13e3d702577","inputWidgets":{},"title":""}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"overflow":false,"datasetInfos":[],"data":[["dbfs:/mnt/twitter_folder/AI/","AI/",0,0],["dbfs:/mnt/twitter_folder/BankofCanada/","BankofCanada/",0,0],["dbfs:/mnt/twitter_folder/BlackFriday/","BlackFriday/",0,0],["dbfs:/mnt/twitter_folder/CERB/","CERB/",0,0],["dbfs:/mnt/twitter_folder/CSIS/","CSIS/",0,0],["dbfs:/mnt/twitter_folder/CanadaHousing/","CanadaHousing/",0,0],["dbfs:/mnt/twitter_folder/ElonMusk/","ElonMusk/",0,0],["dbfs:/mnt/twitter_folder/Flames/","Flames/",0,0],["dbfs:/mnt/twitter_folder/Inflation/","Inflation/",0,0],["dbfs:/mnt/twitter_folder/Interest_rate/","Interest_rate/",0,0],["dbfs:/mnt/twitter_folder/Iran/","Iran/",0,0],["dbfs:/mnt/twitter_folder/MTA/","MTA/",0,0],["dbfs:/mnt/twitter_folder/StudentLoanRelief/","StudentLoanRelief/",0,0],["dbfs:/mnt/twitter_folder/WorldCup/","WorldCup/",0,0],["dbfs:/mnt/twitter_folder/cancer/","cancer/",0,0],["dbfs:/mnt/twitter_folder/greenbelt/","greenbelt/",0,0],["dbfs:/mnt/twitter_folder/thanksgiving/","thanksgiving/",0,0],["dbfs:/mnt/twitter_folder/twitter/","twitter/",0,0],["dbfs:/mnt/twitter_folder/wecan/","wecan/",0,0]],"plotOptions":{"displayType":"table","customPlotOptions":{},"pivotColumns":null,"pivotAggregation":null,"xColumns":null,"yColumns":null},"columnCustomDisplayInfos":{},"aggType":"","isJsonSchema":true,"removedWidgets":[],"aggSchema":[],"schema":[{"name":"path","type":"\"string\"","metadata":"{}"},{"name":"name","type":"\"string\"","metadata":"{}"},{"name":"size","type":"\"long\"","metadata":"{}"},{"name":"modificationTime","type":"\"long\"","metadata":"{}"}],"aggError":"","aggData":[],"addedWidgets":{},"metadata":{"isDbfsCommandResult":false},"dbfsResultPath":null,"type":"table","aggOverflow":false,"aggSeriesLimitReached":false,"arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n","  .table-result-container {\n","    max-height: 300px;\n","    overflow: auto;\n","  }\n","  table, th, td {\n","    border: 1px solid black;\n","    border-collapse: collapse;\n","  }\n","  th, td {\n","    padding: 5px;\n","  }\n","  th {\n","    text-align: left;\n","  }\n","</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>path</th><th>name</th><th>size</th><th>modificationTime</th></tr></thead><tbody><tr><td>dbfs:/mnt/twitter_folder/AI/</td><td>AI/</td><td>0</td><td>0</td></tr><tr><td>dbfs:/mnt/twitter_folder/BankofCanada/</td><td>BankofCanada/</td><td>0</td><td>0</td></tr><tr><td>dbfs:/mnt/twitter_folder/BlackFriday/</td><td>BlackFriday/</td><td>0</td><td>0</td></tr><tr><td>dbfs:/mnt/twitter_folder/CERB/</td><td>CERB/</td><td>0</td><td>0</td></tr><tr><td>dbfs:/mnt/twitter_folder/CSIS/</td><td>CSIS/</td><td>0</td><td>0</td></tr><tr><td>dbfs:/mnt/twitter_folder/CanadaHousing/</td><td>CanadaHousing/</td><td>0</td><td>0</td></tr><tr><td>dbfs:/mnt/twitter_folder/ElonMusk/</td><td>ElonMusk/</td><td>0</td><td>0</td></tr><tr><td>dbfs:/mnt/twitter_folder/Flames/</td><td>Flames/</td><td>0</td><td>0</td></tr><tr><td>dbfs:/mnt/twitter_folder/Inflation/</td><td>Inflation/</td><td>0</td><td>0</td></tr><tr><td>dbfs:/mnt/twitter_folder/Interest_rate/</td><td>Interest_rate/</td><td>0</td><td>0</td></tr><tr><td>dbfs:/mnt/twitter_folder/Iran/</td><td>Iran/</td><td>0</td><td>0</td></tr><tr><td>dbfs:/mnt/twitter_folder/MTA/</td><td>MTA/</td><td>0</td><td>0</td></tr><tr><td>dbfs:/mnt/twitter_folder/StudentLoanRelief/</td><td>StudentLoanRelief/</td><td>0</td><td>0</td></tr><tr><td>dbfs:/mnt/twitter_folder/WorldCup/</td><td>WorldCup/</td><td>0</td><td>0</td></tr><tr><td>dbfs:/mnt/twitter_folder/cancer/</td><td>cancer/</td><td>0</td><td>0</td></tr><tr><td>dbfs:/mnt/twitter_folder/greenbelt/</td><td>greenbelt/</td><td>0</td><td>0</td></tr><tr><td>dbfs:/mnt/twitter_folder/thanksgiving/</td><td>thanksgiving/</td><td>0</td><td>0</td></tr><tr><td>dbfs:/mnt/twitter_folder/twitter/</td><td>twitter/</td><td>0</td><td>0</td></tr><tr><td>dbfs:/mnt/twitter_folder/wecan/</td><td>wecan/</td><td>0</td><td>0</td></tr></tbody></table></div>"]}}],"execution_count":0},{"cell_type":"code","source":["# file path for tweets about Black Friday on November 24th, 2022 \n# Due to limited computing power, we will choose tweets from only the day before Black Friday (11/24).\nfilePath = '/mnt/twitter_folder/BlackFriday/2022/11/24/*/*'"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"3e3fab33-35fa-4517-a7b1-c2f90c85f247","inputWidgets":{},"title":""}},"outputs":[],"execution_count":0},{"cell_type":"markdown","source":["#### Create Spark Session"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"bf207b0d-536c-405a-b511-20962aa51f12","inputWidgets":{},"title":""}}},{"cell_type":"code","source":["spark = (SparkSession\n        .builder\n        .appName('df')\n        .getOrCreate())\n\nprint('Session created')"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"42451235-2aee-4993-b384-7990c499cb5f","inputWidgets":{},"title":""}},"outputs":[{"output_type":"stream","output_type":"stream","name":"stdout","text":["Session created\n"]}],"execution_count":0},{"cell_type":"code","source":["sc = spark.sparkContext"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"1d2d40a2-4eb3-4641-be97-ce0158a158d2","inputWidgets":{},"title":""}},"outputs":[],"execution_count":0},{"cell_type":"code","source":["# Define schema\nschema = StructType([\n    StructField('id', StringType(), True),\n    StructField('name', StringType(), True),\n    StructField('username', StringType(), True),\n    StructField('tweet', StringType(), True),\n    StructField('followers_count', StringType(), True),\n    StructField('location', StringType(), True),\n    StructField('geo', StringType(), True),\n    StructField('created_at', StringType(), True)\n])"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"4f60019e-1050-4550-8d75-28efd3a2e9e7","inputWidgets":{},"title":""}},"outputs":[],"execution_count":0},{"cell_type":"code","source":["# read data from the selected file path\ndf = (spark.read.schema(schema).option('delimiter','\\t').csv(filePath))"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"34a79ea3-4724-45f2-98a6-a63deba05421","inputWidgets":{},"title":""}},"outputs":[],"execution_count":0},{"cell_type":"code","source":["# cache the dataframe for faster iteration\ndf.cache()\n\n# run the count action to materialize the cache\ndf.count()"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"9ef868cf-5d6b-4ac1-904e-a88ea8f5f8bd","inputWidgets":{},"title":""}},"outputs":[{"output_type":"stream","output_type":"stream","name":"stdout","text":["Out[7]: 113814"]}],"execution_count":0},{"cell_type":"code","source":["display(df.head(10))"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"166a6b98-9608-4861-a7ce-df0d9c9e8759","inputWidgets":{},"title":""}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"overflow":false,"datasetInfos":[],"data":[["1595930122003320832","Alan L. Stewart - unetomaterouge","unetomaterouge","RT @ProgIntl: BREAKING: Tomorrow, Amazon workers at 18 warehouses are going on strike in France and Germany to #MakeAmazonPay, with many ot…","2394","None","None","Thu Nov 24 23:59:26 +0000 2022"],["1595930122124550144","Bismillah BigWin","CityxWin0","RT @ChiefElrond: $50 | 24 Hours   🥏 RT &amp; Follow:    @CryptoCoinCoach   +  @NeblioTeam  (BE ACTIVE ON PROFILE)  Tweet #NEBL  NEXT GEM ON BLA…","262","None","None","Thu Nov 24 23:59:26 +0000 2022"],["1595930122682478592","Fuck You I Quit","fuckyouiquit","This is the only Black Friday ad you need to see https://t.co/7Hj3CR4YlS","272510","Corporate Accounts Payable","None","Thu Nov 24 23:59:26 +0000 2022"],["1595930123337105417","kumiii🐻 • freetag","chunvrwin","RT @jihanicorn: $150 | 2.250.000 IDR • 24 Hours 💜  - RT &amp; Follow @CryptoCoinCoach + @NeblioTeam  ____________________________ (BE ACTIVE ON…","53","🍀","None","Thu Nov 24 23:59:27 +0000 2022"],["1595930124075294722","D_Adrian","DexaWinWin","RT @HeiraCrypto: $100 ~ Ends in 24 hrs.   🔶 RT - Follow:    @CryptoCoinCoach  +  @NeblioTeam  (BE ACTIVE ON PROFILE)  Tweet #NEBL  NEXT GEM…","45","None","None","Thu Nov 24 23:59:27 +0000 2022"],["1595930124305649664","$NEST 🎄$TTC freetag sepuasnya🍀","desnumber1","RT @May7ven: $50 — 24 Hours —   ➖RT &amp; Follow:    @CryptoCoinCoach   +   @NeblioTeam  (BE ACTIVE ON PROFILE)  Tweet #NEBL  NEXT GEM ON BLACK…","189","kwangya ","None","Thu Nov 24 23:59:27 +0000 2022"],["1595930125413285890","bigwin 🔥 $COOKIES 🍪","0xbigwinasn","RT @Beast_Cryptox: 💰 $70 ~ 24 HOURS 🐄🦥  ➖ RT &amp; Follow  @CryptoCoinCoach @NeblioTeam ----------------- (BE ACTIVE ON PROFILE)  Tweet - #NEBL…","42","Jawa Tengah, Indonesia","None","Thu Nov 24 23:59:27 +0000 2022"],["1595930125903990784","Rex00o","Rex00o2","RT @GFuelEnergy: 💛 𝗟𝗜𝗞𝗘 + 𝗥𝗧 to win a #BanjoKazooie x #GFUEL \"HONEY BERRY\" Tub!!! Picking 2 winners tomorrow bc we just RESTOCKED these bab…","23","None","None","Thu Nov 24 23:59:27 +0000 2022"],["1595930126499332099","CJ Tocco","DMAlCo241","@ThisIsKyleR Working for doubletime and a half is MY tradition. Of course, the sweet paycheck gets gobbled up on Bl… https://t.co/kaDhOvmW5n","23","None","None","Thu Nov 24 23:59:27 +0000 2022"],["1595930127816609792","Martin Jones","BackPackJones","If y’all don’t get y’all grandma a tv for Black Friday","616","Dallas, TX","None","Thu Nov 24 23:59:28 +0000 2022"]],"plotOptions":{"displayType":"table","customPlotOptions":{},"pivotColumns":null,"pivotAggregation":null,"xColumns":null,"yColumns":null},"columnCustomDisplayInfos":{},"aggType":"","isJsonSchema":true,"removedWidgets":[],"aggSchema":[],"schema":[{"name":"id","type":"\"string\"","metadata":"{}"},{"name":"name","type":"\"string\"","metadata":"{}"},{"name":"username","type":"\"string\"","metadata":"{}"},{"name":"tweet","type":"\"string\"","metadata":"{}"},{"name":"followers_count","type":"\"string\"","metadata":"{}"},{"name":"location","type":"\"string\"","metadata":"{}"},{"name":"geo","type":"\"string\"","metadata":"{}"},{"name":"created_at","type":"\"string\"","metadata":"{}"}],"aggError":"","aggData":[],"addedWidgets":{},"metadata":{},"dbfsResultPath":null,"type":"table","aggOverflow":false,"aggSeriesLimitReached":false,"arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n","  .table-result-container {\n","    max-height: 300px;\n","    overflow: auto;\n","  }\n","  table, th, td {\n","    border: 1px solid black;\n","    border-collapse: collapse;\n","  }\n","  th, td {\n","    padding: 5px;\n","  }\n","  th {\n","    text-align: left;\n","  }\n","</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>id</th><th>name</th><th>username</th><th>tweet</th><th>followers_count</th><th>location</th><th>geo</th><th>created_at</th></tr></thead><tbody><tr><td>1595930122003320832</td><td>Alan L. Stewart - unetomaterouge</td><td>unetomaterouge</td><td>RT @ProgIntl: BREAKING: Tomorrow, Amazon workers at 18 warehouses are going on strike in France and Germany to #MakeAmazonPay, with many ot…</td><td>2394</td><td>None</td><td>None</td><td>Thu Nov 24 23:59:26 +0000 2022</td></tr><tr><td>1595930122124550144</td><td>Bismillah BigWin</td><td>CityxWin0</td><td>RT @ChiefElrond: $50 | 24 Hours   🥏 RT &amp; Follow:    @CryptoCoinCoach   +  @NeblioTeam  (BE ACTIVE ON PROFILE)  Tweet #NEBL  NEXT GEM ON BLA…</td><td>262</td><td>None</td><td>None</td><td>Thu Nov 24 23:59:26 +0000 2022</td></tr><tr><td>1595930122682478592</td><td>Fuck You I Quit</td><td>fuckyouiquit</td><td>This is the only Black Friday ad you need to see https://t.co/7Hj3CR4YlS</td><td>272510</td><td>Corporate Accounts Payable</td><td>None</td><td>Thu Nov 24 23:59:26 +0000 2022</td></tr><tr><td>1595930123337105417</td><td>kumiii🐻 • freetag</td><td>chunvrwin</td><td>RT @jihanicorn: $150 | 2.250.000 IDR • 24 Hours 💜  - RT &amp; Follow @CryptoCoinCoach + @NeblioTeam  ____________________________ (BE ACTIVE ON…</td><td>53</td><td>🍀</td><td>None</td><td>Thu Nov 24 23:59:27 +0000 2022</td></tr><tr><td>1595930124075294722</td><td>D_Adrian</td><td>DexaWinWin</td><td>RT @HeiraCrypto: $100 ~ Ends in 24 hrs.   🔶 RT - Follow:    @CryptoCoinCoach  +  @NeblioTeam  (BE ACTIVE ON PROFILE)  Tweet #NEBL  NEXT GEM…</td><td>45</td><td>None</td><td>None</td><td>Thu Nov 24 23:59:27 +0000 2022</td></tr><tr><td>1595930124305649664</td><td>$NEST 🎄$TTC freetag sepuasnya🍀</td><td>desnumber1</td><td>RT @May7ven: $50 — 24 Hours —   ➖RT &amp; Follow:    @CryptoCoinCoach   +   @NeblioTeam  (BE ACTIVE ON PROFILE)  Tweet #NEBL  NEXT GEM ON BLACK…</td><td>189</td><td>kwangya </td><td>None</td><td>Thu Nov 24 23:59:27 +0000 2022</td></tr><tr><td>1595930125413285890</td><td>bigwin 🔥 $COOKIES 🍪</td><td>0xbigwinasn</td><td>RT @Beast_Cryptox: 💰 $70 ~ 24 HOURS 🐄🦥  ➖ RT &amp; Follow  @CryptoCoinCoach @NeblioTeam ----------------- (BE ACTIVE ON PROFILE)  Tweet - #NEBL…</td><td>42</td><td>Jawa Tengah, Indonesia</td><td>None</td><td>Thu Nov 24 23:59:27 +0000 2022</td></tr><tr><td>1595930125903990784</td><td>Rex00o</td><td>Rex00o2</td><td>RT @GFuelEnergy: 💛 𝗟𝗜𝗞𝗘 + 𝗥𝗧 to win a #BanjoKazooie x #GFUEL \"HONEY BERRY\" Tub!!! Picking 2 winners tomorrow bc we just RESTOCKED these bab…</td><td>23</td><td>None</td><td>None</td><td>Thu Nov 24 23:59:27 +0000 2022</td></tr><tr><td>1595930126499332099</td><td>CJ Tocco</td><td>DMAlCo241</td><td>@ThisIsKyleR Working for doubletime and a half is MY tradition. Of course, the sweet paycheck gets gobbled up on Bl… https://t.co/kaDhOvmW5n</td><td>23</td><td>None</td><td>None</td><td>Thu Nov 24 23:59:27 +0000 2022</td></tr><tr><td>1595930127816609792</td><td>Martin Jones</td><td>BackPackJones</td><td>If y’all don’t get y’all grandma a tv for Black Friday</td><td>616</td><td>Dallas, TX</td><td>None</td><td>Thu Nov 24 23:59:28 +0000 2022</td></tr></tbody></table></div>"]}}],"execution_count":0},{"cell_type":"code","source":["# Get the shape of the DataFrame\nnum_rows = df.count()\nnum_cols = len(df.columns)\n\nprint(\"Number of rows:\", num_rows)\nprint(\"Number of columns:\", num_cols)"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"da10fcd4-13ee-417c-88c8-6c5cb0333c1c","inputWidgets":{},"title":""}},"outputs":[{"output_type":"stream","output_type":"stream","name":"stdout","text":["Number of rows: 113814\nNumber of columns: 8\n"]}],"execution_count":0},{"cell_type":"code","source":["# check for null values for column 'tweet'\n\n# Count the number of rows with null values in the \"tweet\" column\nnull_count = df.select(sum(col(\"tweet\").isNull().cast(\"int\"))).collect()[0][0]\n\nprint(\"Number of rows with null values in the 'tweet' column:\", null_count)\n"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"60ce383a-f851-4f87-99dd-79675718fd38","inputWidgets":{},"title":""}},"outputs":[{"output_type":"stream","output_type":"stream","name":"stdout","text":["Number of rows with null values in the 'tweet' column: 268\n"]}],"execution_count":0},{"cell_type":"code","source":["# drop those columns\ndf = df.dropna(subset=[\"tweet\"])\ndisplay(df.head(10))"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"78372b52-3b40-453a-8d46-da168be4c1b5","inputWidgets":{},"title":""}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"overflow":false,"datasetInfos":[],"data":[["1595930122003320832","Alan L. Stewart - unetomaterouge","unetomaterouge","RT @ProgIntl: BREAKING: Tomorrow, Amazon workers at 18 warehouses are going on strike in France and Germany to #MakeAmazonPay, with many ot…","2394","None","None","Thu Nov 24 23:59:26 +0000 2022"],["1595930122124550144","Bismillah BigWin","CityxWin0","RT @ChiefElrond: $50 | 24 Hours   🥏 RT &amp; Follow:    @CryptoCoinCoach   +  @NeblioTeam  (BE ACTIVE ON PROFILE)  Tweet #NEBL  NEXT GEM ON BLA…","262","None","None","Thu Nov 24 23:59:26 +0000 2022"],["1595930122682478592","Fuck You I Quit","fuckyouiquit","This is the only Black Friday ad you need to see https://t.co/7Hj3CR4YlS","272510","Corporate Accounts Payable","None","Thu Nov 24 23:59:26 +0000 2022"],["1595930123337105417","kumiii🐻 • freetag","chunvrwin","RT @jihanicorn: $150 | 2.250.000 IDR • 24 Hours 💜  - RT &amp; Follow @CryptoCoinCoach + @NeblioTeam  ____________________________ (BE ACTIVE ON…","53","🍀","None","Thu Nov 24 23:59:27 +0000 2022"],["1595930124075294722","D_Adrian","DexaWinWin","RT @HeiraCrypto: $100 ~ Ends in 24 hrs.   🔶 RT - Follow:    @CryptoCoinCoach  +  @NeblioTeam  (BE ACTIVE ON PROFILE)  Tweet #NEBL  NEXT GEM…","45","None","None","Thu Nov 24 23:59:27 +0000 2022"],["1595930124305649664","$NEST 🎄$TTC freetag sepuasnya🍀","desnumber1","RT @May7ven: $50 — 24 Hours —   ➖RT &amp; Follow:    @CryptoCoinCoach   +   @NeblioTeam  (BE ACTIVE ON PROFILE)  Tweet #NEBL  NEXT GEM ON BLACK…","189","kwangya ","None","Thu Nov 24 23:59:27 +0000 2022"],["1595930125413285890","bigwin 🔥 $COOKIES 🍪","0xbigwinasn","RT @Beast_Cryptox: 💰 $70 ~ 24 HOURS 🐄🦥  ➖ RT &amp; Follow  @CryptoCoinCoach @NeblioTeam ----------------- (BE ACTIVE ON PROFILE)  Tweet - #NEBL…","42","Jawa Tengah, Indonesia","None","Thu Nov 24 23:59:27 +0000 2022"],["1595930125903990784","Rex00o","Rex00o2","RT @GFuelEnergy: 💛 𝗟𝗜𝗞𝗘 + 𝗥𝗧 to win a #BanjoKazooie x #GFUEL \"HONEY BERRY\" Tub!!! Picking 2 winners tomorrow bc we just RESTOCKED these bab…","23","None","None","Thu Nov 24 23:59:27 +0000 2022"],["1595930126499332099","CJ Tocco","DMAlCo241","@ThisIsKyleR Working for doubletime and a half is MY tradition. Of course, the sweet paycheck gets gobbled up on Bl… https://t.co/kaDhOvmW5n","23","None","None","Thu Nov 24 23:59:27 +0000 2022"],["1595930127816609792","Martin Jones","BackPackJones","If y’all don’t get y’all grandma a tv for Black Friday","616","Dallas, TX","None","Thu Nov 24 23:59:28 +0000 2022"]],"plotOptions":{"displayType":"table","customPlotOptions":{},"pivotColumns":null,"pivotAggregation":null,"xColumns":null,"yColumns":null},"columnCustomDisplayInfos":{},"aggType":"","isJsonSchema":true,"removedWidgets":[],"aggSchema":[],"schema":[{"name":"id","type":"\"string\"","metadata":"{}"},{"name":"name","type":"\"string\"","metadata":"{}"},{"name":"username","type":"\"string\"","metadata":"{}"},{"name":"tweet","type":"\"string\"","metadata":"{}"},{"name":"followers_count","type":"\"string\"","metadata":"{}"},{"name":"location","type":"\"string\"","metadata":"{}"},{"name":"geo","type":"\"string\"","metadata":"{}"},{"name":"created_at","type":"\"string\"","metadata":"{}"}],"aggError":"","aggData":[],"addedWidgets":{},"metadata":{},"dbfsResultPath":null,"type":"table","aggOverflow":false,"aggSeriesLimitReached":false,"arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n","  .table-result-container {\n","    max-height: 300px;\n","    overflow: auto;\n","  }\n","  table, th, td {\n","    border: 1px solid black;\n","    border-collapse: collapse;\n","  }\n","  th, td {\n","    padding: 5px;\n","  }\n","  th {\n","    text-align: left;\n","  }\n","</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>id</th><th>name</th><th>username</th><th>tweet</th><th>followers_count</th><th>location</th><th>geo</th><th>created_at</th></tr></thead><tbody><tr><td>1595930122003320832</td><td>Alan L. Stewart - unetomaterouge</td><td>unetomaterouge</td><td>RT @ProgIntl: BREAKING: Tomorrow, Amazon workers at 18 warehouses are going on strike in France and Germany to #MakeAmazonPay, with many ot…</td><td>2394</td><td>None</td><td>None</td><td>Thu Nov 24 23:59:26 +0000 2022</td></tr><tr><td>1595930122124550144</td><td>Bismillah BigWin</td><td>CityxWin0</td><td>RT @ChiefElrond: $50 | 24 Hours   🥏 RT &amp; Follow:    @CryptoCoinCoach   +  @NeblioTeam  (BE ACTIVE ON PROFILE)  Tweet #NEBL  NEXT GEM ON BLA…</td><td>262</td><td>None</td><td>None</td><td>Thu Nov 24 23:59:26 +0000 2022</td></tr><tr><td>1595930122682478592</td><td>Fuck You I Quit</td><td>fuckyouiquit</td><td>This is the only Black Friday ad you need to see https://t.co/7Hj3CR4YlS</td><td>272510</td><td>Corporate Accounts Payable</td><td>None</td><td>Thu Nov 24 23:59:26 +0000 2022</td></tr><tr><td>1595930123337105417</td><td>kumiii🐻 • freetag</td><td>chunvrwin</td><td>RT @jihanicorn: $150 | 2.250.000 IDR • 24 Hours 💜  - RT &amp; Follow @CryptoCoinCoach + @NeblioTeam  ____________________________ (BE ACTIVE ON…</td><td>53</td><td>🍀</td><td>None</td><td>Thu Nov 24 23:59:27 +0000 2022</td></tr><tr><td>1595930124075294722</td><td>D_Adrian</td><td>DexaWinWin</td><td>RT @HeiraCrypto: $100 ~ Ends in 24 hrs.   🔶 RT - Follow:    @CryptoCoinCoach  +  @NeblioTeam  (BE ACTIVE ON PROFILE)  Tweet #NEBL  NEXT GEM…</td><td>45</td><td>None</td><td>None</td><td>Thu Nov 24 23:59:27 +0000 2022</td></tr><tr><td>1595930124305649664</td><td>$NEST 🎄$TTC freetag sepuasnya🍀</td><td>desnumber1</td><td>RT @May7ven: $50 — 24 Hours —   ➖RT &amp; Follow:    @CryptoCoinCoach   +   @NeblioTeam  (BE ACTIVE ON PROFILE)  Tweet #NEBL  NEXT GEM ON BLACK…</td><td>189</td><td>kwangya </td><td>None</td><td>Thu Nov 24 23:59:27 +0000 2022</td></tr><tr><td>1595930125413285890</td><td>bigwin 🔥 $COOKIES 🍪</td><td>0xbigwinasn</td><td>RT @Beast_Cryptox: 💰 $70 ~ 24 HOURS 🐄🦥  ➖ RT &amp; Follow  @CryptoCoinCoach @NeblioTeam ----------------- (BE ACTIVE ON PROFILE)  Tweet - #NEBL…</td><td>42</td><td>Jawa Tengah, Indonesia</td><td>None</td><td>Thu Nov 24 23:59:27 +0000 2022</td></tr><tr><td>1595930125903990784</td><td>Rex00o</td><td>Rex00o2</td><td>RT @GFuelEnergy: 💛 𝗟𝗜𝗞𝗘 + 𝗥𝗧 to win a #BanjoKazooie x #GFUEL \"HONEY BERRY\" Tub!!! Picking 2 winners tomorrow bc we just RESTOCKED these bab…</td><td>23</td><td>None</td><td>None</td><td>Thu Nov 24 23:59:27 +0000 2022</td></tr><tr><td>1595930126499332099</td><td>CJ Tocco</td><td>DMAlCo241</td><td>@ThisIsKyleR Working for doubletime and a half is MY tradition. Of course, the sweet paycheck gets gobbled up on Bl… https://t.co/kaDhOvmW5n</td><td>23</td><td>None</td><td>None</td><td>Thu Nov 24 23:59:27 +0000 2022</td></tr><tr><td>1595930127816609792</td><td>Martin Jones</td><td>BackPackJones</td><td>If y’all don’t get y’all grandma a tv for Black Friday</td><td>616</td><td>Dallas, TX</td><td>None</td><td>Thu Nov 24 23:59:28 +0000 2022</td></tr></tbody></table></div>"]}}],"execution_count":0},{"cell_type":"markdown","source":["#### Create Sentiment column using VADER"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"9b88943b-3811-4396-b9ac-d596761592b2","inputWidgets":{},"title":""}}},{"cell_type":"code","source":["# define a function to get sentiment score using VADER\ndef getSentimentScore(tweetText):\n    sia = SentimentIntensityAnalyzer()\n    ss = sia.polarity_scores(tweetText)\n    return float(ss['compound'])\n\n# define a function to get sentiment\ndef getSentiment(score):\n    return 1 if score >= 0 else 0"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"e17112e1-5278-4d36-89ed-5fb92931ce3c","inputWidgets":{},"title":""}},"outputs":[],"execution_count":0},{"cell_type":"code","source":["# create sentiment score column\nudfss=udf(getSentimentScore, FloatType())\ndf = df.withColumn('sentiment score',udfss('tweet'))"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"2413090c-b267-4d5e-8a33-36fba79aded4","inputWidgets":{},"title":""}},"outputs":[],"execution_count":0},{"cell_type":"code","source":["# create sentiment column - positive:1  negative:0\nudfSentiment = udf(getSentiment, IntegerType())\ndf1 = df.withColumn('sentiment', udfSentiment('sentiment score'))"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"d36ee491-afd6-462b-9b9c-185636c5629d","inputWidgets":{},"title":""}},"outputs":[],"execution_count":0},{"cell_type":"code","source":["display(df1.head(5))\n# df1 is the dataframe with the original data + sentiment + sentiment score"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"c1f759d1-2955-4191-a477-c5a86c69a4d2","inputWidgets":{},"title":""}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"overflow":false,"datasetInfos":[],"data":[["1595930122003320832","Alan L. Stewart - unetomaterouge","unetomaterouge","RT @ProgIntl: BREAKING: Tomorrow, Amazon workers at 18 warehouses are going on strike in France and Germany to #MakeAmazonPay, with many ot…","2394","None","None","Thu Nov 24 23:59:26 +0000 2022",0.051600001752376556,1],["1595930122124550144","Bismillah BigWin","CityxWin0","RT @ChiefElrond: $50 | 24 Hours   🥏 RT &amp; Follow:    @CryptoCoinCoach   +  @NeblioTeam  (BE ACTIVE ON PROFILE)  Tweet #NEBL  NEXT GEM ON BLA…","262","None","None","Thu Nov 24 23:59:26 +0000 2022",0.5318999886512756,1],["1595930122682478592","Fuck You I Quit","fuckyouiquit","This is the only Black Friday ad you need to see https://t.co/7Hj3CR4YlS","272510","Corporate Accounts Payable","None","Thu Nov 24 23:59:26 +0000 2022",0.0,1],["1595930123337105417","kumiii🐻 • freetag","chunvrwin","RT @jihanicorn: $150 | 2.250.000 IDR • 24 Hours 💜  - RT &amp; Follow @CryptoCoinCoach + @NeblioTeam  ____________________________ (BE ACTIVE ON…","53","🍀","None","Thu Nov 24 23:59:27 +0000 2022",0.8240000009536743,1],["1595930124075294722","D_Adrian","DexaWinWin","RT @HeiraCrypto: $100 ~ Ends in 24 hrs.   🔶 RT - Follow:    @CryptoCoinCoach  +  @NeblioTeam  (BE ACTIVE ON PROFILE)  Tweet #NEBL  NEXT GEM…","45","None","None","Thu Nov 24 23:59:27 +0000 2022",0.7034000158309937,1]],"plotOptions":{"displayType":"table","customPlotOptions":{},"pivotColumns":null,"pivotAggregation":null,"xColumns":null,"yColumns":null},"columnCustomDisplayInfos":{},"aggType":"","isJsonSchema":true,"removedWidgets":[],"aggSchema":[],"schema":[{"name":"id","type":"\"string\"","metadata":"{}"},{"name":"name","type":"\"string\"","metadata":"{}"},{"name":"username","type":"\"string\"","metadata":"{}"},{"name":"tweet","type":"\"string\"","metadata":"{}"},{"name":"followers_count","type":"\"string\"","metadata":"{}"},{"name":"location","type":"\"string\"","metadata":"{}"},{"name":"geo","type":"\"string\"","metadata":"{}"},{"name":"created_at","type":"\"string\"","metadata":"{}"},{"name":"sentiment score","type":"\"double\"","metadata":"{}"},{"name":"sentiment","type":"\"long\"","metadata":"{}"}],"aggError":"","aggData":[],"addedWidgets":{},"metadata":{},"dbfsResultPath":null,"type":"table","aggOverflow":false,"aggSeriesLimitReached":false,"arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n","  .table-result-container {\n","    max-height: 300px;\n","    overflow: auto;\n","  }\n","  table, th, td {\n","    border: 1px solid black;\n","    border-collapse: collapse;\n","  }\n","  th, td {\n","    padding: 5px;\n","  }\n","  th {\n","    text-align: left;\n","  }\n","</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>id</th><th>name</th><th>username</th><th>tweet</th><th>followers_count</th><th>location</th><th>geo</th><th>created_at</th><th>sentiment score</th><th>sentiment</th></tr></thead><tbody><tr><td>1595930122003320832</td><td>Alan L. Stewart - unetomaterouge</td><td>unetomaterouge</td><td>RT @ProgIntl: BREAKING: Tomorrow, Amazon workers at 18 warehouses are going on strike in France and Germany to #MakeAmazonPay, with many ot…</td><td>2394</td><td>None</td><td>None</td><td>Thu Nov 24 23:59:26 +0000 2022</td><td>0.051600001752376556</td><td>1</td></tr><tr><td>1595930122124550144</td><td>Bismillah BigWin</td><td>CityxWin0</td><td>RT @ChiefElrond: $50 | 24 Hours   🥏 RT &amp; Follow:    @CryptoCoinCoach   +  @NeblioTeam  (BE ACTIVE ON PROFILE)  Tweet #NEBL  NEXT GEM ON BLA…</td><td>262</td><td>None</td><td>None</td><td>Thu Nov 24 23:59:26 +0000 2022</td><td>0.5318999886512756</td><td>1</td></tr><tr><td>1595930122682478592</td><td>Fuck You I Quit</td><td>fuckyouiquit</td><td>This is the only Black Friday ad you need to see https://t.co/7Hj3CR4YlS</td><td>272510</td><td>Corporate Accounts Payable</td><td>None</td><td>Thu Nov 24 23:59:26 +0000 2022</td><td>0.0</td><td>1</td></tr><tr><td>1595930123337105417</td><td>kumiii🐻 • freetag</td><td>chunvrwin</td><td>RT @jihanicorn: $150 | 2.250.000 IDR • 24 Hours 💜  - RT &amp; Follow @CryptoCoinCoach + @NeblioTeam  ____________________________ (BE ACTIVE ON…</td><td>53</td><td>🍀</td><td>None</td><td>Thu Nov 24 23:59:27 +0000 2022</td><td>0.8240000009536743</td><td>1</td></tr><tr><td>1595930124075294722</td><td>D_Adrian</td><td>DexaWinWin</td><td>RT @HeiraCrypto: $100 ~ Ends in 24 hrs.   🔶 RT - Follow:    @CryptoCoinCoach  +  @NeblioTeam  (BE ACTIVE ON PROFILE)  Tweet #NEBL  NEXT GEM…</td><td>45</td><td>None</td><td>None</td><td>Thu Nov 24 23:59:27 +0000 2022</td><td>0.7034000158309937</td><td>1</td></tr></tbody></table></div>"]}}],"execution_count":0},{"cell_type":"code","source":["# select the sentiment and tweet column for the purpose of this sentiment analysis\ntweets=df1.select('sentiment','tweet')\ndisplay(tweets.head(10))"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"0b46e6da-51eb-4e74-b8f3-aa2178bb9303","inputWidgets":{},"title":""}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"overflow":false,"datasetInfos":[],"data":[[1,"RT @ProgIntl: BREAKING: Tomorrow, Amazon workers at 18 warehouses are going on strike in France and Germany to #MakeAmazonPay, with many ot…"],[1,"RT @ChiefElrond: $50 | 24 Hours   🥏 RT &amp; Follow:    @CryptoCoinCoach   +  @NeblioTeam  (BE ACTIVE ON PROFILE)  Tweet #NEBL  NEXT GEM ON BLA…"],[1,"This is the only Black Friday ad you need to see https://t.co/7Hj3CR4YlS"],[1,"RT @jihanicorn: $150 | 2.250.000 IDR • 24 Hours 💜  - RT &amp; Follow @CryptoCoinCoach + @NeblioTeam  ____________________________ (BE ACTIVE ON…"],[1,"RT @HeiraCrypto: $100 ~ Ends in 24 hrs.   🔶 RT - Follow:    @CryptoCoinCoach  +  @NeblioTeam  (BE ACTIVE ON PROFILE)  Tweet #NEBL  NEXT GEM…"],[1,"RT @May7ven: $50 — 24 Hours —   ➖RT &amp; Follow:    @CryptoCoinCoach   +   @NeblioTeam  (BE ACTIVE ON PROFILE)  Tweet #NEBL  NEXT GEM ON BLACK…"],[1,"RT @Beast_Cryptox: 💰 $70 ~ 24 HOURS 🐄🦥  ➖ RT &amp; Follow  @CryptoCoinCoach @NeblioTeam ----------------- (BE ACTIVE ON PROFILE)  Tweet - #NEBL…"],[1,"RT @GFuelEnergy: 💛 𝗟𝗜𝗞𝗘 + 𝗥𝗧 to win a #BanjoKazooie x #GFUEL \"HONEY BERRY\" Tub!!! Picking 2 winners tomorrow bc we just RESTOCKED these bab…"],[1,"@ThisIsKyleR Working for doubletime and a half is MY tradition. Of course, the sweet paycheck gets gobbled up on Bl… https://t.co/kaDhOvmW5n"],[1,"If y’all don’t get y’all grandma a tv for Black Friday"]],"plotOptions":{"displayType":"table","customPlotOptions":{},"pivotColumns":null,"pivotAggregation":null,"xColumns":null,"yColumns":null},"columnCustomDisplayInfos":{},"aggType":"","isJsonSchema":true,"removedWidgets":[],"aggSchema":[],"schema":[{"name":"sentiment","type":"\"long\"","metadata":"{}"},{"name":"tweet","type":"\"string\"","metadata":"{}"}],"aggError":"","aggData":[],"addedWidgets":{},"metadata":{},"dbfsResultPath":null,"type":"table","aggOverflow":false,"aggSeriesLimitReached":false,"arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n","  .table-result-container {\n","    max-height: 300px;\n","    overflow: auto;\n","  }\n","  table, th, td {\n","    border: 1px solid black;\n","    border-collapse: collapse;\n","  }\n","  th, td {\n","    padding: 5px;\n","  }\n","  th {\n","    text-align: left;\n","  }\n","</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>sentiment</th><th>tweet</th></tr></thead><tbody><tr><td>1</td><td>RT @ProgIntl: BREAKING: Tomorrow, Amazon workers at 18 warehouses are going on strike in France and Germany to #MakeAmazonPay, with many ot…</td></tr><tr><td>1</td><td>RT @ChiefElrond: $50 | 24 Hours   🥏 RT &amp; Follow:    @CryptoCoinCoach   +  @NeblioTeam  (BE ACTIVE ON PROFILE)  Tweet #NEBL  NEXT GEM ON BLA…</td></tr><tr><td>1</td><td>This is the only Black Friday ad you need to see https://t.co/7Hj3CR4YlS</td></tr><tr><td>1</td><td>RT @jihanicorn: $150 | 2.250.000 IDR • 24 Hours 💜  - RT &amp; Follow @CryptoCoinCoach + @NeblioTeam  ____________________________ (BE ACTIVE ON…</td></tr><tr><td>1</td><td>RT @HeiraCrypto: $100 ~ Ends in 24 hrs.   🔶 RT - Follow:    @CryptoCoinCoach  +  @NeblioTeam  (BE ACTIVE ON PROFILE)  Tweet #NEBL  NEXT GEM…</td></tr><tr><td>1</td><td>RT @May7ven: $50 — 24 Hours —   ➖RT &amp; Follow:    @CryptoCoinCoach   +   @NeblioTeam  (BE ACTIVE ON PROFILE)  Tweet #NEBL  NEXT GEM ON BLACK…</td></tr><tr><td>1</td><td>RT @Beast_Cryptox: 💰 $70 ~ 24 HOURS 🐄🦥  ➖ RT &amp; Follow  @CryptoCoinCoach @NeblioTeam ----------------- (BE ACTIVE ON PROFILE)  Tweet - #NEBL…</td></tr><tr><td>1</td><td>RT @GFuelEnergy: 💛 𝗟𝗜𝗞𝗘 + 𝗥𝗧 to win a #BanjoKazooie x #GFUEL \"HONEY BERRY\" Tub!!! Picking 2 winners tomorrow bc we just RESTOCKED these bab…</td></tr><tr><td>1</td><td>@ThisIsKyleR Working for doubletime and a half is MY tradition. Of course, the sweet paycheck gets gobbled up on Bl… https://t.co/kaDhOvmW5n</td></tr><tr><td>1</td><td>If y’all don’t get y’all grandma a tv for Black Friday</td></tr></tbody></table></div>"]}}],"execution_count":0},{"cell_type":"markdown","source":["#### Text Cleaning Preprocessing\n\n`pyspark.sql.functions.regexp_replace` is used to process the text\n\n1. Remove URLs such as `http://cnn.com`\n2. Remove special characters\n3. Substituting multiple spaces with single space\n4. Lowercase all text\n5. Trim the leading/trailing whitespaces"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"200e3203-8b9e-42e5-ac8c-4961ab16dcee","inputWidgets":{},"title":""}}},{"cell_type":"code","source":["# clean the tweets as mentioned above\ntweets_clean = tweets.withColumn('tweet', F.regexp_replace('tweet', r\"http\\S+\", \"\")) \\\n                    .withColumn('tweet', F.regexp_replace('tweet', r\"[^a-zA-z]\", \" \")) \\\n                    .withColumn('tweet', F.regexp_replace('tweet', r\"\\s+\", \" \")) \\\n                    .withColumn('tweet', F.lower('tweet')) \\\n                    .withColumn('tweet', F.trim('tweet')) \ndisplay(tweets_clean.head(5))"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"78fd095a-f50d-49fd-a653-9a0f1c701a45","inputWidgets":{},"title":""}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"overflow":false,"datasetInfos":[],"data":[[1,"rt progintl breaking tomorrow amazon workers at warehouses are going on strike in france and germany to makeamazonpay with many ot"],[1,"rt chiefelrond hours rt amp follow cryptocoincoach neblioteam be active on profile tweet nebl next gem on bla"],[1,"this is the only black friday ad you need to see"],[1,"rt jihanicorn idr hours rt amp follow cryptocoincoach neblioteam ____________________________ be active on"],[1,"rt heiracrypto ends in hrs rt follow cryptocoincoach neblioteam be active on profile tweet nebl next gem"]],"plotOptions":{"displayType":"table","customPlotOptions":{},"pivotColumns":null,"pivotAggregation":null,"xColumns":null,"yColumns":null},"columnCustomDisplayInfos":{},"aggType":"","isJsonSchema":true,"removedWidgets":[],"aggSchema":[],"schema":[{"name":"sentiment","type":"\"long\"","metadata":"{}"},{"name":"tweet","type":"\"string\"","metadata":"{}"}],"aggError":"","aggData":[],"addedWidgets":{},"metadata":{},"dbfsResultPath":null,"type":"table","aggOverflow":false,"aggSeriesLimitReached":false,"arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n","  .table-result-container {\n","    max-height: 300px;\n","    overflow: auto;\n","  }\n","  table, th, td {\n","    border: 1px solid black;\n","    border-collapse: collapse;\n","  }\n","  th, td {\n","    padding: 5px;\n","  }\n","  th {\n","    text-align: left;\n","  }\n","</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>sentiment</th><th>tweet</th></tr></thead><tbody><tr><td>1</td><td>rt progintl breaking tomorrow amazon workers at warehouses are going on strike in france and germany to makeamazonpay with many ot</td></tr><tr><td>1</td><td>rt chiefelrond hours rt amp follow cryptocoincoach neblioteam be active on profile tweet nebl next gem on bla</td></tr><tr><td>1</td><td>this is the only black friday ad you need to see</td></tr><tr><td>1</td><td>rt jihanicorn idr hours rt amp follow cryptocoincoach neblioteam ____________________________ be active on</td></tr><tr><td>1</td><td>rt heiracrypto ends in hrs rt follow cryptocoincoach neblioteam be active on profile tweet nebl next gem</td></tr></tbody></table></div>"]}}],"execution_count":0},{"cell_type":"markdown","source":["#### Feature Transformer: Tokenizer"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"5cf4d106-737f-4e5b-bc4e-559a89bebf70","inputWidgets":{},"title":""}}},{"cell_type":"code","source":["# Tokenize the tweets\ntokenizer = Tokenizer(inputCol=\"tweet\", outputCol=\"tokens\") \ntweets_tokenized = tokenizer.transform(tweets_clean)\n\ndisplay(tweets_tokenized.head(5))"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"55023cad-2166-437d-9811-185fe7b34cdb","inputWidgets":{},"title":""}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"overflow":false,"datasetInfos":[],"data":[[1,"rt progintl breaking tomorrow amazon workers at warehouses are going on strike in france and germany to makeamazonpay with many ot",["rt","progintl","breaking","tomorrow","amazon","workers","at","warehouses","are","going","on","strike","in","france","and","germany","to","makeamazonpay","with","many","ot"]],[1,"rt chiefelrond hours rt amp follow cryptocoincoach neblioteam be active on profile tweet nebl next gem on bla",["rt","chiefelrond","hours","rt","amp","follow","cryptocoincoach","neblioteam","be","active","on","profile","tweet","nebl","next","gem","on","bla"]],[1,"this is the only black friday ad you need to see",["this","is","the","only","black","friday","ad","you","need","to","see"]],[1,"rt jihanicorn idr hours rt amp follow cryptocoincoach neblioteam ____________________________ be active on",["rt","jihanicorn","idr","hours","rt","amp","follow","cryptocoincoach","neblioteam","____________________________","be","active","on"]],[1,"rt heiracrypto ends in hrs rt follow cryptocoincoach neblioteam be active on profile tweet nebl next gem",["rt","heiracrypto","ends","in","hrs","rt","follow","cryptocoincoach","neblioteam","be","active","on","profile","tweet","nebl","next","gem"]]],"plotOptions":{"displayType":"table","customPlotOptions":{},"pivotColumns":null,"pivotAggregation":null,"xColumns":null,"yColumns":null},"columnCustomDisplayInfos":{},"aggType":"","isJsonSchema":true,"removedWidgets":[],"aggSchema":[],"schema":[{"name":"sentiment","type":"\"long\"","metadata":"{}"},{"name":"tweet","type":"\"string\"","metadata":"{}"},{"name":"tokens","type":"{\"type\":\"array\",\"elementType\":\"string\",\"containsNull\":true}","metadata":"{}"}],"aggError":"","aggData":[],"addedWidgets":{},"metadata":{},"dbfsResultPath":null,"type":"table","aggOverflow":false,"aggSeriesLimitReached":false,"arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n","  .table-result-container {\n","    max-height: 300px;\n","    overflow: auto;\n","  }\n","  table, th, td {\n","    border: 1px solid black;\n","    border-collapse: collapse;\n","  }\n","  th, td {\n","    padding: 5px;\n","  }\n","  th {\n","    text-align: left;\n","  }\n","</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>sentiment</th><th>tweet</th><th>tokens</th></tr></thead><tbody><tr><td>1</td><td>rt progintl breaking tomorrow amazon workers at warehouses are going on strike in france and germany to makeamazonpay with many ot</td><td>List(rt, progintl, breaking, tomorrow, amazon, workers, at, warehouses, are, going, on, strike, in, france, and, germany, to, makeamazonpay, with, many, ot)</td></tr><tr><td>1</td><td>rt chiefelrond hours rt amp follow cryptocoincoach neblioteam be active on profile tweet nebl next gem on bla</td><td>List(rt, chiefelrond, hours, rt, amp, follow, cryptocoincoach, neblioteam, be, active, on, profile, tweet, nebl, next, gem, on, bla)</td></tr><tr><td>1</td><td>this is the only black friday ad you need to see</td><td>List(this, is, the, only, black, friday, ad, you, need, to, see)</td></tr><tr><td>1</td><td>rt jihanicorn idr hours rt amp follow cryptocoincoach neblioteam ____________________________ be active on</td><td>List(rt, jihanicorn, idr, hours, rt, amp, follow, cryptocoincoach, neblioteam, ____________________________, be, active, on)</td></tr><tr><td>1</td><td>rt heiracrypto ends in hrs rt follow cryptocoincoach neblioteam be active on profile tweet nebl next gem</td><td>List(rt, heiracrypto, ends, in, hrs, rt, follow, cryptocoincoach, neblioteam, be, active, on, profile, tweet, nebl, next, gem)</td></tr></tbody></table></div>"]}}],"execution_count":0},{"cell_type":"markdown","source":["#### Feature Transformer: Stopword Removal"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"1c8d62d2-00ab-44eb-b924-55ad2afa9605","inputWidgets":{},"title":""}}},{"cell_type":"code","source":["# Remove stopword\nstopword_remover = StopWordsRemover(inputCol=\"tokens\", outputCol=\"filtered\")\ntweets_stopword = stopword_remover.transform(tweets_tokenized)\n\ndisplay(tweets_stopword.head(5))"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"178f9ce3-c794-4368-94dd-94c97e2a6e8d","inputWidgets":{},"title":""}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"overflow":false,"datasetInfos":[],"data":[[1,"rt progintl breaking tomorrow amazon workers at warehouses are going on strike in france and germany to makeamazonpay with many ot",["rt","progintl","breaking","tomorrow","amazon","workers","at","warehouses","are","going","on","strike","in","france","and","germany","to","makeamazonpay","with","many","ot"],["rt","progintl","breaking","tomorrow","amazon","workers","warehouses","going","strike","france","germany","makeamazonpay","many","ot"]],[1,"rt chiefelrond hours rt amp follow cryptocoincoach neblioteam be active on profile tweet nebl next gem on bla",["rt","chiefelrond","hours","rt","amp","follow","cryptocoincoach","neblioteam","be","active","on","profile","tweet","nebl","next","gem","on","bla"],["rt","chiefelrond","hours","rt","amp","follow","cryptocoincoach","neblioteam","active","profile","tweet","nebl","next","gem","bla"]],[1,"this is the only black friday ad you need to see",["this","is","the","only","black","friday","ad","you","need","to","see"],["black","friday","ad","need","see"]],[1,"rt jihanicorn idr hours rt amp follow cryptocoincoach neblioteam ____________________________ be active on",["rt","jihanicorn","idr","hours","rt","amp","follow","cryptocoincoach","neblioteam","____________________________","be","active","on"],["rt","jihanicorn","idr","hours","rt","amp","follow","cryptocoincoach","neblioteam","____________________________","active"]],[1,"rt heiracrypto ends in hrs rt follow cryptocoincoach neblioteam be active on profile tweet nebl next gem",["rt","heiracrypto","ends","in","hrs","rt","follow","cryptocoincoach","neblioteam","be","active","on","profile","tweet","nebl","next","gem"],["rt","heiracrypto","ends","hrs","rt","follow","cryptocoincoach","neblioteam","active","profile","tweet","nebl","next","gem"]]],"plotOptions":{"displayType":"table","customPlotOptions":{},"pivotColumns":null,"pivotAggregation":null,"xColumns":null,"yColumns":null},"columnCustomDisplayInfos":{},"aggType":"","isJsonSchema":true,"removedWidgets":[],"aggSchema":[],"schema":[{"name":"sentiment","type":"\"long\"","metadata":"{}"},{"name":"tweet","type":"\"string\"","metadata":"{}"},{"name":"tokens","type":"{\"type\":\"array\",\"elementType\":\"string\",\"containsNull\":true}","metadata":"{}"},{"name":"filtered","type":"{\"type\":\"array\",\"elementType\":\"string\",\"containsNull\":true}","metadata":"{}"}],"aggError":"","aggData":[],"addedWidgets":{},"metadata":{},"dbfsResultPath":null,"type":"table","aggOverflow":false,"aggSeriesLimitReached":false,"arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n","  .table-result-container {\n","    max-height: 300px;\n","    overflow: auto;\n","  }\n","  table, th, td {\n","    border: 1px solid black;\n","    border-collapse: collapse;\n","  }\n","  th, td {\n","    padding: 5px;\n","  }\n","  th {\n","    text-align: left;\n","  }\n","</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>sentiment</th><th>tweet</th><th>tokens</th><th>filtered</th></tr></thead><tbody><tr><td>1</td><td>rt progintl breaking tomorrow amazon workers at warehouses are going on strike in france and germany to makeamazonpay with many ot</td><td>List(rt, progintl, breaking, tomorrow, amazon, workers, at, warehouses, are, going, on, strike, in, france, and, germany, to, makeamazonpay, with, many, ot)</td><td>List(rt, progintl, breaking, tomorrow, amazon, workers, warehouses, going, strike, france, germany, makeamazonpay, many, ot)</td></tr><tr><td>1</td><td>rt chiefelrond hours rt amp follow cryptocoincoach neblioteam be active on profile tweet nebl next gem on bla</td><td>List(rt, chiefelrond, hours, rt, amp, follow, cryptocoincoach, neblioteam, be, active, on, profile, tweet, nebl, next, gem, on, bla)</td><td>List(rt, chiefelrond, hours, rt, amp, follow, cryptocoincoach, neblioteam, active, profile, tweet, nebl, next, gem, bla)</td></tr><tr><td>1</td><td>this is the only black friday ad you need to see</td><td>List(this, is, the, only, black, friday, ad, you, need, to, see)</td><td>List(black, friday, ad, need, see)</td></tr><tr><td>1</td><td>rt jihanicorn idr hours rt amp follow cryptocoincoach neblioteam ____________________________ be active on</td><td>List(rt, jihanicorn, idr, hours, rt, amp, follow, cryptocoincoach, neblioteam, ____________________________, be, active, on)</td><td>List(rt, jihanicorn, idr, hours, rt, amp, follow, cryptocoincoach, neblioteam, ____________________________, active)</td></tr><tr><td>1</td><td>rt heiracrypto ends in hrs rt follow cryptocoincoach neblioteam be active on profile tweet nebl next gem</td><td>List(rt, heiracrypto, ends, in, hrs, rt, follow, cryptocoincoach, neblioteam, be, active, on, profile, tweet, nebl, next, gem)</td><td>List(rt, heiracrypto, ends, hrs, rt, follow, cryptocoincoach, neblioteam, active, profile, tweet, nebl, next, gem)</td></tr></tbody></table></div>"]}}],"execution_count":0},{"cell_type":"markdown","source":["#### Feature Transformer: CountVectorizer (TF - Term Frequency)"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"a094c370-9add-404b-af2b-00d35558bed4","inputWidgets":{},"title":""}}},{"cell_type":"code","source":["# Apply count vectorizer\ncv = CountVectorizer(vocabSize=2**16, inputCol=\"filtered\", outputCol='cv')\ncv_model = cv.fit(tweets_stopword)\ntweets_cv = cv_model.transform(tweets_stopword)\n\ndisplay(tweets_cv.head(5))"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"f2634910-fb76-40c9-9c12-2b98c393bfef","inputWidgets":{},"title":""}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"overflow":false,"datasetInfos":[],"data":[[1,"rt progintl breaking tomorrow amazon workers at warehouses are going on strike in france and germany to makeamazonpay with many ot",["rt","progintl","breaking","tomorrow","amazon","workers","at","warehouses","are","going","on","strike","in","france","and","germany","to","makeamazonpay","with","many","ot"],["rt","progintl","breaking","tomorrow","amazon","workers","warehouses","going","strike","france","germany","makeamazonpay","many","ot"],{"vectorType":"sparse","length":37185,"indices":[0,20,41,57,252,271,372,552,589,790,797,815,823,847],"values":[1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0]}],[1,"rt chiefelrond hours rt amp follow cryptocoincoach neblioteam be active on profile tweet nebl next gem on bla",["rt","chiefelrond","hours","rt","amp","follow","cryptocoincoach","neblioteam","be","active","on","profile","tweet","nebl","next","gem","on","bla"],["rt","chiefelrond","hours","rt","amp","follow","cryptocoincoach","neblioteam","active","profile","tweet","nebl","next","gem","bla"],{"vectorType":"sparse","length":37185,"indices":[0,3,4,5,6,7,8,9,10,11,12,13,85,88],"values":[2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0]}],[1,"this is the only black friday ad you need to see",["this","is","the","only","black","friday","ad","you","need","to","see"],["black","friday","ad","need","see"],{"vectorType":"sparse","length":37185,"indices":[1,2,129,150,312],"values":[1.0,1.0,1.0,1.0,1.0]}],[1,"rt jihanicorn idr hours rt amp follow cryptocoincoach neblioteam ____________________________ be active on",["rt","jihanicorn","idr","hours","rt","amp","follow","cryptocoincoach","neblioteam","____________________________","be","active","on"],["rt","jihanicorn","idr","hours","rt","amp","follow","cryptocoincoach","neblioteam","____________________________","active"],{"vectorType":"sparse","length":37185,"indices":[0,3,5,6,7,9,10,71,72,73],"values":[2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0]}],[1,"rt heiracrypto ends in hrs rt follow cryptocoincoach neblioteam be active on profile tweet nebl next gem",["rt","heiracrypto","ends","in","hrs","rt","follow","cryptocoincoach","neblioteam","be","active","on","profile","tweet","nebl","next","gem"],["rt","heiracrypto","ends","hrs","rt","follow","cryptocoincoach","neblioteam","active","profile","tweet","nebl","next","gem"],{"vectorType":"sparse","length":37185,"indices":[0,3,4,5,6,7,8,11,12,13,36,45,48],"values":[2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0]}]],"plotOptions":{"displayType":"table","customPlotOptions":{},"pivotColumns":null,"pivotAggregation":null,"xColumns":null,"yColumns":null},"columnCustomDisplayInfos":{},"aggType":"","isJsonSchema":true,"removedWidgets":[],"aggSchema":[],"schema":[{"name":"sentiment","type":"\"long\"","metadata":"{}"},{"name":"tweet","type":"\"string\"","metadata":"{}"},{"name":"tokens","type":"{\"type\":\"array\",\"elementType\":\"string\",\"containsNull\":true}","metadata":"{}"},{"name":"filtered","type":"{\"type\":\"array\",\"elementType\":\"string\",\"containsNull\":true}","metadata":"{}"},{"name":"cv","type":"{\"type\":\"udt\",\"class\":\"org.apache.spark.ml.linalg.VectorUDT\",\"pyClass\":\"pyspark.ml.linalg.VectorUDT\",\"sqlType\":{\"type\":\"struct\",\"fields\":[{\"name\":\"type\",\"type\":\"byte\",\"nullable\":false,\"metadata\":{}},{\"name\":\"size\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"indices\",\"type\":{\"type\":\"array\",\"elementType\":\"integer\",\"containsNull\":false},\"nullable\":true,\"metadata\":{}},{\"name\":\"values\",\"type\":{\"type\":\"array\",\"elementType\":\"double\",\"containsNull\":false},\"nullable\":true,\"metadata\":{}}]}}","metadata":"{}"}],"aggError":"","aggData":[],"addedWidgets":{},"metadata":{},"dbfsResultPath":null,"type":"table","aggOverflow":false,"aggSeriesLimitReached":false,"arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n","  .table-result-container {\n","    max-height: 300px;\n","    overflow: auto;\n","  }\n","  table, th, td {\n","    border: 1px solid black;\n","    border-collapse: collapse;\n","  }\n","  th, td {\n","    padding: 5px;\n","  }\n","  th {\n","    text-align: left;\n","  }\n","</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>sentiment</th><th>tweet</th><th>tokens</th><th>filtered</th><th>cv</th></tr></thead><tbody><tr><td>1</td><td>rt progintl breaking tomorrow amazon workers at warehouses are going on strike in france and germany to makeamazonpay with many ot</td><td>List(rt, progintl, breaking, tomorrow, amazon, workers, at, warehouses, are, going, on, strike, in, france, and, germany, to, makeamazonpay, with, many, ot)</td><td>List(rt, progintl, breaking, tomorrow, amazon, workers, warehouses, going, strike, france, germany, makeamazonpay, many, ot)</td><td>Map(vectorType -> sparse, length -> 37185, indices -> List(0, 20, 41, 57, 252, 271, 372, 552, 589, 790, 797, 815, 823, 847), values -> List(1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0))</td></tr><tr><td>1</td><td>rt chiefelrond hours rt amp follow cryptocoincoach neblioteam be active on profile tweet nebl next gem on bla</td><td>List(rt, chiefelrond, hours, rt, amp, follow, cryptocoincoach, neblioteam, be, active, on, profile, tweet, nebl, next, gem, on, bla)</td><td>List(rt, chiefelrond, hours, rt, amp, follow, cryptocoincoach, neblioteam, active, profile, tweet, nebl, next, gem, bla)</td><td>Map(vectorType -> sparse, length -> 37185, indices -> List(0, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 85, 88), values -> List(2.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0))</td></tr><tr><td>1</td><td>this is the only black friday ad you need to see</td><td>List(this, is, the, only, black, friday, ad, you, need, to, see)</td><td>List(black, friday, ad, need, see)</td><td>Map(vectorType -> sparse, length -> 37185, indices -> List(1, 2, 129, 150, 312), values -> List(1.0, 1.0, 1.0, 1.0, 1.0))</td></tr><tr><td>1</td><td>rt jihanicorn idr hours rt amp follow cryptocoincoach neblioteam ____________________________ be active on</td><td>List(rt, jihanicorn, idr, hours, rt, amp, follow, cryptocoincoach, neblioteam, ____________________________, be, active, on)</td><td>List(rt, jihanicorn, idr, hours, rt, amp, follow, cryptocoincoach, neblioteam, ____________________________, active)</td><td>Map(vectorType -> sparse, length -> 37185, indices -> List(0, 3, 5, 6, 7, 9, 10, 71, 72, 73), values -> List(2.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0))</td></tr><tr><td>1</td><td>rt heiracrypto ends in hrs rt follow cryptocoincoach neblioteam be active on profile tweet nebl next gem</td><td>List(rt, heiracrypto, ends, in, hrs, rt, follow, cryptocoincoach, neblioteam, be, active, on, profile, tweet, nebl, next, gem)</td><td>List(rt, heiracrypto, ends, hrs, rt, follow, cryptocoincoach, neblioteam, active, profile, tweet, nebl, next, gem)</td><td>Map(vectorType -> sparse, length -> 37185, indices -> List(0, 3, 4, 5, 6, 7, 8, 11, 12, 13, 36, 45, 48), values -> List(2.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0))</td></tr></tbody></table></div>"]}}],"execution_count":0},{"cell_type":"markdown","source":["#### Feature Transformer: TF-IDF Vectorization"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"1d8af630-c76a-4b61-8117-c2b3b2759ff7","inputWidgets":{},"title":""}}},{"cell_type":"code","source":["# TF-IDF Vectorization\nidf = IDF(inputCol='cv', outputCol=\"features\", minDocFreq=5) \nidf_model = idf.fit(tweets_cv)\ntweets_idf = idf_model.transform(tweets_cv)\n\ndisplay(tweets_idf.head(5)) # The dataframe is now ready for the following machine learning stage."],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"5246a0f1-68e8-4275-af20-469ba71b797b","inputWidgets":{},"title":""}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"overflow":false,"datasetInfos":[],"data":[[1,"rt progintl breaking tomorrow amazon workers at warehouses are going on strike in france and germany to makeamazonpay with many ot",["rt","progintl","breaking","tomorrow","amazon","workers","at","warehouses","are","going","on","strike","in","france","and","germany","to","makeamazonpay","with","many","ot"],["rt","progintl","breaking","tomorrow","amazon","workers","warehouses","going","strike","france","germany","makeamazonpay","many","ot"],{"vectorType":"sparse","length":37185,"indices":[0,20,41,57,252,271,372,552,589,790,797,815,823,847],"values":[1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0]},{"vectorType":"sparse","length":37185,"indices":[0,20,41,57,252,271,372,552,589,790,797,815,823,847],"values":[0.3676543272010515,3.1175924090018903,3.5883125702634744,3.9732819270253423,5.352113566943643,5.411461123514245,5.751094168772547,6.269334098977765,6.3317044297042235,6.705498193974736,6.727317241369375,6.749622998883674,6.7647748039042765,6.780159722743756]}],[1,"rt chiefelrond hours rt amp follow cryptocoincoach neblioteam be active on profile tweet nebl next gem on bla",["rt","chiefelrond","hours","rt","amp","follow","cryptocoincoach","neblioteam","be","active","on","profile","tweet","nebl","next","gem","on","bla"],["rt","chiefelrond","hours","rt","amp","follow","cryptocoincoach","neblioteam","active","profile","tweet","nebl","next","gem","bla"],{"vectorType":"sparse","length":37185,"indices":[0,3,4,5,6,7,8,9,10,11,12,13,85,88],"values":[2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0]},{"vectorType":"sparse","length":37185,"indices":[0,3,4,5,6,7,8,9,10,11,12,13,85,88],"values":[0.735308654402103,1.3188603556373257,1.3732660863895567,1.5056109771931794,1.5058888597300504,1.5203253570295348,1.540835557564483,1.5713939547037072,1.5846222252600537,1.6124985947883086,1.6561496019170256,1.6958706016840448,4.103608188700917,4.185831048958749]}],[1,"this is the only black friday ad you need to see",["this","is","the","only","black","friday","ad","you","need","to","see"],["black","friday","ad","need","see"],{"vectorType":"sparse","length":37185,"indices":[1,2,129,150,312],"values":[1.0,1.0,1.0,1.0,1.0]},{"vectorType":"sparse","length":37185,"indices":[1,2,129,150,312],"values":[0.4434163164512347,0.46447935985153693,4.599435736889472,4.7272293066122515,5.578515208177411]}],[1,"rt jihanicorn idr hours rt amp follow cryptocoincoach neblioteam ____________________________ be active on",["rt","jihanicorn","idr","hours","rt","amp","follow","cryptocoincoach","neblioteam","____________________________","be","active","on"],["rt","jihanicorn","idr","hours","rt","amp","follow","cryptocoincoach","neblioteam","____________________________","active"],{"vectorType":"sparse","length":37185,"indices":[0,3,5,6,7,9,10,71,72,73],"values":[2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0]},{"vectorType":"sparse","length":37185,"indices":[0,3,5,6,7,9,10,71,72,73],"values":[0.735308654402103,1.3188603556373257,1.5056109771931794,1.5058888597300504,1.5203253570295348,1.5713939547037072,1.5846222252600537,3.987426434411507,3.9979277242321705,3.9984076858444557]}],[1,"rt heiracrypto ends in hrs rt follow cryptocoincoach neblioteam be active on profile tweet nebl next gem",["rt","heiracrypto","ends","in","hrs","rt","follow","cryptocoincoach","neblioteam","be","active","on","profile","tweet","nebl","next","gem"],["rt","heiracrypto","ends","hrs","rt","follow","cryptocoincoach","neblioteam","active","profile","tweet","nebl","next","gem"],{"vectorType":"sparse","length":37185,"indices":[0,3,4,5,6,7,8,11,12,13,36,45,48],"values":[2.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0]},{"vectorType":"sparse","length":37185,"indices":[0,3,4,5,6,7,8,11,12,13,36,45,48],"values":[0.735308654402103,1.3188603556373257,1.3732660863895567,1.5056109771931794,1.5058888597300504,1.5203253570295348,1.540835557564483,1.6124985947883086,1.6561496019170256,1.6958706016840448,3.449340446201889,3.61609213437055,3.6466897990038367]}]],"plotOptions":{"displayType":"table","customPlotOptions":{},"pivotColumns":null,"pivotAggregation":null,"xColumns":null,"yColumns":null},"columnCustomDisplayInfos":{},"aggType":"","isJsonSchema":true,"removedWidgets":[],"aggSchema":[],"schema":[{"name":"sentiment","type":"\"long\"","metadata":"{}"},{"name":"tweet","type":"\"string\"","metadata":"{}"},{"name":"tokens","type":"{\"type\":\"array\",\"elementType\":\"string\",\"containsNull\":true}","metadata":"{}"},{"name":"filtered","type":"{\"type\":\"array\",\"elementType\":\"string\",\"containsNull\":true}","metadata":"{}"},{"name":"cv","type":"{\"type\":\"udt\",\"class\":\"org.apache.spark.ml.linalg.VectorUDT\",\"pyClass\":\"pyspark.ml.linalg.VectorUDT\",\"sqlType\":{\"type\":\"struct\",\"fields\":[{\"name\":\"type\",\"type\":\"byte\",\"nullable\":false,\"metadata\":{}},{\"name\":\"size\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"indices\",\"type\":{\"type\":\"array\",\"elementType\":\"integer\",\"containsNull\":false},\"nullable\":true,\"metadata\":{}},{\"name\":\"values\",\"type\":{\"type\":\"array\",\"elementType\":\"double\",\"containsNull\":false},\"nullable\":true,\"metadata\":{}}]}}","metadata":"{}"},{"name":"features","type":"{\"type\":\"udt\",\"class\":\"org.apache.spark.ml.linalg.VectorUDT\",\"pyClass\":\"pyspark.ml.linalg.VectorUDT\",\"sqlType\":{\"type\":\"struct\",\"fields\":[{\"name\":\"type\",\"type\":\"byte\",\"nullable\":false,\"metadata\":{}},{\"name\":\"size\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"indices\",\"type\":{\"type\":\"array\",\"elementType\":\"integer\",\"containsNull\":false},\"nullable\":true,\"metadata\":{}},{\"name\":\"values\",\"type\":{\"type\":\"array\",\"elementType\":\"double\",\"containsNull\":false},\"nullable\":true,\"metadata\":{}}]}}","metadata":"{}"}],"aggError":"","aggData":[],"addedWidgets":{},"metadata":{},"dbfsResultPath":null,"type":"table","aggOverflow":false,"aggSeriesLimitReached":false,"arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n","  .table-result-container {\n","    max-height: 300px;\n","    overflow: auto;\n","  }\n","  table, th, td {\n","    border: 1px solid black;\n","    border-collapse: collapse;\n","  }\n","  th, td {\n","    padding: 5px;\n","  }\n","  th {\n","    text-align: left;\n","  }\n","</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>sentiment</th><th>tweet</th><th>tokens</th><th>filtered</th><th>cv</th><th>features</th></tr></thead><tbody><tr><td>1</td><td>rt progintl breaking tomorrow amazon workers at warehouses are going on strike in france and germany to makeamazonpay with many ot</td><td>List(rt, progintl, breaking, tomorrow, amazon, workers, at, warehouses, are, going, on, strike, in, france, and, germany, to, makeamazonpay, with, many, ot)</td><td>List(rt, progintl, breaking, tomorrow, amazon, workers, warehouses, going, strike, france, germany, makeamazonpay, many, ot)</td><td>Map(vectorType -> sparse, length -> 37185, indices -> List(0, 20, 41, 57, 252, 271, 372, 552, 589, 790, 797, 815, 823, 847), values -> List(1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0))</td><td>Map(vectorType -> sparse, length -> 37185, indices -> List(0, 20, 41, 57, 252, 271, 372, 552, 589, 790, 797, 815, 823, 847), values -> List(0.3676543272010515, 3.1175924090018903, 3.5883125702634744, 3.9732819270253423, 5.352113566943643, 5.411461123514245, 5.751094168772547, 6.269334098977765, 6.3317044297042235, 6.705498193974736, 6.727317241369375, 6.749622998883674, 6.7647748039042765, 6.780159722743756))</td></tr><tr><td>1</td><td>rt chiefelrond hours rt amp follow cryptocoincoach neblioteam be active on profile tweet nebl next gem on bla</td><td>List(rt, chiefelrond, hours, rt, amp, follow, cryptocoincoach, neblioteam, be, active, on, profile, tweet, nebl, next, gem, on, bla)</td><td>List(rt, chiefelrond, hours, rt, amp, follow, cryptocoincoach, neblioteam, active, profile, tweet, nebl, next, gem, bla)</td><td>Map(vectorType -> sparse, length -> 37185, indices -> List(0, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 85, 88), values -> List(2.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0))</td><td>Map(vectorType -> sparse, length -> 37185, indices -> List(0, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 85, 88), values -> List(0.735308654402103, 1.3188603556373257, 1.3732660863895567, 1.5056109771931794, 1.5058888597300504, 1.5203253570295348, 1.540835557564483, 1.5713939547037072, 1.5846222252600537, 1.6124985947883086, 1.6561496019170256, 1.6958706016840448, 4.103608188700917, 4.185831048958749))</td></tr><tr><td>1</td><td>this is the only black friday ad you need to see</td><td>List(this, is, the, only, black, friday, ad, you, need, to, see)</td><td>List(black, friday, ad, need, see)</td><td>Map(vectorType -> sparse, length -> 37185, indices -> List(1, 2, 129, 150, 312), values -> List(1.0, 1.0, 1.0, 1.0, 1.0))</td><td>Map(vectorType -> sparse, length -> 37185, indices -> List(1, 2, 129, 150, 312), values -> List(0.4434163164512347, 0.46447935985153693, 4.599435736889472, 4.7272293066122515, 5.578515208177411))</td></tr><tr><td>1</td><td>rt jihanicorn idr hours rt amp follow cryptocoincoach neblioteam ____________________________ be active on</td><td>List(rt, jihanicorn, idr, hours, rt, amp, follow, cryptocoincoach, neblioteam, ____________________________, be, active, on)</td><td>List(rt, jihanicorn, idr, hours, rt, amp, follow, cryptocoincoach, neblioteam, ____________________________, active)</td><td>Map(vectorType -> sparse, length -> 37185, indices -> List(0, 3, 5, 6, 7, 9, 10, 71, 72, 73), values -> List(2.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0))</td><td>Map(vectorType -> sparse, length -> 37185, indices -> List(0, 3, 5, 6, 7, 9, 10, 71, 72, 73), values -> List(0.735308654402103, 1.3188603556373257, 1.5056109771931794, 1.5058888597300504, 1.5203253570295348, 1.5713939547037072, 1.5846222252600537, 3.987426434411507, 3.9979277242321705, 3.9984076858444557))</td></tr><tr><td>1</td><td>rt heiracrypto ends in hrs rt follow cryptocoincoach neblioteam be active on profile tweet nebl next gem</td><td>List(rt, heiracrypto, ends, in, hrs, rt, follow, cryptocoincoach, neblioteam, be, active, on, profile, tweet, nebl, next, gem)</td><td>List(rt, heiracrypto, ends, hrs, rt, follow, cryptocoincoach, neblioteam, active, profile, tweet, nebl, next, gem)</td><td>Map(vectorType -> sparse, length -> 37185, indices -> List(0, 3, 4, 5, 6, 7, 8, 11, 12, 13, 36, 45, 48), values -> List(2.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0))</td><td>Map(vectorType -> sparse, length -> 37185, indices -> List(0, 3, 4, 5, 6, 7, 8, 11, 12, 13, 36, 45, 48), values -> List(0.735308654402103, 1.3188603556373257, 1.3732660863895567, 1.5056109771931794, 1.5058888597300504, 1.5203253570295348, 1.540835557564483, 1.6124985947883086, 1.6561496019170256, 1.6958706016840448, 3.449340446201889, 3.61609213437055, 3.6466897990038367))</td></tr></tbody></table></div>"]}}],"execution_count":0},{"cell_type":"markdown","source":["### Model Training: Logistic Regression"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"1d61ca6f-1a3a-43b6-84f3-2d73dd402de7","inputWidgets":{},"title":""}}},{"cell_type":"code","source":["# rename column 'sentiment' to 'label'\ntweets_idf = tweets_idf.withColumnRenamed(\"sentiment\", \"label\")"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"14a0da9c-7396-40d5-9673-133cadbaf98f","inputWidgets":{},"title":""}},"outputs":[],"execution_count":0},{"cell_type":"code","source":["# split the data into traning and test sets\ntrain_data, test_data = tweets_idf.randomSplit([0.7, 0.3], seed=1234)\n\nlr = LogisticRegression(maxIter=100)\n\nlr_model = lr.fit(train_data)\n\npredictions = lr_model.transform(test_data)\n\ndisplay(predictions.head(5))"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"e6f54b88-ac4c-4ee5-a992-fcb115d41c9c","inputWidgets":{},"title":""}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"overflow":false,"datasetInfos":[],"data":[[0,"a distraction from shopping in the black friday sales an epic thread combines gynaecological anatomy with vague",["a","distraction","from","shopping","in","the","black","friday","sales","an","epic","thread","combines","gynaecological","anatomy","with","vague"],["distraction","shopping","black","friday","sales","epic","thread","combines","gynaecological","anatomy","vague"],{"vectorType":"sparse","length":37185,"indices":[1,2,32,64,97,246,15471,16074,29796,30875,31219],"values":[1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0]},{"vectorType":"sparse","length":37185,"indices":[1,2,32,64,97,246,15471,16074,29796,30875,31219],"values":[0.4434163164512347,0.46447935985153693,3.4413326718080586,3.8492760959306898,4.286249796705796,5.313822653950329,0.0,0.0,0.0,0.0,0.0]},{"vectorType":"dense","length":2,"values":[-5.241514243446955,5.241514243446955]},{"vectorType":"dense","length":2,"values":[0.005264376700873776,0.9947356232991262]},1.0],[0,"a dizzy nendroid man",["a","dizzy","nendroid","man"],["dizzy","nendroid","man"],{"vectorType":"sparse","length":37185,"indices":[683,13757,36833],"values":[1.0,1.0,1.0]},{"vectorType":"sparse","length":37185,"indices":[683,13757,36833],"values":[6.527984338748885,0.0,0.0]},{"vectorType":"dense","length":2,"values":[-2.7838985014207074,2.7838985014207074]},{"vectorType":"dense","length":2,"values":[0.058200497792373386,0.9417995022076266]},1.0],[0,"adorepixsxoxo give me my black friday shopping allowance im going crazy this year findom findomme fins",["adorepixsxoxo","give","me","my","black","friday","shopping","allowance","im","going","crazy","this","year","findom","findomme","fins"],["adorepixsxoxo","give","black","friday","shopping","allowance","im","going","crazy","year","findom","findomme","fins"],{"vectorType":"sparse","length":37185,"indices":[1,2,41,64,78,108,114,398,548,3915,5535,6957,29373],"values":[1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0]},{"vectorType":"sparse","length":37185,"indices":[1,2,41,64,78,108,114,398,548,3915,5535,6957,29373],"values":[0.4434163164512347,0.46447935985153693,3.5883125702634744,3.8492760959306898,4.052661621082812,4.381559976510121,4.4924128559159735,5.884229913518515,6.283385852433415,8.749600369209263,9.242076854307058,9.560530585425592,0.0]},{"vectorType":"dense","length":2,"values":[2.0453207302256597,-2.0453207302256597]},{"vectorType":"dense","length":2,"values":[0.88547394998761,0.11452605001239002]},0.0],[0,"amazon workers across the world plan black friday strike to demand the tech giant pays fairly and ceases awful un",["amazon","workers","across","the","world","plan","black","friday","strike","to","demand","the","tech","giant","pays","fairly","and","ceases","awful","un"],["amazon","workers","across","world","plan","black","friday","strike","demand","tech","giant","pays","fairly","ceases","awful","un"],{"vectorType":"sparse","length":37185,"indices":[1,2,57,271,369,372,389,419,809,2013,2115,2202,2995,5508,6377,6576],"values":[1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0]},{"vectorType":"sparse","length":37185,"indices":[1,2,57,271,369,372,389,419,809,2013,2115,2202,2995,5508,6377,6576],"values":[0.4434163164512347,0.46447935985153693,3.9732819270253423,5.411461123514245,5.753868095655272,5.751094168772547,5.881070353228147,5.963218324837146,6.742132327154517,7.90230250882206,7.951092672991492,8.029054214461203,8.381875589083945,9.242076854307058,9.442747549769209,9.442747549769209]},{"vectorType":"dense","length":2,"values":[47.24168322895689,-47.24168322895689]},{"vectorType":"dense","length":2,"values":[1.0,0.0]},0.0],[0,"another bullshit from wbd firedavidzaslav",["another","bullshit","from","wbd","firedavidzaslav"],["another","bullshit","wbd","firedavidzaslav"],{"vectorType":"sparse","length":37185,"indices":[365,3116,21514,26094],"values":[1.0,1.0,1.0,1.0]},{"vectorType":"sparse","length":37185,"indices":[365,3116,21514,26094],"values":[5.756649738617149,8.461918296757482,0.0,0.0]},{"vectorType":"dense","length":2,"values":[27.301697931623202,-27.301697931623202]},{"vectorType":"dense","length":2,"values":[0.99999999999861,1.389999226830696E-12]},0.0]],"plotOptions":{"displayType":"table","customPlotOptions":{},"pivotColumns":null,"pivotAggregation":null,"xColumns":null,"yColumns":null},"columnCustomDisplayInfos":{},"aggType":"","isJsonSchema":true,"removedWidgets":[],"aggSchema":[],"schema":[{"name":"label","type":"\"long\"","metadata":"{}"},{"name":"tweet","type":"\"string\"","metadata":"{}"},{"name":"tokens","type":"{\"type\":\"array\",\"elementType\":\"string\",\"containsNull\":true}","metadata":"{}"},{"name":"filtered","type":"{\"type\":\"array\",\"elementType\":\"string\",\"containsNull\":true}","metadata":"{}"},{"name":"cv","type":"{\"type\":\"udt\",\"class\":\"org.apache.spark.ml.linalg.VectorUDT\",\"pyClass\":\"pyspark.ml.linalg.VectorUDT\",\"sqlType\":{\"type\":\"struct\",\"fields\":[{\"name\":\"type\",\"type\":\"byte\",\"nullable\":false,\"metadata\":{}},{\"name\":\"size\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"indices\",\"type\":{\"type\":\"array\",\"elementType\":\"integer\",\"containsNull\":false},\"nullable\":true,\"metadata\":{}},{\"name\":\"values\",\"type\":{\"type\":\"array\",\"elementType\":\"double\",\"containsNull\":false},\"nullable\":true,\"metadata\":{}}]}}","metadata":"{}"},{"name":"features","type":"{\"type\":\"udt\",\"class\":\"org.apache.spark.ml.linalg.VectorUDT\",\"pyClass\":\"pyspark.ml.linalg.VectorUDT\",\"sqlType\":{\"type\":\"struct\",\"fields\":[{\"name\":\"type\",\"type\":\"byte\",\"nullable\":false,\"metadata\":{}},{\"name\":\"size\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"indices\",\"type\":{\"type\":\"array\",\"elementType\":\"integer\",\"containsNull\":false},\"nullable\":true,\"metadata\":{}},{\"name\":\"values\",\"type\":{\"type\":\"array\",\"elementType\":\"double\",\"containsNull\":false},\"nullable\":true,\"metadata\":{}}]}}","metadata":"{}"},{"name":"rawPrediction","type":"{\"type\":\"udt\",\"class\":\"org.apache.spark.ml.linalg.VectorUDT\",\"pyClass\":\"pyspark.ml.linalg.VectorUDT\",\"sqlType\":{\"type\":\"struct\",\"fields\":[{\"name\":\"type\",\"type\":\"byte\",\"nullable\":false,\"metadata\":{}},{\"name\":\"size\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"indices\",\"type\":{\"type\":\"array\",\"elementType\":\"integer\",\"containsNull\":false},\"nullable\":true,\"metadata\":{}},{\"name\":\"values\",\"type\":{\"type\":\"array\",\"elementType\":\"double\",\"containsNull\":false},\"nullable\":true,\"metadata\":{}}]}}","metadata":"{}"},{"name":"probability","type":"{\"type\":\"udt\",\"class\":\"org.apache.spark.ml.linalg.VectorUDT\",\"pyClass\":\"pyspark.ml.linalg.VectorUDT\",\"sqlType\":{\"type\":\"struct\",\"fields\":[{\"name\":\"type\",\"type\":\"byte\",\"nullable\":false,\"metadata\":{}},{\"name\":\"size\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"indices\",\"type\":{\"type\":\"array\",\"elementType\":\"integer\",\"containsNull\":false},\"nullable\":true,\"metadata\":{}},{\"name\":\"values\",\"type\":{\"type\":\"array\",\"elementType\":\"double\",\"containsNull\":false},\"nullable\":true,\"metadata\":{}}]}}","metadata":"{}"},{"name":"prediction","type":"\"double\"","metadata":"{}"}],"aggError":"","aggData":[],"addedWidgets":{},"metadata":{},"dbfsResultPath":null,"type":"table","aggOverflow":false,"aggSeriesLimitReached":false,"arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n","  .table-result-container {\n","    max-height: 300px;\n","    overflow: auto;\n","  }\n","  table, th, td {\n","    border: 1px solid black;\n","    border-collapse: collapse;\n","  }\n","  th, td {\n","    padding: 5px;\n","  }\n","  th {\n","    text-align: left;\n","  }\n","</style><div class='table-result-container'><table class='table-result'><thead style='background-color: white'><tr><th>label</th><th>tweet</th><th>tokens</th><th>filtered</th><th>cv</th><th>features</th><th>rawPrediction</th><th>probability</th><th>prediction</th></tr></thead><tbody><tr><td>0</td><td>a distraction from shopping in the black friday sales an epic thread combines gynaecological anatomy with vague</td><td>List(a, distraction, from, shopping, in, the, black, friday, sales, an, epic, thread, combines, gynaecological, anatomy, with, vague)</td><td>List(distraction, shopping, black, friday, sales, epic, thread, combines, gynaecological, anatomy, vague)</td><td>Map(vectorType -> sparse, length -> 37185, indices -> List(1, 2, 32, 64, 97, 246, 15471, 16074, 29796, 30875, 31219), values -> List(1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0))</td><td>Map(vectorType -> sparse, length -> 37185, indices -> List(1, 2, 32, 64, 97, 246, 15471, 16074, 29796, 30875, 31219), values -> List(0.4434163164512347, 0.46447935985153693, 3.4413326718080586, 3.8492760959306898, 4.286249796705796, 5.313822653950329, 0.0, 0.0, 0.0, 0.0, 0.0))</td><td>Map(vectorType -> dense, length -> 2, values -> List(-5.241514243446955, 5.241514243446955))</td><td>Map(vectorType -> dense, length -> 2, values -> List(0.005264376700873776, 0.9947356232991262))</td><td>1.0</td></tr><tr><td>0</td><td>a dizzy nendroid man</td><td>List(a, dizzy, nendroid, man)</td><td>List(dizzy, nendroid, man)</td><td>Map(vectorType -> sparse, length -> 37185, indices -> List(683, 13757, 36833), values -> List(1.0, 1.0, 1.0))</td><td>Map(vectorType -> sparse, length -> 37185, indices -> List(683, 13757, 36833), values -> List(6.527984338748885, 0.0, 0.0))</td><td>Map(vectorType -> dense, length -> 2, values -> List(-2.7838985014207074, 2.7838985014207074))</td><td>Map(vectorType -> dense, length -> 2, values -> List(0.058200497792373386, 0.9417995022076266))</td><td>1.0</td></tr><tr><td>0</td><td>adorepixsxoxo give me my black friday shopping allowance im going crazy this year findom findomme fins</td><td>List(adorepixsxoxo, give, me, my, black, friday, shopping, allowance, im, going, crazy, this, year, findom, findomme, fins)</td><td>List(adorepixsxoxo, give, black, friday, shopping, allowance, im, going, crazy, year, findom, findomme, fins)</td><td>Map(vectorType -> sparse, length -> 37185, indices -> List(1, 2, 41, 64, 78, 108, 114, 398, 548, 3915, 5535, 6957, 29373), values -> List(1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0))</td><td>Map(vectorType -> sparse, length -> 37185, indices -> List(1, 2, 41, 64, 78, 108, 114, 398, 548, 3915, 5535, 6957, 29373), values -> List(0.4434163164512347, 0.46447935985153693, 3.5883125702634744, 3.8492760959306898, 4.052661621082812, 4.381559976510121, 4.4924128559159735, 5.884229913518515, 6.283385852433415, 8.749600369209263, 9.242076854307058, 9.560530585425592, 0.0))</td><td>Map(vectorType -> dense, length -> 2, values -> List(2.0453207302256597, -2.0453207302256597))</td><td>Map(vectorType -> dense, length -> 2, values -> List(0.88547394998761, 0.11452605001239002))</td><td>0.0</td></tr><tr><td>0</td><td>amazon workers across the world plan black friday strike to demand the tech giant pays fairly and ceases awful un</td><td>List(amazon, workers, across, the, world, plan, black, friday, strike, to, demand, the, tech, giant, pays, fairly, and, ceases, awful, un)</td><td>List(amazon, workers, across, world, plan, black, friday, strike, demand, tech, giant, pays, fairly, ceases, awful, un)</td><td>Map(vectorType -> sparse, length -> 37185, indices -> List(1, 2, 57, 271, 369, 372, 389, 419, 809, 2013, 2115, 2202, 2995, 5508, 6377, 6576), values -> List(1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0))</td><td>Map(vectorType -> sparse, length -> 37185, indices -> List(1, 2, 57, 271, 369, 372, 389, 419, 809, 2013, 2115, 2202, 2995, 5508, 6377, 6576), values -> List(0.4434163164512347, 0.46447935985153693, 3.9732819270253423, 5.411461123514245, 5.753868095655272, 5.751094168772547, 5.881070353228147, 5.963218324837146, 6.742132327154517, 7.90230250882206, 7.951092672991492, 8.029054214461203, 8.381875589083945, 9.242076854307058, 9.442747549769209, 9.442747549769209))</td><td>Map(vectorType -> dense, length -> 2, values -> List(47.24168322895689, -47.24168322895689))</td><td>Map(vectorType -> dense, length -> 2, values -> List(1.0, 0.0))</td><td>0.0</td></tr><tr><td>0</td><td>another bullshit from wbd firedavidzaslav</td><td>List(another, bullshit, from, wbd, firedavidzaslav)</td><td>List(another, bullshit, wbd, firedavidzaslav)</td><td>Map(vectorType -> sparse, length -> 37185, indices -> List(365, 3116, 21514, 26094), values -> List(1.0, 1.0, 1.0, 1.0))</td><td>Map(vectorType -> sparse, length -> 37185, indices -> List(365, 3116, 21514, 26094), values -> List(5.756649738617149, 8.461918296757482, 0.0, 0.0))</td><td>Map(vectorType -> dense, length -> 2, values -> List(27.301697931623202, -27.301697931623202))</td><td>Map(vectorType -> dense, length -> 2, values -> List(0.99999999999861, 1.389999226830696E-12))</td><td>0.0</td></tr></tbody></table></div>"]}}],"execution_count":0},{"cell_type":"markdown","source":["#### Model Evaluation"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"2adacc7f-d41b-4d9b-a091-2912a0052e17","inputWidgets":{},"title":""}}},{"cell_type":"code","source":["# Evaluate the model using binary classification evaluation \nevaluator = BinaryClassificationEvaluator(rawPredictionCol=\"rawPrediction\") \nroc_auc = evaluator.evaluate(predictions)\naccuracy = predictions.filter(predictions.label == predictions.prediction).count() / float(predictions.count())\n\nprint(\"Accuracy Score: {0:.4f}\".format(accuracy))\nprint(\"ROC-AUC: {0:.4f}\".format(roc_auc))"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"1d0e7cdc-93e5-4eeb-abb9-b710edb88961","inputWidgets":{},"title":""}},"outputs":[{"output_type":"stream","output_type":"stream","name":"stdout","text":["Accuracy Score: 0.9588\nROC-AUC: 0.9290\n"]}],"execution_count":0},{"cell_type":"markdown","source":["##### Note:\n\n* Due to the size of the dataset and the limited computing power, we didn't compare performance of different ML classification models or tune hyperparameters for better performance. \n* With our logistic regression model, we have an accuracy score of 0.9588, and a ROC-AUC score of 0.929."],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"0da87cb0-a632-4fe0-81da-b536bcbc5ffa","inputWidgets":{},"title":""}}},{"cell_type":"markdown","source":["#### Save the data and the predictions into my bucket"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"1b710b28-fea0-4ffb-a7e8-329d28dc9781","inputWidgets":{},"title":""}}},{"cell_type":"code","source":["# mount my s3 bucket\nmount_s3_bucket(ACCESS_KEY, SECRET_ACCESS_KEY, 'ptb2-effy', 'my_bucket')"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"02abc602-5df1-4530-928a-11e7dfe20803","inputWidgets":{},"title":""}},"outputs":[{"output_type":"stream","output_type":"stream","name":"stdout","text":["Mounting ptb2-effy\n/mnt/my_bucket has been unmounted.\nThe bucket ptb2-effy was mounted to my_bucket \n\n"]}],"execution_count":0},{"cell_type":"code","source":["# save the data\ndf.write.option('header','false').csv('/mnt/my_bucket/demo/data.csv') # remove header for athena"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"40038c90-bd52-46e7-ad98-0475195ba9e8","inputWidgets":{},"title":""}},"outputs":[],"execution_count":0},{"cell_type":"code","source":["# save the predictions as a Parquet file\npredictions.write.parquet('/mnt/my_bucket/demo/predictions.parquet')"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"df160452-a576-420e-a89b-05a2b3ce0a98","inputWidgets":{},"title":""}},"outputs":[],"execution_count":0},{"cell_type":"markdown","source":["## Conclusion\n\nThis project is focused on analyzing Twitter data during Black Friday using machine learning techniques, specifically sentiment analysis. The aim is to gain insights into consumer behavior and preferences during this significant shopping event. \n\nThe project follows a step-by-step approach, starting with mounting the data on tweets from the WeCloudData public dataset bucket, creating a Spark session and Spark DataFrame, creating a sentiment column, cleaning the text, performing feature transformation, and model training and evaluation. Finally, the data and predictions are saved to the user's bucket. \n\nThis project demonstrates the power of machine learning in analyzing unstructured data from social media platforms like Twitter to provide valuable insights into consumer behavior, which can help businesses and policymakers make informed decisions."],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"bad4652a-d903-4641-bf8c-ce04e9006f3a","inputWidgets":{},"title":""}}}],"metadata":{"application/vnd.databricks.v1+notebook":{"notebookName":"Big Data Project  - Copy","dashboards":[],"notebookMetadata":{"pythonIndentUnit":4,"mostRecentlyExecutedCommandWithImplicitDF":{"commandId":3369833116920540,"dataframes":["_sqldf"]}},"language":"python","widgets":{},"notebookOrigID":1056953587383902}},"nbformat":4,"nbformat_minor":0}