lexical-processing-nlp-zomato-review.ipynb

{"cells":[{"source":"<a href=\"https://www.kaggle.com/code/sharanharsoor/lexical-processing-nlp-zomato-review?scriptVersionId=122397213\" target=\"_blank\"><img align=\"left\" alt=\"Kaggle\" title=\"Open in Kaggle\" src=\"https://kaggle.com/static/images/open-in-kaggle.svg\"></a>","metadata":{},"cell_type":"markdown","outputs":[],"execution_count":0},{"cell_type":"markdown","id":"e89e045d","metadata":{"papermill":{"duration":0.012765,"end_time":"2023-03-16T20:37:26.492216","exception":false,"start_time":"2023-03-16T20:37:26.479451","status":"completed"},"tags":[]},"source":["# 0. Introduction.\n","\n","In this Notebook, will touch base on basics of NLP. extract the Zomato customer reviews from a given data file and will perform various text pre-processing steps:\n","* Removing unnecessary elements like HTML tags, URLs and emojis\n","* Text encoding\n","* Removing special characters and symbols\n","* Converting the text to lower case\n","* Removing stopwords\n","* Stemming and lemmatisation"]},{"cell_type":"markdown","id":"8fcb9c1e","metadata":{"papermill":{"duration":0.011199,"end_time":"2023-03-16T20:37:26.515793","exception":false,"start_time":"2023-03-16T20:37:26.504594","status":"completed"},"tags":[]},"source":["# 1. Importing the libraries"]},{"cell_type":"code","execution_count":1,"id":"338ce332","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:26.541334Z","iopub.status.busy":"2023-03-16T20:37:26.540694Z","iopub.status.idle":"2023-03-16T20:37:28.608842Z","shell.execute_reply":"2023-03-16T20:37:28.607402Z"},"papermill":{"duration":2.088878,"end_time":"2023-03-16T20:37:28.616109","exception":false,"start_time":"2023-03-16T20:37:26.527231","status":"completed"},"tags":[]},"outputs":[{"name":"stderr","output_type":"stream","text":["[nltk_data] Downloading package stopwords to /usr/share/nltk_data...\n","[nltk_data]   Package stopwords is already up-to-date!\n"]},{"data":{"text/plain":["True"]},"execution_count":1,"metadata":{},"output_type":"execute_result"}],"source":["#Load the libraries\n","import numpy as np\n","import pandas as pd\n","import seaborn as sns\n","import matplotlib.pyplot as plt\n","import matplotlib.cm as cm\n","import nltk\n","\n","import os\n","import warnings\n","warnings.filterwarnings('ignore')\n","nltk.download('stopwords')"]},{"cell_type":"markdown","id":"be701965","metadata":{"papermill":{"duration":0.011297,"end_time":"2023-03-16T20:37:28.642473","exception":false,"start_time":"2023-03-16T20:37:28.631176","status":"completed"},"tags":[]},"source":["# 2. Reading the input data"]},{"cell_type":"code","execution_count":2,"id":"734675d5","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:28.668083Z","iopub.status.busy":"2023-03-16T20:37:28.667657Z","iopub.status.idle":"2023-03-16T20:37:28.730754Z","shell.execute_reply":"2023-03-16T20:37:28.729072Z"},"papermill":{"duration":0.079296,"end_time":"2023-03-16T20:37:28.733602","exception":false,"start_time":"2023-03-16T20:37:28.654306","status":"completed"},"tags":[]},"outputs":[{"name":"stdout","output_type":"stream","text":["(5479, 2)\n"]},{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>rating</th>\n","      <th>review</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>5</td>\n","      <td>nice</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>5</td>\n","      <td>best biryani , so supportive staff of outlet ,...</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>4</td>\n","      <td>delivery boy was very decent and supportive.👌👍</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>1</td>\n","      <td>worst biryani i have tasted in my life, half o...</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>5</td>\n","      <td>all food is good and tasty . will order again ...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   rating                                             review\n","0       5                                               nice\n","1       5  best biryani , so supportive staff of outlet ,...\n","2       4     delivery boy was very decent and supportive.👌👍\n","3       1  worst biryani i have tasted in my life, half o...\n","4       5  all food is good and tasty . will order again ..."]},"execution_count":2,"metadata":{},"output_type":"execute_result"}],"source":["zomato = pd.read_csv('/kaggle/input/zomato-reviews-ratings/zomato_reviews.csv',index_col='Unnamed: 0')\n","print(zomato.shape)\n","zomato.head(5)"]},{"cell_type":"markdown","id":"296cb508","metadata":{"papermill":{"duration":0.011562,"end_time":"2023-03-16T20:37:28.757532","exception":false,"start_time":"2023-03-16T20:37:28.74597","status":"completed"},"tags":[]},"source":["# 3. Basic Pre-processing reviews text"]},{"cell_type":"code","execution_count":3,"id":"daf0700c","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:28.783348Z","iopub.status.busy":"2023-03-16T20:37:28.782941Z","iopub.status.idle":"2023-03-16T20:37:28.789066Z","shell.execute_reply":"2023-03-16T20:37:28.787729Z"},"papermill":{"duration":0.022131,"end_time":"2023-03-16T20:37:28.791594","exception":false,"start_time":"2023-03-16T20:37:28.769463","status":"completed"},"tags":[]},"outputs":[{"name":"stdout","output_type":"stream","text":["<div><h1><Title>The apple π was [*][AMAZING][*] and YuMmY too😂! You can Checkout the entire Menu in https://www.zomato.com/chennai/top-restaurantshello </div></h1></Title>\n"]}],"source":["text = u'<div><h1><Title>The apple π was [*][AMAZING][*] and YuMmY too\\U0001f602! You can Checkout the entire Menu in https://www.zomato.com/chennai/top-restaurantshello </div></h1></Title>'\n","print(text)"]},{"cell_type":"markdown","id":"2dd641dd","metadata":{"papermill":{"duration":0.011585,"end_time":"2023-03-16T20:37:28.815732","exception":false,"start_time":"2023-03-16T20:37:28.804147","status":"completed"},"tags":[]},"source":["## 3.1 Removing HTML strips\n","In case any of the reviews have got any HTML tags (ex : < html >\") remove the same."]},{"cell_type":"code","execution_count":4,"id":"cf264491","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:28.842817Z","iopub.status.busy":"2023-03-16T20:37:28.842304Z","iopub.status.idle":"2023-03-16T20:37:29.03854Z","shell.execute_reply":"2023-03-16T20:37:29.037286Z"},"papermill":{"duration":0.213688,"end_time":"2023-03-16T20:37:29.041428","exception":false,"start_time":"2023-03-16T20:37:28.82774","status":"completed"},"tags":[]},"outputs":[],"source":["from bs4 import BeautifulSoup\n","\n","def strip_html(text):\n","    soup = BeautifulSoup(text, \"html.parser\")\n","    return soup.get_text()"]},{"cell_type":"code","execution_count":5,"id":"d605ebf4","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:29.067589Z","iopub.status.busy":"2023-03-16T20:37:29.06695Z","iopub.status.idle":"2023-03-16T20:37:29.074229Z","shell.execute_reply":"2023-03-16T20:37:29.072528Z"},"papermill":{"duration":0.023105,"end_time":"2023-03-16T20:37:29.076675","exception":false,"start_time":"2023-03-16T20:37:29.05357","status":"completed"},"tags":[]},"outputs":[{"name":"stdout","output_type":"stream","text":["The apple π was [*][AMAZING][*] and YuMmY too😂! You can Checkout the entire Menu in https://www.zomato.com/chennai/top-restaurantshello \n"]}],"source":["text = strip_html(text)\n","print(text)"]},{"cell_type":"markdown","id":"5368a960","metadata":{"papermill":{"duration":0.011954,"end_time":"2023-03-16T20:37:29.100612","exception":false,"start_time":"2023-03-16T20:37:29.088658","status":"completed"},"tags":[]},"source":["## 3.2 Removing URLs\n","After parsing the text and removing the html tags, let's now remove all hyper links (for example, urls containing \"https://\") from the reviews text if that may have. We will use regular expressions to do so."]},{"cell_type":"code","execution_count":6,"id":"a690b21a","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:29.12768Z","iopub.status.busy":"2023-03-16T20:37:29.12721Z","iopub.status.idle":"2023-03-16T20:37:29.13503Z","shell.execute_reply":"2023-03-16T20:37:29.133211Z"},"papermill":{"duration":0.024544,"end_time":"2023-03-16T20:37:29.137604","exception":false,"start_time":"2023-03-16T20:37:29.11306","status":"completed"},"tags":[]},"outputs":[{"name":"stdout","output_type":"stream","text":["The apple π was [*][AMAZING][*] and YuMmY too😂! You can Checkout the entire Menu in  \n"]}],"source":["import re\n","\n","text = re.sub(r\"http\\S+\", \"\", text)\n","print(text)"]},{"cell_type":"markdown","id":"7eabf320","metadata":{"papermill":{"duration":0.012072,"end_time":"2023-03-16T20:37:29.162591","exception":false,"start_time":"2023-03-16T20:37:29.150519","status":"completed"},"tags":[]},"source":["## 3.3 Removing Emojis\n","Now, we will define a function to remove different types of emojis like smileys, symbols, gifs, flags and map symbols."]},{"cell_type":"code","execution_count":7,"id":"9e8c5854","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:29.189879Z","iopub.status.busy":"2023-03-16T20:37:29.189414Z","iopub.status.idle":"2023-03-16T20:37:29.19594Z","shell.execute_reply":"2023-03-16T20:37:29.194469Z"},"papermill":{"duration":0.023186,"end_time":"2023-03-16T20:37:29.198353","exception":false,"start_time":"2023-03-16T20:37:29.175167","status":"completed"},"tags":[]},"outputs":[],"source":["def deEmojify(text):\n","    regrex_pattern = re.compile(pattern = \"[\"\n","        u\"\\U0001F600-\\U0001F64F\"  # emoticons\n","        u\"\\U0001F300-\\U0001F5FF\"  # symbols & pictographs\n","        u\"\\U0001F680-\\U0001F6FF\"  # transport & map symbols\n","        u\"\\U0001F1E0-\\U0001F1FF\"  # flags (iOS)\n","                           \"]+\", flags = re.UNICODE)\n","    return regrex_pattern.sub(r'',text)"]},{"cell_type":"code","execution_count":8,"id":"e72ed12a","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:29.225799Z","iopub.status.busy":"2023-03-16T20:37:29.225289Z","iopub.status.idle":"2023-03-16T20:37:29.232733Z","shell.execute_reply":"2023-03-16T20:37:29.231174Z"},"papermill":{"duration":0.024015,"end_time":"2023-03-16T20:37:29.235319","exception":false,"start_time":"2023-03-16T20:37:29.211304","status":"completed"},"tags":[]},"outputs":[{"name":"stdout","output_type":"stream","text":["The apple π was [*][AMAZING][*] and YuMmY too! You can Checkout the entire Menu in  \n"]}],"source":["text = deEmojify(text)\n","print(text)"]},{"cell_type":"markdown","id":"6ae3e197","metadata":{"papermill":{"duration":0.011898,"end_time":"2023-03-16T20:37:29.261619","exception":false,"start_time":"2023-03-16T20:37:29.249721","status":"completed"},"tags":[]},"source":["# 3.4 Text encoding\n","Let us consider some examples of different encodings - UTF-8, ASCII and typical string.\n","\n","Notice the 'b' in front of the text; which basically says the data is in bytes format.\n","\n","Typically we prefer UTF-8 encoding, as it is devoid of foreign characters.\n","\n","In this case we will first remove the \"pi\" symbol and then convert the text to UTF-8 format."]},{"cell_type":"code","execution_count":9,"id":"6c2a7b23","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:29.288084Z","iopub.status.busy":"2023-03-16T20:37:29.287574Z","iopub.status.idle":"2023-03-16T20:37:29.294873Z","shell.execute_reply":"2023-03-16T20:37:29.294011Z"},"papermill":{"duration":0.023192,"end_time":"2023-03-16T20:37:29.297003","exception":false,"start_time":"2023-03-16T20:37:29.273811","status":"completed"},"tags":[]},"outputs":[{"data":{"text/plain":["b'The apple \\xcf\\x80 was [*][AMAZING][*] and YuMmY too! You can Checkout the entire Menu in  '"]},"execution_count":9,"metadata":{},"output_type":"execute_result"}],"source":["text.encode('utf-8', 'ignore')"]},{"cell_type":"markdown","id":"c937bdd5","metadata":{"papermill":{"duration":0.012571,"end_time":"2023-03-16T20:37:29.322164","exception":false,"start_time":"2023-03-16T20:37:29.309593","status":"completed"},"tags":[]},"source":["In this example, if we print the text as unicode-encoded, we can notice the \"pi\" symbol is translated to some xml tags."]},{"cell_type":"code","execution_count":10,"id":"fda0b278","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:29.349552Z","iopub.status.busy":"2023-03-16T20:37:29.349043Z","iopub.status.idle":"2023-03-16T20:37:29.356082Z","shell.execute_reply":"2023-03-16T20:37:29.354517Z"},"papermill":{"duration":0.023595,"end_time":"2023-03-16T20:37:29.358327","exception":false,"start_time":"2023-03-16T20:37:29.334732","status":"completed"},"tags":[]},"outputs":[{"name":"stdout","output_type":"stream","text":["b'The apple  was [*][AMAZING][*] and YuMmY too! You can Checkout the entire Menu in  '\n"]}],"source":["#In our context, we would want to remove the 'pi' symbol from the review, hence we considered ascii encoding.\n","text = text.encode('ascii', 'ignore')\n","print(text)"]},{"cell_type":"code","execution_count":11,"id":"cffed817","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:29.386883Z","iopub.status.busy":"2023-03-16T20:37:29.386357Z","iopub.status.idle":"2023-03-16T20:37:29.394052Z","shell.execute_reply":"2023-03-16T20:37:29.392537Z"},"papermill":{"duration":0.025202,"end_time":"2023-03-16T20:37:29.396503","exception":false,"start_time":"2023-03-16T20:37:29.371301","status":"completed"},"tags":[]},"outputs":[],"source":["def to_unicode(text):\n","    if isinstance(text, float):\n","        text = str(text)\n","    if isinstance(text, int):\n","        text = str(text)\n","    if not isinstance(text, str):\n","        text = text.decode('utf-8', 'ignore')\n","    return text"]},{"cell_type":"code","execution_count":12,"id":"e6558cec","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:29.424191Z","iopub.status.busy":"2023-03-16T20:37:29.423682Z","iopub.status.idle":"2023-03-16T20:37:29.430263Z","shell.execute_reply":"2023-03-16T20:37:29.428744Z"},"papermill":{"duration":0.023626,"end_time":"2023-03-16T20:37:29.432638","exception":false,"start_time":"2023-03-16T20:37:29.409012","status":"completed"},"tags":[]},"outputs":[{"name":"stdout","output_type":"stream","text":["The apple  was [*][AMAZING][*] and YuMmY too! You can Checkout the entire Menu in  \n"]}],"source":["#As you noticed above, there was a 'b' character before the text because it is in byte format after ascii encoding, \n","#so let us convert back to utf-8 encoding\n","text = to_unicode(text)\n","print(text)"]},{"cell_type":"markdown","id":"4a3a8abb","metadata":{"papermill":{"duration":0.012906,"end_time":"2023-03-16T20:37:29.457976","exception":false,"start_time":"2023-03-16T20:37:29.44507","status":"completed"},"tags":[]},"source":["## 3.5 Removing symbols\n"]},{"cell_type":"code","execution_count":13,"id":"f1fdaf7e","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:29.485877Z","iopub.status.busy":"2023-03-16T20:37:29.48536Z","iopub.status.idle":"2023-03-16T20:37:29.493117Z","shell.execute_reply":"2023-03-16T20:37:29.49202Z"},"papermill":{"duration":0.025187,"end_time":"2023-03-16T20:37:29.495895","exception":false,"start_time":"2023-03-16T20:37:29.470708","status":"completed"},"tags":[]},"outputs":[],"source":["#Removing the square brackets, symbols\n","import re,string\n","\n","def remove_between_square_brackets(text):\n","    return re.sub('\\[[^]]*\\]', '', text)"]},{"cell_type":"code","execution_count":14,"id":"789fc0ec","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:29.526175Z","iopub.status.busy":"2023-03-16T20:37:29.525611Z","iopub.status.idle":"2023-03-16T20:37:29.532627Z","shell.execute_reply":"2023-03-16T20:37:29.531222Z"},"papermill":{"duration":0.025679,"end_time":"2023-03-16T20:37:29.535313","exception":false,"start_time":"2023-03-16T20:37:29.509634","status":"completed"},"tags":[]},"outputs":[{"name":"stdout","output_type":"stream","text":["The apple  was  and YuMmY too! You can Checkout the entire Menu in  \n"]}],"source":["text = remove_between_square_brackets(text)\n","print(text)"]},{"cell_type":"markdown","id":"3338dc22","metadata":{"papermill":{"duration":0.014584,"end_time":"2023-03-16T20:37:29.56457","exception":false,"start_time":"2023-03-16T20:37:29.549986","status":"completed"},"tags":[]},"source":["## 3.6 Removing special characters"]},{"cell_type":"code","execution_count":15,"id":"53ec4111","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:29.594335Z","iopub.status.busy":"2023-03-16T20:37:29.59394Z","iopub.status.idle":"2023-03-16T20:37:29.59914Z","shell.execute_reply":"2023-03-16T20:37:29.597936Z"},"papermill":{"duration":0.022224,"end_time":"2023-03-16T20:37:29.601236","exception":false,"start_time":"2023-03-16T20:37:29.579012","status":"completed"},"tags":[]},"outputs":[],"source":["#Function for removing special characters\n","def remove_special_characters(text, remove_digits=True):\n","    pattern=r'[^a-zA-z0-9\\s]'\n","    text=re.sub(pattern,'',text)\n","    return text"]},{"cell_type":"code","execution_count":16,"id":"6df59205","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:29.629386Z","iopub.status.busy":"2023-03-16T20:37:29.628977Z","iopub.status.idle":"2023-03-16T20:37:29.634302Z","shell.execute_reply":"2023-03-16T20:37:29.633424Z"},"papermill":{"duration":0.022842,"end_time":"2023-03-16T20:37:29.637205","exception":false,"start_time":"2023-03-16T20:37:29.614363","status":"completed"},"tags":[]},"outputs":[{"name":"stdout","output_type":"stream","text":["The apple  was  and YuMmY too You can Checkout the entire Menu in  \n"]}],"source":["text = remove_special_characters(text)\n","print(text)"]},{"cell_type":"markdown","id":"5f8a4862","metadata":{"papermill":{"duration":0.012688,"end_time":"2023-03-16T20:37:29.66379","exception":false,"start_time":"2023-03-16T20:37:29.651102","status":"completed"},"tags":[]},"source":["## 3.7 Lowercase conversion\n"]},{"cell_type":"code","execution_count":17,"id":"8f852820","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:29.692725Z","iopub.status.busy":"2023-03-16T20:37:29.691629Z","iopub.status.idle":"2023-03-16T20:37:29.697893Z","shell.execute_reply":"2023-03-16T20:37:29.696998Z"},"papermill":{"duration":0.023367,"end_time":"2023-03-16T20:37:29.700506","exception":false,"start_time":"2023-03-16T20:37:29.677139","status":"completed"},"tags":[]},"outputs":[{"name":"stdout","output_type":"stream","text":["the apple  was  and yummy too you can checkout the entire menu in  \n"]}],"source":["#Converting text to lowercase for standardisation\n","text = text.lower()\n","print(text)"]},{"cell_type":"markdown","id":"947122a7","metadata":{"papermill":{"duration":0.012707,"end_time":"2023-03-16T20:37:29.7262","exception":false,"start_time":"2023-03-16T20:37:29.713493","status":"completed"},"tags":[]},"source":["## 3.8 Function to denoise"]},{"cell_type":"code","execution_count":18,"id":"548214e5","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:29.754894Z","iopub.status.busy":"2023-03-16T20:37:29.754321Z","iopub.status.idle":"2023-03-16T20:37:29.765154Z","shell.execute_reply":"2023-03-16T20:37:29.761487Z"},"papermill":{"duration":0.029631,"end_time":"2023-03-16T20:37:29.768713","exception":false,"start_time":"2023-03-16T20:37:29.739082","status":"completed"},"tags":[]},"outputs":[],"source":["# defining a func to apply all pre-processing steps on all the reviews\n","\n","def denoise_text(text):\n","    text = to_unicode(text)\n","    text = strip_html(text)\n","    text = re.sub(r\"http\\S+\", \"\", text)\n","    text = deEmojify(text)\n","    text = text.encode('ascii', 'ignore')\n","    text = to_unicode(text)\n","    text = remove_between_square_brackets(text)\n","    text = remove_special_characters(text)\n","    text = text.lower()\n","    return text\n"]},{"cell_type":"code","execution_count":19,"id":"3c6c590c","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:29.799729Z","iopub.status.busy":"2023-03-16T20:37:29.799266Z","iopub.status.idle":"2023-03-16T20:37:29.808725Z","shell.execute_reply":"2023-03-16T20:37:29.807695Z"},"papermill":{"duration":0.026415,"end_time":"2023-03-16T20:37:29.810978","exception":false,"start_time":"2023-03-16T20:37:29.784563","status":"completed"},"tags":[]},"outputs":[{"data":{"text/plain":["'Thank you for delivering 🙂🙂'"]},"execution_count":19,"metadata":{},"output_type":"execute_result"}],"source":["# checking a random review \n","zomato['review'][556]"]},{"cell_type":"code","execution_count":20,"id":"6c4d0204","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:29.83899Z","iopub.status.busy":"2023-03-16T20:37:29.838504Z","iopub.status.idle":"2023-03-16T20:37:30.327006Z","shell.execute_reply":"2023-03-16T20:37:30.32558Z"},"papermill":{"duration":0.505951,"end_time":"2023-03-16T20:37:30.329914","exception":false,"start_time":"2023-03-16T20:37:29.823963","status":"completed"},"tags":[]},"outputs":[{"data":{"text/plain":["0                                                 nice\n","1    best biryani  so supportive staff of outlet  p...\n","2          delivery boy was very decent and supportive\n","3    worst biryani i have tasted in my life half of...\n","4    all food is good and tasty  will order again a...\n","Name: review, dtype: object"]},"execution_count":20,"metadata":{},"output_type":"execute_result"}],"source":["# IMP : Apply function on review column\n","zomato['review']=zomato['review'].apply(denoise_text)\n","zomato['review'].head()"]},{"cell_type":"code","execution_count":21,"id":"0d05768b","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:30.359349Z","iopub.status.busy":"2023-03-16T20:37:30.358956Z","iopub.status.idle":"2023-03-16T20:37:30.366551Z","shell.execute_reply":"2023-03-16T20:37:30.365401Z"},"papermill":{"duration":0.025282,"end_time":"2023-03-16T20:37:30.368813","exception":false,"start_time":"2023-03-16T20:37:30.343531","status":"completed"},"tags":[]},"outputs":[{"data":{"text/plain":["'thank you for delivering '"]},"execution_count":21,"metadata":{},"output_type":"execute_result"}],"source":["# Processed example of randomly selected review and the emojies in the text is removed. \n","zomato['review'][556]"]},{"cell_type":"markdown","id":"35994d27","metadata":{"papermill":{"duration":0.014078,"end_time":"2023-03-16T20:37:30.396799","exception":false,"start_time":"2023-03-16T20:37:30.382721","status":"completed"},"tags":[]},"source":["# 4. Removing stopwords\n"]},{"cell_type":"code","execution_count":22,"id":"f13ffe60","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:30.425617Z","iopub.status.busy":"2023-03-16T20:37:30.425231Z","iopub.status.idle":"2023-03-16T20:37:30.431375Z","shell.execute_reply":"2023-03-16T20:37:30.430252Z"},"papermill":{"duration":0.023587,"end_time":"2023-03-16T20:37:30.433959","exception":false,"start_time":"2023-03-16T20:37:30.410372","status":"completed"},"tags":[]},"outputs":[],"source":["from nltk.corpus import stopwords\n","from nltk.tokenize.toktok import ToktokTokenizer"]},{"cell_type":"code","execution_count":23,"id":"eb03ebde","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:30.461782Z","iopub.status.busy":"2023-03-16T20:37:30.461363Z","iopub.status.idle":"2023-03-16T20:37:30.467362Z","shell.execute_reply":"2023-03-16T20:37:30.465886Z"},"papermill":{"duration":0.022661,"end_time":"2023-03-16T20:37:30.469808","exception":false,"start_time":"2023-03-16T20:37:30.447147","status":"completed"},"tags":[]},"outputs":[],"source":["#Tokenization of text\n","tokenizer=ToktokTokenizer() "]},{"cell_type":"code","execution_count":24,"id":"e80cb357","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:30.498177Z","iopub.status.busy":"2023-03-16T20:37:30.497746Z","iopub.status.idle":"2023-03-16T20:37:30.506205Z","shell.execute_reply":"2023-03-16T20:37:30.504658Z"},"papermill":{"duration":0.025989,"end_time":"2023-03-16T20:37:30.508837","exception":false,"start_time":"2023-03-16T20:37:30.482848","status":"completed"},"tags":[]},"outputs":[],"source":["#Setting English stopwords\n","stopword_list=nltk.corpus.stopwords.words('english')"]},{"cell_type":"code","execution_count":25,"id":"a2ab8030","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:30.538552Z","iopub.status.busy":"2023-03-16T20:37:30.536891Z","iopub.status.idle":"2023-03-16T20:37:30.549304Z","shell.execute_reply":"2023-03-16T20:37:30.547205Z"},"papermill":{"duration":0.031296,"end_time":"2023-03-16T20:37:30.553312","exception":false,"start_time":"2023-03-16T20:37:30.522016","status":"completed"},"tags":[]},"outputs":[{"name":"stdout","output_type":"stream","text":["{'you', 'before', 'a', 'against', \"you'll\", 'doesn', \"doesn't\", 'isn', 'few', 'after', 'your', 'for', 'm', 'are', 'our', 'only', 'any', 'so', 'not', 'should', 'until', 'wouldn', \"isn't\", \"that'll\", 'theirs', 'an', 'into', \"aren't\", 'ours', 'himself', 'ourselves', 'who', 'herself', 'ain', 'more', 'll', \"hadn't\", 'hadn', 'had', 'having', 'i', 'above', 'were', 'here', \"she's\", 'down', 'all', 'about', 'haven', 'his', 'or', 'wasn', 'if', \"won't\", 'she', 'both', 'hasn', 'between', \"couldn't\", 'been', \"hasn't\", 'mightn', 'did', 'during', 's', 'out', 'those', 'why', \"it's\", \"needn't\", 'myself', 'o', 'they', 'can', 'him', 'them', 'most', 'with', 'what', 'of', 'd', 'than', 'through', 't', 'no', 'themselves', \"should've\", 'while', 'how', 'her', 'now', 'mustn', 'same', \"shouldn't\", \"weren't\", 'couldn', 'when', 'the', 'by', 'won', 'is', 'on', 'y', 'doing', \"wasn't\", \"wouldn't\", 'there', 'being', 'whom', \"haven't\", 'shan', 'nor', 'yourself', \"you'd\", 'has', 'these', 'does', 'their', 'it', 'will', 'from', 'such', 'where', 'below', 'further', \"don't\", 'yours', 'am', 'because', 'as', 'at', 're', 'which', 'some', \"mightn't\", 'itself', 'was', 'we', 'in', \"shan't\", \"you're\", 'again', 'just', 'off', 'under', 'own', 'didn', 'this', 'and', 'hers', 'me', 'he', 'aren', 'over', 'its', 'but', 'very', 'that', 'each', 'ma', 'my', \"you've\", 'needn', \"mustn't\", 'shouldn', 'do', 'be', 'have', \"didn't\", 'don', 'up', 'weren', 'yourselves', 'too', 'to', 've', 'other', 'then', 'once'}\n"]}],"source":["#Removing standard english stopwords like prepositions, adverbs\n","from nltk.tokenize import word_tokenize,sent_tokenize\n","\n","stop=set(stopwords.words('english'))\n","print(stop)\n","\n","#Removing the stopwords\n","def remove_stopwords(text, is_lower_case=False):\n","    tokens = tokenizer.tokenize(text)\n","    tokens = [token.strip() for token in tokens]\n","    if is_lower_case:\n","        filtered_tokens = [token for token in tokens if token not in stopword_list]\n","    else:\n","        filtered_tokens = [token for token in tokens if token.lower() not in stopword_list]\n","    filtered_text = ' '.join(filtered_tokens)    \n","    return filtered_text\n"]},{"cell_type":"code","execution_count":26,"id":"1079f0d9","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:30.582743Z","iopub.status.busy":"2023-03-16T20:37:30.582328Z","iopub.status.idle":"2023-03-16T20:37:30.590656Z","shell.execute_reply":"2023-03-16T20:37:30.589139Z"},"papermill":{"duration":0.026243,"end_time":"2023-03-16T20:37:30.593408","exception":false,"start_time":"2023-03-16T20:37:30.567165","status":"completed"},"tags":[]},"outputs":[{"data":{"text/plain":["'it took 1 hour to assign valvet and thn prepare food like 30 mins to deliver also 4 valvet was near by cant do delivery and ur help chat talking to 34 ppl cant  still get my delivery on timei didnt got support from support chat system also restaurant was non responsive to cook order on time and this valet system assigning late and any mistake happens ur chat support system credit d voucher seriously its was waste on time to order from ds app n restaurant'"]},"execution_count":26,"metadata":{},"output_type":"execute_result"}],"source":["#Raw example of randomly selected review text\n","zomato['review'][5475]"]},{"cell_type":"code","execution_count":27,"id":"88a2d42a","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:30.622607Z","iopub.status.busy":"2023-03-16T20:37:30.622119Z","iopub.status.idle":"2023-03-16T20:37:31.022522Z","shell.execute_reply":"2023-03-16T20:37:31.021287Z"},"papermill":{"duration":0.418393,"end_time":"2023-03-16T20:37:31.025654","exception":false,"start_time":"2023-03-16T20:37:30.607261","status":"completed"},"tags":[]},"outputs":[],"source":["#Apply function on review column\n","zomato['review']=zomato['review'].apply(remove_stopwords)"]},{"cell_type":"code","execution_count":28,"id":"08486556","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:31.055918Z","iopub.status.busy":"2023-03-16T20:37:31.055496Z","iopub.status.idle":"2023-03-16T20:37:31.062626Z","shell.execute_reply":"2023-03-16T20:37:31.061475Z"},"papermill":{"duration":0.025785,"end_time":"2023-03-16T20:37:31.065572","exception":false,"start_time":"2023-03-16T20:37:31.039787","status":"completed"},"tags":[]},"outputs":[{"data":{"text/plain":["'took 1 hour assign valvet thn prepare food like 30 mins deliver also 4 valvet near cant delivery ur help chat talking 34 ppl cant still get delivery timei didnt got support support chat system also restaurant non responsive cook order time valet system assigning late mistake happens ur chat support system credit voucher seriously waste time order ds app n restaurant'"]},"execution_count":28,"metadata":{},"output_type":"execute_result"}],"source":["#Processed example of randomly selected review text\n","zomato['review'][5475]"]},{"cell_type":"markdown","id":"8ad10aa9","metadata":{"papermill":{"duration":0.012908,"end_time":"2023-03-16T20:37:31.092234","exception":false,"start_time":"2023-03-16T20:37:31.079326","status":"completed"},"tags":[]},"source":["# 5. Stemming and Lemmatization\n"]},{"cell_type":"code","execution_count":29,"id":"f730ed27","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:31.121514Z","iopub.status.busy":"2023-03-16T20:37:31.120253Z","iopub.status.idle":"2023-03-16T20:37:31.196527Z","shell.execute_reply":"2023-03-16T20:37:31.193996Z"},"papermill":{"duration":0.093975,"end_time":"2023-03-16T20:37:31.199419","exception":false,"start_time":"2023-03-16T20:37:31.105444","status":"completed"},"tags":[]},"outputs":[{"name":"stderr","output_type":"stream","text":["[nltk_data] Downloading package wordnet to /usr/share/nltk_data...\n","[nltk_data]   Package wordnet is already up-to-date!\n"]}],"source":["from nltk.stem import WordNetLemmatizer,SnowballStemmer\n","from nltk.stem.porter import PorterStemmer\n","nltk.download('wordnet')\n","\n","def simple_stemmer(text):\n","    ps=SnowballStemmer(language='english')\n","    return ' '.join([ps.stem(word) for word in tokenizer.tokenize(text)])"]},{"cell_type":"code","execution_count":30,"id":"33ebe536","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:31.228178Z","iopub.status.busy":"2023-03-16T20:37:31.227691Z","iopub.status.idle":"2023-03-16T20:37:31.235714Z","shell.execute_reply":"2023-03-16T20:37:31.23455Z"},"papermill":{"duration":0.025662,"end_time":"2023-03-16T20:37:31.238397","exception":false,"start_time":"2023-03-16T20:37:31.212735","status":"completed"},"tags":[]},"outputs":[{"data":{"text/plain":["'took 1 hour assign valvet thn prepare food like 30 mins deliver also 4 valvet near cant delivery ur help chat talking 34 ppl cant still get delivery timei didnt got support support chat system also restaurant non responsive cook order time valet system assigning late mistake happens ur chat support system credit voucher seriously waste time order ds app n restaurant'"]},"execution_count":30,"metadata":{},"output_type":"execute_result"}],"source":["# random text to verify stemming\n","zomato['review'][5475]"]},{"cell_type":"code","execution_count":31,"id":"01eb9723","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:31.26992Z","iopub.status.busy":"2023-03-16T20:37:31.268604Z","iopub.status.idle":"2023-03-16T20:37:31.277789Z","shell.execute_reply":"2023-03-16T20:37:31.276663Z"},"papermill":{"duration":0.027349,"end_time":"2023-03-16T20:37:31.280159","exception":false,"start_time":"2023-03-16T20:37:31.25281","status":"completed"},"tags":[]},"outputs":[{"data":{"text/plain":["'took 1 hour assign valvet thn prepar food like 30 min deliv also 4 valvet near cant deliveri ur help chat talk 34 ppl cant still get deliveri timei didnt got support support chat system also restaur non respons cook order time valet system assign late mistak happen ur chat support system credit voucher serious wast time order ds app n restaur'"]},"execution_count":31,"metadata":{},"output_type":"execute_result"}],"source":["# after stemming some of the words are changed, some words might not make sense as it's rule based apporach.\n","simple_stemmer(zomato['review'][5475])"]},{"cell_type":"code","execution_count":32,"id":"491aabd3","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:31.311514Z","iopub.status.busy":"2023-03-16T20:37:31.310054Z","iopub.status.idle":"2023-03-16T20:37:31.429858Z","shell.execute_reply":"2023-03-16T20:37:31.428379Z"},"papermill":{"duration":0.138193,"end_time":"2023-03-16T20:37:31.4328","exception":false,"start_time":"2023-03-16T20:37:31.294607","status":"completed"},"tags":[]},"outputs":[{"name":"stderr","output_type":"stream","text":["[nltk_data] Downloading package punkt to /usr/share/nltk_data...\n","[nltk_data]   Package punkt is already up-to-date!\n","[nltk_data] Downloading package averaged_perceptron_tagger to\n","[nltk_data]     /usr/share/nltk_data...\n","[nltk_data]   Package averaged_perceptron_tagger is already up-to-\n","[nltk_data]       date!\n"]}],"source":["from nltk.tag import pos_tag\n","from nltk.tokenize import word_tokenize\n","nltk.download('punkt')\n","nltk.download('averaged_perceptron_tagger')\n","#Lemmatizer example\n","def lemmatize_all(sentence):\n","    wnl = WordNetLemmatizer()\n","    for word, tag in pos_tag(word_tokenize(sentence)):\n","        if tag.startswith(\"NN\"):\n","            yield wnl.lemmatize(word, pos='n')\n","        elif tag.startswith('VB'):\n","            yield wnl.lemmatize(word, pos='v')\n","        elif tag.startswith('JJ'):\n","            yield wnl.lemmatize(word, pos='a')\n","        else:\n","            yield word\n","            \n","def lemmatize_text(text):\n","    return ' '.join(lemmatize_all(text))"]},{"cell_type":"code","execution_count":33,"id":"d20beb74","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:31.463833Z","iopub.status.busy":"2023-03-16T20:37:31.463393Z","iopub.status.idle":"2023-03-16T20:37:31.472728Z","shell.execute_reply":"2023-03-16T20:37:31.471364Z"},"papermill":{"duration":0.028493,"end_time":"2023-03-16T20:37:31.475588","exception":false,"start_time":"2023-03-16T20:37:31.447095","status":"completed"},"tags":[]},"outputs":[{"data":{"text/plain":["'took 1 hour assign valvet thn prepare food like 30 mins deliver also 4 valvet near cant delivery ur help chat talking 34 ppl cant still get delivery timei didnt got support support chat system also restaurant non responsive cook order time valet system assigning late mistake happens ur chat support system credit voucher seriously waste time order ds app n restaurant'"]},"execution_count":33,"metadata":{},"output_type":"execute_result"}],"source":["zomato['review'][5475]"]},{"cell_type":"code","execution_count":34,"id":"bc0590d5","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:31.506875Z","iopub.status.busy":"2023-03-16T20:37:31.506479Z","iopub.status.idle":"2023-03-16T20:37:31.686573Z","shell.execute_reply":"2023-03-16T20:37:31.685075Z"},"papermill":{"duration":0.199654,"end_time":"2023-03-16T20:37:31.689454","exception":false,"start_time":"2023-03-16T20:37:31.4898","status":"completed"},"tags":[]},"outputs":[{"name":"stderr","output_type":"stream","text":["[nltk_data] Downloading package omw-1.4 to /usr/share/nltk_data...\n"]},{"data":{"text/plain":["True"]},"execution_count":34,"metadata":{},"output_type":"execute_result"}],"source":["import nltk\n","nltk.download('omw-1.4')"]},{"cell_type":"code","execution_count":35,"id":"aa3ada34","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:31.720171Z","iopub.status.busy":"2023-03-16T20:37:31.719671Z","iopub.status.idle":"2023-03-16T20:37:33.826803Z","shell.execute_reply":"2023-03-16T20:37:33.825177Z"},"papermill":{"duration":2.126109,"end_time":"2023-03-16T20:37:33.830012","exception":false,"start_time":"2023-03-16T20:37:31.703903","status":"completed"},"tags":[]},"outputs":[{"data":{"text/plain":["'take 1 hour assign valvet thn prepare food like 30 min deliver also 4 valvet near cant delivery ur help chat talk 34 ppl cant still get delivery timei didnt get support support chat system also restaurant non responsive cook order time valet system assign late mistake happen ur chat support system credit voucher seriously waste time order d app n restaurant'"]},"execution_count":35,"metadata":{},"output_type":"execute_result"}],"source":["# Lemmatization gives the root of any word ex: Playing -> play, gone -> go, am -> be etc..\n","lemmatize_text(zomato['review'][5475])"]},{"cell_type":"code","execution_count":36,"id":"edc1d5f4","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:33.860941Z","iopub.status.busy":"2023-03-16T20:37:33.860504Z","iopub.status.idle":"2023-03-16T20:37:33.86879Z","shell.execute_reply":"2023-03-16T20:37:33.867385Z"},"papermill":{"duration":0.026799,"end_time":"2023-03-16T20:37:33.871326","exception":false,"start_time":"2023-03-16T20:37:33.844527","status":"completed"},"tags":[]},"outputs":[{"data":{"text/plain":["'took 1 hour assign valvet thn prepare food like 30 mins deliver also 4 valvet near cant delivery ur help chat talking 34 ppl cant still get delivery timei didnt got support support chat system also restaurant non responsive cook order time valet system assigning late mistake happens ur chat support system credit voucher seriously waste time order ds app n restaurant'"]},"execution_count":36,"metadata":{},"output_type":"execute_result"}],"source":["#Raw example of randomly selected review text\n","zomato['review'][5475]"]},{"cell_type":"code","execution_count":37,"id":"e7ac805c","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:33.903659Z","iopub.status.busy":"2023-03-16T20:37:33.903245Z","iopub.status.idle":"2023-03-16T20:37:38.546348Z","shell.execute_reply":"2023-03-16T20:37:38.54504Z"},"papermill":{"duration":4.662524,"end_time":"2023-03-16T20:37:38.549237","exception":false,"start_time":"2023-03-16T20:37:33.886713","status":"completed"},"tags":[]},"outputs":[],"source":["zomato['review'] = zomato['review'].apply(lemmatize_text)"]},{"cell_type":"code","execution_count":38,"id":"549e628d","metadata":{"execution":{"iopub.execute_input":"2023-03-16T20:37:38.579575Z","iopub.status.busy":"2023-03-16T20:37:38.579162Z","iopub.status.idle":"2023-03-16T20:37:38.587943Z","shell.execute_reply":"2023-03-16T20:37:38.586181Z"},"papermill":{"duration":0.027638,"end_time":"2023-03-16T20:37:38.591157","exception":false,"start_time":"2023-03-16T20:37:38.563519","status":"completed"},"tags":[]},"outputs":[{"data":{"text/plain":["'take 1 hour assign valvet thn prepare food like 30 min deliver also 4 valvet near cant delivery ur help chat talk 34 ppl cant still get delivery timei didnt get support support chat system also restaurant non responsive cook order time valet system assign late mistake happen ur chat support system credit voucher seriously waste time order d app n restaurant'"]},"execution_count":38,"metadata":{},"output_type":"execute_result"}],"source":["# Processed example of randomly selected review text \n","# in below text some changes are after applying lemmatization are talking -> talk, assigning -> assign ,happens-> happen.. etc\n","zomato['review'][5475]"]},{"cell_type":"markdown","id":"ff378563","metadata":{"papermill":{"duration":0.013641,"end_time":"2023-03-16T20:37:38.619231","exception":false,"start_time":"2023-03-16T20:37:38.60559","status":"completed"},"tags":[]},"source":["# 6. Conclusion\n","As you can see in the output of the code given above, the review still doesn't make complete grammatical sense. However, you can still realise the value of text preprocessing here. Several unwanted characters have been removed. You can perform a decent amount of text analysis based on this preprocessed text."]},{"cell_type":"code","execution_count":null,"id":"984be5ae","metadata":{"papermill":{"duration":0.013715,"end_time":"2023-03-16T20:37:38.647001","exception":false,"start_time":"2023-03-16T20:37:38.633286","status":"completed"},"tags":[]},"outputs":[],"source":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.12"},"papermill":{"default_parameters":{},"duration":23.54796,"end_time":"2023-03-16T20:37:39.887274","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2023-03-16T20:37:16.339314","version":"2.3.4"}},"nbformat":4,"nbformat_minor":5}