webscraping_workshop.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/samifriedrich/webscraping_workshop/blob/main/webscraping_workshop.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GRa_1D3KYaqm"
   },
   "source": [
    "# Web Scraping in python: From HTML Soup to Tidy Data\n",
    "\n",
    "Thursday, Nov 19, 2020\n",
    "\n",
    "[BioData Club](https://biodata-club.github.io/) Workshop Series\n",
    "\n",
    "- Author: Sami Friedrich\n",
    "- Created: Nov 11, 2020\n",
    "- Updated Nov 19, 2020\n",
    "- Libraries used:\n",
    "  - `requests`\n",
    "  - `BeautifulSoup4`\n",
    "  - `pandas`\n",
    "- Additional tools used:\n",
    "  - the browser Inspector/Inspect tool\n",
    "\n",
    "\n",
    "## Workshop overview\n",
    "The internet is overflowing with data ripe for harvesting. The challenge is that not all of that data is formatted neatly or easily accessible. Enter the web scraping multitool! With the power of web scraping, the contents of virtually any webpage can be transformed into analysis-ready data. During this workshop, you’ll learn using python how to:\n",
    " \n",
    "1. Scavenge the contents of an HTML webpage\n",
    "2. Extract only the data you want\n",
    "3. Format the data into a table\n",
    "\n",
    "### Before we get started\n",
    "1. Some basic python knowledge (looping through list elements, passing arguments to functions, writing basic functions) is a prerequisite for this workshop. \n",
    "  - If you are new to python or want to brush up on these topics before the workshop, check out these free tutorials:\n",
    "    - http://introtopython.org/introducing_functions.html\n",
    "    - http://introtopython.org/lists_tuples.html#Lists-and-Looping\n",
    "2. We will also be working with HTML, and no prior experience is necessary.  However, it will be helpful to have a surface-level understanding of HTML elements - namely, their open/close tag structure, and how they nest within each other.\n",
    "  - If you are not familiar with HTML elements or tags, please take a look at [this short overview on HTML Basics](https://developer.mozilla.org/en-US/docs/Learn/Getting_started_with_the_web/HTML_basics) before beginning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4khpJzLdfpCq"
   },
   "source": [
    "## Intro to Jupyter Notebooks\n",
    "We'll be working through exercises in a [Jupyter Notebook](https://jupyter.org/) hosted on Google CoLabs. A Jupyter Notebook is a coding environment that contains two types of cells - cells with narrative/explanation, like this one you're reading now, and cells with executable code, like the next cell down. Run the code in the cell below by clicking on the cell and pressing the \"play\" button to the left of the cell, or typing `SHIFT+ENTER`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "H55kCttA2RWy"
   },
   "outputs": [],
   "source": [
    "# This is a comment and will not affect the output\n",
    "# Run this code cell\n",
    "message = \"Welcome to web scraping\"\n",
    "print(message)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0GDiOhbf2qh0"
   },
   "source": [
    "The output of a code cell is displayed below it - in this case, the output is the printed string `Welcome to web scraping`. \n",
    "\n",
    "### A note on printing vs displaying in Notebooks\n",
    "Usually in python environments you need to use the `print()` function to see the contents of a variable. One nice feature of Jupyter Notebooks is a shortcut where you simply type the name of the variable to display it below the code cell. \n",
    "\n",
    "### Exercise: Display a variable in a Jupyter Notebook\n",
    "In the cell below, type the variable `message` on the second line and run the cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "dgp2BVgx3rDG"
   },
   "outputs": [],
   "source": [
    "message = \"Welcome to web scraping\"\n",
    "# Write your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "G99Z0vP7Yaqs"
   },
   "source": [
    "That's all you need to know about notebooks for today, but head over to [Jupyter Project](https://jupyter.org/) to learn more.\n",
    "\n",
    "# Step 1: Loading the webpage HTML into python\n",
    "\n",
    "## 1.1 Sending a GET request\n",
    "\n",
    "First, we need to get a webpage's HTML into our python environment. We can do that using the `requests` library.\n",
    "\n",
    "Whenever you type a URL into your browser bar or click a link, your browser makes what is called a `GET` request to the web server hosting the webpage. What your browser receives back from the server is the information and resources to load that webpage, including the HTML, which it then renders in your browser window.\n",
    "\n",
    "We can use the `requests` library to send `GET` requests from within python.\n",
    "\n",
    "### Exercise 1.1.1: Make a GET request using the `requests` library\n",
    "Run the code block below by hitting the \"Run\" button, or pressing SHIFT+RETURN."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "wP3s237rYaqt"
   },
   "outputs": [],
   "source": [
    "# Run this code cell\n",
    "import requests\n",
    "url = \"http://dataquestio.github.io/web-scraping-pages/simple.html\"\n",
    "page = requests.get(url)\n",
    "page"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QUoTd0FFYaq3"
   },
   "source": [
    "What we get back is a Response object, which when printed displays a `status_code` that tells us how our request was handled. A status code of 200 means everything went smoothly and the webpage data was received. \n",
    "\n",
    "You're probably familiar with error codes like \"404\" - the \"not found error.\" We won't get into these other codes today but you can read more about them [here](https://en.wikipedia.org/wiki/List_of_HTTP_status_codes)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IoZ6rmN1Yaq4"
   },
   "source": [
    "### Exercise 1.1.2: Extract the HTML from the `page` object\n",
    "Take a look at the methods available to the `page` variable by typing `page.` and scrolling through the list in the pop-up box. **Pay particular attention to the methods with the blue box symbol.** Try running one of these on the `page` variable. Experiment until you find a method that outputs something that looks like the page's HTML, and assign the output to a variable named `contents`.\n",
    "\n",
    "*Hint: The HTML should have the word \"doctype\" towards the top*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "LIaWaETYYaq4"
   },
   "outputs": [],
   "source": [
    "# Experiment here by trying out different methods on the page object\n",
    "# Write your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RON4UJyLYaq8"
   },
   "source": [
    "Now we have the tools we need from the `requests` library to retrieve a webpage, extract the HTML, and save it as a variable in our python environment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "o4DuLlu9Yaq9"
   },
   "source": [
    "## 1.2 Our business case: \"One of your favorite beers is on tap!\"\n",
    "\n",
    "Let's pretend you're on a developer team that is working to build an app for beer enthusiasts. The team wants to design a feature that sends a notification to the user whenever a beer from the user's favorite beer list pops up on tap around town. It's your job as the PDX Data Wrangler to gather data about what's on tap at bars around Portland.\n",
    "\n",
    "To achieve this goal, the data we need to capture for each bar is:\n",
    "- the name of the beers on tap\n",
    "- the name of the breweries for those beers\n",
    "\n",
    "We'll start by choosing one bar to build our scraper around: Belmont Station. \n",
    "\n",
    "It just so happens that Belmont Station uses a service, TapHunter, to display their rotating tap list on their [bar's website](https://www.belmont-station.com/on-tap). This tap list is also available on [TapHunter](https://www.taphunter.com/location/belmont-station/6549318358269952), as are tap lists from other bars. \n",
    "\n",
    "We could scrape either the bar's page, or the TapHunter page. The benefit of designing our web scraper around the TapHunter page is that we're able to **reuse our code to scrape data for multiple bars** from TapHunter. \n",
    "\n",
    "\n",
    "### Exercise 1.2: Load in the HTML for Belmont Station's Taphunter page\n",
    "\n",
    "To avoid overloading the web servers over at TapHunter and ensure we're all looking at the exact same HTML, I downloaded the HTML of Belmont Station's TapHunter page this morning (Nov 19), then uploaded that .html file to a GitHub repository. Because this .html file is still a webpage (now hosted on GitHub servers), we can use the same approach we did above to retrieve its HTML. \n",
    "\n",
    "Using the `requests` library method we learned above, retrieve the webpage from the URL below, and save the HTML content to a variable named `contents`. Display the `contents` variable by typing it on the last line of your code block, then run the code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "LEzd9sqVpVSG"
   },
   "outputs": [],
   "source": [
    "# Retrieve the web content from the url below and save the HTML to the variable \"contents\"\n",
    "url = \"https://raw.githubusercontent.com/samifriedrich/webscraping_workshop/main/taphunter_belmont_station.html\"\n",
    "# Write your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3AJT-GSzYarA"
   },
   "source": [
    "As it stands, the `contents` variable we created is one, long, unweildy stream of characters with the type `str` (string). We could try to parse it as is, but why brute force it when there are pre-built handy tools to help us make sense of this soup?\n",
    "\n",
    "Enter the `BeautifulSoup` library!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0MYAvLruYarB"
   },
   "source": [
    "## 1.3 Time to make soup\n",
    "\n",
    "Our next step is to transform the long unwieldy HTML string contained in our `contents` variable into a `BeautifulSoup` object. \n",
    "\n",
    "We first import the `BeautifulSoup` data structure from the bs4 (BeautifulSoup4) library. Run the code block below to do this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "g9e1yX_tYarB"
   },
   "outputs": [],
   "source": [
    "# Run this code cell\n",
    "from bs4 import BeautifulSoup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YsVr0GX2YarE"
   },
   "source": [
    "### Upgrading `contents` from string to soup\n",
    "\n",
    "With the BeautifulSoup library loaded, next we \"pour\" our `contents` variable into a BeautifulSoup object, thus creating a new variable. Under the hood, BeautifulSoup will parse the `contents` variable, devising its structure based on rules of HTML, and making it a much more accessible and interactive object.\n",
    "\n",
    "Though it may seem confusing at first, BeautifulSoup is the name of a web scraping library but it is also the name of a function, `BeautifulSoup()`, within that library. \n",
    "\n",
    "You can think of the `BeautifulSoup()` constructor as a function that puts a wrapper around whatever object we pass it. Our original `contents` string will remain intact, but it will be transformed into a BeautifulSoup object that is decorated and **supercharged with methods** beyond what base python offers us. These are the benefits of upgrading `contents` from a `str` type object to a `BeautifulSoup` type object.\n",
    "\n",
    "We create a BeautifulSoup object by passing `contents` to the `BeautifulSoup()` constructor function, and saving the output to a new variable.\n",
    "\n",
    "Because BeautifulSoup can parse other kinds of documents, such as XML, we also need to pass in one additional argument, the string `\"html.parser\"`, to tell `BeautifulSoup()` to parse our string as HTML.\n",
    "\n",
    "Here's an example of how to use the `BeautifulSoup()` constructor:\n",
    "```python\n",
    "bs_object = BeautifulSoup(html_var, \"html.parser\")\n",
    "```\n",
    "### Exercise 1.3: Creating a BeautifulSoup object\n",
    "\n",
    "Call the `BeautifulSoup()` constructor function on the `contents` variable, and pass the string `\"html.parser\"` as the second argument to the function. Save this to a variable named `soup`, and display `soup`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "shdgfW2WYarE"
   },
   "outputs": [],
   "source": [
    "# Transform the contents variable into a BeautifulSoup object \n",
    "# Write your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "t7-T1vi7YarN"
   },
   "source": [
    "## 1.4 Printing pretty with `.prettify()`\n",
    "\n",
    "With our new, supercharged BeautifulSoup object, `soup`, we can use methods that didn't exist for the `contents` string-type variable.\n",
    "\n",
    "One great feature of BeautifulSoup object is its `.prettify()` method, which formats a BeautifulSoup object with indentation, making it easier to read and see how HTML elements are nested. \n",
    "\n",
    "### Exercise 1.4: Print some pretty soup\n",
    "Try using the `.prettify()` method on our `soup` BeautifulSoup object. \n",
    "\n",
    "This method **requires open and closed parentheses** after it.\n",
    "\n",
    "To see the effects of this method, the \"prettified\" object **must be printed**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "_UbmQwKOYarN"
   },
   "outputs": [],
   "source": [
    "# Use the .prettify method on soup and print the result\n",
    "# Write your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "q1z44hwcYarQ"
   },
   "source": [
    "# Step 2: Straining the data from the soup\n",
    "\n",
    "Congrats! We have successfully transferred a webpage's HTML into a pythonic BeautifulSoup object.\n",
    "\n",
    "The next step is to strain the soup - that is, extract the data we're interested in. \n",
    "\n",
    "As we move through this section, keep in mind that there are multiple ways to go about extracting the data we want, not just the one outlined here. If another route makes more sense to you, do that!\n",
    "\n",
    "There are lots of ways to navigate the BeautifulSoup object, including moving linearly through it one element at a time, much like iterating over every element in a list. There are also shortcuts to navigate to specific elements if we already have an idea of what we're looking for.\n",
    "\n",
    "Today, we're going to focus on how to pull out specific elements using a few BeautifulSoup methods: `.find()` and `.find_all()`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6qR0c6HmYarR"
   },
   "source": [
    "## 2.1 The CTRL+F of BeautifulSoup: `.find_all()`\n",
    "\n",
    "`.find_all()` is probably the method you'll use most often when extracting data from a BeautifulSoup object. This method searches the BeautifulSoup object for an element or string, and returns the results as a ResultSet object. We can treat the ResultSet object just like a regular python list that has some bonus methods attached to it.\n",
    "\n",
    "#### A note on `.find()` vs `.find_all()`:\n",
    "\n",
    "BeautifulSoup also has a `.find()` method. The main difference is that `.find()` returns only the first search result as a single element, while `.find_all()` returns all results as a list of elements. \n",
    "\n",
    "#### Searching for HTML tags\n",
    "\n",
    "The `.find_all()` method is quite flexible in terms of what it can search for, but one common way to use `.find_all()` is to extract all instances of a given HTML tag type (`<body>`, `<p>`, `<div>`, etc.). \n",
    "\n",
    "For example, this code would extract all the `div` headers:\n",
    "\n",
    "```python\n",
    "soup.find_all(\"div\")\n",
    "```\n",
    "When using this approach, you do **not** need to include the `<>` characters - just pass in the tag label. \n",
    "\n",
    "### Exercise 2.1: Extracting all hyperlink tags\n",
    "Use the `.find_all()` method to extract all the hyperlinks from our BeautifulSoup object `soup`. Assign the result to `hyperlinks` and display it. \n",
    "\n",
    "*Hint: Hyperlinks are denoted by `<a>` tags.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "k-ijzchcYarR"
   },
   "outputs": [],
   "source": [
    "# Extract all hyperlink tags and print the result\n",
    "# Write your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xAIsP_8TYarV"
   },
   "source": [
    "We can use this same `.find_all(\"tag\")` method to extract all elements of any tag type."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0DIfjMQbYarW"
   },
   "source": [
    "## 2.2 Extracting tags with specific attributes or classes\n",
    "\n",
    "HTML opening tags often contain more than just the tag name itself. These extra bits are called attributes and classes. \n",
    "- Attributes follow the tag name and often have an X=Y structure, e.g. `<h1 lang=\"en\">`. \n",
    "- Classes follow the same structure and serve as a shorthand for multiple attributes defined elsewhere, e.g. `<h1 class=\"breaking-headline\">`. \n",
    "\n",
    "I like to think of of tag classes as wrappers that can be applied to HTML tags to style the elements they contain.\n",
    "\n",
    "In addition to passing in HTML tag types to `.find_all()`, we can also specify attributes or classes associated with tags to further filter our results.\n",
    "\n",
    "For example, we may have 3 different flavors of `<div>` tags throughout our HTML:\n",
    "- `<div class=\"stawberry\">`\n",
    "- `<div class=\"chocolate\">`\n",
    "- `<div class=\"pistacchio\">`\n",
    "\n",
    "We're only interested in the pistacchio `<div>` tags. If we run `.find_all('div')`, we will get back all the `<div>` tags, including the strawberry and chocolate ones. \n",
    "\n",
    "To return only the \"pistacchio\" class `<div>` tags, we pass an additional argument to `.find_all()` - the `class_` argument. \n",
    "\n",
    "#### **Notice the underscore in this `class_` parameter, which delineates it from `class`, which is a reserved word in base python.**\n",
    "\n",
    "To get just the \"pistacchio\" `<div>` elements, we can run:\n",
    "```python\n",
    "soup.find_all(\"div\", class_=\"pistacchio\")\n",
    "```\n",
    "\n",
    "which would return only the `<div class=\"pistacchio\">` tags.\n",
    "\n",
    "### Exercise 2.2: Extract all hyperlinks of a specific class\n",
    "\n",
    "Extract all hyperlink tags from `soup` that have the class \"gtm-link\", and save them to a variable named `a_gtm_link`. Display the `a_gtm_link` variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "zvokbyvLYarW"
   },
   "outputs": [],
   "source": [
    "# Extract all hyperlink elements of class \"gtm-link\" and save to variable a_gtm_link\n",
    "# Write your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mV8LNPpeYarZ"
   },
   "source": [
    "Notice that the first element has multiple classes: `class=\"btn btn-primary gtm-link\"`, one of which matches our `class_` parameter \"gtm-link.\" When you search for a tag that matches a certain class, **you’re matching against any of its classes.**\n",
    "\n",
    "There are BeautifulSoup methods that let us search for elements that match multiple classes and much, much more, but we won't cover that today. If you're interested in this powerful method, check out the [BeautifulSoup documentation](https://www.crummy.com/software/BeautifulSoup/bs4/doc/#css-selectors) on selecting CSS classes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "w9n4-DNhYarb"
   },
   "source": [
    "## 2.3 Locating target data within the HTML\n",
    "\n",
    "Now we understand how to use the `.find_all()` method to extract specific elements based on their tag type and class. The next step is to apply this method to extract the data we want.\n",
    "\n",
    "To extract our target data (the name of the beer and brewery), we first have to figure out **where that data lives in the webpage's HTML**. There are lots of ways to go about this. We could experiment with different searches using `.find_all()`, or we could open and `CTRL+F` the HTML text file in a word processor. \n",
    "\n",
    "Personally, I prefer to use an easy and interactive method that makes use of your browser's built-in **inspector** tool.\n",
    "\n",
    "## The broswer inspector\n",
    "A **browser inspector** is a tool included in most browsers that lets you interactively examine the HTML code underlying webpage elements. What makes using the inspector intuitive is that when you right click on a web page element and choose \"Inspect\", it highlights the HTML corresponding to that element. The inspector will also live-update to highlight corresponding visual elements on the webpage as you mouse over the HTML. The inspector is an essential component of the web scraping multitool!\n",
    "\n",
    "### Exercise 2.3 Exploring HTML using a browser inspector\n",
    "Let's try it now on the TapHunter page for [Belmont Station](https://www.taphunter.com/location/belmont-station/6549318358269952). Open the page in your browser, right click on a beer or brewery, and select \"Inspect\" to open the inspector window. \n",
    "\n",
    "**NOTE:** Keep in mind the tap list may have changed since we downloaded it, but the overall layout of the webpage and HTML will be the same.\n",
    "\n",
    "While we're poking around using the inspector, let's try to figure out which HTML tag types contain our target data. As a reminder, the data we want to capture is:\n",
    "- the name of the beer\n",
    "- the name of the brewery\n",
    "\n",
    "Where do these data show up on the webpage? **Which HTML tag type contains *both* the name of the beer and the brewery name?** Are there attributes or classes that might help us target those tags for extraction using `.find_all()`?\n",
    "\n",
    "*Hint: Repeated visual elements will share repeated HTML code structure.*\n",
    "\n",
    "*Hint: Look for a `<div>` with a named class.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GBWMkGwrYarc"
   },
   "source": [
    "## 2.4 Extracting data-containing elements\n",
    "\n",
    "At this point, we've identified where in the HTML our data lives. Using the extraction methods we learned earlier, we can isolate those elements from the soup.\n",
    "\n",
    "### Exercise 2.4.1 Extracting chunks of data\n",
    "\n",
    "Now that we know what tag and class contains all the info related to each beer, let's use `.find_all()` to extract only those elements from our `soup` object. Save those elements to a variable named `tap_list`, and display `tap_list`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "lwVpV1NQYarc"
   },
   "outputs": [],
   "source": [
    "# Extract the element containing each beer's info,\n",
    "# and name the result list tap_list\n",
    "# Write your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NnYMvuXRYart"
   },
   "source": [
    "### Exercise 2.4.2 Printing extracted elements\n",
    "\n",
    "Because of the BeautifulSoup features attached to `tap_list`, we can use some of the same methods on individual elements (beer entries) in `tap_list` that we used earlier on our `soup` object.\n",
    "\n",
    "Assign the **third** list element of `tap_list` to a variable named `third_beer_entry`, then use `.prettify()` to print it.\n",
    "\n",
    "*Hint: python uses 0-based indexing,* e.g. `my_list[0]` is the first element in `my_list`, and `my_list[1]` is the second element."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "AdcfDzK1Yart"
   },
   "outputs": [],
   "source": [
    "# Pretty-print the third element of tap_list\n",
    "# Write your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aDlQLdQIYary"
   },
   "source": [
    "With indentation, it's much easier to see the HTML structure of these individual beer entries.\n",
    "\n",
    "If we printed each list element (beer entry) in `tap_list`, they would all share similar structure to what we see above for `third_beer_entry`. You can also verify this by returning to the browswer inspector and noticing patterns in the HTML for each beer entry.\n",
    "\n",
    "### Exercise 2.4.3: Identifying enclosing tags\n",
    "Before we move on, take another look at `third_beer_entry` printed above and locate the **tags that *immediately* contain**:\n",
    "- the beer name\n",
    "- the brewery name\n",
    "\n",
    "What kinds of tags are they?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZqSYCXFIYary"
   },
   "source": [
    "## 2.5 Straining extracted elements\n",
    "\n",
    "Clearly, there's a lot of extra data in each of these `tap_list` elements that we don't need, like the ABV and type of beer. We need to pull out:\n",
    "- the beer name\n",
    "- the brewery name\n",
    "\n",
    "Because elements extracted from `soup` inherit some of the special methods of BeautifulSoup objects, we can use `.find_all()` on each list element in `tap_list` to further search within each beer entry.\n",
    "\n",
    "As we discovered when printing the `third_beer_entry` above, the name of the beer sits inside a hyperlink or `<a>` tag. Lucky for us, there is only one hyperlink per beer entry. Because `<a>` is a unique tag within each beer entry:\n",
    "- we can us the `<a>` tag to extract the name of the beer\n",
    "- we can use `.find()` instead of `.find_all()` to return only one result\n",
    "\n",
    "Let's try that now on the `third_beer_entry` variable.\n",
    "\n",
    "### Exercise 2.5.1: Extracting the beer name\n",
    "\n",
    "Use `.find()` to extract the first (and only) hyperlink element in `third_beer_entry`. Save the results to a variable named `a_tag`, and print it.\n",
    "\n",
    "*Hint: hyperlinks use the `<a>` tag.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "LPw-mn-9Yarz"
   },
   "outputs": [],
   "source": [
    "# Extract one hyperlink from the 'beer_entry' object\n",
    "# Write your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aybXshAPYar1"
   },
   "source": [
    "Once you extract it, check out the data type for `a_tag`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4t6B0dedYar1"
   },
   "outputs": [],
   "source": [
    "# Run this code cell\n",
    "type(a_tag)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ocq3TN-zYar4"
   },
   "source": [
    "Notice it's not a string, but rather a BeautifulSoup object called a Tag. Just like `third_beer_entry` has BeautifulSoup methods, `a_tag` also has BeautifulSoup methods, all inherited from our original BeautifulSoup object, `soup`. You can learn more about the Tag object [here](https://www.crummy.com/software/BeautifulSoup/bs4/doc/#tag)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TwQ2JE2CYar4"
   },
   "source": [
    "###  2.5.2 Stripping the tags\n",
    "\n",
    "Now we've reached the smallest unit - the single HTML element - that immediately encloses the name of the beer:\n",
    "\n",
    "`<a href=\"/beer/barley-browns-pallet-jack-ipa/5064484871733248\">Barley Brown's Pallet Jack IPA</a>`\n",
    "\n",
    "We used tags as signposts to navigate through our `soup` and `tap_list` objects until we reached our target data. However, we don't want to include the openeing and closing tags or attributes in our final data table. We only want the name of the beer, which is the **text** between the opening and closing `<a>` tags, `Barley Brown's Pallet Jack IPA`.\n",
    "\n",
    "The way to extract only the text from a tag object is the `.get_text()` method, e.g. \n",
    "```python\n",
    "bsTag.get_text()\n",
    "```\n",
    "\n",
    "### Exercise 2.5.2 Retriving the beer name as a simple string\n",
    "\n",
    "Use the `.get_text()` method to extract the text from `a_tag`. Assign the result to `beer_name` and display `beer_name`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "2bvRZDNKYar5"
   },
   "outputs": [],
   "source": [
    "# Write code to extract the text from the a_tag object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wDhg-LsiYar8"
   },
   "source": [
    "## 2.5.3 Combining our extraction steps into a for loop\n",
    "\n",
    "We've written code to extract the beer name from a *single* beer entry of the list object we generated called `tap_list`. Now it's time to apply the same process to the rest of the entries in `tap_list`.\n",
    "\n",
    "### Exercise 2.5.3: Putting it all together with loops\n",
    "\n",
    "Let's write some code to extract all the beer names by looping through our `tap_list`. \n",
    "\n",
    "Construct a `for` loop using the code bits we've written above to find the tag containing the beer name and extract just the text. Add a final step in the loop that appends the beer name to the list `beer_names`, which has been initialized below. Finally, display the `beer_names` list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "S-OmhibdYar8"
   },
   "outputs": [],
   "source": [
    "# Fill in the '--'s to create a loop that extracts all the beer names from tap_list\n",
    "beer_names = []\n",
    "\n",
    "for beer in tap_list:\n",
    "  a_tag = beer.--\n",
    "  beer_name = a_tag.--\n",
    "  beer_names.append(--)\n",
    "\n",
    "beer_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9XYB5uIZYar-"
   },
   "source": [
    "Congrats! You've just extracted a data type from a webpage and never once used copy/paste shortcuts!\n",
    "\n",
    "Before we move on to extracting the brewery name, notice that the last two entries in `beer_names` are wineries, not breweries. You can ignore those two for now - we'll filter them out later. Just keep in mind that the last two entries of `tap_list` are wine, not beer, so their associated entries in `tap_list` might look a little different than the beer entries."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ks8IRxiWYar_"
   },
   "source": [
    "## 2.5.3 Extracting the brewery name\n",
    "\n",
    "Let's repeat the process we just completed for beer names to now extract the brewery names.\n",
    "\n",
    "### Exercise 2.5.3 Taking another look at one beer entry's HTML\n",
    "Execute the code block below, then take another look at the tags within the `third_beer_entry` from `tap_list`. Identify the type of tag flanking the brewery name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "U3DK0FahYar_"
   },
   "outputs": [],
   "source": [
    "# Run this code cell\n",
    "third_beer_entry = tap_list[2]\n",
    "print(third_beer_entry.prettify())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "v0aedzCoYasB"
   },
   "source": [
    "### Exercise 2.5.2 Extract the brewery name from a single beer entry\n",
    "\n",
    "Unlike the unique `<a>` tag that we used to extract the beer name, the brewery name is enclosed in a `<span>` tag. As evidenced in the printed entry above, `<span>` tags appear multiple times within a single beer entry. This means extracting the brewery name will be a little trickier than extracting the beer name.\n",
    "\n",
    "Before we build a loop, let's first extract the `<span>` tags from a single beer entry. \n",
    "\n",
    "In terms of strategy, we'll need to:\n",
    "- Use `.find_all()` instead of .`find()`, which returns a list of results\n",
    "- Use list indices to select the `<span>` element containing the brewery name from the list of results\n",
    "- Assign the **text only** of the single tag containing the brewery name to a variable called `brewery_name`, and display it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "m2wEqdcwYasB"
   },
   "outputs": [],
   "source": [
    "# Fill in the '--'s to extract the brewery name\n",
    "span_list = third_beer_entry.--\n",
    "brewery_span = span_list[--]\n",
    "brewery_name = brewery_span.--\n",
    "brewery_name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-y_WV1SIYasD"
   },
   "source": [
    "### Exercise 2.5.2: Extract the brewery names for all beer entries in `tap_list`\n",
    "\n",
    "Adapting the code above for a single beer entry, write a `for` loop to extract all the brewery names and append them to a new list named `breweries`. Don't forget to disregard the last two entries of `tap_list` that are wine and not beer.\n",
    "\n",
    "**NOTE:** If you don't get the results you expect, how might you adapt your code to handle slight variations in the tag structure of each beer entry in `tap_list`?\n",
    "\n",
    "*Hint: There are two directions you can move through a list - from the beginning or from the end.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "XYGaNIfjYasD"
   },
   "outputs": [],
   "source": [
    "# Fill in the '--'s to loop through tap_list, extract brewery names, and assign to \"breweries\" list\n",
    "breweries = []\n",
    "\n",
    "for beer in tap_list:\n",
    "  span_list = beer.--\n",
    "  brewery_span = span_list[--]\n",
    "  brewery_name = brewery_span.--\n",
    "  breweries.append(--)\n",
    "\n",
    "breweries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Zb_3TRhaYasF"
   },
   "source": [
    "Is it too early to crack a beer? Because we now have all the data we aimed to collect - beer name, and brewery name - in two neat lists!\n",
    "\n",
    "# Step 3: Formatting the data \n",
    "\n",
    "This last part is a breeze in that it's very strightforward. We'll use the `pandas` library to turn our lists into a dataframe.\n",
    "\n",
    "First, we import the `pandas` library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "OAupV7r8YasG"
   },
   "outputs": [],
   "source": [
    "# Run this code cell\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6ui74jO4YasI"
   },
   "source": [
    "Because we plan to populate our database for the beer alert feature with tap lists from other breweries, it's important to indicate the bar name for each beer entry. Let's create a list that will become the data column for bar names.\n",
    "\n",
    "### Exercise 3.1 Create a bar name list\n",
    "Because all of our data came from the same bar - Belmont Station - we need to generate a list that repeats the bar name for the same number of times as we have beer entries.\n",
    "\n",
    "Create a list called `bars` that is the same length as our `beer_names` and `breweries` lists. Each element of the `bars` list should be the same - the string, \"Belmont Station\".\n",
    "\n",
    "*Hint: The multiplication function can be used on lists.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ydb2t98eYasJ"
   },
   "outputs": [],
   "source": [
    "# Fill in the '--' to create a list called bars of the same length as \n",
    "# beer_names. Every list element of bars should be \"Belmont Station\".\n",
    "num_beers = len(--)\n",
    "bars = ['Belmont Station'] * num_beers\n",
    "bars"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RY-jGDeXYasL"
   },
   "source": [
    "### Exercise 3.2 Build the data table\n",
    "\n",
    "We'll now use `pandas`' DataFrame constructor to build our data table. \n",
    "\n",
    "Let's call the DataFrame `active_taps`, and name our columns \"beer_name\", \"brewery\", and \"bar_name\".\n",
    "\n",
    "*Hint: There are a couple ways to do this, but I like to pass in a dictionary to the DataFrame constructor where the keys are the data column names, and the values are the data lists. For example:*\n",
    "```python\n",
    "pd.DataFrame({'col1_name': list1, \n",
    "              'col2_name': list2, \n",
    "              'col3_name': list3})\n",
    "```\n",
    "\n",
    "Complete the code block below. The column names are provided for you. Fill in the list name that goes with each of these column names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "l4aCLSczYasM"
   },
   "outputs": [],
   "source": [
    "# Fill in the '--'s with our data lists to construct a DataFrame\n",
    "active_taps = pd.DataFrame({\n",
    "    \"beer_name\": --,\n",
    "    \"brewery\": --,\n",
    "    \"bar_name\": --\n",
    "})\n",
    "\n",
    "active_taps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ATsVTTBSYasP"
   },
   "source": [
    "### Exercise 3.3: Toss the wine rows\n",
    "Finally, let's ditch our last two rows, which contain wine data, not beer data.\n",
    "\n",
    "Drop the last two rows and save it back to the `active_taps` DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "iA0lYxuLYasP"
   },
   "outputs": [],
   "source": [
    "# Fill in '--' to drop the last two rows of the DataFrame\n",
    "active_taps = active_taps[--]\n",
    "active_taps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KL5wY6G6YasR"
   },
   "source": [
    "There you have it, from soup to tidy data!\n",
    "\n",
    "From here, you could export the data to a .xlsx or .csv, or build a SQL database, etc., that our hypothetical beer app could then pull in data from to perform our \"favorite beer alert\" feature.\n",
    "\n",
    "# Additional exercises\n",
    "If you had fun web scraping, here's a few more exercises you can try on your own using this Taphunter HTML:\n",
    "- extract the beer type\n",
    "- extract the ABV\n",
    "- extract the brewery location\n",
    "- extract the bar's hours and phone number\n",
    "- turn each extraction step into a function\n",
    "- write a TapHunter web scraping script and try it out on another bar's TapHunter page\n",
    "\n",
    "I highly encourage you to try out your web scraping skills on other websites, as every website's HTML will be as unique as the developers that built it.\n",
    "\n",
    "## Big thanks!\n",
    "I hope you had fun learning how to data-ify the internet! Thanks so much for your time and attention today. \n",
    "\n",
    "I also want to thank the BioData Club organizers (Robin Champeaux, Ted Laderas, Marijane White, and Eric Earl) for this opportunity as well as their support & feedback.\n",
    "\n",
    "# Resources\n",
    "\n",
    "- [requests](https://requests.readthedocs.io/en/master/) library documentation]\n",
    "- [BeautifulSoup4](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) library documentation\n",
    "- [pandas](https://pandas.pydata.org/docs/user_guide/index.html) library documentation\n",
    "- If you like how-to coding books, check out *Web Scraping with Python* by Ryan Mitchell ([link](https://books.google.com/books/about/Web_Scraping_with_Python.html?id=v_k6jwEACAAJ))"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "include_colab_link": true,
   "name": "webscraping_workshop.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}