extraction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scrapfly Extraction API\n",
    "\n",
    "Hi, this is a quick introduction to Scrapfly's Extraction API which a service for extracting data from HTML and text documents using AI and schema-based parsing.\n",
    "\n",
    "It's designed to be simple but powerful, and in this intro we'll take a look at three of it's components:\n",
    "\n",
    "1. LLM based extraction - which allows to parse data using AI prompts.\n",
    "2. AI Auto Extractor - which extracts predefined objects like products, reviews and real estate listings.\n",
    "3. Extraction rules - traditional data parsing simplified through JSON.\n",
    "\n",
    "Before we begin, make sure to grab your API key from the [scrapfly dashboard](https://scrapfly.io/dashboard)\n",
    "\n",
    "To illustrate the examples, we'll be using Python with HTTPX library but the Extraction API can be accessed using any HTTP client like Fetch in javascript or cURL and operates through simple URL parameters. \n",
    "\n",
    "To start, setup the http client with some default values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import httpx\n",
    "\n",
    "client = httpx.Client(\n",
    "    # tip: raise min timeout as long LLM queries can take a few seconds.\n",
    "    timeout=60,  \n",
    "    params={\n",
    "        # your Scrapfly API key\n",
    "        \"key\": \"YOUR SCRAPFLY KEY\",  \n",
    "    },\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by taking a look at LLM extraction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LLM based extraction\n",
    "\n",
    "Extraction API can be used to extract data from HTML pages using LLM (Large Language model) parsers. For this HTML content is sent to ScrapFly with a text prompt of what to extract."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<!DOCTYPE html>\n",
       "<html>\n",
       "  <body>\n",
       "    <h1>Product</h1>\n",
       "    </p>\n",
       "      This product is available \n",
       "      in every store location\n",
       "      for 16.99 USD for limited time!\n",
       "    </p>\n",
       "  </body>\n",
       "</html>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Example of a very simple HTML product page\n",
    "html = \"\"\"\n",
    "<!DOCTYPE html>\n",
    "<html>\n",
    "  <body>\n",
    "    <h1>Product</h1>\n",
    "    </p>\n",
    "      This product is available \n",
    "      in every store location\n",
    "      for 16.99 USD for limited time!\n",
    "    </p>\n",
    "  </body>\n",
    "</html>\n",
    "\"\"\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://api.scrapfly.io/extraction?key=scp-test-14644f57089d47c0aeb705b776792366&extraction_prompt=extract%20price%20and%20currency%20in%20json%20format\n"
     ]
    },
    {
     "data": {
      "application/json": {
       "content_type": "application/json",
       "data": {
        "currency": "USD",
        "price": "16.99"
       }
      },
      "text/plain": [
       "<IPython.core.display.JSON object>"
      ]
     },
     "execution_count": 26,
     "metadata": {
      "application/json": {
       "expanded": false,
       "root": "root"
      }
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "resp = client.post(\n",
    "  \"https://api.scrapfly.io/extraction\",\n",
    "  data=html,\n",
    "  headers={\n",
    "    \"content-type\": \"text/html\",\n",
    "  },\n",
    "  params={\n",
    "    \"extraction_prompt\": \"extract price and currency in json format\"\n",
    "  }\n",
    ")\n",
    "print(resp.url)\n",
    "display.JSON(resp.json())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Integration with Scraper API\n",
    "\n",
    "If you don't have access to HTML datasets yet see [Scrapfly's scraper API](https://scrapfly.io/docs/scrape-api/getting-started) which integrates with extraction API directly and can scrape the data for you."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prompts can be any question\n",
    "\n",
    "The Prompts are not limited to extraction and the AI can be used to answer general questions based on the provided data input or summarize the entire content.\n",
    "\n",
    "In this example, we ask a freeform question about the product's availability using a very simple prompt of \"what locations is this product available in?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://api.scrapfly.io/extraction?key=scp-test-14644f57089d47c0aeb705b776792366&extraction_prompt=what%20locations%20is%20this%20product%20available%20in%3F\n"
     ]
    },
    {
     "data": {
      "application/json": {
       "content_type": "text/plain",
       "data": "The document states that the product is available in every store location."
      },
      "text/plain": [
       "<IPython.core.display.JSON object>"
      ]
     },
     "execution_count": 2,
     "metadata": {
      "application/json": {
       "expanded": false,
       "root": "root"
      }
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "html = \"\"\"\n",
    "<!DOCTYPE html>\n",
    "  <body>\n",
    "    <h1>Product</h1>\n",
    "    </p>\n",
    "      This product is available \n",
    "      in every store location\n",
    "      for 16.99 USD for limited time!\n",
    "    </p>\n",
    "  </body>\n",
    "</html>\n",
    "\"\"\"\n",
    "resp = client.post(\n",
    "    \"https://api.scrapfly.io/extraction\",\n",
    "    data=html,\n",
    "    headers={\n",
    "        \"content-type\": \"text/html\",\n",
    "    },\n",
    "    params={\n",
    "        \"extraction_prompt\": \"what locations is this product \\\n",
    "                              available in?\"\n",
    "    }\n",
    ")\n",
    "print(resp.url)\n",
    "display.JSON(resp.json())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we can see that the AI directly responded to us with a text answer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Real Life Example\n",
    "\n",
    "In this example, we POST the page HTML with prompt asking to extract product specification in JSON format.\n",
    "\n",
    "In response the API returns us a JSON with what it thinks counts as product specification like description, features, packaging and price data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://web-scraping.dev/product/1\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"800\"\n",
       "            src=\"https://web-scraping.dev/product/1\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "            \n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x7f4901cef7d0>"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# html from https://web-scraping.dev/product/1\n",
    "html = Path('product_1.html').read_text()\n",
    "print(\"https://web-scraping.dev/product/1\")\n",
    "display.IFrame(\"https://web-scraping.dev/product/1\", 1000, 800)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://api.scrapfly.io/extraction?key=scp-test-14644f57089d47c0aeb705b776792366&extraction_prompt=extract%20product%20specification%20in%20JSON%20format\n"
     ]
    },
    {
     "data": {
      "application/json": {
       "content_type": "application/json",
       "data": {
        "description": "Indulge your sweet tooth with our Box of Chocolate Candy. Each box contains an assortment of rich, flavorful chocolates with a smooth, creamy filling. Choose from a variety of flavors including zesty orange and sweet cherry. Whether you're looking for the perfect gift or just want to treat yourself, our Box of Chocolate Candy is sure to satisfy.",
        "features": {
         "brand": "ChocoDelight",
         "care instructions": "Store in a cool, dry place",
         "flavors": "Available in Orange and Cherry flavors",
         "material": "Premium quality chocolate",
         "purpose": "Ideal for gifting or self-indulgence",
         "sizes": "Available in small, medium, and large boxes"
        },
        "packs": [
         {
          "deliveryType": "1 Day shipping",
          "packageDimension": "100x230 cm",
          "packageWeight": "1,00 kg",
          "variants": "6 available",
          "version": "Pack 1"
         },
         {
          "deliveryType": "1 Day shipping",
          "packageDimension": "200x460 cm",
          "packageWeight": "2,11 kg",
          "variants": "6 available",
          "version": "Pack 2"
         },
         {
          "deliveryType": "1 Day shipping",
          "packageDimension": "300x690 cm",
          "packageWeight": "3,22 kg",
          "variants": "6 available",
          "version": "Pack 3"
         },
         {
          "deliveryType": "1 Day shipping",
          "packageDimension": "400x920 cm",
          "packageWeight": "4,33 kg",
          "variants": "6 available",
          "version": "Pack 4"
         },
         {
          "deliveryType": "1 Day shipping",
          "packageDimension": "500x1150 cm",
          "packageWeight": "5,44 kg",
          "variants": "6 available",
          "version": "Pack 5"
         }
        ],
        "price": "$9.99 from $12.99",
        "variants": [
         "orange, small",
         "orange, medium",
         "orange, large",
         "cherry, small",
         "cherry, medium",
         "cherry, large"
        ]
       }
      },
      "text/plain": [
       "<IPython.core.display.JSON object>"
      ]
     },
     "execution_count": 14,
     "metadata": {
      "application/json": {
       "expanded": false,
       "root": "root"
      }
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's ask it to extract data as JSON\n",
    "resp = client.post(\n",
    "    \"https://api.scrapfly.io/extraction\",\n",
    "    data=html,\n",
    "    headers={\n",
    "        \"content-type\": \"text/html\",\n",
    "    },\n",
    "    params={\n",
    "        \"extraction_prompt\": \"extract product specification \\\n",
    "                              in JSON format\"\n",
    "    }\n",
    ")\n",
    "print(resp.url)\n",
    "display.JSON(resp.json())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Further, the prompts can be as complex and detailed as you'd like. For example, here to extract reviews from our web-scraping.dev example we can mix structured data extraction and freeform questions allowing AI to do a bit of reasoning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://api.scrapfly.io/extraction?key=scp-test-14644f57089d47c0aeb705b776792366&extraction_prompt=%0A%20%20%20%20%20%20%20%20extract%20review%20data%20in%20JSON%20format%20for%20fields%3A%20date%2C%20text%0A%20%20%20%20%20%20%20%20also%20include%20tone%20field%20which%20estimates%20the%20tone%20of%20the%20review%20%0A%20%20%20%20%20%20%20%20\n"
     ]
    },
    {
     "data": {
      "application/json": {
       "content_type": "application/json",
       "data": [
        {
         "date": "2023-05-18",
         "text": "Unique flavor and great energy boost. It's the perfect gamer's drink!",
         "tone": "positive"
        },
        {
         "date": "2023-05-17",
         "text": "Good flavor and keeps me energized. The bottle design is really fun.",
         "tone": "positive"
        },
        {
         "date": "2023-05-16",
         "text": "Excellent energy drink for gamers. The tropical flavor is refreshing.",
         "tone": "positive"
        },
        {
         "date": "2023-05-15",
         "text": "It’s fun, tastes good, and the energy boost is helpful during intense gaming sessions.",
         "tone": "positive"
        },
        {
         "date": "2023-05-15",
         "text": "The best sneakers I've bought in a long time. Stylish, comfortable and the leather quality is top-notch.",
         "tone": "positive"
        },
        {
         "date": "2023-05-12",
         "text": "The cherry cola flavor is a win. Keeps me energized and focused.",
         "tone": "positive"
        },
        {
         "date": "2023-05-10",
         "text": "These shoes are a hit with my twins. They love the lights and they seem to be quite durable.",
         "tone": "positive"
        },
        {
         "date": "2023-05-01",
         "text": "Excellent boots for outdoor adventures. They offer great support and are very comfortable.",
         "tone": "positive"
        },
        {
         "date": "2023-05-01",
         "text": "Nice design and good quality, but a bit too tight for me. Otherwise, it's a pretty cool beanie.",
         "tone": "neutral"
        },
        {
         "date": "2023-04-25",
         "text": "Really enjoyed the citrus flavor and the energy boost it gives.",
         "tone": "positive"
        },
        {
         "date": "2023-04-25",
         "text": "Great concept, but the sizing is a bit off. Order a size larger.",
         "tone": "neutral"
        },
        {
         "date": "2023-04-18",
         "text": "Delicious chocolates, and the box is pretty substantial. It'd make a nice gift.",
         "tone": "positive"
        },
        {
         "date": "2023-04-12",
         "text": "The boots are durable, but the laces could be better quality.",
         "tone": "neutral"
        },
        {
         "date": "2023-04-11",
         "text": "This potion is a game changer. Love the energy boost and the flavor.",
         "tone": "positive"
        },
        {
         "date": "2023-04-10",
         "text": "Light, comfortable, and nice design. I'm buying another pair.",
         "tone": "positive"
        },
        {
         "date": "2023-04-10",
         "text": "Highly recommend! These shoes are great for evening walks. Kids love them!",
         "tone": "positive"
        },
        {
         "date": "2023-04-09",
         "text": "It's like a health potion for gamers! The energy boost is spot on.",
         "tone": "positive"
        },
        {
         "date": "2023-04-07",
         "text": "Really helps me focus during intense gaming marathons. The teal color is a nice touch.",
         "tone": "positive"
        },
        {
         "date": "2023-04-07",
         "text": "The shoes are nice but they didn't fit me well. I had to exchange for a larger size.",
         "tone": "neutral"
        },
        {
         "date": "2023-04-05",
         "text": "Great taste, and the energy kick is awesome. Feels just like a strength potion.",
         "tone": "positive"
        }
       ]
      },
      "text/plain": [
       "<IPython.core.display.JSON object>"
      ]
     },
     "execution_count": 6,
     "metadata": {
      "application/json": {
       "expanded": false,
       "root": "root"
      }
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# html from https://web-scraping.dev/reviews\n",
    "html = Path('reviews_1.html').read_text()\n",
    "resp = client.post(\n",
    "    \"https://api.scrapfly.io/extraction\",\n",
    "    data=html,\n",
    "    headers={\n",
    "        \"content-type\": \"text/html\",\n",
    "    },\n",
    "    params={\n",
    "        \"extraction_prompt\": \"\"\"\n",
    "        extract review data in JSON format for fields: \n",
    "        date, text also include tone field \n",
    "        which estimates the tone of the review\n",
    "        which can be: positive, negative or neutral\n",
    "        \"\"\"\n",
    "    }\n",
    ")\n",
    "print(resp.url)\n",
    "from IPython import display\n",
    "display.JSON(resp.json())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we asked the model to extract structured data of review date and text and evaluate it for review sentiment tone. This gives us both exact data on the page and AI generated evaluation."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "As for extracting urls the `url` parameter can be used\n",
    "For more structured data we can extract URLs from the page and the `url` parameter can be used to turn the extracted URLs into absolute urls."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'content_type': 'application/json', 'data': ['https://www.example.com/product1', 'https://www.example.com/product2', 'https://www.example.com/product3']}\n"
     ]
    },
    {
     "data": {
      "application/json": {
       "content_type": "application/json",
       "data": [
        "https://www.example.com/product1",
        "https://www.example.com/product2",
        "https://www.example.com/product3"
       ]
      },
      "text/plain": [
       "<IPython.core.display.JSON object>"
      ]
     },
     "execution_count": 8,
     "metadata": {
      "application/json": {
       "expanded": false,
       "root": "root"
      }
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "html = \"\"\"\n",
    "<!DOCTYPE html>\n",
    "<html>\n",
    "  <body>\n",
    "    <h1>Products</h1>\n",
    "    <ul>\n",
    "      <li><a href=\"/product/1.html\">Product 1</a></li>\n",
    "      <li><a href=\"/product/2.html\">Product 2</a></li>\n",
    "      <li><a href=\"/product/3.html\">Product 3</a></li>\n",
    "    </ul>\n",
    "  </body>\n",
    "</html>\n",
    "\"\"\"\n",
    "resp = client.post(\n",
    "    \"https://api.scrapfly.io/extraction\",\n",
    "    data=html,\n",
    "    headers={\n",
    "        \"content-type\": \"text/html\",\n",
    "    },\n",
    "    params={\n",
    "        \"extraction_prompt\": \"extract product links in json format\",\n",
    "        \"url\": \"https://web-scraping.dev/product/1\"  # base url\n",
    "    }\n",
    ")\n",
    "print(resp.json())\n",
    "from IPython import display\n",
    "display.JSON(resp.json())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next let's take a look at a different AI extraction method - predefined auto parsing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AI Auto Parser\n",
    "\n",
    "Scrapfly also includes some pre-build extraction models for common data parsing scenarios like parsing products, reviews, articles or real estate listings.\n",
    "\n",
    "For this example let's take a look at web-scraping.dev/product/1 page which contains product and review data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/json": {
       "content_type": "application/json",
       "data": {
        "aggregate_rating": null,
        "brand": "ChocoDelight",
        "breadcrumbs": null,
        "canonical_url": null,
        "color": null,
        "description": "Indulge your sweet tooth with our Box of Chocolate Candy. Each box contains an assortment of rich, flavorful chocolates with a smooth, creamy filling. Choose from a variety of flavors including zesty orange and sweet cherry. Whether you're looking for the perfect gift or just want to treat yourself, our Box of Chocolate Candy is sure to satisfy.",
        "identifiers": {
         "ean13": null,
         "gtin14": null,
         "gtin8": null,
         "isbn10": null,
         "isbn13": null,
         "ismn": null,
         "issn": null,
         "mpn": null,
         "sku": null,
         "upc": null
        },
        "images": [],
        "main_category": "Products",
        "main_image": "https://web-scraping.dev/assets/products/orange-chocolate-box-small-1.webp",
        "name": "Box of Chocolate Candy",
        "offers": [
         {
          "availability": "available",
          "currency": "$",
          "price": 9.99,
          "regular_price": 12.99
         }
        ],
        "related_products": [
         {
          "availability": "available",
          "description": null,
          "images": [
           {
            "url": "https://web-scraping.dev/assets/products/hiking-boots-1.webp"
           }
          ],
          "link": "https://web-scraping.dev/product/19",
          "name": "Hiking Boots for Outdoor Adventures",
          "price": {
           "amount": 89.99,
           "currency": null,
           "raw": "89.99"
          }
         },
         {
          "availability": "available",
          "description": null,
          "images": [
           {
            "url": "https://web-scraping.dev/assets/products/classic-leather-sneakers-white.webp"
           }
          ],
          "link": "https://web-scraping.dev/product/11",
          "name": "Classic Leather Sneakers",
          "price": {
           "amount": 79.99,
           "currency": null,
           "raw": "79.99"
          }
         },
         {
          "availability": "available",
          "description": null,
          "images": [
           {
            "url": "https://web-scraping.dev/assets/products/teal-potion.webp"
           }
          ],
          "link": "https://web-scraping.dev/product/15",
          "name": "Teal Energy Potion",
          "price": {
           "amount": 4.99,
           "currency": null,
           "raw": "4.99"
          }
         },
         {
          "availability": "available",
          "description": null,
          "images": [
           {
            "url": "https://web-scraping.dev/assets/products/hiking-boots-1.webp"
           }
          ],
          "link": "https://web-scraping.dev/product/7",
          "name": "Hiking Boots for Outdoor Adventures",
          "price": {
           "amount": 89.99,
           "currency": null,
           "raw": "89.99"
          }
         }
        ],
        "secondary_category": null,
        "size": null,
        "specifications": [
         {
          "name": "material",
          "value": "Premium quality chocolate"
         },
         {
          "name": "flavors",
          "value": "Available in Orange and Cherry flavors"
         },
         {
          "name": "sizes",
          "value": "Available in small, medium, and large boxes"
         },
         {
          "name": "brand",
          "value": "ChocoDelight"
         },
         {
          "name": "care instructions",
          "value": "Store in a cool, dry place"
         },
         {
          "name": "purpose",
          "value": "Ideal for gifting or self-indulgence"
         }
        ],
        "style": null,
        "url": "https://web-scraping.dev/",
        "variants": [
         {
          "color": "orange",
          "offers": [
           {
            "availability": "available",
            "price": {
             "amount": null,
             "currency": null,
             "raw": null
            }
           }
          ],
          "sku": null,
          "url": "https://web-scraping.dev/product/1?variant=orange-small"
         },
         {
          "color": "orange",
          "offers": [
           {
            "availability": "available",
            "price": {
             "amount": null,
             "currency": null,
             "raw": null
            }
           }
          ],
          "sku": null,
          "url": "https://web-scraping.dev/product/1?variant=orange-medium"
         },
         {
          "color": "orange",
          "offers": [
           {
            "availability": "available",
            "price": {
             "amount": null,
             "currency": null,
             "raw": null
            }
           }
          ],
          "sku": null,
          "url": "https://web-scraping.dev/product/1?variant=orange-large"
         },
         {
          "color": "cherry",
          "offers": [
           {
            "availability": "available",
            "price": {
             "amount": null,
             "currency": null,
             "raw": null
            }
           }
          ],
          "sku": null,
          "url": "https://web-scraping.dev/product/1?variant=cherry-small"
         },
         {
          "color": "cherry",
          "offers": [
           {
            "availability": "available",
            "price": {
             "amount": null,
             "currency": null,
             "raw": null
            }
           }
          ],
          "sku": null,
          "url": "https://web-scraping.dev/product/1?variant=cherry-medium"
         },
         {
          "color": "cherry",
          "offers": [
           {
            "availability": "available",
            "price": {
             "amount": null,
             "currency": null,
             "raw": null
            }
           }
          ],
          "sku": null,
          "url": "https://web-scraping.dev/product/1?variant=cherry-large"
         }
        ]
       },
       "data_quality": {
        "errors": [
         "related_products[0].price.currency: Input should be a valid string",
         "related_products[1].price.currency: Input should be a valid string",
         "related_products[2].price.currency: Input should be a valid string",
         "related_products[3].price.currency: Input should be a valid string",
         "identifiers.sku: Input should be a valid string"
        ],
        "fulfilled": false,
        "fulfillment_percent": 63.16
       }
      },
      "text/plain": [
       "<IPython.core.display.JSON object>"
      ]
     },
     "execution_count": 6,
     "metadata": {
      "application/json": {
       "expanded": false,
       "root": "root"
      }
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# html from https://web-scraping.dev/product/1\n",
    "html = Path('product_1.html').read_text()\n",
    "\n",
    "resp = client.post(\n",
    "    \"https://api.scrapfly.io/extraction\",\n",
    "    data=html,\n",
    "    headers={\n",
    "        \"content-type\": \"text/html\",\n",
    "    },\n",
    "    params={\n",
    "        \"extraction_model\": \"product\",\n",
    "    }\n",
    ")\n",
    "\n",
    "from IPython import display\n",
    "display.JSON(resp.json())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we set the `extraction_model` parameter to `product` value and in this case Scrapfly AI tries to extract each field of a predefined product schema which includes all of the standard product fields like price, description, brand names and so on.\n",
    "\n",
    "You can find the [full schema definitions in Scrapfly docs](https://scrapfly.io/docs/extraction-api/automatic-ai#models).\n",
    "\n",
    "The advantages of auto parsing over LLMs is that the structure is static and predictable and also doesn't require any prompt engineering. The `data_quality` meta field also includes an estimate of how much data was found."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/json": {
       "content_type": "application/json",
       "data": {
        "paginationNext": null,
        "paginationPrevious": null,
        "rating": null,
        "reviews": [],
        "url": "https://web-scraping.dev/"
       },
       "data_quality": {
        "errors": [],
        "fulfilled": true,
        "fulfillment_percent": 40
       }
      },
      "text/plain": [
       "<IPython.core.display.JSON object>"
      ]
     },
     "execution_count": 9,
     "metadata": {
      "application/json": {
       "expanded": false,
       "root": "root"
      }
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# html from https://web-scraping.dev/product/1\n",
    "html = Path('product_1.html').read_text()\n",
    "resp = client.post(\n",
    "    \"https://api.scrapfly.io/extraction\",\n",
    "    data=html,\n",
    "    headers={\n",
    "        \"content-type\": \"text/html\",\n",
    "    },\n",
    "    params={\n",
    "        \"extraction_model\": \"review_list\",\n",
    "    }\n",
    ")\n",
    "from IPython import display\n",
    "display.JSON(resp.json())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To have even more control over the data extraction, let's take a look at the extraction rules feature next."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extraction Rules\n",
    "\n",
    "Extraction Rules allow to specify exact data parsing instructions. This replicates common data parsing tools as a unified parsing system expressed through a single JSON definition.\n",
    "\n",
    "For example, here we're scraping the reviews page and define a `date posted` field for finding date objects. We define the elements location using CSS selectors and apply formatters to reformat the dates in a desired format. \n",
    "\n",
    "This gives us full control over data extraction and once we define our ruleset we can apply it to any document.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://api.scrapfly.io/extraction?key=scp-test-14644f57089d47c0aeb705b776792366&url=https%3A%2F%2Fweb-scraping.dev&extraction_template=ephemeral%3ACnsgIAogICJzb3VyY2UiOiAiaHRtbCIsCiAgInNlbGVjdG9ycyI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAiZGF0ZV9wb3N0ZWQiLAogICAgICAidHlwZSI6ICJjc3MiLAogICAgICAicXVlcnkiOiAiW2RhdGEtdGVzdGlkPSdyZXZpZXctZGF0ZSddOjp0ZXh0IiwKICAgICAgIm11bHRpcGxlIjogdHJ1ZSwKICAgICAgImZvcm1hdHRlcnMiOiBbIHsKICAgICAgICAibmFtZSI6ICJkYXRldGltZSIsCiAgICAgICAgImFyZ3MiOiB7ImZvcm1hdCI6ICIlWSwgJWIgJWQg4oCUICVBIn0KICAgICAgfSBdCiAgICB9CiAgXQp9Cg%3D%3D\n"
     ]
    },
    {
     "data": {
      "application/json": {
       "content_type": "application/json",
       "data": {
        "date_posted": [
         "2023, May 18 — Thursday",
         "2023, May 17 — Wednesday",
         "2023, May 16 — Tuesday",
         "2023, May 15 — Monday",
         "2023, May 15 — Monday",
         "2023, May 12 — Friday",
         "2023, May 10 — Wednesday",
         "2023, May 01 — Monday",
         "2023, May 01 — Monday",
         "2023, Apr 25 — Tuesday",
         "2023, Apr 25 — Tuesday",
         "2023, Apr 18 — Tuesday",
         "2023, Apr 12 — Wednesday",
         "2023, Apr 11 — Tuesday",
         "2023, Apr 10 — Monday",
         "2023, Apr 10 — Monday",
         "2023, Apr 09 — Sunday",
         "2023, Apr 07 — Friday",
         "2023, Apr 07 — Friday",
         "2023, Apr 05 — Wednesday"
        ]
       }
      },
      "text/plain": [
       "<IPython.core.display.JSON object>"
      ]
     },
     "execution_count": 44,
     "metadata": {
      "application/json": {
       "expanded": false,
       "root": "root"
      }
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# html from https://web-scraping.dev/reviews (with js render)\n",
    "html = Path('reviews_1.html').read_text()\n",
    "\n",
    "# 1. create extaction template as JSON\n",
    "rules = \"\"\"\n",
    "{  \n",
    "  \"source\": \"html\",\n",
    "  \"selectors\": [\n",
    "    {\n",
    "      \"name\": \"date_posted\",\n",
    "      \"type\": \"css\",\n",
    "      \"query\": \"[data-testid='review-date']::text\",\n",
    "      \"multiple\": true,\n",
    "      \"formatters\": [ {\n",
    "        \"name\": \"datetime\",\n",
    "        \"args\": {\"format\": \"%Y, %b %d — %A\"}\n",
    "      } ]\n",
    "    }\n",
    "  ]\n",
    "}\n",
    "\"\"\"\n",
    "\n",
    "# 2. base64 encode the template\n",
    "from base64 import urlsafe_b64encode\n",
    "rules = urlsafe_b64encode(rules.encode()).decode()\n",
    "\n",
    "resp = client.post(\n",
    "    \"https://api.scrapfly.io/extraction\",\n",
    "    data=html,\n",
    "    headers={\n",
    "        \"content-type\": \"text/html\",\n",
    "    },\n",
    "    params={\n",
    "        # use ephemeral: prefix for on-demand templates (templates can also be defined in dashboard soon)\n",
    "        \"extraction_template\": \"ephemeral:\" + rules,\n",
    "    }\n",
    ")\n",
    "print(resp.url)\n",
    "display.JSON(resp.json())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is just the tip of the iceberg when it comes to extraction rule features and for more in-depth examples see the full documentation page."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This short intro to Scrapfly's Extraction API should get you started with your data parsing journey and for more see our [full documentation at scrapfly.io/docs](https://scrapfly.io/docs/extraction-api/getting-started)\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (sf-demo)",
   "language": "python",
   "name": "sf-demo"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}