lessons/Scraper_REST_APIs.ipynb

{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "view-in-github"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/redadmiral/python-for-journalists/blob/main/lessons/Scraper_REST_APIs.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0PyylNGwlQ0_"
      },
      "source": [
        "# Scraper: Offene REST-APIs nutzen\n",
        "\n",
        "Die bequemste Art Informationen aus dem Internet zu beziehen, die nicht direkt als Download bereitgestellt werden, ist es herauszufinden, ob die Seite von der wir Informationen brauchen direkt auf eine Schnittstelle zugreift, die wir auch benutzen können.\n",
        "\n",
        "Die Schnittstellen nennen sich auch APIs (Application Programming Interfaces) und sind der Weg, wie Computer im Internet miteinander reden. Die meisten Schnittstellen sind heute nach dem REST-Prinzip gestaltet - das heißt dass sie Daten in maschinenlesbarer Form übermittelt werden und die Antwort der API stabil bleibt.\n",
        "\n",
        "Wahrscheinlich ohne es zu wissen, benutzt ihr ständig eine solche API: Auch euer Browser arbeitet mit Requests zu REST-APIs. Jede Webseite, die ihr aufruft wird über einen sogenannten GET-Request angefordert und übermittelt. \n",
        "\n",
        "## GET-Requests\n",
        "\n",
        "Ganz einfach gesprochen ruft der Browser beim Server an, sagt was dem Server anhand der URL was er haben will und der Server schickt es ihm dann zurück.\n",
        "\n",
        "Solche GET-Requests sind die einfachsten, weil alle Informationen direkt in der URL angegeben werden. Ein Beispiel dafür findest du auf meiner Homepage unter [api.marco-lehner.de](http://api.marco-lehner.de).\n",
        "\n",
        "Die GET-Requests werden benutzt, um Internetseiten aufrufen zu können, aber auch von Computern bzw. Programmen, um Informationen auszutauschen. \n",
        "\n",
        "Hier ein kleines Beispiel:\n",
        "\n",
        "Wenn ihr die folgende URL in eurem Browser aufruft:\n",
        "\n",
        "[http://api.marco-lehner.de/get_json?number=1&multiplier=3](http://api.marco-lehner.de/get_json?number=1&multiplier=3)\n",
        "\n",
        "Bekommt ihr folgende Antwort:\n",
        "\n",
        "```\n",
        "{\"result\":3}\n",
        "```\n",
        "\n",
        "Die API multipliziert einfach die beiden Zahlen die ihr in der URL angebt:\n",
        "\n",
        "[http://api.marco-lehner.de/get_json?number=2&multiplier=3](http://api.marco-lehner.de/get_json?number=2&multiplier=3)\n",
        "\n",
        "Bekommt ihr folgende Antwort:\n",
        "\n",
        "```\n",
        "{\"result\":6}\n",
        "```\n",
        "\n",
        "Diese Anfrage könnt ihr nicht nur mit eurem Browser machen, sondern auch mit Python. Dafür gibt es die `requests`-Bibliothek.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 42,
      "metadata": {
        "id": "2Z_aL5rmD9E7"
      },
      "outputs": [],
      "source": [
        "import requests"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "185wIwdoEGtW"
      },
      "source": [
        "Und Requests stellt uns die Funktion `get()` zur Verfügung um damit solche GET-Requests ausführen zu können."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 43,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PZK1uFcHlK3J",
        "outputId": "ec294944-c5d3-4e3c-ad48-c82e77cef9b3"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<Response [200]>"
            ]
          },
          "execution_count": 43,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "response = requests.get(\"http://api.marco-lehner.de/get_json?number=2&multiplier=3\")\n",
        "response"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "45D-HZEYEPpb"
      },
      "source": [
        "Die Antwort ist nicht sehr vielsagend. Wir erfahren dass ein Response-Objekt zurückkommt und die Zahl 200.\n",
        "\n",
        "Die 200 ist der Status-Code. [Hier](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status) findet ihr eine Liste mit allen möglichen Status-Codes. Die wichtisten sind folgende:\n",
        "\n",
        "- 200: Alles okay, der Request hat geklappt.\n",
        "- 404: Die Seite konnte nicht gefunden werden.\n",
        "- 500: Server kaputt.\n",
        "\n",
        "Die Zahl 200 im Request sagt uns also dass alles geklappt hat.+\n",
        "\n",
        "Wenn wir sehen wollen, was die API uns eigentlich zurückgegeben hat, müssen wir den Response mit der `.json()`-Methode aufrufen:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 44,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NT27pQSGE5KV",
        "outputId": "483b997c-d308-4061-ab6f-f1e77d837796"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'result': 6}"
            ]
          },
          "execution_count": 44,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "response.json()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9UsP_wqMICOy"
      },
      "source": [
        "Wenn der GET-Request kein JSON zurückliefert, wie in dem Fall wenn wir den folgenden Endpunkt ansprechen, wirft die `.json()`-Methode einen Fehler.\n",
        "\n",
        "Hier rufen wir einen anderen Endpunkt auf, der uns ein kleines HTML-Dokument zurückgibt. Kopier den Link gerne in deinen Browser.\n",
        "\n",
        "[http://api.marco-lehner.de/get_html?number=2](http://api.marco-lehner.de/get_html?number=2)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 45,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 320
        },
        "id": "ezqJk5UwFZ71",
        "outputId": "0d61d494-0009-42f9-edd0-bb9d8284192d"
      },
      "outputs": [
        {
          "ename": "JSONDecodeError",
          "evalue": "ignored",
          "output_type": "error",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mJSONDecodeError\u001b[0m                           Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-45-f48f1ce7fc3a>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mresponse\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrequests\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"http://api.marco-lehner.de/get_html?number=2&multiplier=3\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mresponse\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjson\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/requests/models.py\u001b[0m in \u001b[0;36mjson\u001b[0;34m(self, **kwargs)\u001b[0m\n\u001b[1;32m    898\u001b[0m                     \u001b[0;31m# used.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    899\u001b[0m                     \u001b[0;32mpass\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 900\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mcomplexjson\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mloads\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtext\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    901\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    902\u001b[0m     \u001b[0;34m@\u001b[0m\u001b[0mproperty\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/lib/python3.8/json/__init__.py\u001b[0m in \u001b[0;36mloads\u001b[0;34m(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)\u001b[0m\n\u001b[1;32m    355\u001b[0m             \u001b[0mparse_int\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mparse_float\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;32mand\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    356\u001b[0m             parse_constant is None and object_pairs_hook is None and not kw):\n\u001b[0;32m--> 357\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0m_default_decoder\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdecode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    358\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mcls\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    359\u001b[0m         \u001b[0mcls\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mJSONDecoder\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/lib/python3.8/json/decoder.py\u001b[0m in \u001b[0;36mdecode\u001b[0;34m(self, s, _w)\u001b[0m\n\u001b[1;32m    335\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    336\u001b[0m         \"\"\"\n\u001b[0;32m--> 337\u001b[0;31m         \u001b[0mobj\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mend\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mraw_decode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0midx\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0m_w\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    338\u001b[0m         \u001b[0mend\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_w\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mend\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    339\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mend\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/lib/python3.8/json/decoder.py\u001b[0m in \u001b[0;36mraw_decode\u001b[0;34m(self, s, idx)\u001b[0m\n\u001b[1;32m    353\u001b[0m             \u001b[0mobj\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mend\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mscan_once\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0midx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    354\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mStopIteration\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0merr\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 355\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0mJSONDecodeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Expecting value\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0ms\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0merr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    356\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mobj\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mend\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mJSONDecodeError\u001b[0m: Expecting value: line 1 column 1 (char 0)"
          ]
        }
      ],
      "source": [
        "response = requests.get(\"http://api.marco-lehner.de/get_html?number=2&multiplier=3\")\n",
        "response.json()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vrO5qXvlI2u_"
      },
      "source": [
        "Dann können wir direkt auf das `.content`-Attribut zugreifen:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 46,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "B0fMDWKMI8Qv",
        "outputId": "4242f9e5-d6cd-467e-b861-3e7ceecdc723"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "b'<h1>Result</h1> <p> 3 times 2 is 6</p>'"
            ]
          },
          "execution_count": 46,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "response.content"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9koZzLbXJOe3"
      },
      "source": [
        "Der Response ist jetzt noch als Bytestring formatiert:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 47,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "34vjvD22JSuQ",
        "outputId": "c6ff3ef8-2da7-4810-a24c-a22c569f9640"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "bytes"
            ]
          },
          "execution_count": 47,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "type(response.content)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KsR5uGllJWj5"
      },
      "source": [
        "Um mit ihm als normalem String arbeiten zu können, müssen wir ihn noch mit der `.decode()`-Methode decodieren. \n",
        "\n",
        "Dafür müssen wir den Zeichensatz angeben in dem der String decodiert ist. Meistens ist das `utf-8`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 48,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 36
        },
        "id": "zNuacxhWJynx",
        "outputId": "67bf8dfa-7f82-4b12-d793-41a9be4ead59"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'<h1>Result</h1> <p> 3 times 2 is 6</p>'"
            ]
          },
          "execution_count": 48,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "response.content.decode(\"utf-8\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X0K9Mk5pJ5oc"
      },
      "source": [
        "Ein Indiz dafür, dass wir den falschen Zeichensatz gewählt haben ist es, wenn Umlaute falsch dargestellt werden. Meistens ist der String dann in `iso-8859-1` encodiert."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rTPGgdyUK4Td"
      },
      "source": [
        "## POST-Requests\n",
        "\n",
        "Ein bisschen schwerer zu verstehen sind POST-Requests. Diese Requests können nicht direkt über den Browser ausgeführt werden. \n",
        "\n",
        "Ein Beispiel dazu findest du unter dieser URL: [http://api.marco-lehner.de/post_json](http://api.marco-lehner.de/post_json).\n",
        "\n",
        "Hier zeigt uns der Browser nur an:\n",
        "\n",
        "```\n",
        "{\"detail\":\"Method Not Allowed\"}\n",
        "```\n",
        "\n",
        "Das liegt daran, dass der Browser einen GET-Request an einen Endpunkt macht, der einen POST-Request erwartet.\n",
        "\n",
        "In Python können wir aber einfach selbst festlegen, welche Art von Request wir ausführen wollen. POST-Requests können wir über die `post()`-Methode ausführen."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 49,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jhcuQDhCNTNj",
        "outputId": "963b3af2-ad6e-4a04-cc30-cb8d1032174b"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<Response [422]>"
            ]
          },
          "execution_count": 49,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "response = requests.post(\"http://api.marco-lehner.de/post_json\")\n",
        "response"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I0rQVSrUNdT7"
      },
      "source": [
        "Hier gibt uns requests einen `422`-Fehler zurück. Das liegt daran, dass POST-Requests eine sogenannte Payload erwarten. \n",
        "\n",
        "Das was wir vorher noch über die URL angegeben haben, schicken wir jetzt in einem JSON-Dokument mit. \n",
        "\n",
        "Dieser Endpunkt erwartet ein JSON-Dokument (oder ein `dict` in Python) mit folgender Struktur:\n",
        "\n",
        "```json\n",
        "{\n",
        "  \"number\": int,\n",
        "  \"multiplier\": int\n",
        "}\n",
        "```\n",
        "\n",
        "Diese payload müssen wir zuerst festlegen und können sie dann der POST-Funktion mitgeben."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 50,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uubG03QzQiYp",
        "outputId": "ef0e1d45-e135-4c80-f5ce-357391763a67"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'result': 6}"
            ]
          },
          "execution_count": 50,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "payload = {\"number\": 2, \"multiplier\": 3}\n",
        "\n",
        "response = requests.post(\"http://api.marco-lehner.de/post_json\", json=payload)\n",
        "response.json()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L1bA5TQ_Q-OV"
      },
      "source": [
        "Das mag jetzt erstmal umständlich erscheinen, aber mit POST-Requests können viel mehr Daten an den Server übermittelt werden als mit einem GET-Request, weil URLs nicht unendlich lang werden können. Deshalb werden sie häufig für komplexere Anfragen verwendet.\n",
        "\n",
        "Es gibt noch eine Reihe weiterer Request-Arten (v.a. PUT und DELETE) aber diese sind für unsere Arbeit nicht sehr wichtig. Im Prinzip funktionieren sie genauso wie POST-Requests, also dass immer ein JSON mitgeschickt werden muss.\n",
        "\n",
        "Man verwendet sie aber für andere Sachen, mit PUT-Requests werden normalerweise Dinge auf den Server hochgeladen und mit DELETE-Requests werden sie vom Server gelöscht."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BXJU3ayNXptU"
      },
      "source": [
        "## Finding Open APIs\n",
        "\n",
        "Jetzt wissen wir wie wir REST-APIs ansprechen können, aber wie finden wir offene REST-APIs?\n",
        "\n",
        "Dafür haben wir die Entwickler-Werkzeuge in unserem Browser, diese tauchen auf wenn ihr die F12-Taste in Chrome drückt. \n",
        "\n",
        "Hier könnt ihr euch anzeigen lassen, was beim Laden der Seite alles vom Server nachgeladen wird. Diese Einträge findet ihr im Reiter Network (Netzwerk).\n",
        "\n",
        "Wenn ihr oben auf \"Fetch/XHR\" klickt, dann bekommt ihr nur noch die Requests angezeigt, bei denen strukturierte Daten über das XHR-Protokoll übertragen wurden. Das ist der für uns interessante Reiter.\n",
        "\n",
        "Hier könnt ihr euch den richtigen Request raussuchen und mit rechts auf \"Copy > Copy as cURL\" klicken.\n",
        "\n",
        "Das Ergebnis könnt ihr dann auf der Seite [https://curlconverter.com/](https://curlconverter.com/) in einen Python-Request umwandeln. \n",
        "\n",
        "In diesem Request ist dann alles enthalten, was auch der Browser an den Server geschickt hat und in den meisten Fällen bekommt ihr so auch die gleiche Antwort wie euer Browser.\n",
        "\n",
        "Ein Beispiel für ein solches kopiertes cURL-Kommando seht ihr hier:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 52,
      "metadata": {
        "id": "I1ZJmKBvVXmw"
      },
      "outputs": [],
      "source": [
        "headers = {\n",
        "    'Accept': 'application/json, text/plain, */*',\n",
        "    'Accept-Language': 'en-US,en;q=0.9,de;q=0.8',\n",
        "    'Authorization': 'Basic ZnJvbnRlbmQ6ZnJvbnRlbmQ=',\n",
        "    'Connection': 'keep-alive',\n",
        "    'Origin': 'https://xn--strungsauskunft-9sb.de',\n",
        "    'Referer': 'https://xn--strungsauskunft-9sb.de/',\n",
        "    'Sec-Fetch-Dest': 'empty',\n",
        "    'Sec-Fetch-Mode': 'cors',\n",
        "    'Sec-Fetch-Site': 'cross-site',\n",
        "    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36',\n",
        "    'sec-ch-ua': '\"Not A(Brand\";v=\"24\", \"Chromium\";v=\"110\"',\n",
        "    'sec-ch-ua-mobile': '?0',\n",
        "    'sec-ch-ua-platform': '\"Linux\"',\n",
        "}\n",
        "\n",
        "params = {\n",
        "    'SectorType': '1',\n",
        "}\n",
        "\n",
        "response = requests.get('https://api-public.stoerungsauskunft.de/api/v1/public/outages', params=params, headers=headers)\n",
        "\n",
        "content = response.json()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qR2aj88nbDzp"
      },
      "source": [
        "Das JSON-Objekt (bzw. das dict) können wir dann, solange es nicht zu verschachtelt ist, mit der funktion `.json_normalize()` in pandas laden und weiter bearbeiten:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 53,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 661
        },
        "id": "E418HpbCVzvt",
        "outputId": "170f11ca-5e59-4f6d-c631-fada9d06ca1a"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-9da30fb3-cee6-4b6c-8777-45edcf98c961\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>operatorId</th>\n",
              "      <th>subsection</th>\n",
              "      <th>type</th>\n",
              "      <th>origin</th>\n",
              "      <th>geoType</th>\n",
              "      <th>dateStart</th>\n",
              "      <th>dateEnd</th>\n",
              "      <th>lastUpdate</th>\n",
              "      <th>postalCode</th>\n",
              "      <th>...</th>\n",
              "      <th>internal</th>\n",
              "      <th>houseNr</th>\n",
              "      <th>photo</th>\n",
              "      <th>containerShape.coordinates</th>\n",
              "      <th>containerShape.envelope</th>\n",
              "      <th>containerShape.envelopeInternal.minX</th>\n",
              "      <th>containerShape.envelopeInternal.minY</th>\n",
              "      <th>containerShape.envelopeInternal.MaxX</th>\n",
              "      <th>containerShape.envelopeInternal.MaxY</th>\n",
              "      <th>containerShape.centerCoordinates</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>3644311</td>\n",
              "      <td>180</td>\n",
              "      <td>None</td>\n",
              "      <td>2</td>\n",
              "      <td>5</td>\n",
              "      <td>2</td>\n",
              "      <td>02/27/2023 09:09:57</td>\n",
              "      <td>02/27/2023 09:54:15</td>\n",
              "      <td>02/27/2023 09:54:15</td>\n",
              "      <td>27580</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>3644358</td>\n",
              "      <td>180</td>\n",
              "      <td>None</td>\n",
              "      <td>2</td>\n",
              "      <td>5</td>\n",
              "      <td>2</td>\n",
              "      <td>02/27/2023 09:09:57</td>\n",
              "      <td>02/27/2023 09:56:14</td>\n",
              "      <td>02/27/2023 09:54:15</td>\n",
              "      <td>27580</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3644669</td>\n",
              "      <td>402</td>\n",
              "      <td>None</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>02/27/2023 14:12:57</td>\n",
              "      <td>02/27/2023 17:31:47</td>\n",
              "      <td>02/27/2023 17:31:47</td>\n",
              "      <td>84371</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>3644563</td>\n",
              "      <td>182</td>\n",
              "      <td>None</td>\n",
              "      <td>2</td>\n",
              "      <td>5</td>\n",
              "      <td>2</td>\n",
              "      <td>02/27/2023 10:36:07</td>\n",
              "      <td>02/27/2023 15:48:54</td>\n",
              "      <td>02/27/2023 15:48:54</td>\n",
              "      <td>28201</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>3644600</td>\n",
              "      <td>182</td>\n",
              "      <td>None</td>\n",
              "      <td>2</td>\n",
              "      <td>5</td>\n",
              "      <td>2</td>\n",
              "      <td>02/27/2023 09:00:07</td>\n",
              "      <td>02/27/2023 15:48:54</td>\n",
              "      <td>02/27/2023 15:48:54</td>\n",
              "      <td>28205</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>447</th>\n",
              "      <td>3644554</td>\n",
              "      <td>42</td>\n",
              "      <td>None</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>02/27/2023 12:50:14</td>\n",
              "      <td>02/27/2023 17:35:12</td>\n",
              "      <td>02/27/2023 17:35:12</td>\n",
              "      <td>14715</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>448</th>\n",
              "      <td>3644558</td>\n",
              "      <td>402</td>\n",
              "      <td>None</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>02/27/2023 13:00:00</td>\n",
              "      <td>02/27/2023 17:31:47</td>\n",
              "      <td>02/27/2023 17:31:47</td>\n",
              "      <td>86938</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>449</th>\n",
              "      <td>3644559</td>\n",
              "      <td>402</td>\n",
              "      <td>None</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>02/27/2023 12:46:00</td>\n",
              "      <td>02/27/2023 17:31:47</td>\n",
              "      <td>02/27/2023 17:31:47</td>\n",
              "      <td>94496</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>450</th>\n",
              "      <td>3644546</td>\n",
              "      <td>402</td>\n",
              "      <td>None</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>02/27/2023 12:30:00</td>\n",
              "      <td>02/27/2023 17:31:47</td>\n",
              "      <td>02/27/2023 17:31:47</td>\n",
              "      <td>94158</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>451</th>\n",
              "      <td>3644553</td>\n",
              "      <td>42</td>\n",
              "      <td>None</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>02/27/2023 12:50:14</td>\n",
              "      <td>02/27/2023 17:35:12</td>\n",
              "      <td>02/27/2023 17:35:12</td>\n",
              "      <td>14715</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>452 rows × 37 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-9da30fb3-cee6-4b6c-8777-45edcf98c961')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-9da30fb3-cee6-4b6c-8777-45edcf98c961 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-9da30fb3-cee6-4b6c-8777-45edcf98c961');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "          id  operatorId subsection  type  origin  geoType  \\\n",
              "0    3644311         180       None     2       5        2   \n",
              "1    3644358         180       None     2       5        2   \n",
              "2    3644669         402       None     1       1        2   \n",
              "3    3644563         182       None     2       5        2   \n",
              "4    3644600         182       None     2       5        2   \n",
              "..       ...         ...        ...   ...     ...      ...   \n",
              "447  3644554          42       None     1       1        2   \n",
              "448  3644558         402       None     2       1        2   \n",
              "449  3644559         402       None     1       1        2   \n",
              "450  3644546         402       None     2       1        2   \n",
              "451  3644553          42       None     1       1        2   \n",
              "\n",
              "               dateStart              dateEnd           lastUpdate postalCode  \\\n",
              "0    02/27/2023 09:09:57  02/27/2023 09:54:15  02/27/2023 09:54:15      27580   \n",
              "1    02/27/2023 09:09:57  02/27/2023 09:56:14  02/27/2023 09:54:15      27580   \n",
              "2    02/27/2023 14:12:57  02/27/2023 17:31:47  02/27/2023 17:31:47      84371   \n",
              "3    02/27/2023 10:36:07  02/27/2023 15:48:54  02/27/2023 15:48:54      28201   \n",
              "4    02/27/2023 09:00:07  02/27/2023 15:48:54  02/27/2023 15:48:54      28205   \n",
              "..                   ...                  ...                  ...        ...   \n",
              "447  02/27/2023 12:50:14  02/27/2023 17:35:12  02/27/2023 17:35:12      14715   \n",
              "448  02/27/2023 13:00:00  02/27/2023 17:31:47  02/27/2023 17:31:47      86938   \n",
              "449  02/27/2023 12:46:00  02/27/2023 17:31:47  02/27/2023 17:31:47      94496   \n",
              "450  02/27/2023 12:30:00  02/27/2023 17:31:47  02/27/2023 17:31:47      94158   \n",
              "451  02/27/2023 12:50:14  02/27/2023 17:35:12  02/27/2023 17:35:12      14715   \n",
              "\n",
              "     ... internal houseNr photo  containerShape.coordinates  \\\n",
              "0    ...        0    None  None                        None   \n",
              "1    ...        0    None  None                        None   \n",
              "2    ...        0    None  None                        None   \n",
              "3    ...        0    None  None                        None   \n",
              "4    ...        0    None  None                        None   \n",
              "..   ...      ...     ...   ...                         ...   \n",
              "447  ...        0    None  None                        None   \n",
              "448  ...        0    None  None                        None   \n",
              "449  ...        0    None  None                        None   \n",
              "450  ...        0    None  None                        None   \n",
              "451  ...        0    None  None                        None   \n",
              "\n",
              "    containerShape.envelope containerShape.envelopeInternal.minX  \\\n",
              "0                      None                                  NaN   \n",
              "1                      None                                  NaN   \n",
              "2                      None                                  NaN   \n",
              "3                      None                                  NaN   \n",
              "4                      None                                  NaN   \n",
              "..                      ...                                  ...   \n",
              "447                    None                                  NaN   \n",
              "448                    None                                  NaN   \n",
              "449                    None                                  NaN   \n",
              "450                    None                                  NaN   \n",
              "451                    None                                  NaN   \n",
              "\n",
              "    containerShape.envelopeInternal.minY containerShape.envelopeInternal.MaxX  \\\n",
              "0                                    NaN                                  NaN   \n",
              "1                                    NaN                                  NaN   \n",
              "2                                    NaN                                  NaN   \n",
              "3                                    NaN                                  NaN   \n",
              "4                                    NaN                                  NaN   \n",
              "..                                   ...                                  ...   \n",
              "447                                  NaN                                  NaN   \n",
              "448                                  NaN                                  NaN   \n",
              "449                                  NaN                                  NaN   \n",
              "450                                  NaN                                  NaN   \n",
              "451                                  NaN                                  NaN   \n",
              "\n",
              "    containerShape.envelopeInternal.MaxY containerShape.centerCoordinates  \n",
              "0                                    NaN                              NaN  \n",
              "1                                    NaN                              NaN  \n",
              "2                                    NaN                              NaN  \n",
              "3                                    NaN                              NaN  \n",
              "4                                    NaN                              NaN  \n",
              "..                                   ...                              ...  \n",
              "447                                  NaN                              NaN  \n",
              "448                                  NaN                              NaN  \n",
              "449                                  NaN                              NaN  \n",
              "450                                  NaN                              NaN  \n",
              "451                                  NaN                              NaN  \n",
              "\n",
              "[452 rows x 37 columns]"
            ]
          },
          "execution_count": 53,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import pandas as pd\n",
        "\n",
        "stromausfaelle = pd.json_normalize(content)\n",
        "stromausfaelle"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "authorship_tag": "ABX9TyNSvGR+Lo0L3HHEI45OX77Y",
      "include_colab_link": true,
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}