36-tfx-custom-components/C4_W3_Lab_2_TFX_Custom_Components.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6x1ypzczQCwy"
      },
      "source": [
        "# Ungraded Lab: Developing Custom TFX Components\n",
        "\n",
        "[Tensorflow Extended (TFX)](https://www.tensorflow.org/tfx/) provides ready made components for typical steps in a machine learning workflow. Other courses in this specialization focus on specific components and in this lab, you will learn how to make your own. This will be useful in case your project has specific needs that fall outside the standard TFX components. It will make your pipelines more flexible while still leveraging the experiment tracking and orchestration that TFX provides. In particular, you will:\n",
        "\n",
        "* build a custom component using Python functions\n",
        "* build a custom component by reusing an existing TFX component\n",
        "* run a TFX pipeline locally using a built-in pipeline orchestrator\n",
        "\n",
        "To demonstrate, you will run the pipeline used in this [official tutorial](https://colab.research.google.com/github/tensorflow/tfx/blob/master/docs/tutorials/tfx/penguin_simple.ipynb) then modify it to have a custom component. Some of the discussions here are also taken from that tutorial to explain the motivation and point to additional resources.\n",
        "\n",
        "Let's begin!\n",
        "\n",
        "*Note: If you haven't taken other courses in this specialization and it's the first time you're using TFX, please see\n",
        "[Understanding TFX Pipelines](https://www.tensorflow.org/tfx/guide/understanding_tfx_pipelines)\n",
        "to get an overview of important concepts.*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fmgi8ZvQkScg"
      },
      "source": [
        "## Set Up\n",
        "We first need to install the TFX Python package and download\n",
        "the dataset which we will use for our model.\n",
        "\n",
        "### Upgrade Pip\n",
        "\n",
        "To avoid upgrading Pip in a system when running locally,\n",
        "check to make sure that we are running in Colab.\n",
        "Local systems can of course be upgraded separately."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "as4OTe2ukSqm"
      },
      "source": [
        "try:\n",
        "  import colab\n",
        "  !pip install --upgrade pip\n",
        "except:\n",
        "  pass"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MZOYTt1RW4TK"
      },
      "source": [
        "### Install TFX\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iyQtljP-qPHY",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "8fb4cb22-73c9-4fc7-f5f3-57203e0f59e6"
      },
      "source": [
        "!pip install -U tfx"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting tfx\n",
            "  Downloading tfx-1.14.0-py3-none-any.whl (2.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.8/2.8 MB\u001b[0m \u001b[31m7.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting ml-pipelines-sdk==1.14.0 (from tfx)\n",
            "  Downloading ml_pipelines_sdk-1.14.0-py3-none-any.whl (1.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.5/1.5 MB\u001b[0m \u001b[31m25.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.10/dist-packages (from tfx) (1.4.0)\n",
            "Collecting ml-metadata<1.15.0,>=1.14.0 (from tfx)\n",
            "  Downloading ml_metadata-1.14.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.5/7.5 MB\u001b[0m \u001b[31m36.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting packaging<21,>=20 (from tfx)\n",
            "  Downloading packaging-20.9-py2.py3-none-any.whl (40 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m40.9/40.9 kB\u001b[0m \u001b[31m3.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: portpicker<2,>=1.3.1 in /usr/local/lib/python3.10/dist-packages (from tfx) (1.5.2)\n",
            "Requirement already satisfied: protobuf<5,>=3.20.3 in /usr/local/lib/python3.10/dist-packages (from tfx) (3.20.3)\n",
            "Collecting docker<5,>=4.1 (from tfx)\n",
            "  Downloading docker-4.4.4-py2.py3-none-any.whl (147 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m147.0/147.0 kB\u001b[0m \u001b[31m9.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting google-apitools<1,>=0.5 (from tfx)\n",
            "  Downloading google_apitools-0.5.32-py3-none-any.whl (135 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m135.7/135.7 kB\u001b[0m \u001b[31m11.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting google-api-python-client<2,>=1.8 (from tfx)\n",
            "  Downloading google_api_python_client-1.12.11-py2.py3-none-any.whl (62 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.1/62.1 kB\u001b[0m \u001b[31m2.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: jinja2<4,>=2.7.3 in /usr/local/lib/python3.10/dist-packages (from tfx) (3.1.2)\n",
            "Requirement already satisfied: typing-extensions<5,>=3.10.0.2 in /usr/local/lib/python3.10/dist-packages (from tfx) (4.5.0)\n",
            "Collecting apache-beam[gcp]<3,>=2.47 (from tfx)\n",
            "  Downloading apache_beam-2.53.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (14.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m14.8/14.8 MB\u001b[0m \u001b[31m16.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting attrs<22,>=19.3.0 (from tfx)\n",
            "  Downloading attrs-21.4.0-py2.py3-none-any.whl (60 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m60.6/60.6 kB\u001b[0m \u001b[31m4.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: click<9,>=7 in /usr/local/lib/python3.10/dist-packages (from tfx) (8.1.7)\n",
            "Requirement already satisfied: google-api-core<3 in /usr/local/lib/python3.10/dist-packages (from tfx) (2.11.1)\n",
            "Requirement already satisfied: google-cloud-aiplatform<2,>=1.6.2 in /usr/local/lib/python3.10/dist-packages (from tfx) (1.38.1)\n",
            "Collecting google-cloud-bigquery<3,>=2.26.0 (from tfx)\n",
            "  Downloading google_cloud_bigquery-2.34.4-py2.py3-none-any.whl (206 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m206.6/206.6 kB\u001b[0m \u001b[31m10.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: grpcio<2,>=1.28.1 in /usr/local/lib/python3.10/dist-packages (from tfx) (1.60.0)\n",
            "Collecting keras-tuner<2,>=1.0.4 (from tfx)\n",
            "  Downloading keras_tuner-1.4.6-py3-none-any.whl (128 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m128.9/128.9 kB\u001b[0m \u001b[31m12.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting kubernetes<13,>=10.0.1 (from tfx)\n",
            "  Downloading kubernetes-12.0.1-py2.py3-none-any.whl (1.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m29.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: numpy<2,>=1.16 in /usr/local/lib/python3.10/dist-packages (from tfx) (1.23.5)\n",
            "Requirement already satisfied: pyarrow<11,>=10 in /usr/local/lib/python3.10/dist-packages (from tfx) (10.0.1)\n",
            "Requirement already satisfied: pyyaml<7,>=6 in /usr/local/lib/python3.10/dist-packages (from tfx) (6.0.1)\n",
            "Collecting tensorflow<2.14,>=2.13.0 (from tfx)\n",
            "  Downloading tensorflow-2.13.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (479.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m479.7/479.7 MB\u001b[0m \u001b[31m2.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting tensorflow-hub<0.14,>=0.9.0 (from tfx)\n",
            "  Downloading tensorflow_hub-0.13.0-py2.py3-none-any.whl (100 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m100.6/100.6 kB\u001b[0m \u001b[31m11.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting tensorflow-data-validation<1.15.0,>=1.14.0 (from tfx)\n",
            "  Downloading tensorflow_data_validation-1.14.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (19.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m19.1/19.1 MB\u001b[0m \u001b[31m69.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting tensorflow-model-analysis<0.46.0,>=0.45.0 (from tfx)\n",
            "  Downloading tensorflow_model_analysis-0.45.0-py3-none-any.whl (1.9 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.9/1.9 MB\u001b[0m \u001b[31m63.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting tensorflow-serving-api!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<3,>=1.15 (from tfx)\n",
            "  Downloading tensorflow_serving_api-2.14.1-py2.py3-none-any.whl (26 kB)\n",
            "Collecting tensorflow-transform<1.15.0,>=1.14.0 (from tfx)\n",
            "  Downloading tensorflow_transform-1.14.0-py3-none-any.whl (447 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m447.8/447.8 kB\u001b[0m \u001b[31m41.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting tfx-bsl<1.15.0,>=1.14.0 (from tfx)\n",
            "  Downloading tfx_bsl-1.14.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (22.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m22.5/22.5 MB\u001b[0m \u001b[31m62.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting crcmod<2.0,>=1.7 (from apache-beam[gcp]<3,>=2.47->tfx)\n",
            "  Downloading crcmod-1.7.tar.gz (89 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m89.7/89.7 kB\u001b[0m \u001b[31m9.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting orjson<4,>=3.9.7 (from apache-beam[gcp]<3,>=2.47->tfx)\n",
            "  Downloading orjson-3.9.10-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (138 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m138.7/138.7 kB\u001b[0m \u001b[31m18.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting dill<0.3.2,>=0.3.1.1 (from apache-beam[gcp]<3,>=2.47->tfx)\n",
            "  Downloading dill-0.3.1.1.tar.gz (151 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m152.0/152.0 kB\u001b[0m \u001b[31m19.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: cloudpickle~=2.2.1 in /usr/local/lib/python3.10/dist-packages (from apache-beam[gcp]<3,>=2.47->tfx) (2.2.1)\n",
            "Collecting fastavro<2,>=0.23.6 (from apache-beam[gcp]<3,>=2.47->tfx)\n",
            "  Downloading fastavro-1.9.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.1/3.1 MB\u001b[0m \u001b[31m67.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting fasteners<1.0,>=0.3 (from apache-beam[gcp]<3,>=2.47->tfx)\n",
            "  Downloading fasteners-0.19-py3-none-any.whl (18 kB)\n",
            "Collecting hdfs<3.0.0,>=2.1.0 (from apache-beam[gcp]<3,>=2.47->tfx)\n",
            "  Downloading hdfs-2.7.3.tar.gz (43 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m43.5/43.5 kB\u001b[0m \u001b[31m5.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: httplib2<0.23.0,>=0.8 in /usr/local/lib/python3.10/dist-packages (from apache-beam[gcp]<3,>=2.47->tfx) (0.22.0)\n",
            "Collecting js2py<1,>=0.74 (from apache-beam[gcp]<3,>=2.47->tfx)\n",
            "  Downloading Js2Py-0.74-py3-none-any.whl (1.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.0/1.0 MB\u001b[0m \u001b[31m51.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: jsonschema<5.0.0,>=4.0.0 in /usr/local/lib/python3.10/dist-packages (from apache-beam[gcp]<3,>=2.47->tfx) (4.19.2)\n",
            "Requirement already satisfied: jsonpickle<4.0.0,>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from apache-beam[gcp]<3,>=2.47->tfx) (3.0.2)\n",
            "Collecting objsize<0.7.0,>=0.6.1 (from apache-beam[gcp]<3,>=2.47->tfx)\n",
            "  Downloading objsize-0.6.1-py3-none-any.whl (9.3 kB)\n",
            "INFO: pip is looking at multiple versions of apache-beam[gcp] to determine which version is compatible with other requirements. This could take a while.\n",
            "Collecting apache-beam[gcp]<3,>=2.47 (from tfx)\n",
            "  Downloading apache_beam-2.52.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (14.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m14.7/14.7 MB\u001b[0m \u001b[31m55.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading apache_beam-2.51.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (14.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m14.7/14.7 MB\u001b[0m \u001b[31m36.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading apache_beam-2.50.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (14.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m14.7/14.7 MB\u001b[0m \u001b[31m32.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pymongo<5.0.0,>=3.8.0 (from apache-beam[gcp]<3,>=2.47->tfx)\n",
            "  Downloading pymongo-4.6.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (677 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m677.1/677.1 kB\u001b[0m \u001b[31m48.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: proto-plus<2,>=1.7.1 in /usr/local/lib/python3.10/dist-packages (from apache-beam[gcp]<3,>=2.47->tfx) (1.23.0)\n",
            "Requirement already satisfied: pydot<2,>=1.2.0 in /usr/local/lib/python3.10/dist-packages (from apache-beam[gcp]<3,>=2.47->tfx) (1.4.2)\n",
            "Requirement already satisfied: python-dateutil<3,>=2.8.0 in /usr/local/lib/python3.10/dist-packages (from apache-beam[gcp]<3,>=2.47->tfx) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2018.3 in /usr/local/lib/python3.10/dist-packages (from apache-beam[gcp]<3,>=2.47->tfx) (2023.3.post1)\n",
            "Requirement already satisfied: regex>=2020.6.8 in /usr/local/lib/python3.10/dist-packages (from apache-beam[gcp]<3,>=2.47->tfx) (2023.6.3)\n",
            "Requirement already satisfied: requests<3.0.0,>=2.24.0 in /usr/local/lib/python3.10/dist-packages (from apache-beam[gcp]<3,>=2.47->tfx) (2.31.0)\n",
            "Collecting zstandard<1,>=0.18.0 (from apache-beam[gcp]<3,>=2.47->tfx)\n",
            "  Downloading zstandard-0.22.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.4 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.4/5.4 MB\u001b[0m \u001b[31m66.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: cachetools<6,>=3.1.0 in /usr/local/lib/python3.10/dist-packages (from apache-beam[gcp]<3,>=2.47->tfx) (5.3.2)\n",
            "Collecting google-apitools<1,>=0.5 (from tfx)\n",
            "  Downloading google-apitools-0.5.31.tar.gz (173 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m173.5/173.5 kB\u001b[0m \u001b[31m23.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: google-auth<3,>=1.18.0 in /usr/local/lib/python3.10/dist-packages (from apache-beam[gcp]<3,>=2.47->tfx) (2.17.3)\n",
            "Requirement already satisfied: google-auth-httplib2<0.2.0,>=0.1.0 in /usr/local/lib/python3.10/dist-packages (from apache-beam[gcp]<3,>=2.47->tfx) (0.1.1)\n",
            "Requirement already satisfied: google-cloud-datastore<3,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from apache-beam[gcp]<3,>=2.47->tfx) (2.15.2)\n",
            "Collecting google-cloud-pubsub<3,>=2.1.0 (from apache-beam[gcp]<3,>=2.47->tfx)\n",
            "  Downloading google_cloud_pubsub-2.19.0-py2.py3-none-any.whl (265 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m265.3/265.3 kB\u001b[0m \u001b[31m31.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting google-cloud-pubsublite<2,>=1.2.0 (from apache-beam[gcp]<3,>=2.47->tfx)\n",
            "  Downloading google_cloud_pubsublite-1.9.0-py2.py3-none-any.whl (287 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m287.3/287.3 kB\u001b[0m \u001b[31m34.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: google-cloud-bigquery-storage<3,>=2.6.3 in /usr/local/lib/python3.10/dist-packages (from apache-beam[gcp]<3,>=2.47->tfx) (2.24.0)\n",
            "Requirement already satisfied: google-cloud-core<3,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from apache-beam[gcp]<3,>=2.47->tfx) (2.3.3)\n",
            "Collecting google-cloud-bigtable<3,>=2.19.0 (from apache-beam[gcp]<3,>=2.47->tfx)\n",
            "  Downloading google_cloud_bigtable-2.22.0-py2.py3-none-any.whl (295 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m295.1/295.1 kB\u001b[0m \u001b[31m32.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting google-cloud-spanner<4,>=3.0.0 (from apache-beam[gcp]<3,>=2.47->tfx)\n",
            "  Downloading google_cloud_spanner-3.40.1-py2.py3-none-any.whl (332 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m332.9/332.9 kB\u001b[0m \u001b[31m34.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting google-cloud-dlp<4,>=3.0.0 (from apache-beam[gcp]<3,>=2.47->tfx)\n",
            "  Downloading google_cloud_dlp-3.14.0-py2.py3-none-any.whl (157 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m157.5/157.5 kB\u001b[0m \u001b[31m20.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: google-cloud-language<3,>=2.0 in /usr/local/lib/python3.10/dist-packages (from apache-beam[gcp]<3,>=2.47->tfx) (2.9.1)\n",
            "Collecting google-cloud-videointelligence<3,>=2.0 (from apache-beam[gcp]<3,>=2.47->tfx)\n",
            "  Downloading google_cloud_videointelligence-2.12.0-py2.py3-none-any.whl (228 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m228.6/228.6 kB\u001b[0m \u001b[31m26.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting google-cloud-vision<4,>=2 (from apache-beam[gcp]<3,>=2.47->tfx)\n",
            "  Downloading google_cloud_vision-3.5.0-py2.py3-none-any.whl (442 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m442.1/442.1 kB\u001b[0m \u001b[31m35.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting google-cloud-recommendations-ai<0.11.0,>=0.1.0 (from apache-beam[gcp]<3,>=2.47->tfx)\n",
            "  Downloading google_cloud_recommendations_ai-0.10.6-py2.py3-none-any.whl (172 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m172.5/172.5 kB\u001b[0m \u001b[31m21.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: six>=1.4.0 in /usr/local/lib/python3.10/dist-packages (from docker<5,>=4.1->tfx) (1.16.0)\n",
            "Requirement already satisfied: websocket-client>=0.32.0 in /usr/local/lib/python3.10/dist-packages (from docker<5,>=4.1->tfx) (1.7.0)\n",
            "Requirement already satisfied: googleapis-common-protos<2.0.dev0,>=1.56.2 in /usr/local/lib/python3.10/dist-packages (from google-api-core<3->tfx) (1.62.0)\n",
            "Collecting uritemplate<4dev,>=3.0.0 (from google-api-python-client<2,>=1.8->tfx)\n",
            "  Downloading uritemplate-3.0.1-py2.py3-none-any.whl (15 kB)\n",
            "Requirement already satisfied: oauth2client>=1.4.12 in /usr/local/lib/python3.10/dist-packages (from google-apitools<1,>=0.5->tfx) (4.1.3)\n",
            "Requirement already satisfied: google-cloud-storage<3.0.0dev,>=1.32.0 in /usr/local/lib/python3.10/dist-packages (from google-cloud-aiplatform<2,>=1.6.2->tfx) (2.8.0)\n",
            "Requirement already satisfied: google-cloud-resource-manager<3.0.0dev,>=1.3.3 in /usr/local/lib/python3.10/dist-packages (from google-cloud-aiplatform<2,>=1.6.2->tfx) (1.11.0)\n",
            "Requirement already satisfied: shapely<3.0.0dev in /usr/local/lib/python3.10/dist-packages (from google-cloud-aiplatform<2,>=1.6.2->tfx) (2.0.2)\n",
            "Requirement already satisfied: google-resumable-media<3.0dev,>=0.6.0 in /usr/local/lib/python3.10/dist-packages (from google-cloud-bigquery<3,>=2.26.0->tfx) (2.7.0)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2<4,>=2.7.3->tfx) (2.1.3)\n",
            "Requirement already satisfied: keras in /usr/local/lib/python3.10/dist-packages (from keras-tuner<2,>=1.0.4->tfx) (2.15.0)\n",
            "Collecting kt-legacy (from keras-tuner<2,>=1.0.4->tfx)\n",
            "  Downloading kt_legacy-1.0.5-py3-none-any.whl (9.6 kB)\n",
            "Requirement already satisfied: certifi>=14.05.14 in /usr/local/lib/python3.10/dist-packages (from kubernetes<13,>=10.0.1->tfx) (2023.11.17)\n",
            "Requirement already satisfied: setuptools>=21.0.0 in /usr/local/lib/python3.10/dist-packages (from kubernetes<13,>=10.0.1->tfx) (67.7.2)\n",
            "Requirement already satisfied: requests-oauthlib in /usr/local/lib/python3.10/dist-packages (from kubernetes<13,>=10.0.1->tfx) (1.3.1)\n",
            "Requirement already satisfied: urllib3>=1.24.2 in /usr/local/lib/python3.10/dist-packages (from kubernetes<13,>=10.0.1->tfx) (2.0.7)\n",
            "Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.10/dist-packages (from packaging<21,>=20->tfx) (3.1.1)\n",
            "Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from portpicker<2,>=1.3.1->tfx) (5.9.5)\n",
            "Requirement already satisfied: astunparse>=1.6.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow<2.14,>=2.13.0->tfx) (1.6.3)\n",
            "Requirement already satisfied: flatbuffers>=23.1.21 in /usr/local/lib/python3.10/dist-packages (from tensorflow<2.14,>=2.13.0->tfx) (23.5.26)\n",
            "Collecting gast<=0.4.0,>=0.2.1 (from tensorflow<2.14,>=2.13.0->tfx)\n",
            "  Downloading gast-0.4.0-py3-none-any.whl (9.8 kB)\n",
            "Requirement already satisfied: google-pasta>=0.1.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow<2.14,>=2.13.0->tfx) (0.2.0)\n",
            "Requirement already satisfied: h5py>=2.9.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow<2.14,>=2.13.0->tfx) (3.9.0)\n",
            "Collecting keras (from keras-tuner<2,>=1.0.4->tfx)\n",
            "  Downloading keras-2.13.1-py3-none-any.whl (1.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m79.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: libclang>=13.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow<2.14,>=2.13.0->tfx) (16.0.6)\n",
            "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.10/dist-packages (from tensorflow<2.14,>=2.13.0->tfx) (3.3.0)\n",
            "Collecting tensorboard<2.14,>=2.13 (from tensorflow<2.14,>=2.13.0->tfx)\n",
            "  Downloading tensorboard-2.13.0-py3-none-any.whl (5.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.6/5.6 MB\u001b[0m \u001b[31m73.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting tensorflow-estimator<2.14,>=2.13.0 (from tensorflow<2.14,>=2.13.0->tfx)\n",
            "  Downloading tensorflow_estimator-2.13.0-py2.py3-none-any.whl (440 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m440.8/440.8 kB\u001b[0m \u001b[31m46.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow<2.14,>=2.13.0->tfx) (2.4.0)\n",
            "Requirement already satisfied: wrapt>=1.11.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow<2.14,>=2.13.0->tfx) (1.14.1)\n",
            "Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow<2.14,>=2.13.0->tfx) (0.35.0)\n",
            "Requirement already satisfied: joblib>=1.2.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow-data-validation<1.15.0,>=1.14.0->tfx) (1.3.2)\n",
            "Requirement already satisfied: pandas<2,>=1.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow-data-validation<1.15.0,>=1.14.0->tfx) (1.5.3)\n",
            "Collecting pyfarmhash<0.4,>=0.2.2 (from tensorflow-data-validation<1.15.0,>=1.14.0->tfx)\n",
            "  Downloading pyfarmhash-0.3.2.tar.gz (99 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m99.9/99.9 kB\u001b[0m \u001b[31m14.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: tensorflow-metadata<1.15,>=1.14.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow-data-validation<1.15.0,>=1.14.0->tfx) (1.14.0)\n",
            "Requirement already satisfied: ipython<8,>=7 in /usr/local/lib/python3.10/dist-packages (from tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (7.34.0)\n",
            "Requirement already satisfied: ipywidgets<8,>=7 in /usr/local/lib/python3.10/dist-packages (from tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (7.7.1)\n",
            "Requirement already satisfied: pillow>=9.4.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (9.4.0)\n",
            "Requirement already satisfied: scipy<2,>=1.4.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (1.11.4)\n",
            "INFO: pip is looking at multiple versions of tensorflow-serving-api to determine which version is compatible with other requirements. This could take a while.\n",
            "Collecting tensorflow-serving-api!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,<3,>=1.15 (from tfx)\n",
            "  Downloading tensorflow_serving_api-2.14.0-py2.py3-none-any.whl (26 kB)\n",
            "  Downloading tensorflow_serving_api-2.13.1-py2.py3-none-any.whl (26 kB)\n",
            "Requirement already satisfied: wheel<1.0,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from astunparse>=1.6.0->tensorflow<2.14,>=2.13.0->tfx) (0.42.0)\n",
            "Requirement already satisfied: grpcio-status<2.0.dev0,>=1.33.2 in /usr/local/lib/python3.10/dist-packages (from google-api-core<3->tfx) (1.48.2)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.18.0->apache-beam[gcp]<3,>=2.47->tfx) (0.3.0)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.18.0->apache-beam[gcp]<3,>=2.47->tfx) (4.9)\n",
            "Requirement already satisfied: grpc-google-iam-v1<1.0.0dev,>=0.12.4 in /usr/local/lib/python3.10/dist-packages (from google-cloud-bigtable<3,>=2.19.0->apache-beam[gcp]<3,>=2.47->tfx) (0.13.0)\n",
            "Collecting overrides<8.0.0,>=6.0.1 (from google-cloud-pubsublite<2,>=1.2.0->apache-beam[gcp]<3,>=2.47->tfx)\n",
            "  Downloading overrides-7.4.0-py3-none-any.whl (17 kB)\n",
            "Requirement already satisfied: sqlparse>=0.4.4 in /usr/local/lib/python3.10/dist-packages (from google-cloud-spanner<4,>=3.0.0->apache-beam[gcp]<3,>=2.47->tfx) (0.4.4)\n",
            "Requirement already satisfied: google-crc32c<2.0dev,>=1.0 in /usr/local/lib/python3.10/dist-packages (from google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3,>=2.26.0->tfx) (1.5.0)\n",
            "Collecting docopt (from hdfs<3.0.0,>=2.1.0->apache-beam[gcp]<3,>=2.47->tfx)\n",
            "  Downloading docopt-0.6.2.tar.gz (25 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting jedi>=0.16 (from ipython<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx)\n",
            "  Downloading jedi-0.19.1-py2.py3-none-any.whl (1.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.6/1.6 MB\u001b[0m \u001b[31m78.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: decorator in /usr/local/lib/python3.10/dist-packages (from ipython<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (4.4.2)\n",
            "Requirement already satisfied: pickleshare in /usr/local/lib/python3.10/dist-packages (from ipython<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (0.7.5)\n",
            "Requirement already satisfied: traitlets>=4.2 in /usr/local/lib/python3.10/dist-packages (from ipython<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (5.7.1)\n",
            "Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from ipython<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (3.0.43)\n",
            "Requirement already satisfied: pygments in /usr/local/lib/python3.10/dist-packages (from ipython<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (2.16.1)\n",
            "Requirement already satisfied: backcall in /usr/local/lib/python3.10/dist-packages (from ipython<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (0.2.0)\n",
            "Requirement already satisfied: matplotlib-inline in /usr/local/lib/python3.10/dist-packages (from ipython<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (0.1.6)\n",
            "Requirement already satisfied: pexpect>4.3 in /usr/local/lib/python3.10/dist-packages (from ipython<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (4.9.0)\n",
            "Requirement already satisfied: ipykernel>=4.5.1 in /usr/local/lib/python3.10/dist-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (5.5.6)\n",
            "Requirement already satisfied: ipython-genutils~=0.2.0 in /usr/local/lib/python3.10/dist-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (0.2.0)\n",
            "Requirement already satisfied: widgetsnbextension~=3.6.0 in /usr/local/lib/python3.10/dist-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (3.6.6)\n",
            "Requirement already satisfied: jupyterlab-widgets>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (3.0.9)\n",
            "Requirement already satisfied: pyasn1>=0.1.7 in /usr/local/lib/python3.10/dist-packages (from oauth2client>=1.4.12->google-apitools<1,>=0.5->tfx) (0.5.1)\n",
            "Collecting dnspython<3.0.0,>=1.16.0 (from pymongo<5.0.0,>=3.8.0->apache-beam[gcp]<3,>=2.47->tfx)\n",
            "  Downloading dnspython-2.4.2-py3-none-any.whl (300 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m300.4/300.4 kB\u001b[0m \u001b[31m31.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.24.0->apache-beam[gcp]<3,>=2.47->tfx) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.24.0->apache-beam[gcp]<3,>=2.47->tfx) (3.6)\n",
            "Collecting google-auth-oauthlib<1.1,>=0.5 (from tensorboard<2.14,>=2.13->tensorflow<2.14,>=2.13.0->tfx)\n",
            "  Downloading google_auth_oauthlib-1.0.0-py2.py3-none-any.whl (18 kB)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.14,>=2.13->tensorflow<2.14,>=2.13.0->tfx) (3.5.1)\n",
            "Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.14,>=2.13->tensorflow<2.14,>=2.13.0->tfx) (0.7.2)\n",
            "Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.14,>=2.13->tensorflow<2.14,>=2.13.0->tfx) (3.0.1)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from requests-oauthlib->kubernetes<13,>=10.0.1->tfx) (3.2.2)\n",
            "Requirement already satisfied: jupyter-client in /usr/local/lib/python3.10/dist-packages (from ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (6.1.12)\n",
            "Requirement already satisfied: tornado>=4.2 in /usr/local/lib/python3.10/dist-packages (from ipykernel>=4.5.1->ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (6.3.2)\n",
            "Requirement already satisfied: parso<0.9.0,>=0.8.3 in /usr/local/lib/python3.10/dist-packages (from jedi>=0.16->ipython<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (0.8.3)\n",
            "Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.10/dist-packages (from pexpect>4.3->ipython<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (0.7.0)\n",
            "Requirement already satisfied: wcwidth in /usr/local/lib/python3.10/dist-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (0.2.12)\n",
            "Requirement already satisfied: notebook>=4.4.1 in /usr/local/lib/python3.10/dist-packages (from widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (6.5.5)\n",
            "Requirement already satisfied: pyzmq<25,>=17 in /usr/local/lib/python3.10/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (23.2.1)\n",
            "Requirement already satisfied: argon2-cffi in /usr/local/lib/python3.10/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (23.1.0)\n",
            "Requirement already satisfied: jupyter-core>=4.6.1 in /usr/local/lib/python3.10/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (5.7.0)\n",
            "Requirement already satisfied: nbformat in /usr/local/lib/python3.10/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (5.9.2)\n",
            "Requirement already satisfied: nbconvert>=5 in /usr/local/lib/python3.10/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (6.5.4)\n",
            "Requirement already satisfied: nest-asyncio>=1.5 in /usr/local/lib/python3.10/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (1.5.8)\n",
            "Requirement already satisfied: Send2Trash>=1.8.0 in /usr/local/lib/python3.10/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (1.8.2)\n",
            "Requirement already satisfied: terminado>=0.8.3 in /usr/local/lib/python3.10/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (0.18.0)\n",
            "Requirement already satisfied: prometheus-client in /usr/local/lib/python3.10/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (0.19.0)\n",
            "Requirement already satisfied: nbclassic>=0.4.7 in /usr/local/lib/python3.10/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (1.0.0)\n",
            "Requirement already satisfied: platformdirs>=2.5 in /usr/local/lib/python3.10/dist-packages (from jupyter-core>=4.6.1->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (4.1.0)\n",
            "Requirement already satisfied: jupyter-server>=1.8 in /usr/local/lib/python3.10/dist-packages (from nbclassic>=0.4.7->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (1.24.0)\n",
            "Requirement already satisfied: notebook-shim>=0.2.3 in /usr/local/lib/python3.10/dist-packages (from nbclassic>=0.4.7->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (0.2.3)\n",
            "Requirement already satisfied: lxml in /usr/local/lib/python3.10/dist-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (4.9.4)\n",
            "Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.10/dist-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (4.11.2)\n",
            "Requirement already satisfied: bleach in /usr/local/lib/python3.10/dist-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (6.1.0)\n",
            "Requirement already satisfied: defusedxml in /usr/local/lib/python3.10/dist-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (0.7.1)\n",
            "Requirement already satisfied: entrypoints>=0.2.2 in /usr/local/lib/python3.10/dist-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (0.4)\n",
            "Requirement already satisfied: jupyterlab-pygments in /usr/local/lib/python3.10/dist-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (0.3.0)\n",
            "Requirement already satisfied: mistune<2,>=0.8.1 in /usr/local/lib/python3.10/dist-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (0.8.4)\n",
            "Requirement already satisfied: nbclient>=0.5.0 in /usr/local/lib/python3.10/dist-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (0.9.0)\n",
            "Requirement already satisfied: pandocfilters>=1.4.1 in /usr/local/lib/python3.10/dist-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (1.5.0)\n",
            "Requirement already satisfied: tinycss2 in /usr/local/lib/python3.10/dist-packages (from nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (1.2.1)\n",
            "Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.10/dist-packages (from nbformat->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (2.19.1)\n",
            "Requirement already satisfied: argon2-cffi-bindings in /usr/local/lib/python3.10/dist-packages (from argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (21.2.0)\n",
            "INFO: pip is looking at multiple versions of jsonschema to determine which version is compatible with other requirements. This could take a while.\n",
            "Collecting jsonschema>=2.6 (from nbformat->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx)\n",
            "  Downloading jsonschema-4.20.0-py3-none-any.whl (84 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m84.7/84.7 kB\u001b[0m \u001b[31m11.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading jsonschema-4.19.1-py3-none-any.whl (83 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m83.3/83.3 kB\u001b[0m \u001b[31m12.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading jsonschema-4.19.0-py3-none-any.whl (83 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m83.4/83.4 kB\u001b[0m \u001b[31m11.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading jsonschema-4.18.6-py3-none-any.whl (83 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m83.2/83.2 kB\u001b[0m \u001b[31m10.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading jsonschema-4.18.5-py3-none-any.whl (82 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m82.5/82.5 kB\u001b[0m \u001b[31m11.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading jsonschema-4.18.4-py3-none-any.whl (80 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m81.0/81.0 kB\u001b[0m \u001b[31m10.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading jsonschema-4.18.3-py3-none-any.whl (80 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m81.0/81.0 kB\u001b[0m \u001b[31m11.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hINFO: pip is looking at multiple versions of jsonschema to determine which version is compatible with other requirements. This could take a while.\n",
            "  Downloading jsonschema-4.18.2-py3-none-any.whl (80 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m80.8/80.8 kB\u001b[0m \u001b[31m10.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading jsonschema-4.18.1-py3-none-any.whl (80 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m80.7/80.7 kB\u001b[0m \u001b[31m11.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading jsonschema-4.18.0-py3-none-any.whl (81 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m81.5/81.5 kB\u001b[0m \u001b[31m11.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Downloading jsonschema-4.17.3-py3-none-any.whl (90 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m90.4/90.4 kB\u001b[0m \u001b[31m12.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 (from jsonschema>=2.6->nbformat->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx)\n",
            "  Downloading pyrsistent-0.20.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (117 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m117.7/117.7 kB\u001b[0m \u001b[31m14.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: anyio<4,>=3.1.0 in /usr/local/lib/python3.10/dist-packages (from jupyter-server>=1.8->nbclassic>=0.4.7->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (3.7.1)\n",
            "Requirement already satisfied: cffi>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from argon2-cffi-bindings->argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (1.16.0)\n",
            "Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.10/dist-packages (from beautifulsoup4->nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (2.5)\n",
            "Requirement already satisfied: webencodings in /usr/local/lib/python3.10/dist-packages (from bleach->nbconvert>=5->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (0.5.1)\n",
            "Requirement already satisfied: sniffio>=1.1 in /usr/local/lib/python3.10/dist-packages (from anyio<4,>=3.1.0->jupyter-server>=1.8->nbclassic>=0.4.7->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (1.3.0)\n",
            "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<4,>=3.1.0->jupyter-server>=1.8->nbclassic>=0.4.7->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (1.2.0)\n",
            "Requirement already satisfied: pycparser in /usr/local/lib/python3.10/dist-packages (from cffi>=1.0.1->argon2-cffi-bindings->argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets<8,>=7->tensorflow-model-analysis<0.46.0,>=0.45.0->tfx) (2.21)\n",
            "Building wheels for collected packages: google-apitools, crcmod, dill, hdfs, pyfarmhash, docopt\n",
            "  Building wheel for google-apitools (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for google-apitools: filename=google_apitools-0.5.31-py3-none-any.whl size=131015 sha256=71e585d8989f37f8fa4a8ce92cfbd767b3d62f66ae0035b059feee051014162a\n",
            "  Stored in directory: /root/.cache/pip/wheels/04/b7/e0/9712f8c23a5da3d9d16fb88216b897bf60e85b12f5470f26ee\n",
            "  Building wheel for crcmod (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for crcmod: filename=crcmod-1.7-cp310-cp310-linux_x86_64.whl size=31406 sha256=316926f8f4831fe93e284c03446bcf97b5f0ed3bdf43dddb49a877f306cc4425\n",
            "  Stored in directory: /root/.cache/pip/wheels/85/4c/07/72215c529bd59d67e3dac29711d7aba1b692f543c808ba9e86\n",
            "  Building wheel for dill (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for dill: filename=dill-0.3.1.1-py3-none-any.whl size=78540 sha256=410b5be3350f9fae41b85e68a0cd7910279d57809833114b8655ee7828eb81e8\n",
            "  Stored in directory: /root/.cache/pip/wheels/ea/e2/86/64980d90e297e7bf2ce588c2b96e818f5399c515c4bb8a7e4f\n",
            "  Building wheel for hdfs (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for hdfs: filename=hdfs-2.7.3-py3-none-any.whl size=34325 sha256=4aa84551c01fcc51ac8d956cd0be2f3a79f57b945189fe6d1b3f3488fbcc26bd\n",
            "  Stored in directory: /root/.cache/pip/wheels/e5/8d/b6/99c1c0a3ac5788c866b0ecd3f48b0134a5910e6ed26011800b\n",
            "  Building wheel for pyfarmhash (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pyfarmhash: filename=pyfarmhash-0.3.2-cp310-cp310-linux_x86_64.whl size=88659 sha256=f78e646445f8752963d4da2fd7e60318360205e6dd0a611833ca2745d34e6818\n",
            "  Stored in directory: /root/.cache/pip/wheels/e0/08/da/f66b1f3258fe3f1e767b2136c5444dbfa9fa3f7944cc5e1983\n",
            "  Building wheel for docopt (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for docopt: filename=docopt-0.6.2-py2.py3-none-any.whl size=13706 sha256=51c326a25ae2ba2cc814ae128043fd3eb534cc17940e0a53feffed451647ef59\n",
            "  Stored in directory: /root/.cache/pip/wheels/fc/ab/d4/5da2067ac95b36618c629a5f93f809425700506f72c9732fac\n",
            "Successfully built google-apitools crcmod dill hdfs pyfarmhash docopt\n",
            "Installing collected packages: pyfarmhash, kt-legacy, docopt, crcmod, zstandard, uritemplate, tensorflow-hub, tensorflow-estimator, pyrsistent, packaging, overrides, orjson, objsize, keras, jedi, gast, fasteners, fastavro, dnspython, dill, attrs, pymongo, ml-metadata, keras-tuner, jsonschema, hdfs, docker, kubernetes, google-auth-oauthlib, google-apitools, apache-beam, tensorboard, google-api-python-client, tensorflow, ml-pipelines-sdk, google-cloud-vision, google-cloud-videointelligence, google-cloud-spanner, google-cloud-recommendations-ai, google-cloud-pubsub, google-cloud-dlp, google-cloud-bigtable, google-cloud-bigquery, tensorflow-serving-api, google-cloud-pubsublite, tfx-bsl, tensorflow-transform, tensorflow-data-validation, tensorflow-model-analysis, tfx\n",
            "  Attempting uninstall: uritemplate\n",
            "    Found existing installation: uritemplate 4.1.1\n",
            "    Uninstalling uritemplate-4.1.1:\n",
            "      Successfully uninstalled uritemplate-4.1.1\n",
            "  Attempting uninstall: tensorflow-hub\n",
            "    Found existing installation: tensorflow-hub 0.15.0\n",
            "    Uninstalling tensorflow-hub-0.15.0:\n",
            "      Successfully uninstalled tensorflow-hub-0.15.0\n",
            "  Attempting uninstall: tensorflow-estimator\n",
            "    Found existing installation: tensorflow-estimator 2.15.0\n",
            "    Uninstalling tensorflow-estimator-2.15.0:\n",
            "      Successfully uninstalled tensorflow-estimator-2.15.0\n",
            "  Attempting uninstall: packaging\n",
            "    Found existing installation: packaging 23.2\n",
            "    Uninstalling packaging-23.2:\n",
            "      Successfully uninstalled packaging-23.2\n",
            "  Attempting uninstall: keras\n",
            "    Found existing installation: keras 2.15.0\n",
            "    Uninstalling keras-2.15.0:\n",
            "      Successfully uninstalled keras-2.15.0\n",
            "  Attempting uninstall: gast\n",
            "    Found existing installation: gast 0.5.4\n",
            "    Uninstalling gast-0.5.4:\n",
            "      Successfully uninstalled gast-0.5.4\n",
            "  Attempting uninstall: attrs\n",
            "    Found existing installation: attrs 23.2.0\n",
            "    Uninstalling attrs-23.2.0:\n",
            "      Successfully uninstalled attrs-23.2.0\n",
            "  Attempting uninstall: jsonschema\n",
            "    Found existing installation: jsonschema 4.19.2\n",
            "    Uninstalling jsonschema-4.19.2:\n",
            "      Successfully uninstalled jsonschema-4.19.2\n",
            "  Attempting uninstall: google-auth-oauthlib\n",
            "    Found existing installation: google-auth-oauthlib 1.2.0\n",
            "    Uninstalling google-auth-oauthlib-1.2.0:\n",
            "      Successfully uninstalled google-auth-oauthlib-1.2.0\n",
            "  Attempting uninstall: tensorboard\n",
            "    Found existing installation: tensorboard 2.15.1\n",
            "    Uninstalling tensorboard-2.15.1:\n",
            "      Successfully uninstalled tensorboard-2.15.1\n",
            "  Attempting uninstall: google-api-python-client\n",
            "    Found existing installation: google-api-python-client 2.84.0\n",
            "    Uninstalling google-api-python-client-2.84.0:\n",
            "      Successfully uninstalled google-api-python-client-2.84.0\n",
            "  Attempting uninstall: tensorflow\n",
            "    Found existing installation: tensorflow 2.15.0\n",
            "    Uninstalling tensorflow-2.15.0:\n",
            "      Successfully uninstalled tensorflow-2.15.0\n",
            "  Attempting uninstall: google-cloud-bigquery\n",
            "    Found existing installation: google-cloud-bigquery 3.12.0\n",
            "    Uninstalling google-cloud-bigquery-3.12.0:\n",
            "      Successfully uninstalled google-cloud-bigquery-3.12.0\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "lida 0.0.10 requires fastapi, which is not installed.\n",
            "lida 0.0.10 requires kaleido, which is not installed.\n",
            "lida 0.0.10 requires python-multipart, which is not installed.\n",
            "lida 0.0.10 requires uvicorn, which is not installed.\n",
            "bigframes 0.18.0 requires google-cloud-bigquery[bqstorage,pandas]>=3.10.0, but you have google-cloud-bigquery 2.34.4 which is incompatible.\n",
            "pandas-gbq 0.19.2 requires google-cloud-bigquery!=2.4.*,<4.0.0dev,>=3.3.5, but you have google-cloud-bigquery 2.34.4 which is incompatible.\n",
            "referencing 0.32.0 requires attrs>=22.2.0, but you have attrs 21.4.0 which is incompatible.\n",
            "statsmodels 0.14.1 requires packaging>=21.3, but you have packaging 20.9 which is incompatible.\n",
            "xarray 2023.7.0 requires packaging>=21.3, but you have packaging 20.9 which is incompatible.\u001b[0m\u001b[31m\n",
            "\u001b[0mSuccessfully installed apache-beam-2.50.0 attrs-21.4.0 crcmod-1.7 dill-0.3.1.1 dnspython-2.4.2 docker-4.4.4 docopt-0.6.2 fastavro-1.9.3 fasteners-0.19 gast-0.4.0 google-api-python-client-1.12.11 google-apitools-0.5.31 google-auth-oauthlib-1.0.0 google-cloud-bigquery-2.34.4 google-cloud-bigtable-2.22.0 google-cloud-dlp-3.14.0 google-cloud-pubsub-2.19.0 google-cloud-pubsublite-1.9.0 google-cloud-recommendations-ai-0.10.6 google-cloud-spanner-3.40.1 google-cloud-videointelligence-2.12.0 google-cloud-vision-3.5.0 hdfs-2.7.3 jedi-0.19.1 jsonschema-4.17.3 keras-2.13.1 keras-tuner-1.4.6 kt-legacy-1.0.5 kubernetes-12.0.1 ml-metadata-1.14.0 ml-pipelines-sdk-1.14.0 objsize-0.6.1 orjson-3.9.10 overrides-7.4.0 packaging-20.9 pyfarmhash-0.3.2 pymongo-4.6.1 pyrsistent-0.20.0 tensorboard-2.13.0 tensorflow-2.13.1 tensorflow-data-validation-1.14.0 tensorflow-estimator-2.13.0 tensorflow-hub-0.13.0 tensorflow-model-analysis-0.45.0 tensorflow-serving-api-2.13.1 tensorflow-transform-1.14.0 tfx-1.14.0 tfx-bsl-1.14.0 uritemplate-3.0.1 zstandard-0.22.0\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.colab-display-data+json": {
              "pip_warning": {
                "packages": [
                  "google",
                  "packaging"
                ]
              }
            }
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install -U google-cloud-aiplatform \"shapely<2\""
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 985
        },
        "id": "q81D7HPBf0xx",
        "outputId": "ab4bf5e2-0f5a-4ede-b3f5-995f5dfacd32"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: google-cloud-aiplatform in /usr/local/lib/python3.10/dist-packages (1.38.1)\n",
            "Collecting google-cloud-aiplatform\n",
            "  Downloading google_cloud_aiplatform-1.39.0-py2.py3-none-any.whl (3.4 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.4/3.4 MB\u001b[0m \u001b[31m23.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting shapely<2\n",
            "  Downloading Shapely-1.8.5.post1-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (2.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.0/2.0 MB\u001b[0m \u001b[31m75.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0 in /usr/local/lib/python3.10/dist-packages (from google-cloud-aiplatform) (2.11.1)\n",
            "Requirement already satisfied: proto-plus<2.0.0dev,>=1.22.0 in /usr/local/lib/python3.10/dist-packages (from google-cloud-aiplatform) (1.23.0)\n",
            "Requirement already satisfied: protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 in /usr/local/lib/python3.10/dist-packages (from google-cloud-aiplatform) (3.20.3)\n",
            "Requirement already satisfied: packaging>=14.3 in /usr/local/lib/python3.10/dist-packages (from google-cloud-aiplatform) (20.9)\n",
            "Requirement already satisfied: google-cloud-storage<3.0.0dev,>=1.32.0 in /usr/local/lib/python3.10/dist-packages (from google-cloud-aiplatform) (2.8.0)\n",
            "Requirement already satisfied: google-cloud-bigquery<4.0.0dev,>=1.15.0 in /usr/local/lib/python3.10/dist-packages (from google-cloud-aiplatform) (2.34.4)\n",
            "Requirement already satisfied: google-cloud-resource-manager<3.0.0dev,>=1.3.3 in /usr/local/lib/python3.10/dist-packages (from google-cloud-aiplatform) (1.11.0)\n",
            "Requirement already satisfied: googleapis-common-protos<2.0.dev0,>=1.56.2 in /usr/local/lib/python3.10/dist-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (1.62.0)\n",
            "Requirement already satisfied: google-auth<3.0.dev0,>=2.14.1 in /usr/local/lib/python3.10/dist-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (2.17.3)\n",
            "Requirement already satisfied: requests<3.0.0.dev0,>=2.18.0 in /usr/local/lib/python3.10/dist-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (2.31.0)\n",
            "Requirement already satisfied: grpcio<2.0dev,>=1.33.2 in /usr/local/lib/python3.10/dist-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (1.60.0)\n",
            "Requirement already satisfied: grpcio-status<2.0.dev0,>=1.33.2 in /usr/local/lib/python3.10/dist-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (1.48.2)\n",
            "Requirement already satisfied: google-cloud-core<3.0.0dev,>=1.4.1 in /usr/local/lib/python3.10/dist-packages (from google-cloud-bigquery<4.0.0dev,>=1.15.0->google-cloud-aiplatform) (2.3.3)\n",
            "Requirement already satisfied: google-resumable-media<3.0dev,>=0.6.0 in /usr/local/lib/python3.10/dist-packages (from google-cloud-bigquery<4.0.0dev,>=1.15.0->google-cloud-aiplatform) (2.7.0)\n",
            "Requirement already satisfied: python-dateutil<3.0dev,>=2.7.2 in /usr/local/lib/python3.10/dist-packages (from google-cloud-bigquery<4.0.0dev,>=1.15.0->google-cloud-aiplatform) (2.8.2)\n",
            "Requirement already satisfied: grpc-google-iam-v1<1.0.0dev,>=0.12.4 in /usr/local/lib/python3.10/dist-packages (from google-cloud-resource-manager<3.0.0dev,>=1.3.3->google-cloud-aiplatform) (0.13.0)\n",
            "Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.10/dist-packages (from packaging>=14.3->google-cloud-aiplatform) (3.1.1)\n",
            "Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from google-auth<3.0.dev0,>=2.14.1->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (5.3.2)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from google-auth<3.0.dev0,>=2.14.1->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (0.3.0)\n",
            "Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.10/dist-packages (from google-auth<3.0.dev0,>=2.14.1->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (1.16.0)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.10/dist-packages (from google-auth<3.0.dev0,>=2.14.1->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (4.9)\n",
            "Requirement already satisfied: google-crc32c<2.0dev,>=1.0 in /usr/local/lib/python3.10/dist-packages (from google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<4.0.0dev,>=1.15.0->google-cloud-aiplatform) (1.5.0)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0.dev0,>=2.18.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0.dev0,>=2.18.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (3.6)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0.dev0,>=2.18.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (2.0.7)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0.dev0,>=2.18.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (2023.11.17)\n",
            "Requirement already satisfied: pyasn1<0.6.0,>=0.4.6 in /usr/local/lib/python3.10/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3.0.dev0,>=2.14.1->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (0.5.1)\n",
            "Installing collected packages: shapely, google-cloud-aiplatform\n",
            "  Attempting uninstall: shapely\n",
            "    Found existing installation: shapely 2.0.2\n",
            "    Uninstalling shapely-2.0.2:\n",
            "      Successfully uninstalled shapely-2.0.2\n",
            "  Attempting uninstall: google-cloud-aiplatform\n",
            "    Found existing installation: google-cloud-aiplatform 1.38.1\n",
            "    Uninstalling google-cloud-aiplatform-1.38.1:\n",
            "      Successfully uninstalled google-cloud-aiplatform-1.38.1\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "lida 0.0.10 requires fastapi, which is not installed.\n",
            "lida 0.0.10 requires kaleido, which is not installed.\n",
            "lida 0.0.10 requires python-multipart, which is not installed.\n",
            "lida 0.0.10 requires uvicorn, which is not installed.\u001b[0m\u001b[31m\n",
            "\u001b[0mSuccessfully installed google-cloud-aiplatform-1.39.0 shapely-1.8.5.post1\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.colab-display-data+json": {
              "pip_warning": {
                "packages": [
                  "google",
                  "shapely"
                ]
              }
            }
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EwT0nov5QO1M"
      },
      "source": [
        "**Note:** *Please do not proceed to the next steps without restarting the Runtime after installing `tfx`. You can do that by either pressing the `Restart Runtime` button at the end of the cell output above, or going to the `Runtime` button at the Colab toolbar above and selecting `Restart Runtime`.*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BDnPgN8UJtzN"
      },
      "source": [
        "Check the TensorFlow and TFX versions."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6jh7vKSRqPHb",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4ecd2c96-249c-40ad-fe1b-edd3f313ce72"
      },
      "source": [
        "import tensorflow as tf\n",
        "print('TensorFlow version: {}'.format(tf.__version__))\n",
        "from tfx import v1 as tfx\n",
        "print('TFX version: {}'.format(tfx.__version__))"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "TensorFlow version: 2.13.1\n",
            "TFX version: 1.14.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aDtLdSkvqPHe"
      },
      "source": [
        "### Set up variables\n",
        "\n",
        "There are some variables used to define a pipeline. You can customize these\n",
        "variables as you want. By default all output from the pipeline will be\n",
        "generated under the current directory."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EcUseqJaE2XN"
      },
      "source": [
        "import os\n",
        "from absl import logging\n",
        "\n",
        "# Pipeline label\n",
        "PIPELINE_NAME = \"penguin-simple\"\n",
        "\n",
        "# Output directory to store artifacts generated from the pipeline.\n",
        "PIPELINE_ROOT = os.path.join('pipelines', PIPELINE_NAME)\n",
        "\n",
        "# Path to a SQLite DB file to use as an MLMD storage.\n",
        "METADATA_PATH = os.path.join('metadata', PIPELINE_NAME, 'metadata.db')\n",
        "\n",
        "# Output directory where created models from the pipeline will be exported.\n",
        "SERVING_MODEL_DIR = os.path.join('serving_model', PIPELINE_NAME)\n",
        "\n",
        "# Set default logging level.\n",
        "logging.set_verbosity(logging.INFO)"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8F2SRwRLSYGa"
      },
      "source": [
        "### Prepare example data\n",
        "\n",
        "The dataset you will use is the\n",
        "[Palmer Penguins dataset](https://allisonhorst.github.io/palmerpenguins/articles/intro.html).\n",
        "\n",
        "There are four numeric features in this dataset:\n",
        "\n",
        "- culmen_length_mm\n",
        "- culmen_depth_mm\n",
        "- flipper_length_mm\n",
        "- body_mass_g\n",
        "\n",
        "All features were already normalized to have range [0,1]. You will build a\n",
        "classification model which predicts the `species` of penguins. The code below creates a\n",
        "directory and copies the dataset to it. After running it, you should see the dataset in the Colab file explorer under the `data` folder."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4fxMs6u86acP",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6348b144-fd6c-41a3-8a66-a057d81ed103"
      },
      "source": [
        "import urllib.request\n",
        "\n",
        "# Create directory\n",
        "DATA_ROOT = 'data'\n",
        "!mkdir {DATA_ROOT}\n",
        "\n",
        "# Copy dataset to directory\n",
        "_data_url = 'https://raw.githubusercontent.com/tensorflow/tfx/master/tfx/examples/penguin/data/labelled/penguins_processed.csv'\n",
        "_data_filepath = os.path.join(DATA_ROOT, \"data.csv\")\n",
        "urllib.request.urlretrieve(_data_url, _data_filepath)"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "('data/data.csv', <http.client.HTTPMessage at 0x7a5e053fcaf0>)"
            ]
          },
          "metadata": {},
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XlcD-gMhqlUv"
      },
      "source": [
        "## Running the pipeline using standard components\n",
        "\n",
        "The pipeline will consist of three essential TFX components and the graph will look like this:\n",
        "\n",
        "```\n",
        "ExampleGen -> Trainer -> Pusher\n",
        "```\n",
        "\n",
        "The pipeline includes the most minimal ML workflow which is\n",
        "importing data (ExampleGen), training a model (Trainer) and exporting the trained model (Pusher)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SD3H-oc9rn1K"
      },
      "source": [
        "### Model training code\n",
        "\n",
        "You will first define the trainer module so the `Trainer` component can build the model and train it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EyinPEAKrm9z"
      },
      "source": [
        "_trainer_module_file = 'penguin_trainer.py'"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jbQOc2VIqe-E",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ad2de4ae-357e-4d24-d097-4271cc509200"
      },
      "source": [
        "%%writefile {_trainer_module_file}\n",
        "\n",
        "from typing import List\n",
        "from absl import logging\n",
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "from tensorflow_transform.tf_metadata import schema_utils\n",
        "\n",
        "from tfx import v1 as tfx\n",
        "from tfx_bsl.public import tfxio\n",
        "from tensorflow_metadata.proto.v0 import schema_pb2\n",
        "\n",
        "_FEATURE_KEYS = [\n",
        "    'culmen_length_mm', 'culmen_depth_mm', 'flipper_length_mm', 'body_mass_g'\n",
        "]\n",
        "_LABEL_KEY = 'species'\n",
        "\n",
        "_TRAIN_BATCH_SIZE = 20\n",
        "_EVAL_BATCH_SIZE = 10\n",
        "\n",
        "# Since we're not generating or creating a schema, we will instead create\n",
        "# a feature spec.  Since there are a fairly small number of features this is\n",
        "# manageable for this dataset.\n",
        "_FEATURE_SPEC = {\n",
        "    **{\n",
        "        feature: tf.io.FixedLenFeature(shape=[1], dtype=tf.float32)\n",
        "           for feature in _FEATURE_KEYS\n",
        "       },\n",
        "    _LABEL_KEY: tf.io.FixedLenFeature(shape=[1], dtype=tf.int64)\n",
        "}\n",
        "\n",
        "\n",
        "def _input_fn(file_pattern: List[str],\n",
        "              data_accessor: tfx.components.DataAccessor,\n",
        "              schema: schema_pb2.Schema,\n",
        "              batch_size: int = 200) -> tf.data.Dataset:\n",
        "  \"\"\"Generates features and label for training.\n",
        "\n",
        "  Args:\n",
        "    file_pattern: List of paths or patterns of input tfrecord files.\n",
        "    data_accessor: DataAccessor for converting input to RecordBatch.\n",
        "    schema: schema of the input data.\n",
        "    batch_size: representing the number of consecutive elements of returned\n",
        "      dataset to combine in a single batch\n",
        "\n",
        "  Returns:\n",
        "    A dataset that contains (features, indices) tuple where features is a\n",
        "      dictionary of Tensors, and indices is a single Tensor of label indices.\n",
        "  \"\"\"\n",
        "  return data_accessor.tf_dataset_factory(\n",
        "      file_pattern,\n",
        "      tfxio.TensorFlowDatasetOptions(\n",
        "          batch_size=batch_size, label_key=_LABEL_KEY),\n",
        "      schema=schema).repeat()\n",
        "\n",
        "\n",
        "def _build_keras_model() -> tf.keras.Model:\n",
        "  \"\"\"Creates a DNN Keras model for classifying penguin data.\n",
        "\n",
        "  Returns:\n",
        "    A Keras Model.\n",
        "  \"\"\"\n",
        "  # The model below is built with Functional API, please refer to\n",
        "  # https://www.tensorflow.org/guide/keras/overview for all API options.\n",
        "  inputs = [keras.layers.Input(shape=(1,), name=f) for f in _FEATURE_KEYS]\n",
        "  d = keras.layers.concatenate(inputs)\n",
        "  for _ in range(2):\n",
        "    d = keras.layers.Dense(8, activation='relu')(d)\n",
        "  outputs = keras.layers.Dense(3)(d)\n",
        "\n",
        "  model = keras.Model(inputs=inputs, outputs=outputs)\n",
        "  model.compile(\n",
        "      optimizer=keras.optimizers.Adam(1e-2),\n",
        "      loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
        "      metrics=[keras.metrics.SparseCategoricalAccuracy()])\n",
        "\n",
        "  model.summary(print_fn=logging.info)\n",
        "  return model\n",
        "\n",
        "\n",
        "# TFX Trainer will call this function.\n",
        "def run_fn(fn_args: tfx.components.FnArgs):\n",
        "  \"\"\"Train the model based on given args.\n",
        "\n",
        "  Args:\n",
        "    fn_args: Holds args used to train the model as name/value pairs.\n",
        "  \"\"\"\n",
        "\n",
        "  # This schema is usually either an output of SchemaGen or a manually-curated\n",
        "  # version provided by pipeline author. A schema can also derived from TFT\n",
        "  # graph if a Transform component is used. In the case when either is missing,\n",
        "  # `schema_from_feature_spec` could be used to generate schema from very simple\n",
        "  # feature_spec, but the schema returned would be very primitive.\n",
        "  schema = schema_utils.schema_from_feature_spec(_FEATURE_SPEC)\n",
        "\n",
        "  train_dataset = _input_fn(\n",
        "      fn_args.train_files,\n",
        "      fn_args.data_accessor,\n",
        "      schema,\n",
        "      batch_size=_TRAIN_BATCH_SIZE)\n",
        "  eval_dataset = _input_fn(\n",
        "      fn_args.eval_files,\n",
        "      fn_args.data_accessor,\n",
        "      schema,\n",
        "      batch_size=_EVAL_BATCH_SIZE)\n",
        "\n",
        "  model = _build_keras_model()\n",
        "  model.fit(\n",
        "      train_dataset,\n",
        "      steps_per_epoch=fn_args.train_steps,\n",
        "      validation_data=eval_dataset,\n",
        "      validation_steps=fn_args.eval_steps)\n",
        "\n",
        "  # The result of the training should be saved in `fn_args.serving_model_dir`\n",
        "  # directory.\n",
        "  model.save(fn_args.serving_model_dir, save_format='tf')"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Writing penguin_trainer.py\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1FiAttwdsXRn"
      },
      "source": [
        "### Write a pipeline definition\n",
        "\n",
        "Next, you will define a function to create a TFX pipeline. A [`Pipeline`](https://www.tensorflow.org/tfx/api_docs/python/tfx/v1/dsl/Pipeline) object represents a TFX pipeline which can be run using one of pipeline orchestration systems that TFX supports."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "78NRGIGNsNPD"
      },
      "source": [
        "def _create_pipeline(pipeline_name: str, pipeline_root: str, data_root: str,\n",
        "                     module_file: str, serving_model_dir: str,\n",
        "                     metadata_path: str) -> tfx.dsl.Pipeline:\n",
        "  \"\"\"Creates a three component penguin pipeline with TFX.\"\"\"\n",
        "  # Brings data into the pipeline.\n",
        "  example_gen = tfx.components.CsvExampleGen(input_base=data_root)\n",
        "\n",
        "  # Uses user-provided Python function that trains a model.\n",
        "  trainer = tfx.components.Trainer(\n",
        "      module_file=module_file,\n",
        "      examples=example_gen.outputs['examples'],\n",
        "      train_args=tfx.proto.TrainArgs(num_steps=100),\n",
        "      eval_args=tfx.proto.EvalArgs(num_steps=5))\n",
        "\n",
        "  # Pushes the model to a filesystem destination.\n",
        "  pusher = tfx.components.Pusher(\n",
        "      model=trainer.outputs['model'],\n",
        "      push_destination=tfx.proto.PushDestination(\n",
        "          filesystem=tfx.proto.PushDestination.Filesystem(\n",
        "              base_directory=serving_model_dir)))\n",
        "\n",
        "  # Following three components will be included in the pipeline.\n",
        "  components = [\n",
        "      example_gen,\n",
        "      trainer,\n",
        "      pusher,\n",
        "  ]\n",
        "\n",
        "  return tfx.dsl.Pipeline(\n",
        "      pipeline_name=pipeline_name,\n",
        "      pipeline_root=pipeline_root,\n",
        "      metadata_connection_config=tfx.orchestration.metadata\n",
        "      .sqlite_metadata_connection_config(metadata_path),\n",
        "      components=components)"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k3A4E8YOsyv4"
      },
      "source": [
        "## Run the pipeline\n",
        "\n",
        "TFX supports multiple orchestrators to run pipelines.\n",
        "In this tutorial we will use `LocalDagRunner` which is included in the TFX\n",
        "Python package and runs pipelines on local environment.\n",
        "We often call TFX pipelines \"DAGs\" which stands for directed acyclic graph.\n",
        "\n",
        "`LocalDagRunner` provides fast iterations for developemnt and debugging.\n",
        "TFX also supports other orchestrators including Kubeflow Pipelines and Apache\n",
        "Airflow which are suitable for production use cases.\n",
        "\n",
        "See\n",
        "[TFX on Cloud AI Platform Pipelines](https://www.tensorflow.org/tfx/tutorials/tfx/cloud-ai-platform-pipelines)\n",
        "or\n",
        "[TFX Airflow Tutorial](https://www.tensorflow.org/tfx/tutorials/tfx/airflow_workshop)\n",
        "to learn more about other orchestration systems.\n",
        "\n",
        "The code below creates a `LocalDagRunner` and passes a `Pipeline` object created from the\n",
        "function you defined earlier. The pipeline runs directly and you can see logs for the progress of the pipeline including ML model training."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S-TCt1Res8rH",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "c15417b8-054d-46ed-f993-9c8aff3ac459"
      },
      "source": [
        "tfx.orchestration.LocalDagRunner().run(\n",
        "  _create_pipeline(\n",
        "      pipeline_name=PIPELINE_NAME,\n",
        "      pipeline_root=PIPELINE_ROOT,\n",
        "      data_root=DATA_ROOT,\n",
        "      module_file=_trainer_module_file,\n",
        "      serving_model_dir=SERVING_MODEL_DIR,\n",
        "      metadata_path=METADATA_PATH))"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:absl:Generating ephemeral wheel package for '/content/penguin_trainer.py' (including modules: ['penguin_trainer']).\n",
            "INFO:absl:User module package has hash fingerprint version a7e2e8dccbb913b74904edeec5549d868a2ea392bcd84fbc1965aba698dce3fc.\n",
            "INFO:absl:Executing: ['/usr/bin/python3', '/tmp/tmpsxxr0wh1/_tfx_generated_setup.py', 'bdist_wheel', '--bdist-dir', '/tmp/tmpn8uzrx8q', '--dist-dir', '/tmp/tmp7abxrh5z']\n",
            "INFO:absl:Successfully built user code wheel distribution at 'pipelines/penguin-simple/_wheels/tfx_user_code_Trainer-0.0+a7e2e8dccbb913b74904edeec5549d868a2ea392bcd84fbc1965aba698dce3fc-py3-none-any.whl'; target user module is 'penguin_trainer'.\n",
            "INFO:absl:Full user module path is 'penguin_trainer@pipelines/penguin-simple/_wheels/tfx_user_code_Trainer-0.0+a7e2e8dccbb913b74904edeec5549d868a2ea392bcd84fbc1965aba698dce3fc-py3-none-any.whl'\n",
            "INFO:absl:Using deployment config:\n",
            " executor_specs {\n",
            "  key: \"CsvExampleGen\"\n",
            "  value {\n",
            "    beam_executable_spec {\n",
            "      python_executor_spec {\n",
            "        class_path: \"tfx.components.example_gen.csv_example_gen.executor.Executor\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "executor_specs {\n",
            "  key: \"Pusher\"\n",
            "  value {\n",
            "    python_class_executable_spec {\n",
            "      class_path: \"tfx.components.pusher.executor.Executor\"\n",
            "    }\n",
            "  }\n",
            "}\n",
            "executor_specs {\n",
            "  key: \"Trainer\"\n",
            "  value {\n",
            "    python_class_executable_spec {\n",
            "      class_path: \"tfx.components.trainer.executor.GenericExecutor\"\n",
            "    }\n",
            "  }\n",
            "}\n",
            "custom_driver_specs {\n",
            "  key: \"CsvExampleGen\"\n",
            "  value {\n",
            "    python_class_executable_spec {\n",
            "      class_path: \"tfx.components.example_gen.driver.FileBasedDriver\"\n",
            "    }\n",
            "  }\n",
            "}\n",
            "metadata_connection_config {\n",
            "  database_connection_config {\n",
            "    sqlite {\n",
            "      filename_uri: \"metadata/penguin-simple/metadata.db\"\n",
            "      connection_mode: READWRITE_OPENCREATE\n",
            "    }\n",
            "  }\n",
            "}\n",
            "\n",
            "INFO:absl:Using connection config:\n",
            " sqlite {\n",
            "  filename_uri: \"metadata/penguin-simple/metadata.db\"\n",
            "  connection_mode: READWRITE_OPENCREATE\n",
            "}\n",
            "\n",
            "INFO:absl:Component CsvExampleGen is running.\n",
            "INFO:absl:Running launcher for node_info {\n",
            "  type {\n",
            "    name: \"tfx.components.example_gen.csv_example_gen.component.CsvExampleGen\"\n",
            "  }\n",
            "  id: \"CsvExampleGen\"\n",
            "}\n",
            "contexts {\n",
            "  contexts {\n",
            "    type {\n",
            "      name: \"pipeline\"\n",
            "    }\n",
            "    name {\n",
            "      field_value {\n",
            "        string_value: \"penguin-simple\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  contexts {\n",
            "    type {\n",
            "      name: \"pipeline_run\"\n",
            "    }\n",
            "    name {\n",
            "      field_value {\n",
            "        string_value: \"2024-01-10T08:33:19.404351\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  contexts {\n",
            "    type {\n",
            "      name: \"node\"\n",
            "    }\n",
            "    name {\n",
            "      field_value {\n",
            "        string_value: \"penguin-simple.CsvExampleGen\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "outputs {\n",
            "  outputs {\n",
            "    key: \"examples\"\n",
            "    value {\n",
            "      artifact_spec {\n",
            "        type {\n",
            "          name: \"Examples\"\n",
            "          properties {\n",
            "            key: \"span\"\n",
            "            value: INT\n",
            "          }\n",
            "          properties {\n",
            "            key: \"split_names\"\n",
            "            value: STRING\n",
            "          }\n",
            "          properties {\n",
            "            key: \"version\"\n",
            "            value: INT\n",
            "          }\n",
            "          base_type: DATASET\n",
            "        }\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "parameters {\n",
            "  parameters {\n",
            "    key: \"input_base\"\n",
            "    value {\n",
            "      field_value {\n",
            "        string_value: \"data\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  parameters {\n",
            "    key: \"input_config\"\n",
            "    value {\n",
            "      field_value {\n",
            "        string_value: \"{\\n  \\\"splits\\\": [\\n    {\\n      \\\"name\\\": \\\"single_split\\\",\\n      \\\"pattern\\\": \\\"*\\\"\\n    }\\n  ]\\n}\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  parameters {\n",
            "    key: \"output_config\"\n",
            "    value {\n",
            "      field_value {\n",
            "        string_value: \"{\\n  \\\"split_config\\\": {\\n    \\\"splits\\\": [\\n      {\\n        \\\"hash_buckets\\\": 2,\\n        \\\"name\\\": \\\"train\\\"\\n      },\\n      {\\n        \\\"hash_buckets\\\": 1,\\n        \\\"name\\\": \\\"eval\\\"\\n      }\\n    ]\\n  }\\n}\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  parameters {\n",
            "    key: \"output_data_format\"\n",
            "    value {\n",
            "      field_value {\n",
            "        int_value: 6\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  parameters {\n",
            "    key: \"output_file_format\"\n",
            "    value {\n",
            "      field_value {\n",
            "        int_value: 5\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "downstream_nodes: \"Trainer\"\n",
            "execution_options {\n",
            "  caching_options {\n",
            "  }\n",
            "}\n",
            "\n",
            "INFO:absl:MetadataStore with DB connection initialized\n",
            "INFO:absl:[CsvExampleGen] Resolved inputs: ({},)\n",
            "INFO:absl:select span and version = (0, None)\n",
            "INFO:absl:latest span and version = (0, None)\n",
            "INFO:absl:MetadataStore with DB connection initialized\n",
            "INFO:absl:Going to run a new execution 1\n",
            "INFO:absl:Going to run a new execution: ExecutionInfo(execution_id=1, input_dict={}, output_dict=defaultdict(<class 'list'>, {'examples': [Artifact(artifact: uri: \"pipelines/penguin-simple/CsvExampleGen/examples/1\"\n",
            "custom_properties {\n",
            "  key: \"input_fingerprint\"\n",
            "  value {\n",
            "    string_value: \"split:single_split,num_files:1,total_bytes:25648,xor_checksum:1704875555,sum_checksum:1704875555\"\n",
            "  }\n",
            "}\n",
            "custom_properties {\n",
            "  key: \"span\"\n",
            "  value {\n",
            "    int_value: 0\n",
            "  }\n",
            "}\n",
            ", artifact_type: name: \"Examples\"\n",
            "properties {\n",
            "  key: \"span\"\n",
            "  value: INT\n",
            "}\n",
            "properties {\n",
            "  key: \"split_names\"\n",
            "  value: STRING\n",
            "}\n",
            "properties {\n",
            "  key: \"version\"\n",
            "  value: INT\n",
            "}\n",
            "base_type: DATASET\n",
            ")]}), exec_properties={'input_config': '{\\n  \"splits\": [\\n    {\\n      \"name\": \"single_split\",\\n      \"pattern\": \"*\"\\n    }\\n  ]\\n}', 'output_data_format': 6, 'input_base': 'data', 'output_file_format': 5, 'output_config': '{\\n  \"split_config\": {\\n    \"splits\": [\\n      {\\n        \"hash_buckets\": 2,\\n        \"name\": \"train\"\\n      },\\n      {\\n        \"hash_buckets\": 1,\\n        \"name\": \"eval\"\\n      }\\n    ]\\n  }\\n}', 'span': 0, 'version': None, 'input_fingerprint': 'split:single_split,num_files:1,total_bytes:25648,xor_checksum:1704875555,sum_checksum:1704875555'}, execution_output_uri='pipelines/penguin-simple/CsvExampleGen/.system/executor_execution/1/executor_output.pb', stateful_working_dir='pipelines/penguin-simple/CsvExampleGen/.system/stateful_working_dir/2024-01-10T08:33:19.404351', tmp_dir='pipelines/penguin-simple/CsvExampleGen/.system/executor_execution/1/.temp/', pipeline_node=node_info {\n",
            "  type {\n",
            "    name: \"tfx.components.example_gen.csv_example_gen.component.CsvExampleGen\"\n",
            "  }\n",
            "  id: \"CsvExampleGen\"\n",
            "}\n",
            "contexts {\n",
            "  contexts {\n",
            "    type {\n",
            "      name: \"pipeline\"\n",
            "    }\n",
            "    name {\n",
            "      field_value {\n",
            "        string_value: \"penguin-simple\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  contexts {\n",
            "    type {\n",
            "      name: \"pipeline_run\"\n",
            "    }\n",
            "    name {\n",
            "      field_value {\n",
            "        string_value: \"2024-01-10T08:33:19.404351\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  contexts {\n",
            "    type {\n",
            "      name: \"node\"\n",
            "    }\n",
            "    name {\n",
            "      field_value {\n",
            "        string_value: \"penguin-simple.CsvExampleGen\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "outputs {\n",
            "  outputs {\n",
            "    key: \"examples\"\n",
            "    value {\n",
            "      artifact_spec {\n",
            "        type {\n",
            "          name: \"Examples\"\n",
            "          properties {\n",
            "            key: \"span\"\n",
            "            value: INT\n",
            "          }\n",
            "          properties {\n",
            "            key: \"split_names\"\n",
            "            value: STRING\n",
            "          }\n",
            "          properties {\n",
            "            key: \"version\"\n",
            "            value: INT\n",
            "          }\n",
            "          base_type: DATASET\n",
            "        }\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "parameters {\n",
            "  parameters {\n",
            "    key: \"input_base\"\n",
            "    value {\n",
            "      field_value {\n",
            "        string_value: \"data\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  parameters {\n",
            "    key: \"input_config\"\n",
            "    value {\n",
            "      field_value {\n",
            "        string_value: \"{\\n  \\\"splits\\\": [\\n    {\\n      \\\"name\\\": \\\"single_split\\\",\\n      \\\"pattern\\\": \\\"*\\\"\\n    }\\n  ]\\n}\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  parameters {\n",
            "    key: \"output_config\"\n",
            "    value {\n",
            "      field_value {\n",
            "        string_value: \"{\\n  \\\"split_config\\\": {\\n    \\\"splits\\\": [\\n      {\\n        \\\"hash_buckets\\\": 2,\\n        \\\"name\\\": \\\"train\\\"\\n      },\\n      {\\n        \\\"hash_buckets\\\": 1,\\n        \\\"name\\\": \\\"eval\\\"\\n      }\\n    ]\\n  }\\n}\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  parameters {\n",
            "    key: \"output_data_format\"\n",
            "    value {\n",
            "      field_value {\n",
            "        int_value: 6\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  parameters {\n",
            "    key: \"output_file_format\"\n",
            "    value {\n",
            "      field_value {\n",
            "        int_value: 5\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "downstream_nodes: \"Trainer\"\n",
            "execution_options {\n",
            "  caching_options {\n",
            "  }\n",
            "}\n",
            ", pipeline_info=id: \"penguin-simple\"\n",
            ", pipeline_run_id='2024-01-10T08:33:19.404351', top_level_pipeline_run_id=None)\n",
            "INFO:absl:Generating examples.\n",
            "WARNING:apache_beam.runners.interactive.interactive_environment:Dependencies required for Interactive Beam PCollection visualization are not available, please use: `pip install apache-beam[interactive]` to install necessary dependencies to enable all data visualization features.\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "\n",
              "        if (typeof window.interactive_beam_jquery == 'undefined') {\n",
              "          var jqueryScript = document.createElement('script');\n",
              "          jqueryScript.src = 'https://code.jquery.com/jquery-3.4.1.slim.min.js';\n",
              "          jqueryScript.type = 'text/javascript';\n",
              "          jqueryScript.onload = function() {\n",
              "            var datatableScript = document.createElement('script');\n",
              "            datatableScript.src = 'https://cdn.datatables.net/1.10.20/js/jquery.dataTables.min.js';\n",
              "            datatableScript.type = 'text/javascript';\n",
              "            datatableScript.onload = function() {\n",
              "              window.interactive_beam_jquery = jQuery.noConflict(true);\n",
              "              window.interactive_beam_jquery(document).ready(function($){\n",
              "                \n",
              "              });\n",
              "            }\n",
              "            document.head.appendChild(datatableScript);\n",
              "          };\n",
              "          document.head.appendChild(jqueryScript);\n",
              "        } else {\n",
              "          window.interactive_beam_jquery(document).ready(function($){\n",
              "            \n",
              "          });\n",
              "        }"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:absl:Processing input csv data data/* to TFExample.\n",
            "WARNING:apache_beam.io.tfrecordio:Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be.\n",
            "INFO:absl:Examples generated.\n",
            "INFO:absl:Value type <class 'NoneType'> of key version in exec_properties is not supported, going to drop it\n",
            "INFO:absl:Value type <class 'list'> of key _beam_pipeline_args in exec_properties is not supported, going to drop it\n",
            "INFO:absl:Cleaning up stateless execution info.\n",
            "INFO:absl:Execution 1 succeeded.\n",
            "INFO:absl:Cleaning up stateful execution info.\n",
            "INFO:absl:Publishing output artifacts defaultdict(<class 'list'>, {'examples': [Artifact(artifact: uri: \"pipelines/penguin-simple/CsvExampleGen/examples/1\"\n",
            "custom_properties {\n",
            "  key: \"input_fingerprint\"\n",
            "  value {\n",
            "    string_value: \"split:single_split,num_files:1,total_bytes:25648,xor_checksum:1704875555,sum_checksum:1704875555\"\n",
            "  }\n",
            "}\n",
            "custom_properties {\n",
            "  key: \"span\"\n",
            "  value {\n",
            "    int_value: 0\n",
            "  }\n",
            "}\n",
            ", artifact_type: name: \"Examples\"\n",
            "properties {\n",
            "  key: \"span\"\n",
            "  value: INT\n",
            "}\n",
            "properties {\n",
            "  key: \"split_names\"\n",
            "  value: STRING\n",
            "}\n",
            "properties {\n",
            "  key: \"version\"\n",
            "  value: INT\n",
            "}\n",
            "base_type: DATASET\n",
            ")]}) for execution 1\n",
            "INFO:absl:MetadataStore with DB connection initialized\n",
            "INFO:absl:Component CsvExampleGen is finished.\n",
            "INFO:absl:Component Trainer is running.\n",
            "INFO:absl:Running launcher for node_info {\n",
            "  type {\n",
            "    name: \"tfx.components.trainer.component.Trainer\"\n",
            "    base_type: TRAIN\n",
            "  }\n",
            "  id: \"Trainer\"\n",
            "}\n",
            "contexts {\n",
            "  contexts {\n",
            "    type {\n",
            "      name: \"pipeline\"\n",
            "    }\n",
            "    name {\n",
            "      field_value {\n",
            "        string_value: \"penguin-simple\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  contexts {\n",
            "    type {\n",
            "      name: \"pipeline_run\"\n",
            "    }\n",
            "    name {\n",
            "      field_value {\n",
            "        string_value: \"2024-01-10T08:33:19.404351\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  contexts {\n",
            "    type {\n",
            "      name: \"node\"\n",
            "    }\n",
            "    name {\n",
            "      field_value {\n",
            "        string_value: \"penguin-simple.Trainer\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "inputs {\n",
            "  inputs {\n",
            "    key: \"examples\"\n",
            "    value {\n",
            "      channels {\n",
            "        producer_node_query {\n",
            "          id: \"CsvExampleGen\"\n",
            "        }\n",
            "        context_queries {\n",
            "          type {\n",
            "            name: \"pipeline\"\n",
            "          }\n",
            "          name {\n",
            "            field_value {\n",
            "              string_value: \"penguin-simple\"\n",
            "            }\n",
            "          }\n",
            "        }\n",
            "        context_queries {\n",
            "          type {\n",
            "            name: \"pipeline_run\"\n",
            "          }\n",
            "          name {\n",
            "            field_value {\n",
            "              string_value: \"2024-01-10T08:33:19.404351\"\n",
            "            }\n",
            "          }\n",
            "        }\n",
            "        context_queries {\n",
            "          type {\n",
            "            name: \"node\"\n",
            "          }\n",
            "          name {\n",
            "            field_value {\n",
            "              string_value: \"penguin-simple.CsvExampleGen\"\n",
            "            }\n",
            "          }\n",
            "        }\n",
            "        artifact_query {\n",
            "          type {\n",
            "            name: \"Examples\"\n",
            "            base_type: DATASET\n",
            "          }\n",
            "        }\n",
            "        output_key: \"examples\"\n",
            "      }\n",
            "      min_count: 1\n",
            "    }\n",
            "  }\n",
            "}\n",
            "outputs {\n",
            "  outputs {\n",
            "    key: \"model\"\n",
            "    value {\n",
            "      artifact_spec {\n",
            "        type {\n",
            "          name: \"Model\"\n",
            "          base_type: MODEL\n",
            "        }\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  outputs {\n",
            "    key: \"model_run\"\n",
            "    value {\n",
            "      artifact_spec {\n",
            "        type {\n",
            "          name: \"ModelRun\"\n",
            "        }\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "parameters {\n",
            "  parameters {\n",
            "    key: \"custom_config\"\n",
            "    value {\n",
            "      field_value {\n",
            "        string_value: \"null\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  parameters {\n",
            "    key: \"eval_args\"\n",
            "    value {\n",
            "      field_value {\n",
            "        string_value: \"{\\n  \\\"num_steps\\\": 5\\n}\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  parameters {\n",
            "    key: \"module_path\"\n",
            "    value {\n",
            "      field_value {\n",
            "        string_value: \"penguin_trainer@pipelines/penguin-simple/_wheels/tfx_user_code_Trainer-0.0+a7e2e8dccbb913b74904edeec5549d868a2ea392bcd84fbc1965aba698dce3fc-py3-none-any.whl\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  parameters {\n",
            "    key: \"train_args\"\n",
            "    value {\n",
            "      field_value {\n",
            "        string_value: \"{\\n  \\\"num_steps\\\": 100\\n}\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "upstream_nodes: \"CsvExampleGen\"\n",
            "downstream_nodes: \"Pusher\"\n",
            "execution_options {\n",
            "  caching_options {\n",
            "  }\n",
            "}\n",
            "\n",
            "INFO:absl:MetadataStore with DB connection initialized\n",
            "WARNING:absl:ArtifactQuery.property_predicate is not supported.\n",
            "INFO:absl:[Trainer] Resolved inputs: ({'examples': [Artifact(artifact: id: 1\n",
            "type_id: 15\n",
            "uri: \"pipelines/penguin-simple/CsvExampleGen/examples/1\"\n",
            "properties {\n",
            "  key: \"split_names\"\n",
            "  value {\n",
            "    string_value: \"[\\\"train\\\", \\\"eval\\\"]\"\n",
            "  }\n",
            "}\n",
            "custom_properties {\n",
            "  key: \"file_format\"\n",
            "  value {\n",
            "    string_value: \"tfrecords_gzip\"\n",
            "  }\n",
            "}\n",
            "custom_properties {\n",
            "  key: \"input_fingerprint\"\n",
            "  value {\n",
            "    string_value: \"split:single_split,num_files:1,total_bytes:25648,xor_checksum:1704875555,sum_checksum:1704875555\"\n",
            "  }\n",
            "}\n",
            "custom_properties {\n",
            "  key: \"is_external\"\n",
            "  value {\n",
            "    int_value: 0\n",
            "  }\n",
            "}\n",
            "custom_properties {\n",
            "  key: \"payload_format\"\n",
            "  value {\n",
            "    string_value: \"FORMAT_TF_EXAMPLE\"\n",
            "  }\n",
            "}\n",
            "custom_properties {\n",
            "  key: \"span\"\n",
            "  value {\n",
            "    int_value: 0\n",
            "  }\n",
            "}\n",
            "custom_properties {\n",
            "  key: \"tfx_version\"\n",
            "  value {\n",
            "    string_value: \"1.14.0\"\n",
            "  }\n",
            "}\n",
            "state: LIVE\n",
            "type: \"Examples\"\n",
            "create_time_since_epoch: 1704875601028\n",
            "last_update_time_since_epoch: 1704875601028\n",
            ", artifact_type: id: 15\n",
            "name: \"Examples\"\n",
            "properties {\n",
            "  key: \"span\"\n",
            "  value: INT\n",
            "}\n",
            "properties {\n",
            "  key: \"split_names\"\n",
            "  value: STRING\n",
            "}\n",
            "properties {\n",
            "  key: \"version\"\n",
            "  value: INT\n",
            "}\n",
            "base_type: DATASET\n",
            ")]},)\n",
            "INFO:absl:MetadataStore with DB connection initialized\n",
            "INFO:absl:Going to run a new execution 2\n",
            "INFO:absl:Going to run a new execution: ExecutionInfo(execution_id=2, input_dict={'examples': [Artifact(artifact: id: 1\n",
            "type_id: 15\n",
            "uri: \"pipelines/penguin-simple/CsvExampleGen/examples/1\"\n",
            "properties {\n",
            "  key: \"split_names\"\n",
            "  value {\n",
            "    string_value: \"[\\\"train\\\", \\\"eval\\\"]\"\n",
            "  }\n",
            "}\n",
            "custom_properties {\n",
            "  key: \"file_format\"\n",
            "  value {\n",
            "    string_value: \"tfrecords_gzip\"\n",
            "  }\n",
            "}\n",
            "custom_properties {\n",
            "  key: \"input_fingerprint\"\n",
            "  value {\n",
            "    string_value: \"split:single_split,num_files:1,total_bytes:25648,xor_checksum:1704875555,sum_checksum:1704875555\"\n",
            "  }\n",
            "}\n",
            "custom_properties {\n",
            "  key: \"is_external\"\n",
            "  value {\n",
            "    int_value: 0\n",
            "  }\n",
            "}\n",
            "custom_properties {\n",
            "  key: \"payload_format\"\n",
            "  value {\n",
            "    string_value: \"FORMAT_TF_EXAMPLE\"\n",
            "  }\n",
            "}\n",
            "custom_properties {\n",
            "  key: \"span\"\n",
            "  value {\n",
            "    int_value: 0\n",
            "  }\n",
            "}\n",
            "custom_properties {\n",
            "  key: \"tfx_version\"\n",
            "  value {\n",
            "    string_value: \"1.14.0\"\n",
            "  }\n",
            "}\n",
            "state: LIVE\n",
            "type: \"Examples\"\n",
            "create_time_since_epoch: 1704875601028\n",
            "last_update_time_since_epoch: 1704875601028\n",
            ", artifact_type: id: 15\n",
            "name: \"Examples\"\n",
            "properties {\n",
            "  key: \"span\"\n",
            "  value: INT\n",
            "}\n",
            "properties {\n",
            "  key: \"split_names\"\n",
            "  value: STRING\n",
            "}\n",
            "properties {\n",
            "  key: \"version\"\n",
            "  value: INT\n",
            "}\n",
            "base_type: DATASET\n",
            ")]}, output_dict=defaultdict(<class 'list'>, {'model_run': [Artifact(artifact: uri: \"pipelines/penguin-simple/Trainer/model_run/2\"\n",
            ", artifact_type: name: \"ModelRun\"\n",
            ")], 'model': [Artifact(artifact: uri: \"pipelines/penguin-simple/Trainer/model/2\"\n",
            ", artifact_type: name: \"Model\"\n",
            "base_type: MODEL\n",
            ")]}), exec_properties={'train_args': '{\\n  \"num_steps\": 100\\n}', 'module_path': 'penguin_trainer@pipelines/penguin-simple/_wheels/tfx_user_code_Trainer-0.0+a7e2e8dccbb913b74904edeec5549d868a2ea392bcd84fbc1965aba698dce3fc-py3-none-any.whl', 'custom_config': 'null', 'eval_args': '{\\n  \"num_steps\": 5\\n}'}, execution_output_uri='pipelines/penguin-simple/Trainer/.system/executor_execution/2/executor_output.pb', stateful_working_dir='pipelines/penguin-simple/Trainer/.system/stateful_working_dir/2024-01-10T08:33:19.404351', tmp_dir='pipelines/penguin-simple/Trainer/.system/executor_execution/2/.temp/', pipeline_node=node_info {\n",
            "  type {\n",
            "    name: \"tfx.components.trainer.component.Trainer\"\n",
            "    base_type: TRAIN\n",
            "  }\n",
            "  id: \"Trainer\"\n",
            "}\n",
            "contexts {\n",
            "  contexts {\n",
            "    type {\n",
            "      name: \"pipeline\"\n",
            "    }\n",
            "    name {\n",
            "      field_value {\n",
            "        string_value: \"penguin-simple\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  contexts {\n",
            "    type {\n",
            "      name: \"pipeline_run\"\n",
            "    }\n",
            "    name {\n",
            "      field_value {\n",
            "        string_value: \"2024-01-10T08:33:19.404351\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  contexts {\n",
            "    type {\n",
            "      name: \"node\"\n",
            "    }\n",
            "    name {\n",
            "      field_value {\n",
            "        string_value: \"penguin-simple.Trainer\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "inputs {\n",
            "  inputs {\n",
            "    key: \"examples\"\n",
            "    value {\n",
            "      channels {\n",
            "        producer_node_query {\n",
            "          id: \"CsvExampleGen\"\n",
            "        }\n",
            "        context_queries {\n",
            "          type {\n",
            "            name: \"pipeline\"\n",
            "          }\n",
            "          name {\n",
            "            field_value {\n",
            "              string_value: \"penguin-simple\"\n",
            "            }\n",
            "          }\n",
            "        }\n",
            "        context_queries {\n",
            "          type {\n",
            "            name: \"pipeline_run\"\n",
            "          }\n",
            "          name {\n",
            "            field_value {\n",
            "              string_value: \"2024-01-10T08:33:19.404351\"\n",
            "            }\n",
            "          }\n",
            "        }\n",
            "        context_queries {\n",
            "          type {\n",
            "            name: \"node\"\n",
            "          }\n",
            "          name {\n",
            "            field_value {\n",
            "              string_value: \"penguin-simple.CsvExampleGen\"\n",
            "            }\n",
            "          }\n",
            "        }\n",
            "        artifact_query {\n",
            "          type {\n",
            "            name: \"Examples\"\n",
            "            base_type: DATASET\n",
            "          }\n",
            "        }\n",
            "        output_key: \"examples\"\n",
            "      }\n",
            "      min_count: 1\n",
            "    }\n",
            "  }\n",
            "}\n",
            "outputs {\n",
            "  outputs {\n",
            "    key: \"model\"\n",
            "    value {\n",
            "      artifact_spec {\n",
            "        type {\n",
            "          name: \"Model\"\n",
            "          base_type: MODEL\n",
            "        }\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  outputs {\n",
            "    key: \"model_run\"\n",
            "    value {\n",
            "      artifact_spec {\n",
            "        type {\n",
            "          name: \"ModelRun\"\n",
            "        }\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "parameters {\n",
            "  parameters {\n",
            "    key: \"custom_config\"\n",
            "    value {\n",
            "      field_value {\n",
            "        string_value: \"null\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  parameters {\n",
            "    key: \"eval_args\"\n",
            "    value {\n",
            "      field_value {\n",
            "        string_value: \"{\\n  \\\"num_steps\\\": 5\\n}\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  parameters {\n",
            "    key: \"module_path\"\n",
            "    value {\n",
            "      field_value {\n",
            "        string_value: \"penguin_trainer@pipelines/penguin-simple/_wheels/tfx_user_code_Trainer-0.0+a7e2e8dccbb913b74904edeec5549d868a2ea392bcd84fbc1965aba698dce3fc-py3-none-any.whl\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  parameters {\n",
            "    key: \"train_args\"\n",
            "    value {\n",
            "      field_value {\n",
            "        string_value: \"{\\n  \\\"num_steps\\\": 100\\n}\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "upstream_nodes: \"CsvExampleGen\"\n",
            "downstream_nodes: \"Pusher\"\n",
            "execution_options {\n",
            "  caching_options {\n",
            "  }\n",
            "}\n",
            ", pipeline_info=id: \"penguin-simple\"\n",
            ", pipeline_run_id='2024-01-10T08:33:19.404351', top_level_pipeline_run_id=None)\n",
            "INFO:absl:Train on the 'train' split when train_args.splits is not set.\n",
            "INFO:absl:Evaluate on the 'eval' split when eval_args.splits is not set.\n",
            "INFO:absl:udf_utils.get_fn {'train_args': '{\\n  \"num_steps\": 100\\n}', 'module_path': 'penguin_trainer@pipelines/penguin-simple/_wheels/tfx_user_code_Trainer-0.0+a7e2e8dccbb913b74904edeec5549d868a2ea392bcd84fbc1965aba698dce3fc-py3-none-any.whl', 'custom_config': 'null', 'eval_args': '{\\n  \"num_steps\": 5\\n}'} 'run_fn'\n",
            "INFO:absl:Installing 'pipelines/penguin-simple/_wheels/tfx_user_code_Trainer-0.0+a7e2e8dccbb913b74904edeec5549d868a2ea392bcd84fbc1965aba698dce3fc-py3-none-any.whl' to a temporary directory.\n",
            "INFO:absl:Executing: ['/usr/bin/python3', '-m', 'pip', 'install', '--target', '/tmp/tmp930qyg6z', 'pipelines/penguin-simple/_wheels/tfx_user_code_Trainer-0.0+a7e2e8dccbb913b74904edeec5549d868a2ea392bcd84fbc1965aba698dce3fc-py3-none-any.whl']\n",
            "INFO:absl:Successfully installed 'pipelines/penguin-simple/_wheels/tfx_user_code_Trainer-0.0+a7e2e8dccbb913b74904edeec5549d868a2ea392bcd84fbc1965aba698dce3fc-py3-none-any.whl'.\n",
            "INFO:absl:Training model.\n",
            "INFO:absl:Feature body_mass_g has a shape dim {\n",
            "  size: 1\n",
            "}\n",
            ". Setting to DenseTensor.\n",
            "INFO:absl:Feature culmen_depth_mm has a shape dim {\n",
            "  size: 1\n",
            "}\n",
            ". Setting to DenseTensor.\n",
            "INFO:absl:Feature culmen_length_mm has a shape dim {\n",
            "  size: 1\n",
            "}\n",
            ". Setting to DenseTensor.\n",
            "INFO:absl:Feature flipper_length_mm has a shape dim {\n",
            "  size: 1\n",
            "}\n",
            ". Setting to DenseTensor.\n",
            "INFO:absl:Feature species has a shape dim {\n",
            "  size: 1\n",
            "}\n",
            ". Setting to DenseTensor.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.10/dist-packages/tfx_bsl/tfxio/tf_example_record.py:343: parse_example_dataset (from tensorflow.python.data.experimental.ops.parsing_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use `tf.data.Dataset.map(tf.io.parse_example(...))` instead.\n",
            "INFO:absl:Feature body_mass_g has a shape dim {\n",
            "  size: 1\n",
            "}\n",
            ". Setting to DenseTensor.\n",
            "INFO:absl:Feature culmen_depth_mm has a shape dim {\n",
            "  size: 1\n",
            "}\n",
            ". Setting to DenseTensor.\n",
            "INFO:absl:Feature culmen_length_mm has a shape dim {\n",
            "  size: 1\n",
            "}\n",
            ". Setting to DenseTensor.\n",
            "INFO:absl:Feature flipper_length_mm has a shape dim {\n",
            "  size: 1\n",
            "}\n",
            ". Setting to DenseTensor.\n",
            "INFO:absl:Feature species has a shape dim {\n",
            "  size: 1\n",
            "}\n",
            ". Setting to DenseTensor.\n",
            "INFO:absl:Feature body_mass_g has a shape dim {\n",
            "  size: 1\n",
            "}\n",
            ". Setting to DenseTensor.\n",
            "INFO:absl:Feature culmen_depth_mm has a shape dim {\n",
            "  size: 1\n",
            "}\n",
            ". Setting to DenseTensor.\n",
            "INFO:absl:Feature culmen_length_mm has a shape dim {\n",
            "  size: 1\n",
            "}\n",
            ". Setting to DenseTensor.\n",
            "INFO:absl:Feature flipper_length_mm has a shape dim {\n",
            "  size: 1\n",
            "}\n",
            ". Setting to DenseTensor.\n",
            "INFO:absl:Feature species has a shape dim {\n",
            "  size: 1\n",
            "}\n",
            ". Setting to DenseTensor.\n",
            "INFO:absl:Feature body_mass_g has a shape dim {\n",
            "  size: 1\n",
            "}\n",
            ". Setting to DenseTensor.\n",
            "INFO:absl:Feature culmen_depth_mm has a shape dim {\n",
            "  size: 1\n",
            "}\n",
            ". Setting to DenseTensor.\n",
            "INFO:absl:Feature culmen_length_mm has a shape dim {\n",
            "  size: 1\n",
            "}\n",
            ". Setting to DenseTensor.\n",
            "INFO:absl:Feature flipper_length_mm has a shape dim {\n",
            "  size: 1\n",
            "}\n",
            ". Setting to DenseTensor.\n",
            "INFO:absl:Feature species has a shape dim {\n",
            "  size: 1\n",
            "}\n",
            ". Setting to DenseTensor.\n",
            "INFO:absl:Model: \"model\"\n",
            "INFO:absl:__________________________________________________________________________________________________\n",
            "INFO:absl: Layer (type)                Output Shape                 Param #   Connected to                  \n",
            "INFO:absl:==================================================================================================\n",
            "INFO:absl: culmen_length_mm (InputLay  [(None, 1)]                  0         []                            \n",
            "INFO:absl: er)                                                                                              \n",
            "INFO:absl:                                                                                                  \n",
            "INFO:absl: culmen_depth_mm (InputLaye  [(None, 1)]                  0         []                            \n",
            "INFO:absl: r)                                                                                               \n",
            "INFO:absl:                                                                                                  \n",
            "INFO:absl: flipper_length_mm (InputLa  [(None, 1)]                  0         []                            \n",
            "INFO:absl: yer)                                                                                             \n",
            "INFO:absl:                                                                                                  \n",
            "INFO:absl: body_mass_g (InputLayer)    [(None, 1)]                  0         []                            \n",
            "INFO:absl:                                                                                                  \n",
            "INFO:absl: concatenate (Concatenate)   (None, 4)                    0         ['culmen_length_mm[0][0]',    \n",
            "INFO:absl:                                                                     'culmen_depth_mm[0][0]',     \n",
            "INFO:absl:                                                                     'flipper_length_mm[0][0]',   \n",
            "INFO:absl:                                                                     'body_mass_g[0][0]']         \n",
            "INFO:absl:                                                                                                  \n",
            "INFO:absl: dense (Dense)               (None, 8)                    40        ['concatenate[0][0]']         \n",
            "INFO:absl:                                                                                                  \n",
            "INFO:absl: dense_1 (Dense)             (None, 8)                    72        ['dense[0][0]']               \n",
            "INFO:absl:                                                                                                  \n",
            "INFO:absl: dense_2 (Dense)             (None, 3)                    27        ['dense_1[0][0]']             \n",
            "INFO:absl:                                                                                                  \n",
            "INFO:absl:==================================================================================================\n",
            "INFO:absl:Total params: 139 (556.00 Byte)\n",
            "INFO:absl:Trainable params: 139 (556.00 Byte)\n",
            "INFO:absl:Non-trainable params: 0 (0.00 Byte)\n",
            "INFO:absl:__________________________________________________________________________________________________\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "100/100 [==============================] - 2s 6ms/step - loss: 0.4978 - sparse_categorical_accuracy: 0.8030 - val_loss: 0.2406 - val_sparse_categorical_accuracy: 0.9000\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:absl:Function `_wrapped_model` contains input name(s) resource with unsupported characters which will be renamed to model_dense_2_biasadd_readvariableop_resource in the SavedModel.\n",
            "INFO:absl:Found untraced functions such as _update_step_xla while saving (showing 1 of 1). These functions will not be directly callable after loading.\n",
            "INFO:absl:Writing fingerprint to pipelines/penguin-simple/Trainer/model/2/Format-Serving/fingerprint.pb\n",
            "INFO:absl:Training complete. Model written to pipelines/penguin-simple/Trainer/model/2/Format-Serving. ModelRun written to pipelines/penguin-simple/Trainer/model_run/2\n",
            "INFO:absl:Cleaning up stateless execution info.\n",
            "INFO:absl:Execution 2 succeeded.\n",
            "INFO:absl:Cleaning up stateful execution info.\n",
            "INFO:absl:Publishing output artifacts defaultdict(<class 'list'>, {'model_run': [Artifact(artifact: uri: \"pipelines/penguin-simple/Trainer/model_run/2\"\n",
            ", artifact_type: name: \"ModelRun\"\n",
            ")], 'model': [Artifact(artifact: uri: \"pipelines/penguin-simple/Trainer/model/2\"\n",
            ", artifact_type: name: \"Model\"\n",
            "base_type: MODEL\n",
            ")]}) for execution 2\n",
            "INFO:absl:MetadataStore with DB connection initialized\n",
            "INFO:absl:Component Trainer is finished.\n",
            "INFO:absl:Component Pusher is running.\n",
            "INFO:absl:Running launcher for node_info {\n",
            "  type {\n",
            "    name: \"tfx.components.pusher.component.Pusher\"\n",
            "    base_type: DEPLOY\n",
            "  }\n",
            "  id: \"Pusher\"\n",
            "}\n",
            "contexts {\n",
            "  contexts {\n",
            "    type {\n",
            "      name: \"pipeline\"\n",
            "    }\n",
            "    name {\n",
            "      field_value {\n",
            "        string_value: \"penguin-simple\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  contexts {\n",
            "    type {\n",
            "      name: \"pipeline_run\"\n",
            "    }\n",
            "    name {\n",
            "      field_value {\n",
            "        string_value: \"2024-01-10T08:33:19.404351\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  contexts {\n",
            "    type {\n",
            "      name: \"node\"\n",
            "    }\n",
            "    name {\n",
            "      field_value {\n",
            "        string_value: \"penguin-simple.Pusher\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "inputs {\n",
            "  inputs {\n",
            "    key: \"model\"\n",
            "    value {\n",
            "      channels {\n",
            "        producer_node_query {\n",
            "          id: \"Trainer\"\n",
            "        }\n",
            "        context_queries {\n",
            "          type {\n",
            "            name: \"pipeline\"\n",
            "          }\n",
            "          name {\n",
            "            field_value {\n",
            "              string_value: \"penguin-simple\"\n",
            "            }\n",
            "          }\n",
            "        }\n",
            "        context_queries {\n",
            "          type {\n",
            "            name: \"pipeline_run\"\n",
            "          }\n",
            "          name {\n",
            "            field_value {\n",
            "              string_value: \"2024-01-10T08:33:19.404351\"\n",
            "            }\n",
            "          }\n",
            "        }\n",
            "        context_queries {\n",
            "          type {\n",
            "            name: \"node\"\n",
            "          }\n",
            "          name {\n",
            "            field_value {\n",
            "              string_value: \"penguin-simple.Trainer\"\n",
            "            }\n",
            "          }\n",
            "        }\n",
            "        artifact_query {\n",
            "          type {\n",
            "            name: \"Model\"\n",
            "            base_type: MODEL\n",
            "          }\n",
            "        }\n",
            "        output_key: \"model\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "outputs {\n",
            "  outputs {\n",
            "    key: \"pushed_model\"\n",
            "    value {\n",
            "      artifact_spec {\n",
            "        type {\n",
            "          name: \"PushedModel\"\n",
            "          base_type: MODEL\n",
            "        }\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "parameters {\n",
            "  parameters {\n",
            "    key: \"custom_config\"\n",
            "    value {\n",
            "      field_value {\n",
            "        string_value: \"null\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  parameters {\n",
            "    key: \"push_destination\"\n",
            "    value {\n",
            "      field_value {\n",
            "        string_value: \"{\\n  \\\"filesystem\\\": {\\n    \\\"base_directory\\\": \\\"serving_model/penguin-simple\\\"\\n  }\\n}\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "upstream_nodes: \"Trainer\"\n",
            "execution_options {\n",
            "  caching_options {\n",
            "  }\n",
            "}\n",
            "\n",
            "INFO:absl:MetadataStore with DB connection initialized\n",
            "WARNING:absl:ArtifactQuery.property_predicate is not supported.\n",
            "INFO:absl:[Pusher] Resolved inputs: ({'model': [Artifact(artifact: id: 3\n",
            "type_id: 18\n",
            "uri: \"pipelines/penguin-simple/Trainer/model/2\"\n",
            "custom_properties {\n",
            "  key: \"is_external\"\n",
            "  value {\n",
            "    int_value: 0\n",
            "  }\n",
            "}\n",
            "custom_properties {\n",
            "  key: \"tfx_version\"\n",
            "  value {\n",
            "    string_value: \"1.14.0\"\n",
            "  }\n",
            "}\n",
            "state: LIVE\n",
            "type: \"Model\"\n",
            "create_time_since_epoch: 1704875611024\n",
            "last_update_time_since_epoch: 1704875611024\n",
            ", artifact_type: id: 18\n",
            "name: \"Model\"\n",
            "base_type: MODEL\n",
            ")]},)\n",
            "INFO:absl:MetadataStore with DB connection initialized\n",
            "INFO:absl:Going to run a new execution 3\n",
            "INFO:absl:Going to run a new execution: ExecutionInfo(execution_id=3, input_dict={'model': [Artifact(artifact: id: 3\n",
            "type_id: 18\n",
            "uri: \"pipelines/penguin-simple/Trainer/model/2\"\n",
            "custom_properties {\n",
            "  key: \"is_external\"\n",
            "  value {\n",
            "    int_value: 0\n",
            "  }\n",
            "}\n",
            "custom_properties {\n",
            "  key: \"tfx_version\"\n",
            "  value {\n",
            "    string_value: \"1.14.0\"\n",
            "  }\n",
            "}\n",
            "state: LIVE\n",
            "type: \"Model\"\n",
            "create_time_since_epoch: 1704875611024\n",
            "last_update_time_since_epoch: 1704875611024\n",
            ", artifact_type: id: 18\n",
            "name: \"Model\"\n",
            "base_type: MODEL\n",
            ")]}, output_dict=defaultdict(<class 'list'>, {'pushed_model': [Artifact(artifact: uri: \"pipelines/penguin-simple/Pusher/pushed_model/3\"\n",
            ", artifact_type: name: \"PushedModel\"\n",
            "base_type: MODEL\n",
            ")]}), exec_properties={'custom_config': 'null', 'push_destination': '{\\n  \"filesystem\": {\\n    \"base_directory\": \"serving_model/penguin-simple\"\\n  }\\n}'}, execution_output_uri='pipelines/penguin-simple/Pusher/.system/executor_execution/3/executor_output.pb', stateful_working_dir='pipelines/penguin-simple/Pusher/.system/stateful_working_dir/2024-01-10T08:33:19.404351', tmp_dir='pipelines/penguin-simple/Pusher/.system/executor_execution/3/.temp/', pipeline_node=node_info {\n",
            "  type {\n",
            "    name: \"tfx.components.pusher.component.Pusher\"\n",
            "    base_type: DEPLOY\n",
            "  }\n",
            "  id: \"Pusher\"\n",
            "}\n",
            "contexts {\n",
            "  contexts {\n",
            "    type {\n",
            "      name: \"pipeline\"\n",
            "    }\n",
            "    name {\n",
            "      field_value {\n",
            "        string_value: \"penguin-simple\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  contexts {\n",
            "    type {\n",
            "      name: \"pipeline_run\"\n",
            "    }\n",
            "    name {\n",
            "      field_value {\n",
            "        string_value: \"2024-01-10T08:33:19.404351\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  contexts {\n",
            "    type {\n",
            "      name: \"node\"\n",
            "    }\n",
            "    name {\n",
            "      field_value {\n",
            "        string_value: \"penguin-simple.Pusher\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "inputs {\n",
            "  inputs {\n",
            "    key: \"model\"\n",
            "    value {\n",
            "      channels {\n",
            "        producer_node_query {\n",
            "          id: \"Trainer\"\n",
            "        }\n",
            "        context_queries {\n",
            "          type {\n",
            "            name: \"pipeline\"\n",
            "          }\n",
            "          name {\n",
            "            field_value {\n",
            "              string_value: \"penguin-simple\"\n",
            "            }\n",
            "          }\n",
            "        }\n",
            "        context_queries {\n",
            "          type {\n",
            "            name: \"pipeline_run\"\n",
            "          }\n",
            "          name {\n",
            "            field_value {\n",
            "              string_value: \"2024-01-10T08:33:19.404351\"\n",
            "            }\n",
            "          }\n",
            "        }\n",
            "        context_queries {\n",
            "          type {\n",
            "            name: \"node\"\n",
            "          }\n",
            "          name {\n",
            "            field_value {\n",
            "              string_value: \"penguin-simple.Trainer\"\n",
            "            }\n",
            "          }\n",
            "        }\n",
            "        artifact_query {\n",
            "          type {\n",
            "            name: \"Model\"\n",
            "            base_type: MODEL\n",
            "          }\n",
            "        }\n",
            "        output_key: \"model\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "outputs {\n",
            "  outputs {\n",
            "    key: \"pushed_model\"\n",
            "    value {\n",
            "      artifact_spec {\n",
            "        type {\n",
            "          name: \"PushedModel\"\n",
            "          base_type: MODEL\n",
            "        }\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "parameters {\n",
            "  parameters {\n",
            "    key: \"custom_config\"\n",
            "    value {\n",
            "      field_value {\n",
            "        string_value: \"null\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  parameters {\n",
            "    key: \"push_destination\"\n",
            "    value {\n",
            "      field_value {\n",
            "        string_value: \"{\\n  \\\"filesystem\\\": {\\n    \\\"base_directory\\\": \\\"serving_model/penguin-simple\\\"\\n  }\\n}\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "upstream_nodes: \"Trainer\"\n",
            "execution_options {\n",
            "  caching_options {\n",
            "  }\n",
            "}\n",
            ", pipeline_info=id: \"penguin-simple\"\n",
            ", pipeline_run_id='2024-01-10T08:33:19.404351', top_level_pipeline_run_id=None)\n",
            "WARNING:absl:Pusher is going to push the model without validation. Consider using Evaluator or InfraValidator in your pipeline.\n",
            "INFO:absl:Model version: 1704875611\n",
            "INFO:absl:Model written to serving path serving_model/penguin-simple/1704875611.\n",
            "INFO:absl:Model pushed to pipelines/penguin-simple/Pusher/pushed_model/3.\n",
            "INFO:absl:Cleaning up stateless execution info.\n",
            "INFO:absl:Execution 3 succeeded.\n",
            "INFO:absl:Cleaning up stateful execution info.\n",
            "INFO:absl:Publishing output artifacts defaultdict(<class 'list'>, {'pushed_model': [Artifact(artifact: uri: \"pipelines/penguin-simple/Pusher/pushed_model/3\"\n",
            ", artifact_type: name: \"PushedModel\"\n",
            "base_type: MODEL\n",
            ")]}) for execution 3\n",
            "INFO:absl:MetadataStore with DB connection initialized\n",
            "INFO:absl:Component Pusher is finished.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YBkiw0wUtdBU"
      },
      "source": [
        "You should see \"INFO:absl:Component Pusher is finished.\" at the end of the\n",
        "logs if the pipeline finished successfully. Because `Pusher` component is the\n",
        "last component of the pipeline. You can also see that the `metadata` (metadata store), `pipelines` (pipeline root), and `serving_model` directories have been populated with the metadata, artifacts, and models that the pipeline generates.\n",
        "\n",
        "Now that you've ran this simple pipeline, you will see in the next sections how you can modify it to use custom components."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8WNr5JfHt-Sn"
      },
      "source": [
        "## Building Custom Components\n",
        "\n",
        "Let's say you want to modify the pipeline above to filter the data first before running the trainer. Without using a TFX component, you would run something like the code below. This will just get the rows where the `culmen_length_mm` feature is greater than `0.3`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k2EAJyXD3za_",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 424
        },
        "outputId": "3ded012c-96a4-41b6-e2bb-8418203c55b6"
      },
      "source": [
        "import pandas as pd\n",
        "import glob\n",
        "\n",
        "# search directory for one or more CSVs\n",
        "files = glob.glob(f'{DATA_ROOT}/*.csv')\n",
        "\n",
        "# filter the dataset\n",
        "for file in files:\n",
        "  df = pd.read_csv(file, index_col=False)\n",
        "  filtered_df = df[df['culmen_length_mm'] > 0.3].reset_index(drop=True)\n",
        "\n",
        "# print latest modified file\n",
        "filtered_df"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "     species  culmen_length_mm  culmen_depth_mm  flipper_length_mm  \\\n",
              "0          0          0.327273         0.535714           0.169492   \n",
              "1          0          0.378182         0.904762           0.423729   \n",
              "2          0          0.505455         1.000000           0.372881   \n",
              "3          0          0.309091         0.654762           0.186441   \n",
              "4          0          0.305455         0.571429           0.254237   \n",
              "..       ...               ...              ...                ...   \n",
              "227        2          0.549091         0.071429           0.711864   \n",
              "228        2          0.534545         0.142857           0.728814   \n",
              "229        2          0.665455         0.309524           0.847458   \n",
              "230        2          0.476364         0.202381           0.677966   \n",
              "231        2          0.647273         0.357143           0.694915   \n",
              "\n",
              "     body_mass_g  \n",
              "0       0.138889  \n",
              "1       0.500000  \n",
              "2       0.416667  \n",
              "3       0.236111  \n",
              "4       0.138889  \n",
              "..           ...  \n",
              "227     0.618056  \n",
              "228     0.597222  \n",
              "229     0.847222  \n",
              "230     0.694444  \n",
              "231     0.750000  \n",
              "\n",
              "[232 rows x 5 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-15d2e393-e372-4e8e-9a72-b5005112bb07\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>species</th>\n",
              "      <th>culmen_length_mm</th>\n",
              "      <th>culmen_depth_mm</th>\n",
              "      <th>flipper_length_mm</th>\n",
              "      <th>body_mass_g</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>0.327273</td>\n",
              "      <td>0.535714</td>\n",
              "      <td>0.169492</td>\n",
              "      <td>0.138889</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0</td>\n",
              "      <td>0.378182</td>\n",
              "      <td>0.904762</td>\n",
              "      <td>0.423729</td>\n",
              "      <td>0.500000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0</td>\n",
              "      <td>0.505455</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.372881</td>\n",
              "      <td>0.416667</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0</td>\n",
              "      <td>0.309091</td>\n",
              "      <td>0.654762</td>\n",
              "      <td>0.186441</td>\n",
              "      <td>0.236111</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0</td>\n",
              "      <td>0.305455</td>\n",
              "      <td>0.571429</td>\n",
              "      <td>0.254237</td>\n",
              "      <td>0.138889</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>227</th>\n",
              "      <td>2</td>\n",
              "      <td>0.549091</td>\n",
              "      <td>0.071429</td>\n",
              "      <td>0.711864</td>\n",
              "      <td>0.618056</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>228</th>\n",
              "      <td>2</td>\n",
              "      <td>0.534545</td>\n",
              "      <td>0.142857</td>\n",
              "      <td>0.728814</td>\n",
              "      <td>0.597222</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>229</th>\n",
              "      <td>2</td>\n",
              "      <td>0.665455</td>\n",
              "      <td>0.309524</td>\n",
              "      <td>0.847458</td>\n",
              "      <td>0.847222</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>230</th>\n",
              "      <td>2</td>\n",
              "      <td>0.476364</td>\n",
              "      <td>0.202381</td>\n",
              "      <td>0.677966</td>\n",
              "      <td>0.694444</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>231</th>\n",
              "      <td>2</td>\n",
              "      <td>0.647273</td>\n",
              "      <td>0.357143</td>\n",
              "      <td>0.694915</td>\n",
              "      <td>0.750000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>232 rows × 5 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-15d2e393-e372-4e8e-9a72-b5005112bb07')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-15d2e393-e372-4e8e-9a72-b5005112bb07 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-15d2e393-e372-4e8e-9a72-b5005112bb07');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-36511d5d-5c3a-4206-adf3-314b1400c346\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-36511d5d-5c3a-4206-adf3-314b1400c346')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-36511d5d-5c3a-4206-adf3-314b1400c346 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Cqr7pZ3lyYUd"
      },
      "source": [
        "You can save the dataset above to a different directory and point the TFX pipeline to it. That definitely works but you can include this code in a custom component as well so it's part of the TFX pipeline.\n",
        "\n",
        "As mentioned in the lectures, a TFX component has a driver, executor, and publisher. The driver and publisher interacts with the metadata store while the executor runs the actual processing. More often than not, you only need to modify the executor and that's what you'll be doing in the next sections."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8n6HkPOEzaSN"
      },
      "source": [
        "## Custom components through Python functions\n",
        "\n",
        "TFX provides a way to define an executor by just using a `component` decorator on your function. If you've done the Kubeflow Pipelines ungraded lab earlier this week, this will look very familiar. It uses the same concepts such as defining the inputs and outputs using type annotations then using those parameters in the function body.\n",
        "\n",
        "The component below basically uses the same filtering code you saw earlier but wraps it in a Python component. It defines two input parameters and outputs a dictionary with a String artifact (see function docstring for description). Because this component uses the same publisher as a standard TFX component, the output artifact will be saved in the metadata store as you'll see later."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SwLgLLY3m51s"
      },
      "source": [
        "#import artifact type that you will use\n",
        "from tfx.types.standard_artifacts import String\n",
        "\n",
        "# import the decorator\n",
        "from tfx.dsl.component.experimental.decorators import component\n",
        "\n",
        "# import type annotations\n",
        "from tfx.dsl.component.experimental.annotations import InputArtifact, OutputArtifact, Parameter, OutputDict\n",
        "\n",
        "@component\n",
        "def CustomFilterComponent(input_base: Parameter[str],\n",
        "                    output_base: Parameter[str],\n",
        "                    ) -> OutputDict(output_path=str):\n",
        "  '''\n",
        "  Args:\n",
        "    input_base - location of the raw CSV\n",
        "    output_base - location where you want to save the filtered CSV\n",
        "\n",
        "  Returns:\n",
        "    OutputDict:\n",
        "      output_path - String artifact that just holds the `output_base` value\n",
        "  '''\n",
        "  import pandas as pd\n",
        "  import glob\n",
        "  import os\n",
        "\n",
        "  # create the output base if it does not exist yet\n",
        "  if not os.path.exists(output_base):\n",
        "      os.mkdir(output_base)\n",
        "\n",
        "  # search for CSVs in the input base\n",
        "  files = glob.glob(f'{input_base}/*.csv')\n",
        "\n",
        "  # loop through CSVs\n",
        "  for file in files:\n",
        "\n",
        "    # read the CSV\n",
        "    df = pd.read_csv(file, index_col=False)\n",
        "\n",
        "    # filter the data\n",
        "    filtered_df = df[df['culmen_length_mm'] > 0.3].reset_index(drop=True)\n",
        "\n",
        "    # compose output filename\n",
        "    filename = os.path.basename(file).replace('.csv','')\n",
        "    filtered_filename = f'{filename}_filtered.csv'\n",
        "\n",
        "    # save filtered CSV to output base\n",
        "    filtered_df.to_csv(f'{output_base}/{filtered_filename}', index=False)\n",
        "\n",
        "  # define the output artifact\n",
        "  return {'output_path': output_base}"
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FjKA_IWm29UA"
      },
      "source": [
        "You can now run your newly built component. You will just run a single node pipeline to prove that it works."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XGRzLRR-ukuN",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "567ac4a9-0048-414d-8639-fc12da670b89"
      },
      "source": [
        "# define a filter task\n",
        "filter_task = CustomFilterComponent(input_base=DATA_ROOT,\n",
        "                                        output_base=f'{DATA_ROOT}_filtered')\n",
        "\n",
        "# include the task\n",
        "components = [filter_task]\n",
        "\n",
        "# define a pipeline with only the single component\n",
        "pipeline = tfx.dsl.Pipeline(\n",
        "      pipeline_name=PIPELINE_NAME,\n",
        "      pipeline_root=PIPELINE_ROOT,\n",
        "      metadata_connection_config=tfx.orchestration.metadata\n",
        "      .sqlite_metadata_connection_config(METADATA_PATH),\n",
        "      components=components)\n",
        "\n",
        "# run the pipeline\n",
        "tfx.orchestration.LocalDagRunner().run(pipeline)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:absl:Using deployment config:\n",
            " executor_specs {\n",
            "  key: \"CustomFilterComponent\"\n",
            "  value {\n",
            "    python_class_executable_spec {\n",
            "      class_path: \"__main__.CustomFilterComponent_Executor\"\n",
            "    }\n",
            "  }\n",
            "}\n",
            "metadata_connection_config {\n",
            "  database_connection_config {\n",
            "    sqlite {\n",
            "      filename_uri: \"metadata/penguin-simple/metadata.db\"\n",
            "      connection_mode: READWRITE_OPENCREATE\n",
            "    }\n",
            "  }\n",
            "}\n",
            "\n",
            "INFO:absl:Using connection config:\n",
            " sqlite {\n",
            "  filename_uri: \"metadata/penguin-simple/metadata.db\"\n",
            "  connection_mode: READWRITE_OPENCREATE\n",
            "}\n",
            "\n",
            "INFO:absl:Component CustomFilterComponent is running.\n",
            "INFO:absl:Running launcher for node_info {\n",
            "  type {\n",
            "    name: \"__main__.CustomFilterComponent\"\n",
            "  }\n",
            "  id: \"CustomFilterComponent\"\n",
            "}\n",
            "contexts {\n",
            "  contexts {\n",
            "    type {\n",
            "      name: \"pipeline\"\n",
            "    }\n",
            "    name {\n",
            "      field_value {\n",
            "        string_value: \"penguin-simple\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  contexts {\n",
            "    type {\n",
            "      name: \"pipeline_run\"\n",
            "    }\n",
            "    name {\n",
            "      field_value {\n",
            "        string_value: \"2024-01-10T08:35:04.805360\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  contexts {\n",
            "    type {\n",
            "      name: \"node\"\n",
            "    }\n",
            "    name {\n",
            "      field_value {\n",
            "        string_value: \"penguin-simple.CustomFilterComponent\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "outputs {\n",
            "  outputs {\n",
            "    key: \"output_path\"\n",
            "    value {\n",
            "      artifact_spec {\n",
            "        type {\n",
            "          name: \"String\"\n",
            "        }\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "parameters {\n",
            "  parameters {\n",
            "    key: \"input_base\"\n",
            "    value {\n",
            "      field_value {\n",
            "        string_value: \"data\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  parameters {\n",
            "    key: \"output_base\"\n",
            "    value {\n",
            "      field_value {\n",
            "        string_value: \"data_filtered\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "execution_options {\n",
            "  caching_options {\n",
            "  }\n",
            "}\n",
            "\n",
            "INFO:absl:MetadataStore with DB connection initialized\n",
            "INFO:absl:[CustomFilterComponent] Resolved inputs: ({},)\n",
            "INFO:absl:MetadataStore with DB connection initialized\n",
            "INFO:absl:Going to run a new execution 4\n",
            "INFO:absl:Going to run a new execution: ExecutionInfo(execution_id=4, input_dict={}, output_dict=defaultdict(<class 'list'>, {'output_path': [Artifact(artifact: uri: \"pipelines/penguin-simple/CustomFilterComponent/output_path/4/value\"\n",
            ", artifact_type: name: \"String\"\n",
            ")]}), exec_properties={'input_base': 'data', 'output_base': 'data_filtered'}, execution_output_uri='pipelines/penguin-simple/CustomFilterComponent/.system/executor_execution/4/executor_output.pb', stateful_working_dir='pipelines/penguin-simple/CustomFilterComponent/.system/stateful_working_dir/2024-01-10T08:35:04.805360', tmp_dir='pipelines/penguin-simple/CustomFilterComponent/.system/executor_execution/4/.temp/', pipeline_node=node_info {\n",
            "  type {\n",
            "    name: \"__main__.CustomFilterComponent\"\n",
            "  }\n",
            "  id: \"CustomFilterComponent\"\n",
            "}\n",
            "contexts {\n",
            "  contexts {\n",
            "    type {\n",
            "      name: \"pipeline\"\n",
            "    }\n",
            "    name {\n",
            "      field_value {\n",
            "        string_value: \"penguin-simple\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  contexts {\n",
            "    type {\n",
            "      name: \"pipeline_run\"\n",
            "    }\n",
            "    name {\n",
            "      field_value {\n",
            "        string_value: \"2024-01-10T08:35:04.805360\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  contexts {\n",
            "    type {\n",
            "      name: \"node\"\n",
            "    }\n",
            "    name {\n",
            "      field_value {\n",
            "        string_value: \"penguin-simple.CustomFilterComponent\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "outputs {\n",
            "  outputs {\n",
            "    key: \"output_path\"\n",
            "    value {\n",
            "      artifact_spec {\n",
            "        type {\n",
            "          name: \"String\"\n",
            "        }\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "parameters {\n",
            "  parameters {\n",
            "    key: \"input_base\"\n",
            "    value {\n",
            "      field_value {\n",
            "        string_value: \"data\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  parameters {\n",
            "    key: \"output_base\"\n",
            "    value {\n",
            "      field_value {\n",
            "        string_value: \"data_filtered\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "execution_options {\n",
            "  caching_options {\n",
            "  }\n",
            "}\n",
            ", pipeline_info=id: \"penguin-simple\"\n",
            ", pipeline_run_id='2024-01-10T08:35:04.805360', top_level_pipeline_run_id=None)\n",
            "INFO:absl:Cleaning up stateless execution info.\n",
            "INFO:absl:Execution 4 succeeded.\n",
            "INFO:absl:Cleaning up stateful execution info.\n",
            "INFO:absl:Publishing output artifacts defaultdict(<class 'list'>, {'output_path': [Artifact(artifact: uri: \"pipelines/penguin-simple/CustomFilterComponent/output_path/4/value\"\n",
            ", artifact_type: name: \"String\"\n",
            ")]}) for execution 4\n",
            "INFO:absl:MetadataStore with DB connection initialized\n",
            "INFO:absl:Component CustomFilterComponent is finished.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y2CxoUcu4Ne5"
      },
      "source": [
        "You should now see the filtered CSV in the `data_filtered` folder in the file explorer. As expected, you can also see that it has less lines than the original because of the filtering."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0cDdNBhku0j9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9b087796-17dc-4ea3-cbed-389b64955c48"
      },
      "source": [
        "# number of rows in original csv\n",
        "!cat data/data.csv | wc -l\n",
        "\n",
        "# number of rows in filtered csv\n",
        "!cat data_filtered/data_filtered.csv | wc -l"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "335\n",
            "233\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pUUTdRPU4xmZ"
      },
      "source": [
        "If you navigate to `pipelines/penguin-simple/CustomFilterComponent/output_path/`, you will also see a directory with the run id of the pipeline (e.g. `4`). If you click on the `value` file beneath it, you'll see the string value you saved which is just the `output_base` (i.e. `data_filtered`). You will want to feed this value to the `CsvExampleGen` component and that's what you'll do next."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JNkcASQ25Vgm"
      },
      "source": [
        "### Building a custom component using standard components\n",
        "\n",
        "If we try to use our new component with CsvExampleGen, **you will encounter an error** as shown below:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L3WzxHyA56M8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "622c9b34-702d-4d2f-c6cf-63ceb1f45e0b"
      },
      "source": [
        "# Define filter task\n",
        "filter_task = CustomFilterComponent(input_base=DATA_ROOT,\n",
        "                                        output_base=f'{DATA_ROOT}_filtered')\n",
        "\n",
        "# Try using the custom component with CsvExampleGen. This code will expectedly throw an error.\n",
        "try:\n",
        "  example_gen = tfx.components.CsvExampleGen(input_base=filter_task.outputs['output_path'])\n",
        "\n",
        "except Exception as e:\n",
        "  print(\"Error thrown as expected!\")\n",
        "  print(e)"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Error thrown as expected!\n",
            "Expected type <class 'str'> for parameter 'input_base' but got OutputChannel(artifact_type=String, producer_component_id=CustomFilterComponent, output_key=output_path, additional_properties={}, additional_custom_properties={}, _input_trigger=None instead.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OaZ9X-SH7QXG"
      },
      "source": [
        "As you can see, the output of our custom component cannot be accepted into `CsvExampleGen` because it is configured to accept a primitive string. TFX components generate and consume [`Channel`](https://www.tensorflow.org/tfx/api_docs/python/tfx/v1/dsl/Channel) objects and that's what our custom component outputs. For that, we need to build a custom data ingestion component that reuses the `CsvExampleGen` code but accepts a `Channel`. You will do that in the following sections."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K8U2wy1L76H7"
      },
      "source": [
        "### Standard ExampleGen code\n",
        "\n",
        "TFX is open source so the code for standard components can easily be found the public [repo](https://github.com/tensorflow/tfx/tree/89d3cb6c59acd0d487916bff703711815f1506b5/tfx). We placed links in the following sections of the actual files that you'll be modifying/overriding in case you want to compare what was changed for your custom ExampleGen.\n",
        "\n",
        "The class heirarchy for these components is pretty deep but in summary, you will only need to modify three:\n",
        "\n",
        "* Component Spec - this describes the inputs, outputs, and parameters passed on to the component\n",
        "* Executor - code for processing the inputs, outputs, and parameters\n",
        "* Component Class - puts everything together so your code can be run by the TFX runner."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ptdXl9mx8au2"
      },
      "source": [
        "### Modify the Component Spec\n",
        "\n",
        "First, you will need to modify the Component Spec. This file describes the parameters, inputs, and outputs of the standard components. Parameters are values supplied at runtime while inputs and outputs are values read from the metadata store. The original for ExampleGen is found [here](https://github.com/tensorflow/tfx/blob/89d3cb6c59acd0d487916bff703711815f1506b5/tfx/types/standard_component_specs.py#L187).\n",
        "\n",
        "You will need to implement the same for our custom ExampleGen but it should accept a `Channel` parameter The revised version is shown below. Notice that we revised the `INPUTS` dictionary to have a `ChannelParameter` whereas in the original, all are just in the `PARAMETERS` dictionary."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aZlH_YRp0mVY"
      },
      "source": [
        "from tfx.types.component_spec import ChannelParameter\n",
        "from tfx.types.component_spec import ExecutionParameter\n",
        "from tfx.types.component_spec import ComponentSpec\n",
        "from tfx.types import standard_artifacts\n",
        "from tfx.proto import example_gen_pb2\n",
        "from tfx.proto import range_config_pb2\n",
        "\n",
        "\n",
        "# Key for example_gen input that we want to use\n",
        "INPUT_BASE_KEY = 'input_base'\n",
        "\n",
        "# Other keys\n",
        "INPUT_CONFIG_KEY = 'input_config'\n",
        "OUTPUT_CONFIG_KEY = 'output_config'\n",
        "OUTPUT_DATA_FORMAT_KEY = 'output_data_format'\n",
        "RANGE_CONFIG_KEY = 'range_config'\n",
        "CUSTOM_CONFIG_KEY = 'custom_config'\n",
        "EXAMPLES_KEY = 'examples'\n",
        "\n",
        "class MyCustomExampleGenSpec(ComponentSpec):\n",
        "  \"\"\"File-based ExampleGen component spec.\"\"\"\n",
        "\n",
        "  PARAMETERS = {\n",
        "      INPUT_CONFIG_KEY:\n",
        "          ExecutionParameter(type=example_gen_pb2.Input),\n",
        "      OUTPUT_CONFIG_KEY:\n",
        "          ExecutionParameter(type=example_gen_pb2.Output),\n",
        "      OUTPUT_DATA_FORMAT_KEY:\n",
        "          ExecutionParameter(type=int),\n",
        "      CUSTOM_CONFIG_KEY:\n",
        "          ExecutionParameter(type=example_gen_pb2.CustomConfig, optional=True),\n",
        "      RANGE_CONFIG_KEY:\n",
        "          ExecutionParameter(type=range_config_pb2.RangeConfig, optional=True),\n",
        "  }\n",
        "\n",
        "  # Now accepts a channel\n",
        "  INPUTS = {\n",
        "      INPUT_BASE_KEY:\n",
        "          ChannelParameter(type=standard_artifacts.String),\n",
        "  }\n",
        "\n",
        "  OUTPUTS = {\n",
        "      EXAMPLES_KEY: ChannelParameter(type=standard_artifacts.Examples),\n",
        "  }"
      ],
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v_YXOyJc-3Tu"
      },
      "source": [
        "### Customize the Executor\n",
        "\n",
        "With that, you should now modify the executor code to take note of this change of input types. Instead of looking at just the parameters, it should also look into Channel inputs passed onto the component.\n",
        "\n",
        "Executor classes are executed by TFX starting with the `Do` function so you will need to modify that. The original Executor for `CsvExampleGen` can be found [here](https://github.com/tensorflow/tfx/blob/89d3cb6c59acd0d487916bff703711815f1506b5/tfx/components/example_gen/csv_example_gen/executor.py) and it inherits the base class [here](https://github.com/tensorflow/tfx/blob/89d3cb6c59acd0d487916bff703711815f1506b5/tfx/components/example_gen/base_example_gen_executor.py#L132). The base class includes the `Do()` function and that's what you'll be overriding in your new custom executor below.\n",
        "\n",
        "Basically, you're using all the functions defined in the standard component but you're modifying it so it can find the `input_base` value from the Channel inputs.\n",
        "\n",
        "*Take note that this tutorial prioritizes code brevity. In your projects, you may take a different approach such as modifying the [`_CsvToExample`](https://github.com/tensorflow/tfx/blob/89d3cb6c59acd0d487916bff703711815f1506b5/tfx/components/example_gen/csv_example_gen/executor.py#L124) code to look for the string value in the `input_dict` instead of `exec_properties`. Doing that here in this Colab will result in very long code blocks so the shorter approach is taken.*"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c_agzAOu42v_"
      },
      "source": [
        "from typing import Any, Dict, Iterable, List, Text, Union\n",
        "\n",
        "from tfx.components.example_gen.csv_example_gen.executor import Executor as CsvExampleGenExecutor\n",
        "from tfx.types import standard_component_specs\n",
        "from tfx.types import artifact_utils\n",
        "from tfx import types\n",
        "\n",
        "class MyCustomExecutor(CsvExampleGenExecutor):\n",
        "  \"\"\"Generic TFX CSV example gen executor.\"\"\"\n",
        "\n",
        "  def Do(\n",
        "      self,\n",
        "      input_dict: Dict[Text, List[types.Artifact]],\n",
        "      output_dict: Dict[Text, List[types.Artifact]],\n",
        "      exec_properties: Dict[Text, Any],\n",
        "  ) -> None:\n",
        "    \"\"\"Take input data source and generates serialized data splits.\n",
        "    The output is intended to be serialized tf.train.Examples or\n",
        "    tf.train.SequenceExamples protocol buffer in gzipped TFRecord format,\n",
        "    but subclasses can choose to override to write to any serialized records\n",
        "    payload into gzipped TFRecord as specified, so long as downstream\n",
        "    component can consume it. The format of payload is added to\n",
        "    `payload_format` custom property of the output Example artifact.\n",
        "    Args:\n",
        "      input_dict: Input dict from input key to a list of Artifacts. Depends on\n",
        "        detailed example gen implementation.\n",
        "        - input_base: an external directory containing the data files.\n",
        "      output_dict: Output dict from output key to a list of Artifacts.\n",
        "        - examples: splits of serialized records.\n",
        "      exec_properties: A dict of execution properties. Depends on detailed\n",
        "        example gen implementation.\n",
        "        - input_config: JSON string of example_gen_pb2.Input instance,\n",
        "          providing input configuration.\n",
        "        - output_config: JSON string of example_gen_pb2.Output instance,\n",
        "          providing output configuration.\n",
        "        - output_data_format: Payload format of generated data in output\n",
        "          artifact, one of example_gen_pb2.PayloadFormat enum.\n",
        "    Returns:\n",
        "      None\n",
        "    \"\"\"\n",
        "    self._log_startup(input_dict, output_dict, exec_properties)\n",
        "\n",
        "    # Get the artifact from the Channel input\n",
        "    filter_component_artifact = artifact_utils.get_single_instance(\n",
        "        input_dict[standard_component_specs.INPUT_BASE_KEY])\n",
        "\n",
        "    # Put the input string value into the exec_properties fictionary\n",
        "    exec_properties[standard_component_specs.INPUT_BASE_KEY] = filter_component_artifact.value\n",
        "\n",
        "    # execute superclass\n",
        "    super(MyCustomExecutor, self).Do(input_dict=input_dict, output_dict=output_dict, exec_properties=exec_properties)"
      ],
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "inMjz8qDAJyz"
      },
      "source": [
        "### Define the Component class\n",
        "\n",
        "Lastly, you will need to put everything together in a class. This will be the one you instantiate so you can run the component later. For comparison, the original `CsvExampleGen` component class is found [here](https://github.com/tensorflow/tfx/blob/89d3cb6c59acd0d487916bff703711815f1506b5/tfx/components/example_gen/csv_example_gen/component.py) and it inherits the `FileBasedExampleGen` from [here](https://github.com/tensorflow/tfx/blob/89d3cb6c59acd0d487916bff703711815f1506b5/tfx/components/example_gen/component.py#L115). The revised version is shown below:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wOlYnQumyWfX"
      },
      "source": [
        "from typing import Any, Dict, Optional, Text, Union\n",
        "\n",
        "from tfx.dsl.components.base import base_beam_component\n",
        "from tfx.dsl.components.base import executor_spec\n",
        "\n",
        "from tfx.proto import example_gen_pb2\n",
        "from tfx.proto import range_config_pb2\n",
        "\n",
        "from tfx import types\n",
        "from tfx.components.example_gen import utils\n",
        "\n",
        "class MyCustomExampleGen(base_beam_component.BaseBeamComponent):\n",
        "\n",
        "  # Define the Spec class and executor spec using the functions and\n",
        "  # classes you defined earlier.\n",
        "  SPEC_CLASS = MyCustomExampleGenSpec\n",
        "  EXECUTOR_SPEC = executor_spec.BeamExecutorSpec(MyCustomExecutor)\n",
        "\n",
        "  # Define init function. Notice that `input_base` now accepts a Channel.\n",
        "  def __init__(self,\n",
        "               input_base: types.Channel = None,\n",
        "               input_config: Optional[Union[example_gen_pb2.Input,\n",
        "                                            Dict[Text, Any]]] = None,\n",
        "               output_config: Optional[Union[example_gen_pb2.Output,\n",
        "                                             Dict[Text, Any]]] = None,\n",
        "               range_config: Optional[Union[range_config_pb2.RangeConfig,\n",
        "                                            Dict[Text, Any]]] = None,\n",
        "               output_data_format: Optional[int] = example_gen_pb2.FORMAT_TF_EXAMPLE):\n",
        "    \"\"\"Customized ExampleGen component.\n",
        "    Args:\n",
        "      input_base: an external directory containing the CSV files. Accepts a Channel\n",
        "        from a previous TFX component.\n",
        "      input_config: An example_gen_pb2.Input instance, providing input\n",
        "        configuration. If unset, the files under input_base will be treated as a\n",
        "        single split. If any field is provided as a RuntimeParameter,\n",
        "        input_config should be constructed as a dict with the same field names\n",
        "        as Input proto message.\n",
        "      output_config: An example_gen_pb2.Output instance, providing output\n",
        "        configuration. If unset, default splits will be 'train' and 'eval' with\n",
        "        size 2:1. If any field is provided as a RuntimeParameter, output_config\n",
        "        should be constructed as a dict with the same field names as Output\n",
        "        proto message.\n",
        "      range_config: An optional range_config_pb2.RangeConfig instance,\n",
        "        specifying the range of span values to consider. If unset, driver will\n",
        "        default to searching for latest span with no restrictions.\n",
        "    \"\"\"\n",
        "\n",
        "    # Configure inputs and outputs.\n",
        "    input_config = input_config or utils.make_default_input_config()\n",
        "    output_config = output_config or utils.make_default_output_config(\n",
        "        input_config)\n",
        "\n",
        "    # Define output type.\n",
        "    example_artifacts = types.Channel(type=standard_artifacts.Examples)\n",
        "\n",
        "    # Pass input arguments to your custom ExampleGen spec.\n",
        "    spec = MyCustomExampleGenSpec(\n",
        "        input_base=input_base,\n",
        "        input_config=input_config,\n",
        "        output_config=output_config,\n",
        "        range_config=range_config,\n",
        "        output_data_format=output_data_format,\n",
        "        examples=example_artifacts)\n",
        "\n",
        "    # This will check if the values passed are the correct type else\n",
        "    # it will throw the error you saw earlier.\n",
        "    super(MyCustomExampleGen, self).__init__(\n",
        "        spec=spec)\n",
        ""
      ],
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n2oFBiFZggLS"
      },
      "source": [
        "You can now use the custom component (`MyCustomExampleGen`) in your code as shown below. It will no longer get an error because you reconfigured the `input_base` to accept a channel."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DXOlqQ_9y-rL",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "971f975b-1893-4950-81e2-454ae29efe75"
      },
      "source": [
        "# Filter the dataset\n",
        "filter_task = CustomFilterComponent(input_base=DATA_ROOT,\n",
        "                                        output_base=f'{DATA_ROOT}_filtered')\n",
        "\n",
        "# Use the output of filter_task to know the input_base for this custom ExampleGen\n",
        "custom_example_gen_task = MyCustomExampleGen(input_base=filter_task.outputs['output_path'])\n",
        "\n",
        "# Define components to include\n",
        "components = [filter_task,\n",
        "              custom_example_gen_task]\n",
        "\n",
        "# Create the pipeline\n",
        "pipeline = tfx.dsl.Pipeline(\n",
        "      pipeline_name=PIPELINE_NAME,\n",
        "      pipeline_root=PIPELINE_ROOT,\n",
        "      metadata_connection_config=tfx.orchestration.metadata\n",
        "      .sqlite_metadata_connection_config(METADATA_PATH),\n",
        "      components=components)\n",
        "\n",
        "# Run the pipeline\n",
        "tfx.orchestration.LocalDagRunner().run(pipeline)"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:absl:Using deployment config:\n",
            " executor_specs {\n",
            "  key: \"CustomFilterComponent\"\n",
            "  value {\n",
            "    python_class_executable_spec {\n",
            "      class_path: \"__main__.CustomFilterComponent_Executor\"\n",
            "    }\n",
            "  }\n",
            "}\n",
            "executor_specs {\n",
            "  key: \"MyCustomExampleGen\"\n",
            "  value {\n",
            "    beam_executable_spec {\n",
            "      python_executor_spec {\n",
            "        class_path: \"__main__.MyCustomExecutor\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "metadata_connection_config {\n",
            "  database_connection_config {\n",
            "    sqlite {\n",
            "      filename_uri: \"metadata/penguin-simple/metadata.db\"\n",
            "      connection_mode: READWRITE_OPENCREATE\n",
            "    }\n",
            "  }\n",
            "}\n",
            "\n",
            "INFO:absl:Using connection config:\n",
            " sqlite {\n",
            "  filename_uri: \"metadata/penguin-simple/metadata.db\"\n",
            "  connection_mode: READWRITE_OPENCREATE\n",
            "}\n",
            "\n",
            "INFO:absl:Component CustomFilterComponent is running.\n",
            "INFO:absl:Running launcher for node_info {\n",
            "  type {\n",
            "    name: \"__main__.CustomFilterComponent\"\n",
            "  }\n",
            "  id: \"CustomFilterComponent\"\n",
            "}\n",
            "contexts {\n",
            "  contexts {\n",
            "    type {\n",
            "      name: \"pipeline\"\n",
            "    }\n",
            "    name {\n",
            "      field_value {\n",
            "        string_value: \"penguin-simple\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  contexts {\n",
            "    type {\n",
            "      name: \"pipeline_run\"\n",
            "    }\n",
            "    name {\n",
            "      field_value {\n",
            "        string_value: \"2024-01-10T08:37:35.818630\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  contexts {\n",
            "    type {\n",
            "      name: \"node\"\n",
            "    }\n",
            "    name {\n",
            "      field_value {\n",
            "        string_value: \"penguin-simple.CustomFilterComponent\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "outputs {\n",
            "  outputs {\n",
            "    key: \"output_path\"\n",
            "    value {\n",
            "      artifact_spec {\n",
            "        type {\n",
            "          name: \"String\"\n",
            "        }\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "parameters {\n",
            "  parameters {\n",
            "    key: \"input_base\"\n",
            "    value {\n",
            "      field_value {\n",
            "        string_value: \"data\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  parameters {\n",
            "    key: \"output_base\"\n",
            "    value {\n",
            "      field_value {\n",
            "        string_value: \"data_filtered\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "downstream_nodes: \"MyCustomExampleGen\"\n",
            "execution_options {\n",
            "  caching_options {\n",
            "  }\n",
            "}\n",
            "\n",
            "INFO:absl:MetadataStore with DB connection initialized\n",
            "INFO:absl:[CustomFilterComponent] Resolved inputs: ({},)\n",
            "INFO:absl:MetadataStore with DB connection initialized\n",
            "INFO:absl:Going to run a new execution 5\n",
            "INFO:absl:Going to run a new execution: ExecutionInfo(execution_id=5, input_dict={}, output_dict=defaultdict(<class 'list'>, {'output_path': [Artifact(artifact: uri: \"pipelines/penguin-simple/CustomFilterComponent/output_path/5/value\"\n",
            ", artifact_type: name: \"String\"\n",
            ")]}), exec_properties={'output_base': 'data_filtered', 'input_base': 'data'}, execution_output_uri='pipelines/penguin-simple/CustomFilterComponent/.system/executor_execution/5/executor_output.pb', stateful_working_dir='pipelines/penguin-simple/CustomFilterComponent/.system/stateful_working_dir/2024-01-10T08:37:35.818630', tmp_dir='pipelines/penguin-simple/CustomFilterComponent/.system/executor_execution/5/.temp/', pipeline_node=node_info {\n",
            "  type {\n",
            "    name: \"__main__.CustomFilterComponent\"\n",
            "  }\n",
            "  id: \"CustomFilterComponent\"\n",
            "}\n",
            "contexts {\n",
            "  contexts {\n",
            "    type {\n",
            "      name: \"pipeline\"\n",
            "    }\n",
            "    name {\n",
            "      field_value {\n",
            "        string_value: \"penguin-simple\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  contexts {\n",
            "    type {\n",
            "      name: \"pipeline_run\"\n",
            "    }\n",
            "    name {\n",
            "      field_value {\n",
            "        string_value: \"2024-01-10T08:37:35.818630\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  contexts {\n",
            "    type {\n",
            "      name: \"node\"\n",
            "    }\n",
            "    name {\n",
            "      field_value {\n",
            "        string_value: \"penguin-simple.CustomFilterComponent\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "outputs {\n",
            "  outputs {\n",
            "    key: \"output_path\"\n",
            "    value {\n",
            "      artifact_spec {\n",
            "        type {\n",
            "          name: \"String\"\n",
            "        }\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "parameters {\n",
            "  parameters {\n",
            "    key: \"input_base\"\n",
            "    value {\n",
            "      field_value {\n",
            "        string_value: \"data\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  parameters {\n",
            "    key: \"output_base\"\n",
            "    value {\n",
            "      field_value {\n",
            "        string_value: \"data_filtered\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "downstream_nodes: \"MyCustomExampleGen\"\n",
            "execution_options {\n",
            "  caching_options {\n",
            "  }\n",
            "}\n",
            ", pipeline_info=id: \"penguin-simple\"\n",
            ", pipeline_run_id='2024-01-10T08:37:35.818630', top_level_pipeline_run_id=None)\n",
            "INFO:absl:Cleaning up stateless execution info.\n",
            "INFO:absl:Execution 5 succeeded.\n",
            "INFO:absl:Cleaning up stateful execution info.\n",
            "INFO:absl:Publishing output artifacts defaultdict(<class 'list'>, {'output_path': [Artifact(artifact: uri: \"pipelines/penguin-simple/CustomFilterComponent/output_path/5/value\"\n",
            ", artifact_type: name: \"String\"\n",
            ")]}) for execution 5\n",
            "INFO:absl:MetadataStore with DB connection initialized\n",
            "INFO:absl:Component CustomFilterComponent is finished.\n",
            "INFO:absl:Component MyCustomExampleGen is running.\n",
            "INFO:absl:Running launcher for node_info {\n",
            "  type {\n",
            "    name: \"__main__.MyCustomExampleGen\"\n",
            "  }\n",
            "  id: \"MyCustomExampleGen\"\n",
            "}\n",
            "contexts {\n",
            "  contexts {\n",
            "    type {\n",
            "      name: \"pipeline\"\n",
            "    }\n",
            "    name {\n",
            "      field_value {\n",
            "        string_value: \"penguin-simple\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  contexts {\n",
            "    type {\n",
            "      name: \"pipeline_run\"\n",
            "    }\n",
            "    name {\n",
            "      field_value {\n",
            "        string_value: \"2024-01-10T08:37:35.818630\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  contexts {\n",
            "    type {\n",
            "      name: \"node\"\n",
            "    }\n",
            "    name {\n",
            "      field_value {\n",
            "        string_value: \"penguin-simple.MyCustomExampleGen\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "inputs {\n",
            "  inputs {\n",
            "    key: \"input_base\"\n",
            "    value {\n",
            "      channels {\n",
            "        producer_node_query {\n",
            "          id: \"CustomFilterComponent\"\n",
            "        }\n",
            "        context_queries {\n",
            "          type {\n",
            "            name: \"pipeline\"\n",
            "          }\n",
            "          name {\n",
            "            field_value {\n",
            "              string_value: \"penguin-simple\"\n",
            "            }\n",
            "          }\n",
            "        }\n",
            "        context_queries {\n",
            "          type {\n",
            "            name: \"pipeline_run\"\n",
            "          }\n",
            "          name {\n",
            "            field_value {\n",
            "              string_value: \"2024-01-10T08:37:35.818630\"\n",
            "            }\n",
            "          }\n",
            "        }\n",
            "        context_queries {\n",
            "          type {\n",
            "            name: \"node\"\n",
            "          }\n",
            "          name {\n",
            "            field_value {\n",
            "              string_value: \"penguin-simple.CustomFilterComponent\"\n",
            "            }\n",
            "          }\n",
            "        }\n",
            "        artifact_query {\n",
            "          type {\n",
            "            name: \"String\"\n",
            "          }\n",
            "        }\n",
            "        output_key: \"output_path\"\n",
            "      }\n",
            "      min_count: 1\n",
            "    }\n",
            "  }\n",
            "}\n",
            "outputs {\n",
            "  outputs {\n",
            "    key: \"examples\"\n",
            "    value {\n",
            "      artifact_spec {\n",
            "        type {\n",
            "          name: \"Examples\"\n",
            "          properties {\n",
            "            key: \"span\"\n",
            "            value: INT\n",
            "          }\n",
            "          properties {\n",
            "            key: \"split_names\"\n",
            "            value: STRING\n",
            "          }\n",
            "          properties {\n",
            "            key: \"version\"\n",
            "            value: INT\n",
            "          }\n",
            "          base_type: DATASET\n",
            "        }\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "parameters {\n",
            "  parameters {\n",
            "    key: \"input_config\"\n",
            "    value {\n",
            "      field_value {\n",
            "        string_value: \"{\\n  \\\"splits\\\": [\\n    {\\n      \\\"name\\\": \\\"single_split\\\",\\n      \\\"pattern\\\": \\\"*\\\"\\n    }\\n  ]\\n}\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  parameters {\n",
            "    key: \"output_config\"\n",
            "    value {\n",
            "      field_value {\n",
            "        string_value: \"{\\n  \\\"split_config\\\": {\\n    \\\"splits\\\": [\\n      {\\n        \\\"hash_buckets\\\": 2,\\n        \\\"name\\\": \\\"train\\\"\\n      },\\n      {\\n        \\\"hash_buckets\\\": 1,\\n        \\\"name\\\": \\\"eval\\\"\\n      }\\n    ]\\n  }\\n}\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  parameters {\n",
            "    key: \"output_data_format\"\n",
            "    value {\n",
            "      field_value {\n",
            "        int_value: 6\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "upstream_nodes: \"CustomFilterComponent\"\n",
            "execution_options {\n",
            "  caching_options {\n",
            "  }\n",
            "}\n",
            "\n",
            "INFO:absl:MetadataStore with DB connection initialized\n",
            "WARNING:absl:ArtifactQuery.property_predicate is not supported.\n",
            "INFO:absl:[MyCustomExampleGen] Resolved inputs: ({'input_base': [Artifact(artifact: id: 6\n",
            "type_id: 22\n",
            "uri: \"pipelines/penguin-simple/CustomFilterComponent/output_path/5/value\"\n",
            "custom_properties {\n",
            "  key: \"__is_null__\"\n",
            "  value {\n",
            "    int_value: 0\n",
            "  }\n",
            "}\n",
            "custom_properties {\n",
            "  key: \"is_external\"\n",
            "  value {\n",
            "    int_value: 0\n",
            "  }\n",
            "}\n",
            "custom_properties {\n",
            "  key: \"tfx_version\"\n",
            "  value {\n",
            "    string_value: \"1.14.0\"\n",
            "  }\n",
            "}\n",
            "state: LIVE\n",
            "type: \"String\"\n",
            "create_time_since_epoch: 1704875855922\n",
            "last_update_time_since_epoch: 1704875855922\n",
            ", artifact_type: id: 22\n",
            "name: \"String\"\n",
            ")]},)\n",
            "INFO:absl:MetadataStore with DB connection initialized\n",
            "INFO:absl:Going to run a new execution 6\n",
            "INFO:absl:Going to run a new execution: ExecutionInfo(execution_id=6, input_dict={'input_base': [Artifact(artifact: id: 6\n",
            "type_id: 22\n",
            "uri: \"pipelines/penguin-simple/CustomFilterComponent/output_path/5/value\"\n",
            "custom_properties {\n",
            "  key: \"__is_null__\"\n",
            "  value {\n",
            "    int_value: 0\n",
            "  }\n",
            "}\n",
            "custom_properties {\n",
            "  key: \"is_external\"\n",
            "  value {\n",
            "    int_value: 0\n",
            "  }\n",
            "}\n",
            "custom_properties {\n",
            "  key: \"tfx_version\"\n",
            "  value {\n",
            "    string_value: \"1.14.0\"\n",
            "  }\n",
            "}\n",
            "state: LIVE\n",
            "type: \"String\"\n",
            "create_time_since_epoch: 1704875855922\n",
            "last_update_time_since_epoch: 1704875855922\n",
            ", artifact_type: id: 22\n",
            "name: \"String\"\n",
            ")]}, output_dict=defaultdict(<class 'list'>, {'examples': [Artifact(artifact: uri: \"pipelines/penguin-simple/MyCustomExampleGen/examples/6\"\n",
            ", artifact_type: name: \"Examples\"\n",
            "properties {\n",
            "  key: \"span\"\n",
            "  value: INT\n",
            "}\n",
            "properties {\n",
            "  key: \"split_names\"\n",
            "  value: STRING\n",
            "}\n",
            "properties {\n",
            "  key: \"version\"\n",
            "  value: INT\n",
            "}\n",
            "base_type: DATASET\n",
            ")]}), exec_properties={'output_data_format': 6, 'input_config': '{\\n  \"splits\": [\\n    {\\n      \"name\": \"single_split\",\\n      \"pattern\": \"*\"\\n    }\\n  ]\\n}', 'output_config': '{\\n  \"split_config\": {\\n    \"splits\": [\\n      {\\n        \"hash_buckets\": 2,\\n        \"name\": \"train\"\\n      },\\n      {\\n        \"hash_buckets\": 1,\\n        \"name\": \"eval\"\\n      }\\n    ]\\n  }\\n}'}, execution_output_uri='pipelines/penguin-simple/MyCustomExampleGen/.system/executor_execution/6/executor_output.pb', stateful_working_dir='pipelines/penguin-simple/MyCustomExampleGen/.system/stateful_working_dir/2024-01-10T08:37:35.818630', tmp_dir='pipelines/penguin-simple/MyCustomExampleGen/.system/executor_execution/6/.temp/', pipeline_node=node_info {\n",
            "  type {\n",
            "    name: \"__main__.MyCustomExampleGen\"\n",
            "  }\n",
            "  id: \"MyCustomExampleGen\"\n",
            "}\n",
            "contexts {\n",
            "  contexts {\n",
            "    type {\n",
            "      name: \"pipeline\"\n",
            "    }\n",
            "    name {\n",
            "      field_value {\n",
            "        string_value: \"penguin-simple\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  contexts {\n",
            "    type {\n",
            "      name: \"pipeline_run\"\n",
            "    }\n",
            "    name {\n",
            "      field_value {\n",
            "        string_value: \"2024-01-10T08:37:35.818630\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  contexts {\n",
            "    type {\n",
            "      name: \"node\"\n",
            "    }\n",
            "    name {\n",
            "      field_value {\n",
            "        string_value: \"penguin-simple.MyCustomExampleGen\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "inputs {\n",
            "  inputs {\n",
            "    key: \"input_base\"\n",
            "    value {\n",
            "      channels {\n",
            "        producer_node_query {\n",
            "          id: \"CustomFilterComponent\"\n",
            "        }\n",
            "        context_queries {\n",
            "          type {\n",
            "            name: \"pipeline\"\n",
            "          }\n",
            "          name {\n",
            "            field_value {\n",
            "              string_value: \"penguin-simple\"\n",
            "            }\n",
            "          }\n",
            "        }\n",
            "        context_queries {\n",
            "          type {\n",
            "            name: \"pipeline_run\"\n",
            "          }\n",
            "          name {\n",
            "            field_value {\n",
            "              string_value: \"2024-01-10T08:37:35.818630\"\n",
            "            }\n",
            "          }\n",
            "        }\n",
            "        context_queries {\n",
            "          type {\n",
            "            name: \"node\"\n",
            "          }\n",
            "          name {\n",
            "            field_value {\n",
            "              string_value: \"penguin-simple.CustomFilterComponent\"\n",
            "            }\n",
            "          }\n",
            "        }\n",
            "        artifact_query {\n",
            "          type {\n",
            "            name: \"String\"\n",
            "          }\n",
            "        }\n",
            "        output_key: \"output_path\"\n",
            "      }\n",
            "      min_count: 1\n",
            "    }\n",
            "  }\n",
            "}\n",
            "outputs {\n",
            "  outputs {\n",
            "    key: \"examples\"\n",
            "    value {\n",
            "      artifact_spec {\n",
            "        type {\n",
            "          name: \"Examples\"\n",
            "          properties {\n",
            "            key: \"span\"\n",
            "            value: INT\n",
            "          }\n",
            "          properties {\n",
            "            key: \"split_names\"\n",
            "            value: STRING\n",
            "          }\n",
            "          properties {\n",
            "            key: \"version\"\n",
            "            value: INT\n",
            "          }\n",
            "          base_type: DATASET\n",
            "        }\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "parameters {\n",
            "  parameters {\n",
            "    key: \"input_config\"\n",
            "    value {\n",
            "      field_value {\n",
            "        string_value: \"{\\n  \\\"splits\\\": [\\n    {\\n      \\\"name\\\": \\\"single_split\\\",\\n      \\\"pattern\\\": \\\"*\\\"\\n    }\\n  ]\\n}\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  parameters {\n",
            "    key: \"output_config\"\n",
            "    value {\n",
            "      field_value {\n",
            "        string_value: \"{\\n  \\\"split_config\\\": {\\n    \\\"splits\\\": [\\n      {\\n        \\\"hash_buckets\\\": 2,\\n        \\\"name\\\": \\\"train\\\"\\n      },\\n      {\\n        \\\"hash_buckets\\\": 1,\\n        \\\"name\\\": \\\"eval\\\"\\n      }\\n    ]\\n  }\\n}\"\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "  parameters {\n",
            "    key: \"output_data_format\"\n",
            "    value {\n",
            "      field_value {\n",
            "        int_value: 6\n",
            "      }\n",
            "    }\n",
            "  }\n",
            "}\n",
            "upstream_nodes: \"CustomFilterComponent\"\n",
            "execution_options {\n",
            "  caching_options {\n",
            "  }\n",
            "}\n",
            ", pipeline_info=id: \"penguin-simple\"\n",
            ", pipeline_run_id='2024-01-10T08:37:35.818630', top_level_pipeline_run_id=None)\n",
            "INFO:absl:Generating examples.\n",
            "INFO:absl:Processing input csv data data_filtered/* to TFExample.\n",
            "INFO:absl:Examples generated.\n",
            "INFO:absl:Value type <class 'list'> of key _beam_pipeline_args in exec_properties is not supported, going to drop it\n",
            "INFO:absl:Cleaning up stateless execution info.\n",
            "INFO:absl:Execution 6 succeeded.\n",
            "INFO:absl:Cleaning up stateful execution info.\n",
            "INFO:absl:Publishing output artifacts defaultdict(<class 'list'>, {'examples': [Artifact(artifact: uri: \"pipelines/penguin-simple/MyCustomExampleGen/examples/6\"\n",
            ", artifact_type: name: \"Examples\"\n",
            "properties {\n",
            "  key: \"span\"\n",
            "  value: INT\n",
            "}\n",
            "properties {\n",
            "  key: \"split_names\"\n",
            "  value: STRING\n",
            "}\n",
            "properties {\n",
            "  key: \"version\"\n",
            "  value: INT\n",
            "}\n",
            "base_type: DATASET\n",
            ")]}) for execution 6\n",
            "INFO:absl:MetadataStore with DB connection initialized\n",
            "INFO:absl:Component MyCustomExampleGen is finished.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y8nfx3annFTC"
      },
      "source": [
        "As a sanity check, you can compute the number of examples for both the training and eval splits. It should equal the number of examples found in your filtered CSV. You can use the code below by replacing the `EXECUTION_ID` with the ID shown in your latest run. You can see it in the last three lines of the output cell above. For example:\n",
        "\n",
        "```\n",
        ")]}) for execution 16  # ---> **16 is the EXECUTION ID**\n",
        "INFO:absl:MetadataStore with DB connection initialized\n",
        "INFO:absl:Component MyCustomExampleGen is finished.\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6DxCauSIPl_X",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "aee8e482-2f23-4b3f-a0f0-84e5527fe92d"
      },
      "source": [
        "EXECUTION_ID = 6 # PLACE THE EXECUTION ID HERE\n",
        "\n",
        "# Create a `TFRecordDataset` to read these files\n",
        "train_dataset = tf.data.TFRecordDataset(f'{PIPELINE_ROOT}/MyCustomExampleGen/examples/{EXECUTION_ID}/Split-train/data_tfrecord-00000-of-00001.gz', compression_type=\"GZIP\")\n",
        "eval_dataset = tf.data.TFRecordDataset(f'{PIPELINE_ROOT}/MyCustomExampleGen/examples/{EXECUTION_ID}/Split-eval/data_tfrecord-00000-of-00001.gz', compression_type=\"GZIP\")\n",
        "\n",
        "# Get number of records for each dataset (only use for small datasets to avoid memory issues)\n",
        "num_train_data = len(list(train_dataset))\n",
        "num_eval_data = len(list(eval_dataset))\n",
        "\n",
        "# Get the total\n",
        "total_examples = num_train_data + num_eval_data\n",
        "\n",
        "print(f'total number of examples: {total_examples}')"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "total number of examples: 232\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V8jE62T0ry43"
      },
      "source": [
        "## Wrap Up\n",
        "\n",
        "In this lab, you were able to use custom components to create a pipeline. This shows that you can go outside the standard TFX components if your project calls for it. To know more about custom components, you can read more [here](https://www.tensorflow.org/tfx/guide/understanding_custom_components) and see the examples [here](https://github.com/tensorflow/tfx/tree/2e41786328f5b69720e90ec4d9ecae500f5c157a/tfx/examples/custom_components)."
      ]
    }
  ]
}