27-permutation-feature-importance/C3_W5_Lab_2_Permutation_Importance.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tUUVrx70Z5KL"
      },
      "source": [
        "# Ungraded lab: Permutation Feature Importance\n",
        "------------------------\n",
        "\n",
        "Welcome, during this ungraded lab you are going to be perform Permutation Feature Importance on the [wine dataset](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_wine.html#sklearn.datasets.load_wine) using scikit-learn. In particular you will:\n",
        "\n",
        "\n",
        "1. Train a Random Forest classifier on the data.\n",
        "2. Compute the feature importance score by permutating each feature.\n",
        "3. Re-train the model with only the top features.\n",
        "4. Check other classifiers for comparison.\n",
        "\n",
        "Let's get started!\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KxXreDg0tGOF"
      },
      "source": [
        "## Inspect and pre-process the data\n",
        "\n",
        "Begin by upgrading scikit-learn to the latest version:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UsGnA3r9DQkw",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7427ee47-7be5-4513-c8cf-40ca3c6bf628"
      },
      "source": [
        "!pip install -U scikit-learn"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.10/dist-packages (1.2.2)\n",
            "Collecting scikit-learn\n",
            "  Downloading scikit_learn-1.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (10.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m10.8/10.8 MB\u001b[0m \u001b[31m17.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: numpy<2.0,>=1.17.3 in /usr/local/lib/python3.10/dist-packages (from scikit-learn) (1.23.5)\n",
            "Requirement already satisfied: scipy>=1.5.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn) (1.11.4)\n",
            "Requirement already satisfied: joblib>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from scikit-learn) (1.3.2)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn) (3.2.0)\n",
            "Installing collected packages: scikit-learn\n",
            "  Attempting uninstall: scikit-learn\n",
            "    Found existing installation: scikit-learn 1.2.2\n",
            "    Uninstalling scikit-learn-1.2.2:\n",
            "      Successfully uninstalled scikit-learn-1.2.2\n",
            "Successfully installed scikit-learn-1.3.2\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wx9r-xd5tWZO"
      },
      "source": [
        "Now import the required dependencies and load the dataset:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DCRACaLFC-1N",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "outputId": "122353b5-f235-4296-c9fd-0d5e85824d98"
      },
      "source": [
        "import numpy as np\n",
        "from sklearn.datasets import load_wine\n",
        "\n",
        "# as_frame param requires scikit-learn >= 0.23\n",
        "data = load_wine(as_frame=True)\n",
        "\n",
        "# Print first rows of the data\n",
        "data.frame.head()"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   alcohol  malic_acid   ash  alcalinity_of_ash  magnesium  total_phenols  \\\n",
              "0    14.23        1.71  2.43               15.6      127.0           2.80   \n",
              "1    13.20        1.78  2.14               11.2      100.0           2.65   \n",
              "2    13.16        2.36  2.67               18.6      101.0           2.80   \n",
              "3    14.37        1.95  2.50               16.8      113.0           3.85   \n",
              "4    13.24        2.59  2.87               21.0      118.0           2.80   \n",
              "\n",
              "   flavanoids  nonflavanoid_phenols  proanthocyanins  color_intensity   hue  \\\n",
              "0        3.06                  0.28             2.29             5.64  1.04   \n",
              "1        2.76                  0.26             1.28             4.38  1.05   \n",
              "2        3.24                  0.30             2.81             5.68  1.03   \n",
              "3        3.49                  0.24             2.18             7.80  0.86   \n",
              "4        2.69                  0.39             1.82             4.32  1.04   \n",
              "\n",
              "   od280/od315_of_diluted_wines  proline  target  \n",
              "0                          3.92   1065.0       0  \n",
              "1                          3.40   1050.0       0  \n",
              "2                          3.17   1185.0       0  \n",
              "3                          3.45   1480.0       0  \n",
              "4                          2.93    735.0       0  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-d5da72d7-28de-4a9a-98ac-df657d35b39b\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>alcohol</th>\n",
              "      <th>malic_acid</th>\n",
              "      <th>ash</th>\n",
              "      <th>alcalinity_of_ash</th>\n",
              "      <th>magnesium</th>\n",
              "      <th>total_phenols</th>\n",
              "      <th>flavanoids</th>\n",
              "      <th>nonflavanoid_phenols</th>\n",
              "      <th>proanthocyanins</th>\n",
              "      <th>color_intensity</th>\n",
              "      <th>hue</th>\n",
              "      <th>od280/od315_of_diluted_wines</th>\n",
              "      <th>proline</th>\n",
              "      <th>target</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>14.23</td>\n",
              "      <td>1.71</td>\n",
              "      <td>2.43</td>\n",
              "      <td>15.6</td>\n",
              "      <td>127.0</td>\n",
              "      <td>2.80</td>\n",
              "      <td>3.06</td>\n",
              "      <td>0.28</td>\n",
              "      <td>2.29</td>\n",
              "      <td>5.64</td>\n",
              "      <td>1.04</td>\n",
              "      <td>3.92</td>\n",
              "      <td>1065.0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>13.20</td>\n",
              "      <td>1.78</td>\n",
              "      <td>2.14</td>\n",
              "      <td>11.2</td>\n",
              "      <td>100.0</td>\n",
              "      <td>2.65</td>\n",
              "      <td>2.76</td>\n",
              "      <td>0.26</td>\n",
              "      <td>1.28</td>\n",
              "      <td>4.38</td>\n",
              "      <td>1.05</td>\n",
              "      <td>3.40</td>\n",
              "      <td>1050.0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>13.16</td>\n",
              "      <td>2.36</td>\n",
              "      <td>2.67</td>\n",
              "      <td>18.6</td>\n",
              "      <td>101.0</td>\n",
              "      <td>2.80</td>\n",
              "      <td>3.24</td>\n",
              "      <td>0.30</td>\n",
              "      <td>2.81</td>\n",
              "      <td>5.68</td>\n",
              "      <td>1.03</td>\n",
              "      <td>3.17</td>\n",
              "      <td>1185.0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>14.37</td>\n",
              "      <td>1.95</td>\n",
              "      <td>2.50</td>\n",
              "      <td>16.8</td>\n",
              "      <td>113.0</td>\n",
              "      <td>3.85</td>\n",
              "      <td>3.49</td>\n",
              "      <td>0.24</td>\n",
              "      <td>2.18</td>\n",
              "      <td>7.80</td>\n",
              "      <td>0.86</td>\n",
              "      <td>3.45</td>\n",
              "      <td>1480.0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>13.24</td>\n",
              "      <td>2.59</td>\n",
              "      <td>2.87</td>\n",
              "      <td>21.0</td>\n",
              "      <td>118.0</td>\n",
              "      <td>2.80</td>\n",
              "      <td>2.69</td>\n",
              "      <td>0.39</td>\n",
              "      <td>1.82</td>\n",
              "      <td>4.32</td>\n",
              "      <td>1.04</td>\n",
              "      <td>2.93</td>\n",
              "      <td>735.0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-d5da72d7-28de-4a9a-98ac-df657d35b39b')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-d5da72d7-28de-4a9a-98ac-df657d35b39b button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-d5da72d7-28de-4a9a-98ac-df657d35b39b');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-25f8bf77-e015-417e-b350-15d27a4995b1\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-25f8bf77-e015-417e-b350-15d27a4995b1')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-25f8bf77-e015-417e-b350-15d27a4995b1 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ]
          },
          "metadata": {},
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l8opVcVeuFLn"
      },
      "source": [
        "This dataset is made up of 13 numerical features and there are 3 different classes of wine.\n",
        "\n",
        "Now perform the train/test split and normalize the data using [`StandardScaler`](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Aaszrn9CEsIf"
      },
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "\n",
        "# Train / Test split\n",
        "X_train, X_test, y_train, y_test = train_test_split(data.data, data.target, random_state=42)\n",
        "\n",
        "# Instantiate StandardScaler\n",
        "scaler = StandardScaler()\n",
        "\n",
        "# Fit it to the train data\n",
        "scaler.fit(X_train)\n",
        "\n",
        "# Use it to transform the train and test data\n",
        "X_train = scaler.transform(X_train)\n",
        "\n",
        "# Notice that the scaler is trained on the train data to avoid data leakage from the test set\n",
        "X_test = scaler.transform(X_test)"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BVmae4rtvGiA"
      },
      "source": [
        "## Train the classifier\n",
        "\n",
        "Now you will fit a [Random Forest classifier](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) with 10 estimators and compute the mean accuracy achieved:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NK5Dxa70Ir3N",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6142f13d-f22a-432d-b0c6-36188f1449ad"
      },
      "source": [
        "from sklearn.ensemble import RandomForestClassifier\n",
        "\n",
        "# Fit the classifier\n",
        "rf_clf = RandomForestClassifier(n_estimators=10, random_state=42).fit(X_train, y_train)\n",
        "\n",
        "# Print the mean accuracy achieved by the classifier on the test set\n",
        "rf_clf.score(X_test, y_test)"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.9111111111111111"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-exiHBeJwVGh"
      },
      "source": [
        "This model achieved a mean accuracy of 91%. Pretty good for a model without fine tunning."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O_RPyB9Owfms"
      },
      "source": [
        "# Permutation Feature Importance\n",
        "\n",
        "To perform the model inspection technique known as Permutation Feature Importance you will use scikit-learn's built-in function [`permutation_importance`](https://scikit-learn.org/stable/modules/generated/sklearn.inspection.permutation_importance.html#sklearn.inspection.permutation_importance).\n",
        "\n",
        "You will create a function that given a classifier, features and labels computes the feature importance for every feature:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nAvDl_2rJsTA"
      },
      "source": [
        "from sklearn.inspection import permutation_importance\n",
        "\n",
        "def feature_importance(clf, X, y, top_limit=None):\n",
        "\n",
        "  # Retrieve the Bunch object after 50 repeats\n",
        "  # n_repeats is the number of times that each feature was permuted to compute the final score\n",
        "  bunch = permutation_importance(clf, X, y,\n",
        "                                 n_repeats=50, random_state=42)\n",
        "\n",
        "  # Average feature importance\n",
        "  imp_means = bunch.importances_mean\n",
        "\n",
        "  # List that contains the index of each feature in descending order of importance\n",
        "  ordered_imp_means_args = np.argsort(imp_means)[::-1]\n",
        "\n",
        "  # If no limit print all features\n",
        "  if top_limit is None:\n",
        "    top_limit = len(ordered_imp_means_args)\n",
        "\n",
        "  # Print relevant information\n",
        "  for i, _ in zip(ordered_imp_means_args, range(top_limit)):\n",
        "    name = data.feature_names[i]\n",
        "    imp_score = imp_means[i]\n",
        "    imp_std = bunch.importances_std[i]\n",
        "    print(f\"Feature {name} with index {i} has an average importance score of {imp_score:.3f} +/- {imp_std:.3f}\\n\")"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dCHWbK_3yeJ_"
      },
      "source": [
        "The importance score is computed in a way that higher values represent better predictive power. To know exactly how it is computed check out this [link](https://scikit-learn.org/stable/modules/permutation_importance.html#outline-of-the-permutation-importance-algorithm).\n",
        "\n",
        "Now use the `feature_importance` function on the Random Forest classifier and the train set:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wuB6EyTuHT7S",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1a370322-d7c4-4e0a-82d2-a4b7db5c2208"
      },
      "source": [
        "feature_importance(rf_clf, X_train, y_train)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Feature flavanoids with index 6 has an average importance score of 0.227 +/- 0.025\n",
            "\n",
            "Feature proline with index 12 has an average importance score of 0.142 +/- 0.019\n",
            "\n",
            "Feature color_intensity with index 9 has an average importance score of 0.112 +/- 0.023\n",
            "\n",
            "Feature od280/od315_of_diluted_wines with index 11 has an average importance score of 0.007 +/- 0.005\n",
            "\n",
            "Feature total_phenols with index 5 has an average importance score of 0.003 +/- 0.004\n",
            "\n",
            "Feature malic_acid with index 1 has an average importance score of 0.002 +/- 0.004\n",
            "\n",
            "Feature proanthocyanins with index 8 has an average importance score of 0.002 +/- 0.003\n",
            "\n",
            "Feature hue with index 10 has an average importance score of 0.002 +/- 0.003\n",
            "\n",
            "Feature nonflavanoid_phenols with index 7 has an average importance score of 0.000 +/- 0.000\n",
            "\n",
            "Feature magnesium with index 4 has an average importance score of 0.000 +/- 0.000\n",
            "\n",
            "Feature alcalinity_of_ash with index 3 has an average importance score of 0.000 +/- 0.000\n",
            "\n",
            "Feature ash with index 2 has an average importance score of 0.000 +/- 0.000\n",
            "\n",
            "Feature alcohol with index 0 has an average importance score of 0.000 +/- 0.000\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CiYP--tl5623"
      },
      "source": [
        "Looks like many of the features have a fairly low importance score. This points that the predictive power of this dataset is conmdensed in a few features.\n",
        "\n",
        "However it is important to notice that this process was done for the training set, so this feature importance does NOT have into account if the feature might help with the generalization power of the model.\n",
        "\n",
        "To check this, repeat the process for the test set:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iDjYLTDBzfXT",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5dd67c1a-57fb-49e1-fbbe-d18f7fba2eb1"
      },
      "source": [
        "feature_importance(rf_clf, X_test, y_test)"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Feature flavanoids with index 6 has an average importance score of 0.202 +/- 0.047\n",
            "\n",
            "Feature proline with index 12 has an average importance score of 0.143 +/- 0.042\n",
            "\n",
            "Feature color_intensity with index 9 has an average importance score of 0.112 +/- 0.043\n",
            "\n",
            "Feature alcohol with index 0 has an average importance score of 0.024 +/- 0.017\n",
            "\n",
            "Feature magnesium with index 4 has an average importance score of 0.021 +/- 0.015\n",
            "\n",
            "Feature od280/od315_of_diluted_wines with index 11 has an average importance score of 0.015 +/- 0.018\n",
            "\n",
            "Feature hue with index 10 has an average importance score of 0.013 +/- 0.018\n",
            "\n",
            "Feature total_phenols with index 5 has an average importance score of 0.002 +/- 0.016\n",
            "\n",
            "Feature nonflavanoid_phenols with index 7 has an average importance score of 0.000 +/- 0.000\n",
            "\n",
            "Feature alcalinity_of_ash with index 3 has an average importance score of 0.000 +/- 0.000\n",
            "\n",
            "Feature malic_acid with index 1 has an average importance score of -0.002 +/- 0.017\n",
            "\n",
            "Feature ash with index 2 has an average importance score of -0.003 +/- 0.008\n",
            "\n",
            "Feature proanthocyanins with index 8 has an average importance score of -0.021 +/- 0.020\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aclWL_oJ7h3V"
      },
      "source": [
        "Notice that the top most important features are the same for both sets. However features such as **alcohol**, which was considered not important for the training set is much more important when using the testing set. This hints that this feature will contribute to the generalization power of the model.\n",
        "\n",
        "**If a feature is deemed as important for the train set but not for the testing, this feature will probably cause the model to overfit.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DpHC8q3y8byl"
      },
      "source": [
        "## Re-train the model with the most important features\n",
        "\n",
        "Now you will re-train the Random Forest classifier with only the top 3 most important features.\n",
        "\n",
        "In this case they are the same for both sets:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "daZkt0PE1oxc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "64aafd68-7a3e-4ef5-a016-885e7396e9e1"
      },
      "source": [
        "print(\"On TRAIN split:\\n\")\n",
        "feature_importance(rf_clf, X_train, y_train, top_limit=3)\n",
        "\n",
        "print(\"\\nOn TEST split:\\n\")\n",
        "feature_importance(rf_clf, X_test, y_test, top_limit=3)"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "On TRAIN split:\n",
            "\n",
            "Feature flavanoids with index 6 has an average importance score of 0.227 +/- 0.025\n",
            "\n",
            "Feature proline with index 12 has an average importance score of 0.142 +/- 0.019\n",
            "\n",
            "Feature color_intensity with index 9 has an average importance score of 0.112 +/- 0.023\n",
            "\n",
            "\n",
            "On TEST split:\n",
            "\n",
            "Feature flavanoids with index 6 has an average importance score of 0.202 +/- 0.047\n",
            "\n",
            "Feature proline with index 12 has an average importance score of 0.143 +/- 0.042\n",
            "\n",
            "Feature color_intensity with index 9 has an average importance score of 0.112 +/- 0.043\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VOZyil7eqH5-",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "103adbd6-b88b-4e8c-898c-34bf2b16f78e"
      },
      "source": [
        "# Preserve only the top 3 features\n",
        "X_train_top_features = X_train[:,[6, 9, 12]]\n",
        "X_test_top_features = X_test[:,[6, 9, 12]]\n",
        "\n",
        "# Re-train with only these features\n",
        "rf_clf_top = RandomForestClassifier(n_estimators=10, random_state=42).fit(X_train_top_features, y_train)\n",
        "\n",
        "# Compute mean accuracy achieved\n",
        "rf_clf_top.score(X_test_top_features, y_test)"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.9333333333333333"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5CeyUCTi8zIj"
      },
      "source": [
        "Notice that by using only the 3 most important features the model achieved a mean accuracy even higher than the one using all 13 features.\n",
        "\n",
        "\n",
        "Remember that the **alcohol** feature was deemed not important in the train split but you had the hypotheses that it had important information for the generalization of the model.\n",
        "\n",
        "Add this feature and see how the model performs:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "b72FJLpj9Aly",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e5ed7a00-cee2-419b-9c03-6d3654115827"
      },
      "source": [
        "# Preserve only the top 3 features\n",
        "X_train_top_features = X_train[:,[0, 6, 9, 12]]\n",
        "X_test_top_features = X_test[:,[0, 6, 9, 12]]\n",
        "\n",
        "# Re-train with only these features\n",
        "rf_clf_top = RandomForestClassifier(n_estimators=10, random_state=42).fit(X_train_top_features, y_train)\n",
        "\n",
        "# Compute mean accuracy achieved\n",
        "rf_clf_top.score(X_test_top_features, y_test)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1.0"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L4imw5IU9nGx"
      },
      "source": [
        "Wow! By adding this additional feature you know get a mean accuracy of 100%! Quite remarkable! Looks like this feature did in fact provided some important information that helped the model do a better job at generalizing."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pGqKd5c7-Jaw"
      },
      "source": [
        "## Try out other classifiers\n",
        "\n",
        "The process of Permutation Feature Importance is also dependant on the classifier you are using. Since different classifiers follow different rules for classification it is natural to assume they will consider different features to be important or unimportant.\n",
        "\n",
        "To test this, try out other classifiers:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Hv6oXNMUrzmR",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "11873059-512e-471c-e613-71c5491bb8aa"
      },
      "source": [
        "from sklearn.svm import SVC\n",
        "from sklearn.linear_model import Lasso, Ridge\n",
        "from sklearn.tree import DecisionTreeClassifier\n",
        "\n",
        "# Select 4 new classifiers\n",
        "clfs = {\"Laso\": Lasso(alpha=0.05),\n",
        "        \"Ridge\": Ridge(),\n",
        "        \"Decision Tree\": DecisionTreeClassifier(),\n",
        "        \"Support Vector\": SVC()}\n",
        "\n",
        "\n",
        "# Compute feature importance on the test set given a classifier\n",
        "def fit_compute_importance(clf):\n",
        "  clf.fit(X_train, y_train)\n",
        "  print(f\"📏 Mean accuracy score on the test set: {clf.score(X_test, y_test)*100:.2f}%\\n\")\n",
        "  print(\"🔝 Top 4 features when using the test set:\\n\")\n",
        "  feature_importance(clf, X_test, y_test, top_limit=4)\n",
        "\n",
        "\n",
        "# Print results\n",
        "for name, clf in clfs.items():\n",
        "  print(\"=====\"*20)\n",
        "  print(f\"➡️ {name} classifier\\n\")\n",
        "  fit_compute_importance(clf)"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "====================================================================================================\n",
            "➡️ Laso classifier\n",
            "\n",
            "📏 Mean accuracy score on the test set: 86.80%\n",
            "\n",
            "🔝 Top 4 features when using the test set:\n",
            "\n",
            "Feature flavanoids with index 6 has an average importance score of 0.323 +/- 0.055\n",
            "\n",
            "Feature proline with index 12 has an average importance score of 0.203 +/- 0.035\n",
            "\n",
            "Feature od280/od315_of_diluted_wines with index 11 has an average importance score of 0.146 +/- 0.030\n",
            "\n",
            "Feature alcalinity_of_ash with index 3 has an average importance score of 0.038 +/- 0.014\n",
            "\n",
            "====================================================================================================\n",
            "➡️ Ridge classifier\n",
            "\n",
            "📏 Mean accuracy score on the test set: 88.71%\n",
            "\n",
            "🔝 Top 4 features when using the test set:\n",
            "\n",
            "Feature flavanoids with index 6 has an average importance score of 0.445 +/- 0.071\n",
            "\n",
            "Feature proline with index 12 has an average importance score of 0.210 +/- 0.035\n",
            "\n",
            "Feature color_intensity with index 9 has an average importance score of 0.119 +/- 0.029\n",
            "\n",
            "Feature od280/od315_of_diluted_wines with index 11 has an average importance score of 0.111 +/- 0.026\n",
            "\n",
            "====================================================================================================\n",
            "➡️ Decision Tree classifier\n",
            "\n",
            "📏 Mean accuracy score on the test set: 93.33%\n",
            "\n",
            "🔝 Top 4 features when using the test set:\n",
            "\n",
            "Feature flavanoids with index 6 has an average importance score of 0.297 +/- 0.061\n",
            "\n",
            "Feature color_intensity with index 9 has an average importance score of 0.206 +/- 0.046\n",
            "\n",
            "Feature proline with index 12 has an average importance score of 0.198 +/- 0.038\n",
            "\n",
            "Feature malic_acid with index 1 has an average importance score of 0.008 +/- 0.012\n",
            "\n",
            "====================================================================================================\n",
            "➡️ Support Vector classifier\n",
            "\n",
            "📏 Mean accuracy score on the test set: 97.78%\n",
            "\n",
            "🔝 Top 4 features when using the test set:\n",
            "\n",
            "Feature proline with index 12 has an average importance score of 0.069 +/- 0.031\n",
            "\n",
            "Feature flavanoids with index 6 has an average importance score of 0.061 +/- 0.023\n",
            "\n",
            "Feature alcohol with index 0 has an average importance score of 0.044 +/- 0.023\n",
            "\n",
            "Feature ash with index 2 has an average importance score of 0.032 +/- 0.018\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0vLoU0xjDsqg"
      },
      "source": [
        "Looks like **flavanoids** and **proline** are very important across all classifiers. However there is variability from one classifier to the others on what features are considered the most important ones."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cwRlO14nZ_nY"
      },
      "source": [
        "-----------------------------\n",
        "**Congratulations on finishing this ungraded lab!** Now you should have a clearer understanding of what Permutation Feature Importance is, why it is useful and how to implement this technique using scikit-learn.\n",
        "\n",
        "**Keep it up!**"
      ]
    }
  ]
}