05-logistic-regression/4-assessing-coefficients-bootstrapping.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5e9e49bd",
   "metadata": {},
   "source": [
    "# Assessing coefficients with bootstrapping\n",
    "\n",
    "[Machine Learning Interpretability course](https://www.trainindata.com/p/machine-learning-interpretability)\n",
    "\n",
    "In this notebook, we will use bootstrapping to determine the error of the coefficients.\n",
    "\n",
    "if the coefficients are significant, then the contribution of the feature towards the probability is meaningful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b619bd11",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "from scipy import stats\n",
    "\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "750e70b0",
   "metadata": {},
   "source": [
    "### Load data\n",
    "\n",
    "To obtain the data, check the folder `prepare-data` in this repo, or section 2 of the course."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "050acc05",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load titanic dataset\n",
    "\n",
    "df = pd.read_csv('../titanic.csv')\n",
    "\n",
    "# split data\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    df.drop(\"survived\", axis=1), \n",
    "    df[\"survived\"],\n",
    "    test_size=0.15,\n",
    "    random_state=1,\n",
    ")\n",
    "\n",
    "# scale the variables\n",
    "scaler = StandardScaler().set_output(transform=\"pandas\")\n",
    "\n",
    "X_train = scaler.fit_transform(X_train)\n",
    "X_test = scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adfae3e9",
   "metadata": {},
   "source": [
    "## Bootstrapping with Scikit-learn"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94c955a4",
   "metadata": {},
   "source": [
    "In the linear regression section, we used cross-validation and different partitions of the train set to estimate the error of the coefficients. Those are valid approaches and you can use them for logistic regression as well.\n",
    "\n",
    "Here, I will introduce another alternative to estimate the coefficients error: **bootstrapping**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5ed6b7b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train 50 models on different bootstrapped\n",
    "# partitions of the train set\n",
    "\n",
    "s = dict()\n",
    "\n",
    "for i in np.linspace(1, 50, num=50):\n",
    "\n",
    "    # bootstrap with replacement\n",
    "    X_train_b = X_train.sample(frac=0.8, replace=True, random_state=int(i))\n",
    "    y_train_b = y_train.loc[X_train_b.index]\n",
    "\n",
    "    # train model\n",
    "    logit = LogisticRegression(\n",
    "        penalty=None, random_state=0).fit(X_train_b, y_train_b)\n",
    "\n",
    "    # store coefficients\n",
    "    s[str(int(i))] = pd.Series(logit.coef_[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "25b0684e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>...</th>\n",
       "      <th>41</th>\n",
       "      <th>42</th>\n",
       "      <th>43</th>\n",
       "      <th>44</th>\n",
       "      <th>45</th>\n",
       "      <th>46</th>\n",
       "      <th>47</th>\n",
       "      <th>48</th>\n",
       "      <th>49</th>\n",
       "      <th>50</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>pclass</th>\n",
       "      <td>-0.534509</td>\n",
       "      <td>-0.624764</td>\n",
       "      <td>-0.351549</td>\n",
       "      <td>-0.438266</td>\n",
       "      <td>-0.512061</td>\n",
       "      <td>-0.440676</td>\n",
       "      <td>-0.578593</td>\n",
       "      <td>-0.472116</td>\n",
       "      <td>-0.615872</td>\n",
       "      <td>-0.542379</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.438867</td>\n",
       "      <td>-0.382899</td>\n",
       "      <td>-0.621534</td>\n",
       "      <td>-0.416289</td>\n",
       "      <td>-0.466371</td>\n",
       "      <td>-0.528710</td>\n",
       "      <td>-0.393777</td>\n",
       "      <td>-0.406558</td>\n",
       "      <td>-0.589398</td>\n",
       "      <td>-0.394367</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sibsp</th>\n",
       "      <td>-0.064679</td>\n",
       "      <td>-0.177056</td>\n",
       "      <td>-0.328538</td>\n",
       "      <td>-0.077019</td>\n",
       "      <td>-0.084303</td>\n",
       "      <td>-0.406636</td>\n",
       "      <td>-0.178394</td>\n",
       "      <td>-0.083164</td>\n",
       "      <td>-0.206828</td>\n",
       "      <td>-0.327789</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.171062</td>\n",
       "      <td>-0.243986</td>\n",
       "      <td>-0.297647</td>\n",
       "      <td>-0.402791</td>\n",
       "      <td>-0.336380</td>\n",
       "      <td>-0.346853</td>\n",
       "      <td>-0.140565</td>\n",
       "      <td>-0.400153</td>\n",
       "      <td>-0.185640</td>\n",
       "      <td>-0.285593</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>parch</th>\n",
       "      <td>0.015369</td>\n",
       "      <td>0.071580</td>\n",
       "      <td>0.062171</td>\n",
       "      <td>-0.032246</td>\n",
       "      <td>-0.005300</td>\n",
       "      <td>0.032669</td>\n",
       "      <td>0.179450</td>\n",
       "      <td>0.000523</td>\n",
       "      <td>-0.147484</td>\n",
       "      <td>0.049313</td>\n",
       "      <td>...</td>\n",
       "      <td>0.096691</td>\n",
       "      <td>0.301907</td>\n",
       "      <td>0.024190</td>\n",
       "      <td>0.140074</td>\n",
       "      <td>0.053208</td>\n",
       "      <td>-0.002931</td>\n",
       "      <td>0.073487</td>\n",
       "      <td>0.099207</td>\n",
       "      <td>0.100552</td>\n",
       "      <td>0.050043</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sex_female</th>\n",
       "      <td>1.047560</td>\n",
       "      <td>1.216410</td>\n",
       "      <td>1.220586</td>\n",
       "      <td>1.208595</td>\n",
       "      <td>1.168361</td>\n",
       "      <td>1.304313</td>\n",
       "      <td>1.194548</td>\n",
       "      <td>1.290748</td>\n",
       "      <td>1.190516</td>\n",
       "      <td>1.244512</td>\n",
       "      <td>...</td>\n",
       "      <td>1.166265</td>\n",
       "      <td>1.081790</td>\n",
       "      <td>1.183863</td>\n",
       "      <td>1.295611</td>\n",
       "      <td>1.251984</td>\n",
       "      <td>1.338715</td>\n",
       "      <td>1.173924</td>\n",
       "      <td>1.102490</td>\n",
       "      <td>1.361943</td>\n",
       "      <td>1.141719</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>embarked_S</th>\n",
       "      <td>0.074015</td>\n",
       "      <td>0.135938</td>\n",
       "      <td>0.141950</td>\n",
       "      <td>0.168767</td>\n",
       "      <td>-0.015130</td>\n",
       "      <td>0.201774</td>\n",
       "      <td>-0.066315</td>\n",
       "      <td>0.039502</td>\n",
       "      <td>0.050337</td>\n",
       "      <td>-0.025995</td>\n",
       "      <td>...</td>\n",
       "      <td>0.009892</td>\n",
       "      <td>-0.134835</td>\n",
       "      <td>0.180129</td>\n",
       "      <td>-0.044966</td>\n",
       "      <td>0.145100</td>\n",
       "      <td>0.075629</td>\n",
       "      <td>-0.089000</td>\n",
       "      <td>-0.084067</td>\n",
       "      <td>0.014126</td>\n",
       "      <td>-0.034973</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 50 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                   1         2         3         4         5         6  \\\n",
       "pclass     -0.534509 -0.624764 -0.351549 -0.438266 -0.512061 -0.440676   \n",
       "sibsp      -0.064679 -0.177056 -0.328538 -0.077019 -0.084303 -0.406636   \n",
       "parch       0.015369  0.071580  0.062171 -0.032246 -0.005300  0.032669   \n",
       "sex_female  1.047560  1.216410  1.220586  1.208595  1.168361  1.304313   \n",
       "embarked_S  0.074015  0.135938  0.141950  0.168767 -0.015130  0.201774   \n",
       "\n",
       "                   7         8         9        10  ...        41        42  \\\n",
       "pclass     -0.578593 -0.472116 -0.615872 -0.542379  ... -0.438867 -0.382899   \n",
       "sibsp      -0.178394 -0.083164 -0.206828 -0.327789  ... -0.171062 -0.243986   \n",
       "parch       0.179450  0.000523 -0.147484  0.049313  ...  0.096691  0.301907   \n",
       "sex_female  1.194548  1.290748  1.190516  1.244512  ...  1.166265  1.081790   \n",
       "embarked_S -0.066315  0.039502  0.050337 -0.025995  ...  0.009892 -0.134835   \n",
       "\n",
       "                  43        44        45        46        47        48  \\\n",
       "pclass     -0.621534 -0.416289 -0.466371 -0.528710 -0.393777 -0.406558   \n",
       "sibsp      -0.297647 -0.402791 -0.336380 -0.346853 -0.140565 -0.400153   \n",
       "parch       0.024190  0.140074  0.053208 -0.002931  0.073487  0.099207   \n",
       "sex_female  1.183863  1.295611  1.251984  1.338715  1.173924  1.102490   \n",
       "embarked_S  0.180129 -0.044966  0.145100  0.075629 -0.089000 -0.084067   \n",
       "\n",
       "                  49        50  \n",
       "pclass     -0.589398 -0.394367  \n",
       "sibsp      -0.185640 -0.285593  \n",
       "parch       0.100552  0.050043  \n",
       "sex_female  1.361943  1.141719  \n",
       "embarked_S  0.014126 -0.034973  \n",
       "\n",
       "[5 rows x 50 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# put coefficients in a dataframe\n",
    "\n",
    "df = pd.concat(s, axis=1)\n",
    "df.index = logit.feature_names_in_\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "8f625002",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>pclass</th>\n",
       "      <td>-0.515202</td>\n",
       "      <td>0.117951</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sibsp</th>\n",
       "      <td>-0.240314</td>\n",
       "      <td>0.101813</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>parch</th>\n",
       "      <td>0.059448</td>\n",
       "      <td>0.097416</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sex_female</th>\n",
       "      <td>1.218624</td>\n",
       "      <td>0.091900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>embarked_S</th>\n",
       "      <td>0.012063</td>\n",
       "      <td>0.130206</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>embarked_C</th>\n",
       "      <td>0.255133</td>\n",
       "      <td>0.127116</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cabin_B</th>\n",
       "      <td>-0.287461</td>\n",
       "      <td>0.490073</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cabin_C</th>\n",
       "      <td>-0.444941</td>\n",
       "      <td>0.593502</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cabin_E</th>\n",
       "      <td>-0.067307</td>\n",
       "      <td>0.418947</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cabin_D</th>\n",
       "      <td>-0.247478</td>\n",
       "      <td>0.446451</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cabin_A</th>\n",
       "      <td>-0.035592</td>\n",
       "      <td>0.488525</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cabin_M</th>\n",
       "      <td>-0.792789</td>\n",
       "      <td>0.966033</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cabin_G</th>\n",
       "      <td>-0.258165</td>\n",
       "      <td>0.564198</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                mean       std\n",
       "pclass     -0.515202  0.117951\n",
       "sibsp      -0.240314  0.101813\n",
       "parch       0.059448  0.097416\n",
       "sex_female  1.218624  0.091900\n",
       "embarked_S  0.012063  0.130206\n",
       "embarked_C  0.255133  0.127116\n",
       "cabin_B    -0.287461  0.490073\n",
       "cabin_C    -0.444941  0.593502\n",
       "cabin_E    -0.067307  0.418947\n",
       "cabin_D    -0.247478  0.446451\n",
       "cabin_A    -0.035592  0.488525\n",
       "cabin_M    -0.792789  0.966033\n",
       "cabin_G    -0.258165  0.564198"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Summarize variability of coefficients\n",
    "\n",
    "coeff_summary = df.agg([\"mean\", \"std\"], axis=1)\n",
    "\n",
    "coeff_summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "ab5d741a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>z</th>\n",
       "      <th>p_values</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>pclass</th>\n",
       "      <td>-0.515202</td>\n",
       "      <td>0.117951</td>\n",
       "      <td>-4.367951</td>\n",
       "      <td>1.254175e-05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sibsp</th>\n",
       "      <td>-0.240314</td>\n",
       "      <td>0.101813</td>\n",
       "      <td>-2.360338</td>\n",
       "      <td>1.825831e-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>parch</th>\n",
       "      <td>0.059448</td>\n",
       "      <td>0.097416</td>\n",
       "      <td>0.610250</td>\n",
       "      <td>5.416963e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sex_female</th>\n",
       "      <td>1.218624</td>\n",
       "      <td>0.091900</td>\n",
       "      <td>13.260308</td>\n",
       "      <td>3.932762e-40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>embarked_S</th>\n",
       "      <td>0.012063</td>\n",
       "      <td>0.130206</td>\n",
       "      <td>0.092647</td>\n",
       "      <td>9.261837e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>embarked_C</th>\n",
       "      <td>0.255133</td>\n",
       "      <td>0.127116</td>\n",
       "      <td>2.007094</td>\n",
       "      <td>4.473961e-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cabin_B</th>\n",
       "      <td>-0.287461</td>\n",
       "      <td>0.490073</td>\n",
       "      <td>-0.586567</td>\n",
       "      <td>5.574943e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cabin_C</th>\n",
       "      <td>-0.444941</td>\n",
       "      <td>0.593502</td>\n",
       "      <td>-0.749686</td>\n",
       "      <td>4.534437e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cabin_E</th>\n",
       "      <td>-0.067307</td>\n",
       "      <td>0.418947</td>\n",
       "      <td>-0.160658</td>\n",
       "      <td>8.723629e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cabin_D</th>\n",
       "      <td>-0.247478</td>\n",
       "      <td>0.446451</td>\n",
       "      <td>-0.554322</td>\n",
       "      <td>5.793587e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cabin_A</th>\n",
       "      <td>-0.035592</td>\n",
       "      <td>0.488525</td>\n",
       "      <td>-0.072857</td>\n",
       "      <td>9.419199e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cabin_M</th>\n",
       "      <td>-0.792789</td>\n",
       "      <td>0.966033</td>\n",
       "      <td>-0.820664</td>\n",
       "      <td>4.118375e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cabin_G</th>\n",
       "      <td>-0.258165</td>\n",
       "      <td>0.564198</td>\n",
       "      <td>-0.457578</td>\n",
       "      <td>6.472557e-01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                mean       std          z      p_values\n",
       "pclass     -0.515202  0.117951  -4.367951  1.254175e-05\n",
       "sibsp      -0.240314  0.101813  -2.360338  1.825831e-02\n",
       "parch       0.059448  0.097416   0.610250  5.416963e-01\n",
       "sex_female  1.218624  0.091900  13.260308  3.932762e-40\n",
       "embarked_S  0.012063  0.130206   0.092647  9.261837e-01\n",
       "embarked_C  0.255133  0.127116   2.007094  4.473961e-02\n",
       "cabin_B    -0.287461  0.490073  -0.586567  5.574943e-01\n",
       "cabin_C    -0.444941  0.593502  -0.749686  4.534437e-01\n",
       "cabin_E    -0.067307  0.418947  -0.160658  8.723629e-01\n",
       "cabin_D    -0.247478  0.446451  -0.554322  5.793587e-01\n",
       "cabin_A    -0.035592  0.488525  -0.072857  9.419199e-01\n",
       "cabin_M    -0.792789  0.966033  -0.820664  4.118375e-01\n",
       "cabin_G    -0.258165  0.564198  -0.457578  6.472557e-01"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate z and the p-values\n",
    "\n",
    "coeff_summary[\"z\"] = coeff_summary[\"mean\"] / coeff_summary[\"std\"]\n",
    "\n",
    "coeff_summary[\"p_values\"] = stats.norm.sf(abs(coeff_summary[\"z\"]))*2  # two sided\n",
    "\n",
    "coeff_summary"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a249e73b",
   "metadata": {},
   "source": [
    "Experimentally, we have larger errors, and therefore, the significance is smaller. \n",
    "\n",
    "This is probably because the variables are not completely independent, that is, there is some colinearity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "af8250bc",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "fsml",
   "language": "python",
   "name": "fsml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}