05-logistic-regression/1-statistical-parameters.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "35869318",
   "metadata": {},
   "source": [
    "# Statistical parameters\n",
    "\n",
    "[Machine Learning Interpretability course](https://www.trainindata.com/p/machine-learning-interpretability)\n",
    "\n",
    "In this notebook, we will see how to calculate the different functions and statistical parameters that are relevant for logistic regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b26734c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1dac32e0",
   "metadata": {},
   "source": [
    "## Logistic function\n",
    "\n",
    "Change the values of z to understand the output of the logistic function. The value of z is the result of the linear model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f53d9eab",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.05, 0.27, 0.5 , 0.88, 0.95])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# z is the linear equation\n",
    "\n",
    "z = np.array([-3, -1, 0, 2, 3])\n",
    "\n",
    "# the logistic function\n",
    "\n",
    "def sigmoid(z):\n",
    "    \n",
    "    s = 1/(1+np.exp(-z))\n",
    "    \n",
    "    return np.round(s, 2)\n",
    "\n",
    "sigmoid(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18515b75",
   "metadata": {},
   "source": [
    "## Cost function\n",
    "\n",
    "What we are trying to minimize when finding the coefficients for the logistic function.\n",
    "\n",
    "The cost function takes one form when y is 1 and a different form when y is 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "9bd6b360",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.2039728 , 0.69314718, 0.22314355])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# when y = 1\n",
    "\n",
    "p = np.array([0.3, 0.5, 0.8])\n",
    "\n",
    "-np.log(p)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ac251157",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.2039728 , 0.51082562, 0.22314355])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# when y = 0\n",
    "\n",
    "p = np.array([0.7, 0.4, 0.2])\n",
    "\n",
    "-np.log(1-p)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2fb569c",
   "metadata": {},
   "source": [
    "## Log-loss\n",
    "\n",
    "The log-loss is another expression of the cost function, where the 2 components are put together, so we sum all the logs of the probabilities and then divide by -1/m, where m is the number of observations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "023e6546",
   "metadata": {},
   "outputs": [],
   "source": [
    "# target variable\n",
    "y = np.array([0, 0, 1, 1, 1])\n",
    "\n",
    "# Baseline: mean of y\n",
    "p_b = np.array([0.6, 0.6, 0.6, 0.6, 0.6])\n",
    "\n",
    "# Model\n",
    "p_l = np.array([0.2, 0.3, 0.6, 0.7, 0.8])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "5f094896",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import log_loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "69d43555",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6730116670092563"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "log_loss(y, p_b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "2ac9d578",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.334092522854375"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "log_loss(y, p_l)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbad8c65",
   "metadata": {},
   "source": [
    "## Log-likelihood\n",
    "\n",
    "The sum of the log of the probabilities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "311e6660",
   "metadata": {},
   "outputs": [],
   "source": [
    "def ll(p):\n",
    "    return np.sum(y*np.log(p) + (1-y)*np.log(1-p))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "da3de06c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-3.3650583350462817"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ll(p_b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "a91445d0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-1.670462614271875"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ll(p_l)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "513cd1e7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-3.3650583350462817"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# or with sklearn\n",
    "\n",
    "- log_loss(y, p_b, normalize=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "2a27d488",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-1.670462614271875"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# or with sklearn\n",
    "\n",
    "- log_loss(y, p_l, normalize=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a0e161c",
   "metadata": {},
   "source": [
    "## Deviance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "8f3eeec1",
   "metadata": {},
   "outputs": [],
   "source": [
    "def deviance(y_true, y_pred):\n",
    "    return 2 * log_loss(y_true, y_pred, normalize=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "dedb5d8c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6.730116670092563"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "deviance(y, p_b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "bd7b95e3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.34092522854375"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "deviance(y, p_l)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b134a80",
   "metadata": {},
   "source": [
    "## p-value\n",
    "\n",
    "The deviance follows a chi-square distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "c0df9393",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.06562512323458003"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from scipy import stats\n",
    "\n",
    "x2 = deviance(y, p_b) - deviance(y, p_l)\n",
    "\n",
    "df = 1\n",
    "\n",
    "1 - stats.chi2.cdf(x2, df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bde73947",
   "metadata": {},
   "source": [
    "## R-statistic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "b90a2d21",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5035858377626312"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "R = (deviance(y, p_b) - deviance(y, p_l) )/ deviance(y, p_b)\n",
    "\n",
    "R"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f2f4457",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "fsml",
   "language": "python",
   "name": "fsml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}