lecture_5_probability.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create anaconda environment\n",
    "<br>\n",
    "```bash\n",
    "conda create -n ml python=3.7.4 jupyter\n",
    "```\n",
    "Install fastai library\n",
    "<br>\n",
    "```bash\n",
    "conda install -c pytorch -c fastai fastai\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sigma-algebras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let $X$ be a set and $\\mathcal{F} \\subset 2^X$ with properties:\n",
    "- $X \\in \\mathcal{F}$\n",
    "- if $A \\in \\mathcal{F}$ then $X - A \\in \\mathcal{F}$\n",
    "- for every $(A_n)_{n = 1}^{\\infty}$ such that $A_n \\in \\mathcal{F}$ for any $1 \\le n \\le \\infty$: $\\bigcup_{n=1}^{\\infty}A_n \\in \\mathcal{F}$\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The $\\mathcal{F}$ with these properties is called the <b>$\\sigma$-algebra</b> on X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the above properties we can easily conclude that for every $\\sigma$-algebra $\\mathcal{F}$ on the set $X$:\n",
    "- $\\emptyset \\in \\mathcal{F}$\n",
    "- for every $(A_n)_{n = 1}^{\\infty}$ such that $A_n \\in \\mathcal{F}$ for any $1 \\le n \\le \\infty$: $\\bigcap_{n=1}^{\\infty}A_n \\in \\mathcal{F}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For every set $X$ the \"smallest\" $\\sigma$-algebra will be $\\{X, \\emptyset\\}$ and the biggest will be - $2^X$\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The elements of the $\\sigma$-algebra are called measurable sets and the pair $(X, \\mathcal{F})$ is called measurable space"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function $f:(X, \\mathcal{F}_X) \\to (Y, \\mathcal{F}_Y)$ between two measurable spaces is called measurable function if for every $F \\in \\mathcal{F}_Y$ the set $f^{-1}(F) \\in \\mathcal{F}_X$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Measure"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let $(X, \\mathcal{F})$ be a measurable space, the function $\\mu:\\mathcal{F} \\to [0, \\infty]$ is called <b>measure</b> if:\n",
    "- $\\mu(\\emptyset) = 0$\n",
    "- For pairwise disjoint sets $(E_k)_{k=1}^\\infty$ the following holds: $\\mu\\left(\\bigcup_{k=1}^\\infty E_k\\right)=\\sum_{k=1}^\\infty \\mu(E_k)$\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Properties:\n",
    "- if $A \\subset B$ then $\\mu(A) \\le \\mu(B)$\n",
    "<br>\n",
    "Proof: $(B - A) \\cup (A \\cap B) = (B - A) \\cup A = B$ and $(B - A) \\cap (A \\cap B) = \\emptyset$ so $\\mu((B - A) \\cup (A \\cap B)) = \\mu(B - A) + \\mu(A \\cap B) = \\mu(B - A) + \\mu(A) = \\mu(B)$ so $\\mu(A) \\le \\mu(B)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The triple $(X, \\mathcal{F}, \\mu)$ is called measure space"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Probability"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define $\\sigma$-algebra on the sample space $\\mathcal{F} \\subset 2^{\\Omega}$ and measure $P:\\mathcal{F} \\subset 2^{\\Omega} \\to [0, 1] \\subset \\mathbb{R}_+$ measure"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The measure $P:\\mathcal{F} \\subset 2^{\\Omega} \\to [0, \\infty]$ on $\\sigma$-algebra is called probability if $P(\\Omega) = 1$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Properties:\n",
    "- $P(\\emptyset) = 0$\n",
    "- if $A \\subset B$ then $P(A) \\le P(B)$\n",
    "- $0 \\le P(A) \\le 1$\n",
    "- $P(A^c) = 1 - P(A)$ where $A^{c} = \\Omega - A$\n",
    "- $P(A \\cup B) = P(A) + P(B) - P(A \\cap B)$\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For $A \\subset \\Omega$ and $B \\subset \\Omega$ events, we can assume that:\n",
    "<br>\n",
    "$A \\cap B$ is interpreted as $A$ and $B$ events simultaneously and sometimes we denote $AB$ \n",
    "<br>\n",
    "$A \\cup B$ mean $A$ or $B$ events"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example (Uniform probability distribution):\n",
    "Let $\\Omega$ is finite set of samples and let the probability of each outcome equally likely, then for $A \\subset \\Omega$ \n",
    "$$\n",
    "\\frac{|A|}{|\\Omega|}\n",
    "$$\n",
    "<br>\n",
    "For instance, let $\\Omega$ be a toss of a coin, then we have $\\Omega = \\{H, T\\}$ and $|\\Omega| = 2$ and $|\\{H\\}| = 1$, so the probability of head $P(H) = \\frac{|A|}{|\\Omega|} = \\frac{1}{2}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Independent events:\n",
    "\n",
    "If we toss the coin twice, then the probability of two heads is \n",
    "$$P(\\{H, H\\}) = \\frac{1}{2} \\cdot \\frac{1}{2} = \\frac{1}{4}$$\n",
    "<br>\n",
    "#### Definition:\n",
    "Two events $A$ and $B$ are independent if\n",
    "$$P(A \\cap B) = P(A)P(B)$$\n",
    "<br>\n",
    "A set of events $(A_i)_{i \\in I}$ are independent if\n",
    "$$P(\\bigcap_{i \\in I}A_i) = \\prod_{i\\in I}P(A_i)$$\n",
    "<br>\n",
    "We can assume, that two events are independent, for instance when we tossing the coins, we know that coins does not have a memory and thus each toss is independent for others.\n",
    "On the other hand, we can prove that $P(AB) = P(A)P(B)$ and conclude that $A$ and $B$ are independent.\n",
    "<br>\n",
    "Let $\\Omega$ be a tossing the fair dice and let $A = \\{2, 4, 6\\}$ and $B = \\{1, 2, 3, 4\\}$ then $AB = A \\cap B = \\{2, 4\\}$ and $P(AB) = \\frac{2}{6} = \\frac{1}{3}$. But $P(A) = \\frac{3}{6}$ and $P(B) = \\frac{4}{6}$ which iplies $P(AB) = \\frac{1}{3}$ by which we can conclude that this events are independent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Conditional probability:\n",
    "If $P(B) \\gt 0$ then conditional probability of $A$ given $B$ is:\n",
    "$$P(A|B) = \\frac{P(AB)}{P(B)}$$\n",
    "<br>\n",
    "For fixed $B \\subset \\Omega$ such that $P(B) \\gt 0$ let define function $P_B = P(.|B):\\Omega \\to \\mathbb{R}_+$, then $P_B$ is probability measure on $2^{\\Omega}$\n",
    "<br>\n",
    "- $P(\\Omega|B) = 1$ ir contains $P(B|B) = 1$\n",
    "- $P(\\bigcup_{i=1}^{n}A_i|B) = \\sum_{i=1}^{n}P(A_i|B)$ for disjoint sets $(A_i)_{i=1}^{n}$\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example: \n",
    "Let we have a medical desease $D$ test with outcomes $+$ and $-$:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table>\n",
    "    <tr>\n",
    "        <td>\n",
    "        </td>\n",
    "        <td>\n",
    "        $D$\n",
    "        </td>\n",
    "        <td>\n",
    "        $D^c$\n",
    "        </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>\n",
    "        $+$\n",
    "        </td>\n",
    "        <td>\n",
    "        $0.0081$\n",
    "        </td>\n",
    "        <td>\n",
    "        $0.0900$\n",
    "        </td>\n",
    "    <tr>\n",
    "        <td>\n",
    "        $-$\n",
    "        </td>\n",
    "        <td>\n",
    "        $0.0009$\n",
    "        </td>\n",
    "        <td>\n",
    "        $0.9010$\n",
    "        </td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then:\n",
    "$$P(+|D) = \\frac{P(+, D)}{P(D)} = \\frac{0.0081}{0.0081 + 0.0009} = 0.9$$\n",
    "<br>\n",
    "$$P(-|D^c) = \\frac{P(-, D^c)}{P(D^c)} = \\frac{0.9010}{0.0900 + 0.9010} \\approx 0.9$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.9, 0.9091826437941474)"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p1 = (0.0081) / (0.0081 + 0.0009)\n",
    "p2 = (0.9010) / (0.0900 + 0.9010)\n",
    "p1, p2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The test is accurate, sick people get positive with $0.9$ probability or $90%$ accuracy results as well as healthy people get negative result with $0.9$ probability or $90%$ accuracy.\n",
    "<br>\n",
    "The question is, if some person got test result as positive ($+$), what is the probability that they have disease?\n",
    "<br>\n",
    "The first answer is $0.9$ but it is wrong."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "P(D|+) = \\frac{P(D, +)}{P(+)} = \\frac{0.081}{0.081 + 0.0900} = \\frac{0.081}{0.0981} \\approx 0.48\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4736842105263158"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p3 = (0.081) / (0.081 + 0.0900)\n",
    "p3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If $A$ and $B$ are independent events, then \n",
    "$$P(A|B) = \\frac{P(AB)}{P(B)} = \\frac{P(A)P(B)}{P(B)} = P(A)$$\n",
    "<br>\n",
    "and if\n",
    "$$P(A|B) = P(A)$$\n",
    "$A$ and $B$ are independent events"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bayes' theorem:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Partition of set $X$ on subsets $\\{A_i \\subset X|I \\in I\\}$ is $X = \\bigcup_{i \\in I}A_i$ and $A_i \\cap A_j = \\emptyset$ for every pair $i,j \\in I$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each subset $A \\subset X$ we have partition $X = A \\cup A^c$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Total probability theorem:\n",
    "For every partition $A_1, A_2 \\dots, A_k$ of $\\Omega$ and event $B \\subset \\Omega$:\n",
    "$$P(B) = \\sum_{i=1}^{k}P(B|A_i)P(A_i)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Theorem (Bayes' theorem):\n",
    "Let $A_1, A_2 \\dots, A_k$ be a partition of $\\Omega$ such that $P(A_i) > 0$ for each $i \\in \\{1, 2, \\dots, k\\}$. Then for $B \\subset \\Omega$ event, such that $P(B) > 0$, for each $i \\in \\{1, 2, \\dots, k\\}$:\n",
    "$$\n",
    "P(A_i|B) = \\frac{P(B|A_i)P(A_i)}{\\sum_{j=1}^{k}P(B|A_j)P(A_j)}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Note:\n",
    "We call $P(A_i)$ the prior probability and $P(A_i|B)$ the posterior probability"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the events $A$ and $B$ such that $P(B) \\gt 0$ we have:\n",
    "$$\n",
    "P(A|B) = \\frac{P(B|A)P(A)}{P(B)}\n",
    "$$\n",
    "<br>\n",
    "We can consider the partition of $\\Omega$ on $A$ and $A^c$, the from Byes' theorem we have:\n",
    "$$\n",
    "P(A|B) = \\frac{P(B|A)P(A)}{P(B|A)P(A) + P(B|A^c)P(A^c)} = \\text{(by the total probability) }\\frac{P(B|A)P(A)}{P(B)}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example:\n",
    "Divide emails $A_1 = \\text{\"spam\"}$, $A_2 = \\text{\"low priority\"}$ and $A_3 = \\text{\"high priority\"}$ and let: $P(A_1) = 0.7$, $P(A_2) = 0.2$ and $P(A_3) = 0.1$. ($P(A_1) + P(A_2) + P(A_3) = 0.7 + 0.2 + 0.1 = 1$)\n",
    "<br>\n",
    "Let $B$ be the event that email contains the word \"free\" and we know from previous experience that: $P(B|A_1) = 0.9$, $P(B|A_2) = 0.01$ and $P(B|A_3) = 0.01$.\n",
    "<br>\n",
    "If we receive the email with word \"free\" in it, what is the probability, that this email is spam?\n",
    "From Bayes' theorem:\n",
    "$$\n",
    "P(A_1|B) = \\frac{P(B|A_1)P(A_1)}{P(B|A_1)P(A_1) + P(B|A_2)P(A_2) + P(B|A_3)P(A_3)} = \\frac{0.9 \\cdot 0.7}{0.9 \\cdot 0.7 + 0.01 \\cdot 0.2 + 0.01 \\cdot .01} = 0.995\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Random variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Random variable is measurable mapping $X:\\Omega \\to \\mathbb{R}$ which assigns number to each outcome.\n",
    "<br>\n",
    "For instance $X:\\{H, T\\} \\to \\{0, 1\\}$\n",
    "For randoem variable $X$ define:\n",
    "$$\n",
    "P(X \\in A) = P(X^{-1}(A)) = P(\\{\\omega \\in \\Omega |X(\\omega) \\in A\\})\n",
    "$$\n",
    "<br>\n",
    "$$\n",
    "P(X = x) = P(X^{-1}(x)) = P(\\{\\omega \\in \\Omega |X(\\omega) = x\\})\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The cumulative distribution function:\n",
    "$$F_X:\\mathbb{R} \\to [0, 1]$$\n",
    "of random variable $X$ is defined by\n",
    "$$\n",
    "F_X(x) = P(X \\le x)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Theorem:\n",
    "A function $F:\\mathbb{R} \\to [0, 1]$ is CDF if and only if:\n",
    "- $F$ is non decreasing: for each $x_1 \\lt x_2$ we have $F(x_1) \\le F(x_2)$\n",
    "- $F$ is normalized: $\\lim_{x \\to -\\infty}F(x) = 0$ and $\\lim_{x \\to \\infty}F(x) = 1$\n",
    "- $F$ is rights continuous"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Definition:\n",
    "$X$ is discrete if it takes countably many values:\n",
    "$$\\{x_1, x_2, \\dots\\}$$\n",
    "<br>\n",
    "We define probability mass function:\n",
    "$f_X(x)=P(X=x)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the above we have:\n",
    "- $f_X(x) \\ge 0$ for each $x \\in \\mathbb{R}$\n",
    "- $\\sum_{x}f_X(x) = 1$\n",
    "<br>\n",
    "$$\n",
    "F_X(x) = P(X \\le x) = \\sum_{x_i \\le x}f_X(x_i)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Definition:\n",
    "$X$ is continuous if there exists a function $f_X:\\mathbb{R} \\to \\mathbb{R}$ such that:\n",
    "$$f_X(x) \\ge 0 \\text{ for all} x \\in \\mathbb{R}$$\n",
    "$$P(a \\lt x \\lt b) = \\int_{a}^{b}f_X(x)dx$$\n",
    "<br>\n",
    "The function $f_X$ is called probability density function PDF and we have that \n",
    "$$F_X(x)=\\int_{-\\infty}^{x}f_X(t)dt$$\n",
    "and\n",
    "$$f_X(x) = F_X'(x) \\text{ for all points } x \\in \\mathbb{R} \\text{ where } F_X \\text{ is differentiable}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If $X$ continuous random variable, then $P(X = x) = 0$ for each $x \\in \\mathbb{R}$ so $f(x)$ does not mean $P(X = x)$, this only hold in case of discrete variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Some discrete distributions:\n",
    "\n",
    "- Discrete uniform $X \\sim \\operatorname{Uniform}(k)$:\n",
    "$$f(x) = \n",
    "\\begin{cases}\n",
    "\\frac{1}{k}, & \\text{ for } x = 1, 2, \\dots, k \\\\\n",
    "0 & \\text{ otherwise}\n",
    "\\end{cases}$$\n",
    "<br>\n",
    "- Bernoulli $X \\sim \\operatorname{Bernoulli}(p)$: Suppose we have two outcomes with probability $p$ and $1 - p$:\n",
    "$$\n",
    "f(x) = p^x(1- p)^{1 - x} \\text{ for } x \\in \\{0, 1\\}\n",
    "$$\n",
    "Example of flipping the coins \n",
    "<br>\n",
    "- Binomial distribution $X \\sim \\operatorname{Binomial}(n, p)$: Suppose we have $n$ experiments with two outcomes with probability $p$ and $1 - p$ each\n",
    "$$\n",
    "f(x) = \n",
    "\\begin{cases}\n",
    "{n\\choose x} p^x(1-p)^{1-x}, & \\text{ for } x = 0, 1, \\dots, n \\\\\n",
    "0 & \\text{ otherwise}\n",
    "\\end{cases}\n",
    "$$\n",
    "or\n",
    "$$\n",
    "f(x) = \n",
    "\\begin{cases}\n",
    "C_n^x p^x(1-p)^{1-x}, & \\text{ for } x = 0, 1, \\dots, n \\\\\n",
    "0 & \\text{ otherwise}\n",
    "\\end{cases}\n",
    "$$\n",
    "<br>\n",
    "- Geometric distribution $X \\sim \\operatorname{Geom}(p)$:  Suppose we have $n$ experiments with two outcomes with probability $p$ and $1 - p$ each\n",
    "$$\n",
    "P(X=k) = p(1-p)^{k-1}\n",
    "$$\n",
    "Number of flips needed until the first head\n",
    "<br>\n",
    "- Hypergeometric distribution $X \\sim \\operatorname{Hypergeometric}(N,K,n)$:\n",
    "$$\n",
    "p_X(k) = \\Pr(X = k) \n",
    "= \\frac{\\binom{K}{k} \\binom{N - K}{n-k}}{\\binom{N}{n}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Some continouos distributions:\n",
    "\n",
    "- Uniform $X \\sim \\operatorname{Uniform}(a, b)$:\n",
    "$$\n",
    "f(x) = \n",
    "\\begin{cases}\n",
    "\\frac{1}{b-a}, & \\text{ for } x \\in [a, b] \\\\\n",
    "0 & \\text{ otherwise}\n",
    "\\end{cases}\n",
    "$$\n",
    "<br>\n",
    "- Normal (Gaussian) with parameters $(\\mu, \\sigma^2)$ where $\\mu \\in \\mathbb{R}$ and $\\sigma \\gt 0$ $X \\sim \\operatorname{N}(\\mu, \\sigma^2)$:\n",
    "$$\n",
    "f(x) = \\frac{1}{\\sigma \\sqrt{2\\pi}}\\exp{(-\\frac{1}{2\\sigma^2}(x - \\mu)^2)}\n",
    "$$\n",
    "We say that $X$ had standard normal distribution if $\\mu=0$ and $\\sigma=1$.\n",
    "<br>\n",
    "Some properties of normal distributiun:\n",
    "<br>\n",
    "if $X \\sim \\operatorname{N}(\\mu, \\sigma^2)$ then \n",
    "$$Z = \\frac{X - \\mu}{\\sigma} \\sim \\operatorname{N}(0, 1) \\text{ (standardization)}$$\n",
    "<br>\n",
    "if $X = \\sim \\operatorname{N}(0, 1)$ then:\n",
    "$$Z = \\mu + \\sigma X \\sim \\operatorname{N}(\\mu, \\sigma^2)$$\n",
    "<br>\n",
    "If $X_i \\sim \\operatorname{N}(\\mu_i, \\sigma_i^2)$ for $i = 1, 2, \\dots, n$ are independent, then:\n",
    "$$\\sum_{i=1}^n X_i \\sim \\operatorname{N}(\\sum_{i=1}^n\\mu_i, \\sum_{i=1}^n\\sigma_i^2)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Bivariate Distributions:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For two discrete random variables $X$ and $Y$ define the joint mass function:\n",
    "$$f(x, y) = P(X = x and Y = y) = P(X = x, Y =y)$$\n",
    "<br>\n",
    "For two continuous random variables $X$ and $Y$ we call function PDF if:\n",
    "$$f(x, y) \\ge 0 \\text{ for all } (x, y)$$\n",
    "$$\\int_{-\\infty}^{\\infty}\\int_{-\\infty}^{\\infty} f(x, y)\\,dx\\,dy = 1$$\n",
    "<br>\n",
    "and for any $A \\subset \\mathbb{R} \\times \\mathbb{R}$:\n",
    "$$P(X, Y) \\in A = \\iint_A f(x,y) \\,dx\\,dy$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Independent random variables:\n",
    "\n",
    "Two random variables $X$ and $Y$ are independent if\n",
    "$$\n",
    "P(X \\in A, Y \\in B) = P(X \\in A)\\cdot P(Y \\in B)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Conditional probability mass function:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For two discrete random variables $X$ and $Y$ define the conditional probability mass function:\n",
    "$$f_{X|Y}(x|y) = P(X = x | Y = y) = \\frac{P(X = x, Y =y)}{P(Y = y)} = \\frac{f_{X, Y}(x, y)}{f_Y(y)}$$\n",
    "<br>\n",
    "For two continuous random variables $X$ and $Y$ we call conditional PDF if:\n",
    "$$f_{X|Y}(x|y) = \\frac{f_{X, Y}(x, y)}{f_Y(y)}$$\n",
    "assuming that $f_Y(y) \\gt 0$\n",
    "$$P(X \\in A | Y = y) = \\int_A f_{X|Y}(x|y)\\,dx$$\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multivariate distributions:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let $X = (X_1, X_2, \\dots, X_n)$ where $X_1, X_2, \\dots, X_n$ are random variables, we call $X$ a random vector. Define $f(x_1, x_2, \\dots, x_n)$ as PDF function.\n",
    "<br>\n",
    "We say that $X_1, X_2, \\dots, X_n$ are independent if:\n",
    "$$\n",
    "P(X_1 \\in A_1, X_2 \\in A_2, \\dots, X_n \\in A_n) = \\prod_{i=1}^nP(X_i \\in A_i)\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Multinomial (Binomial) $X \\sim \\operatorname{Multinomial}(p)$:\n",
    "$$\n",
    "\\begin{align}\n",
    "f(x_1,\\ldots,x_k;n,p_1,\\ldots,p_k) & {} = \\Pr(X_1 = x_1 \\text{ and } \\dots \\text{ and } X_k = x_k) \\\\\n",
    "& {} = \\begin{cases} { \\displaystyle {n! \\over x_1!\\cdots x_k!}p_1^{x_1}\\times\\cdots\\times p_k^{x_k}}, \\quad &\n",
    "\\text{when } \\sum_{i=1}^k x_i=n \\\\  \\\\\n",
    "0 & \\text{otherwise,} \\end{cases}\n",
    "\\end{align}\n",
    "$$\n",
    "<br>\n",
    "- Multivariate normal $X \\sim \\operatorname{N}(\\mu, \\Sigma)$:\n",
    "$$\n",
    "f_{\\mathbf X}(x) = f_{\\mathbf X}(x_1,\\ldots,x_k) = \\frac{\\exp\\left(-\\frac 1 2 ({\\mathbf x}-{\\boldsymbol\\mu})^\\mathrm{T}{\\boldsymbol\\Sigma}^{-1}({\\mathbf x}-{\\boldsymbol\\mu})\\right)}{\\sqrt{(2\\pi)^k|\\boldsymbol\\Sigma|}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conditional distribution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose $X$ and $Y$ are discrete random variables, we observed that $Y = y$ then conditional probability mass function will be:\n",
    "$$\n",
    "f_{X|Y}(x|y)=P(X=x|Y=y)=\\frac{P(X=x, Y=y)}{P(Y = y)}=\\frac{f_{X,Y}(x, y)}{f_Y(y)} \\text{ if } f_Y(y) \\gt 0\n",
    "$$\n",
    "<br>\n",
    "For continuous random variables $X$ and $Y$ we have PDF:\n",
    "$$\n",
    "f_{X|Y}(x|y) = \\frac{f_{X,Y}(x, y)}{f_Y(y)}\n",
    "$$\n",
    "<br>\n",
    "when $f_Y(y) > 0$ and probability is:\n",
    "$$\n",
    "P(X \\in A|Y=y) = \\int_{A}f_{X|Y}(x|y)\\, dx.\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Expected value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Expected value of random variable $X$ is:\n",
    "\n",
    "$$\\operatorname{E}[X] = \\int_{\\mathbb{R}} x f(x)\\, dx.$$\n",
    "<br>\n",
    "For discrete and continuous random variables:\n",
    "$$\n",
    "\\operatorname{E}[X] = \n",
    "\\begin{cases}\n",
    "\\sum_x xf(x), & \\text{ if } X \\text { is discrete} \\\\\n",
    "\\int_{\\mathbb{R}} x f(x)\\, dx. & \\text{ if } X \\text { is continuous}\n",
    "\\end{cases}\n",
    "$$\n",
    "<br>\n",
    "Or in general for probability measure space $(\\Omega, P, X)$:\n",
    "$$\n",
    "\\operatorname{E} [X]  = \\int_\\Omega X(\\omega)\\,d\\operatorname{P}(\\omega)\n",
    "$$\n",
    "<br>\n",
    "For multidimensional case\n",
    "$$\n",
    "\\operatorname{E}[(X_1,\\ldots,X_n)]=(\\operatorname{E}[X_1],\\ldots,\\operatorname{E}[X_n])\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Properties of expectation:\n",
    "<br>\n",
    "$$\n",
    "\\begin{align}\n",
    "  \\operatorname{E}[X + Y] &=   \\operatorname{E}[X] + \\operatorname{E}[Y], \\\\\n",
    "  \\operatorname{E}[aX]    &= a \\operatorname{E}[X],\n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or in general for any sequence of random variables $(X_1, X_2, \\dots, X_n)$ and numbers $(a_1, a_2, \\dots, a_n)$:\n",
    "$$\\operatorname{E}[\\sum_{i=1}^na_iX_i] = \\sum_{i=1}^n a_i \\operatorname{E}[X_i]$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let $X_1, X_2, \\dots, X_n$ be independent random variables, then:\n",
    "<br>\n",
    "$$\n",
    "\\operatorname{E}[\\prod_{i=1}^nX_i] = \\prod_{i=1}^n \\operatorname{E}[X_i]\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In many cases expectation is denoted by $\\mu$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Variance and covariance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Variance (spread) of a distribution:\n",
    "\n",
    "$$\n",
    "\\sigma^2 = \\operatorname{Var}(X)=  \\operatorname{E}[X - \\operatorname{E}[X]]^2\n",
    "$$\n",
    "<br>\n",
    "Easy to prove\n",
    "$$\n",
    "\\sigma^2 = \\operatorname{E}[X^2] - (\\operatorname{E}[X])^2\n",
    "$$\n",
    "Assuming that expectation exists\n",
    "<br>\n",
    "Denoted by $\\sigma^2$\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Standard deviation:\n",
    "$$\\sigma = \\sqrt{\\operatorname{Var}(X)}$$\n",
    "<br>\n",
    "$$\\sigma = \\sqrt{\\sigma^2}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Properties of variance:\n",
    "- $\\operatorname{Var}(X + Y) = \\operatorname{Var}(X) + \\operatorname{Var}(Y)$\n",
    "- $\\operatorname{Var}(a \\cdot X) = a^2\\operatorname{Var}(X)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Covariance and correlation:\n",
    "Let $X$ and $Y$ be a random variables with $\\mu_X$ and $\\mu_Y$ expectations and $\\sigma_X$ and $\\sigma_Y$ variances, then covariance between them is:\n",
    "$$\n",
    "Cov(X, Y) = \\operatorname{E}[(X - \\mu_X)(Y - \\mu_Y)]\n",
    "$$\n",
    "<br>\n",
    "and correlation is:\n",
    "$$\n",
    "\\rho = \\rho_{X,Y} = \\rho(X, Y) = \\frac{Cov(X, Y)}{\\sigma_X \\sigma_Y}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Properties:\n",
    "$$\n",
    "Cov(X, Y) = \\operatorname{E}[XY] - \\operatorname{E}[X]\\operatorname{E}[Y]\n",
    "$$\n",
    "<br>\n",
    "$$\n",
    "-1 \\le \\rho(X, Y) \\le 1\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Examples:\n",
    "If $Y = aX + b$ then:\n",
    "$$\n",
    "\\rho(X, Y) = 1 \\text{ if } a \\gt 0\n",
    "$$\n",
    "and\n",
    "$$\n",
    "\\rho(X, Y) = -1 \\text{ if } a \\lt 0\n",
    "$$\n",
    "<br>\n",
    "If $X$ and $Y$ are independent then\n",
    "$$\n",
    "\\rho(X, Y) = 0\n",
    "$$\n",
    "<br>\n",
    "But contrary not always correct"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\operatorname{Var}(X + Y)= \\operatorname{Var}(X) + \\operatorname{Var}(Y) + 2Cov(X,Y)\n",
    "$$\n",
    "<br>\n",
    "$$\n",
    "\\operatorname{Var}(X - Y)= \\operatorname{Var}(X) + \\operatorname{Var}(Y) - 2Cov(X,Y)\n",
    "$$\n",
    "<br>\n",
    "In general:\n",
    "$$\n",
    "\\operatorname{Var}(\\sum_{i=1}^{n}a_iX_i) = \\sum_{i=1}^{n}a_I^2\\operatorname{Var}(X_i) + 2\\sum_{i=1}^{n}\\sum_{j\\lt i}a_ia_jCov(X_i, X_j)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conditional Expectation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For discrete and continuous random variables $X$ and $Y$ we can define conditional expectation, what is the mean of $X$ when $Y=y$:\n",
    "<br>\n",
    "$$\n",
    "\\operatorname{E}[X|Y] = \n",
    "\\begin{cases}\n",
    "\\sum_x xf_{X|Y}(x|y), & \\text{ if } X \\text { is discrete} \\\\\n",
    "\\int_{\\mathbb{R}} x f_{X|Y}(x|y)\\, dx. & \\text{ if } X \\text { is continuous}\n",
    "\\end{cases}\n",
    "$$\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The rule of iterated expectation:\n",
    "For random variables $X$ and $Y$ assuming the expectation exists, we have that:\n",
    "$$\n",
    "\\operatorname{E}[\\operatorname{E}[X|Y]] = \\operatorname{E}[Y]\n",
    "$$\n",
    "<br>\n",
    "and\n",
    "<br>\n",
    "$$\n",
    "\\operatorname{E}[\\operatorname{E}[Y|X]] = \\operatorname{E}[X]\n",
    "$$\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convergence in probability:\n",
    "Let $X_1, X_2, \\dots$ be a sequence of random variables, and $X$ be a random variable\n",
    "- Convergence in probability written $X_n \\xrightarrow{P}\\ X \\qquad\\textrm{when}\\ n \\to \\infty$:\n",
    "$$\n",
    "P(|X_n - X| > \\epsilon) \\rightarrow 0 \\text{ when } n \\rightarrow \\infty\n",
    "$$\n",
    "<br>\n",
    "- Convergence in distribution:\n",
    "$$\n",
    "\\lim_{n \\to \\infty}F_n(t) = F(t) \\text{ for all } t \\text{when } F \\text{ is continuous} \n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Law of large numbers:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let $X_1, X_2, \\dots, X_n$ random variables, then\n",
    "$$\n",
    "\\overline{X}_n=\\frac1n(X_1+\\cdots+X_n)\n",
    "$$\n",
    "We call identically distributed independent random variables (IID) if they have the same distribution\n",
    "$$\n",
    "F_{X_i}(x) = F_{X_j}(x) \\text{ for each } i, j \n",
    "$$\n",
    "and are independent\n",
    "$$\n",
    "F_{X_1,\\ldots,X_n}(x_1,\\ldots,x_n) = F_{X_1}(x_1) \\cdot \\ldots \\cdot F_{X_n}(x_n) \\text{ for all } x_1,\\ldots,x_n \\in I\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Weak law of large numbers:\n",
    "Let $X_1, X_2, \\dots, X_n$ be a IID random variables, then:\n",
    "$$\n",
    "\\begin{matrix}{}\\\\\n",
    "    \\overline{X}_n\\ \\xrightarrow{P}\\ \\mu \\qquad\\textrm{when}\\ n \\to \\infty.\n",
    "\\\\{}\\end{matrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}