lecture_14_optimizations.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Basic setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Create anaconda environment\n",
    "<br>\n",
    "```bash\n",
    "conda create -n ml python=3.7.5 jupyter\n",
    "```\n",
    "Install fastai library\n",
    "<br>\n",
    "```bash\n",
    "conda install -c pytorch -c fastai fastai\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Exponential moving average"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Define data as $(y_1, y_2, \\dots, y_n)$ peaks of function $f:X \\to Y$ where for some $(x_1, x_2, \\dots, x_n)$, $f(x_1) = y_1, f(x_2) = y_2, \\dots, f(x_n) = y_n$ and define\n",
    "<br>\n",
    "$$v_1 = (1 - \\beta)y_1,  \\\\\n",
    "v_2 = \\beta v_1 + (1 - \\beta)y_2 \\\\\n",
    "\\dots \\\\\n",
    "v_n = \\beta v_{n-1} + (1 - \\beta)y_n \\\\\n",
    "$$ \n",
    "</br>\n",
    "for $\\beta \\in [0, 1]$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<div>\n",
    "<img src=\"images/opts/ema_1.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now lets try higher $\\beta$\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/ema_2.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Try with lower $\\beta$\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/ema_3.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Bias on early stage:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/ema_4.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For early stages, worm up, better will be if we use, so called bias correction:\n",
    "$$\n",
    "v^{corr}_t =\\frac{v_t}{1 - \\beta^t}\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Different loss functions and noisy gradients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Applied machine learning is hardly experiment base and needs many experiments while achieving the stable result. On the other hand, DL models need \"big\" data for training and full batch processing is almost never possible.\n",
    "<br>\n",
    "For instance $100000$ examples or even more than $1000000$ can't be feat in to the one batch in GPU memory and iteratively calculation is too slow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In batch gradient descent loss goes down per iteration, if not, than maybe learning rate is too big or other:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/bt_1.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "From the different point of view, gradients are pretty similar at each epoch and if we stack in local extrema or saddle point, we can stay there for long:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/bt_2.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Generally data is divided in mini-batches and model is trained on that mini-datasets with batch gradient descent:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/bt_3.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "But to match noise can increase the training time significatly:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/bt_4.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Noisy gradient descent:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/ng_1.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We need to somehow speedup the training time (the amount of experiments might be large):\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/ng_2.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Loss function is often overcomplicated rather that MSE with convex landscape:\n",
    "\n",
    "For probability activations usually softmax is used:\n",
    "$$\n",
    "\\sigma(x) = \\frac{e^{x_j}}{\\sum_i{e^{x_i}}}\n",
    "$$\n",
    "<br>\n",
    "with cross entropy loss:\n",
    "$$\n",
    "C = -\\sum_{i}^{C}t_{i} log (s_{i}) = -log\\left ( \\frac{e^{s_{p}}}{\\sum_{j}^{C} e^{s_{j}}} \\right )\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Training deep learning models might use combination of many loss functions:\n",
    "\n",
    "- Object detection: bounding box regression plus (weighted) cross entropy for image classification.\n",
    "- Instance segmentation: bounding box regression plus (weighted) pixel binary classification plus (weighted) cross entropy for classification "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Different losses combined for different tasks:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/loss_odcs_1.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Detect, segment and classify at once:\n",
    "\n",
    "Different losses combined for different tasks:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/loss_odcs_2.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "On the other hand loss can have a different surfaces, because of many parameters, chance that many of them has the same direction is low:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/loss_ld_1.jpeg\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Loss landscape and optimization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Non smooth surface:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/ls_1.jpeg\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Smooth surface:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/ls_2.jpeg\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Make step optimization (moving average):\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/lsgd_1.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Make surface optimization (landscape):\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/lsgd_2.gif\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Gradient descent with momentum"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Recall gradient descent algorithm, for some cost function $C$ and learning rate $\\alpha$ we do parameters (weights) update by:\n",
    "$$\n",
    "W^l_{i,j} = W^l_{i,j} - \\alpha \\frac{\\partial{C}}{\\partial{W^l_{i,j}}}\n",
    "$$\n",
    "<br>\n",
    "and\n",
    "$$\n",
    "b^l_{j} = b^l_{j} - \\alpha \\frac{\\partial{C}}{\\partial{W^l_{j}}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If the cost function $C$ is defined, denote partial derivative with respect of weights: \n",
    "$$\n",
    "d{W} = \\nabla_{W}C\n",
    "$$\n",
    "<br>\n",
    "and with respect of biases:\n",
    "$$\n",
    "d{b} = \\nabla_{b}C\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "So our gradient descent optimization can be written as:\n",
    "$$\n",
    "W^l_{i,j} = W^l_{i,j} - \\alpha \\partial{W^l_{i,j}}\n",
    "$$\n",
    "<br>\n",
    "and\n",
    "$$\n",
    "b^l_{j} = b^l_{j} - \\alpha \\partial{b^l_{j}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Or in general for batch or mini-batch gradient descent\n",
    "optimization can be written as:\n",
    "$$\n",
    "W = W - \\alpha d{W}\n",
    "$$\n",
    "<br>\n",
    "and\n",
    "$$\n",
    "b = b - \\alpha d{b}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<div>\n",
    "<img src=\"images/opts/gd_1.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For each iteration, compute $d W$ and $d b$ for the current mini-batch "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Here we have a picks and higher is oscillation slower the optimization is.\n",
    "We can use moving average in order to reduce horizontal variance and increase speed of optimization:\n",
    "$$\n",
    "V_{d W} = \\beta V_{d W} + (1 - \\beta)d W\n",
    "$$\n",
    "<br>\n",
    "$$\n",
    "V_{d b} = \\beta V_{d b} + (1 - \\beta)d b\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can use this averages in gradient descent optimization instead of direct gradients:\n",
    "$$\n",
    "W = W - \\alpha V_{d W}\n",
    "$$\n",
    "<br>\n",
    "and\n",
    "$$\n",
    "b = b - \\alpha V_{d b}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<div>\n",
    "<img src=\"images/opts/gd_mom_1.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## RMSProp optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<b>Root Mean Square Prop</b>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For each iteration, compute $d W$ and $d b$ for the current mini-batch "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Calculate momentum with changes:\n",
    "$$\n",
    "S_{d W} = \\beta S_{d W} + (1 - \\beta)d W^2 \\text{ elementwise}\n",
    "$$\n",
    "<br>\n",
    "$$\n",
    "S_{d b} = \\beta S_{d b} + (1 - \\beta)d b^2 \\text{ elementwise}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now update the weights and biases by:\n",
    "$$\n",
    "W = W - \\alpha \\frac{d W}{\\sqrt{S_{d W}}}\n",
    "$$\n",
    "<br>\n",
    "and\n",
    "$$\n",
    "b = b - \\alpha \\frac{d b}{\\sqrt{S_{d b}}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<div>\n",
    "<img src=\"images/opts/rms_1.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "RMSProp steps:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/rms_2.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Here if $S_{d W}$ is large, it means that step will be forward"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "To avoid zero division error (if $\\sqrt{S_{d W}}$ or $\\sqrt{S_{d b}}$ is almost zero) we can add small $\\epsilon$ to the denominators:\n",
    "$$\n",
    "W = W - \\alpha \\frac{d W}{\\sqrt{S_{d W}} + \\epsilon}\n",
    "$$\n",
    "<br>\n",
    "and\n",
    "$$\n",
    "b = b - \\alpha \\frac{d b}{\\sqrt{S_{d b}} + \\epsilon}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Here $\\epsilon = 10^{-8}$ for instance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Adam optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<b>Adaptive Moment Estimation </b>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For Adam optimization algorithm we will combine Momentum and RMSProp together"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Set $V_{d W} = 0$, $S_{d W = 0}$, $V_{d b} = 0$, $S_{d b = 0}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For each iteration, compute $d W$ and $d b$ for the current mini-batch "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "First calculate momentums $V_{d W}$ and $V_{d b}$ with hyperparameter $\\beta_1$:\n",
    "$$\n",
    "V_{d W} = \\beta_1 V_{d W} + (1 - \\beta_1)d W\n",
    "$$\n",
    "<br>\n",
    "$$\n",
    "V_{d b} = \\beta_1 V_{d b} + (1 - \\beta_1)d b\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now calculate $S_{d W}$ and $S_{d b}$ with hyperparameter $\\beta_2$:\n",
    "$$\n",
    "S_{d W} = \\beta_2 S_{d W} + (1 - \\beta_2)d W^2 \\text{ elementwise}\n",
    "$$\n",
    "<br>\n",
    "$$\n",
    "S_{d b} = \\beta_2 S_{d b} + (1 - \\beta_2)d b^2 \\text{ elementwise}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For the current iteration, let's say $t$, calculate so called bias corrections for momentum:\n",
    "$$\n",
    "V^{corr}_{d W} = \\frac {V_{d W}}{1 - \\beta_1^t}\n",
    "$$\n",
    "<br>\n",
    "$$\n",
    "V^{corr}_{d b} = \\frac{V_{d b}}{{1 - \\beta_1^t}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The same correction for RMSProp parameters:\n",
    "$$\n",
    "S^{corr}_{d W} = \\frac {S_{d W}}{1 - \\beta_2^t}\n",
    "$$\n",
    "<br>\n",
    "$$\n",
    "S^{corr}_{d b} = \\frac{S_{d b}}{{1 - \\beta_2^t}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now lets update weights and biases:\n",
    "$$\n",
    "W = W - \\alpha \\frac{V^{corr}_{d W}}{\\sqrt{S^{corr}_{d W}} + \\epsilon}\n",
    "$$\n",
    "<br>\n",
    "and\n",
    "$$\n",
    "b = b - \\alpha \\frac{V^{corr}_{d b}}{\\sqrt{S^{corr}_{d b}} + \\epsilon}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "This algorithms combines the Momentum effect together with RMSProp optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We than have a number of hyperparameters:\n",
    "- The learning rate $\\alpha$\n",
    "- Momentum parameter $\\beta_1$ for derivatives $d W$ and $d b$ with common choice $0.9$\n",
    "- RMSProp parameter $\\beta_2$ for $d W^2$ and $d b^2$ squares with common choice $0.999$\n",
    "- $\\epsilon = 10^{-8}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Almost always $\\alpha$ is tunned and almost never other parameters, the values above was recommended by the authors of the Adam paper and I personally don't remember that other values gave me any positive effect in improvements of performance (optimization) if not negative."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The $\\beta_1$ is considered for first moment and $\\beta_2$ for second moment and that's why optimizer has the name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Visualization of different optimizers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Optimizers on surface:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/gds_1.gif\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Optimizers on projection:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/gds_2.gif\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Learning rate decay"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Near the extrema points, large learning rate might cause so called bouncing gradient:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/lrdc_1.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In order to avoid this problem, we can decrease learning rate per iteration, or epochs:\n",
    "\n",
    "$$\n",
    "\\alpha = \\frac{1}{1 + dr \\cdot ep} \\cdot \\alpha_0\n",
    "$$\n",
    "<br>\n",
    "- $dr$ - is the decay rate\n",
    "- $ep$ - is the epoch number"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Or square root decay:\n",
    "\n",
    "$$\n",
    "\\alpha = \\frac{k}{\\sqrt{ep}} \\cdot \\alpha_0\n",
    "$$\n",
    "<br>\n",
    "- $k$ - is the another hyperparameter\n",
    "- $ep$ - is the epoch number"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Or mini-batch decay:\n",
    "\n",
    "$$\n",
    "\\alpha = \\frac{k}{\\sqrt{t}} \\cdot \\alpha_0\n",
    "$$\n",
    "<br>\n",
    "- $k$ - is the another hyperparameter\n",
    "- $t$ - mini-batch number"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Stepwise decay:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/lrdc_2.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Local optimum and saddle points"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "On early stages of deep learning, people thought that local optimum was the main problem of optimization. But for local optima we need that each \n",
    "$$\\frac{\\partial C}{\\partial W^l_{i,j}}$$ \n",
    "direction to be the same which for instance of $100000000$ or even $200000000$ parameters might have a $2^{-100000000}$ or $2^{-200000000}$ probability"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Local optima in lower dimensional case:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/lo_1.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<b> Analysis in lower dimensions often does not generalize for higher dimensions </b>\n",
    "\n",
    "- The probability of local optima is low\n",
    "- But the probability of saddle points is high"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Local optima vs saddle points in higher dimensions:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/lo_2.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Saddle points and plateaus:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/saddle_1.jpeg\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The main goal of optimization algorithms is to escape plateaus:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/saddle_2.gif\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## One cycle policy CycleLR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In order to avoid saddle point trap, we can increase learning rate during the epoch and then decrease it with some linear or non linear (exponential) function:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/one_cycle_1.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "One cycle policy with non linear learning rate change:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/one_cycle_2.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can use this technique to find initial learning rate before training:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/lr_find_1.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<div>\n",
    "<img src=\"images/opts/lr_find_2.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "One cycle policy speeds up learning process significantly and reduces the chances to get in to the saddle point or bad extrema point trap"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "There are many techniques of learning rate manipulation during the training:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/one_cycle_mx_1.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Or even:\n",
    "\n",
    "<div>\n",
    "<img src=\"images/opts/one_cycle_mx_2.png\"  height=\"600\" width=\"800\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Questions"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}