lecture_13_06_2022_segmentation_detection.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Segmentation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Image patch classifier\n",
    "<img src=\"images/od/semantic_segmentation_sliding_window.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Computationally expensive"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Image patch classifier\n",
    "<img src=\"images/od/semantic_segmentation_sliding_window.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Computationally expensive and hard to classify without additional information of image context"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Fully convolutional model byt at the end layers information is lost\n",
    "<img src=\"images/od/fully_conv.jpeg\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Fully convolutional model size preserving\n",
    "<img src=\"images/od/fully_convolutional_2.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Without down-sampling preserve original image scale would be really expensive and that's why encoder-decoder model is preferred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Down-sampling plus up-sampling (Encoder decoder architecture)\n",
    "<img src=\"images/od/encoder_decoder_1.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Encoder-decoder application:\n",
    "<img src=\"images/od/encoder_decoder_2.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Detects smaller objects on first layers of encoder and last layers on decoder\n",
    "- Detects bigger objects on last layers of encoder and fist layers on decoder\n",
    "- Decodes medium sized objects on medium layers\n",
    "- Example auto-encoders"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/od/autoencoder.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Up-sampling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Unpooling example (non-learnable):\n",
    "<img src=\"images/od/unpooling_1.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Recall convolution with stride:\n",
    "<img src=\"images/od/conv_stride_1.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Transpose convolution:\n",
    "<img src=\"images/od/transpose_1.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Transpose convolution for $1$ dimension:\n",
    "<img src=\"images/od/transpose_2.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "After each layer naturally there are activations, maybe normalization and other layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If we write layer as matrix operation:\n",
    "<img src=\"images/od/transpose_matrix_1.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Transpose convolution because of it's nature of repeating the same value from receptive field, causes the so called \"checkerboard\" artifacts:\n",
    "<img src=\"images/od/checkerboard_1.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Other up-sampling methods:\n",
    "Up-sample with chosen (bilinear) interpolation for instance twice or three times and then run convolution:\n",
    "<br>\n",
    "$$\n",
    "I_r = \\frac{I - F_d + 2 P}{S_d} + 1 = \\frac{I}{S_d} - \\frac{F_d - 2 P}{S_d} + 1\n",
    "$$\n",
    "<br>\n",
    "so we get:\n",
    "$$\n",
    "I_r = \\frac{n \\cdot I - F_d + 2 P}{S_d} + 1 = n \\cdot \\frac{I}{S_d} - \\frac{I}{F_d - 2 P}{S_d} + 1\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "There are bunch of other learnable up-sampling methods, for instance pixel shuffle up-sampling instead of other interpolation before convolution layer:\n",
    "<img src=\"images/od/pixel_shuffle_1.jpg\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The main results are:\n",
    "- Up-sample might be learnable layer\n",
    "- It's pretty similar to convolution layer with activations \n",
    "- Has the same depth for filter as input and outputs no depth\n",
    "- Might generate as match up-sampled feature maps as we configure"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "During the down-sample convolutional layers, information is extracted from the input image independent from the tasks sometimes.\n",
    "<br>\n",
    "Often pre-trained (on ImageNet classification task) encoder (often called backbone) used"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For segmentation we can use $L_2$ or $L_1$ loss:\n",
    "$$\n",
    "L_2 = \\sum_{i = 1}^{h}\\sum_{j=1}^{w}||y_{ij} - \\hat{y}_{ij}||^2\n",
    "$$\n",
    "<br>\n",
    "$$\n",
    "L_1 = \\sum_{i = 1}^{h}\\sum_{j=1}^{w}||y_{ij} - \\hat{y}_{ij}||\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can produce as many feature maps on last layer as many objects we have\n",
    "<br>\n",
    "or different number can denote different objects for labels:\n",
    "$$\\begin{align} Y &= \\begin{pmatrix}\n",
    "           0, 1, \\dots, 0 \\\\\n",
    "           1, 1, \\dots, 0 \\\\\n",
    "           \\vdots \\\\\n",
    "           0, 1, \\dots, 1 \\\\\n",
    "         \\end{pmatrix}\n",
    "  \\end{align}$$\n",
    "  <br>\n",
    "  or:\n",
    "  $$\\begin{align} Y &= \\begin{pmatrix}\n",
    "           1, 2, \\dots, 4 \\\\\n",
    "           0, 4, \\dots, 2 \\\\\n",
    "           \\vdots \\\\\n",
    "           2, 1, \\dots, 0 \\\\\n",
    "         \\end{pmatrix}\n",
    "  \\end{align}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Why not mix encoder-decoder models in segmentation-recognition models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Mix encoder-decoder model layers\n",
    "<img src=\"images/od/mixing_layers_1.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## UNet combine encoder and decoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Lets combing down-sampling and up-sampling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "UNet\n",
    "<img src=\"images/od/unet.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Lets combing down-sampling and up-sampling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "UNet\n",
    "<img src=\"images/od/unet_2.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Single shot detection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Multi model has high performance but at the same time:\n",
    "- they are hard to train\n",
    "- hard to select correct parameters / hyper-parameters\n",
    "- computationally expensive during the training\n",
    "- computationally expensive during the inference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For the real time or near-real time approaches there exists different single shot detectors\n",
    "Which runs bounding box regression and the classification at the same time"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Recall localization and classification algorith:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/od/localization_classification.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In some aspects, this is the main approach, we can not have a undefined output of the feed-forward neural networks (not RNN)\n",
    "<br>\n",
    "So we need somehow localize single object in the image\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "First approach which we saw in the previous presentation was the family of the RCNN models\n",
    "<br>\n",
    "Model first detected blobs with different algorithms (Region proposals, Region proposals on the feature map, decoder encoder-model) and then uses classification + localization algorithm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "On the other hand we can split the image by the smaller parts for instance $10 \\times 10$ or $6 \\times 6$ or even $19 \\times 19$ parts, grid-cells"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/od/yolo_1.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For the object detection we have:\n",
    "- Object classes: $(c_1, c_2, \\dots, c_n)$\n",
    "- Bounding boxes: $((x, y, h, w )_1, (x, y, h, w )_2 \\dots (x, y, h, w )_m)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "And for each grid cell (which is small enough) lets take center and classify is there object or not:\n",
    "- $p$ is the probability is there object or not\n",
    "- Also class label $𝑐_1,𝑐_2,…,𝑐_𝑛$\n",
    "- Bounding boxes $x, y, h, w $ for this instant class"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "So we have:\n",
    "$$\n",
    "\\begin{align}\n",
    "    g_i &= \\begin{bmatrix}\n",
    "           p \\\\\n",
    "           x \\\\\n",
    "           y \\\\\n",
    "           h \\\\\n",
    "           w \\\\\n",
    "           c_1 \\\\\n",
    "           c_2 \\\\\n",
    "           \\vdots \\\\\n",
    "           c_n\n",
    "         \\end{bmatrix}\n",
    "  \\end{align}\n",
    "$$\n",
    "<br>\n",
    "For each grid-cell of image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now let consider it as a tensor with $6 \\times 6$, $14 \\times 14$, $19 \\times 19$, $g \\times g$ or any other, with $g_i$ grid cell vector for each\n",
    "<br>\n",
    "$g \\times g \\times (5 + c)$\n",
    "- So use this as a target for the prediction and run model per grid-cell\n",
    "- Or we can use compose our model with the single model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "So lets create neural network which has a convolutional layers with the our $g \\times g$ model\n",
    "<img src=\"images/od/yolo_2.gif\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## YOLO-v3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Previous YOLO models does not have up-sampling architectures which means that it was fast but it was not accurate enough\n",
    "<br>\n",
    "In YOLOv3 was added DarkNet53 backbone model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "YOLOv3 model architecture with upsampling layers:\n",
    "<img src=\"images/od/yolo_v3_arch.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Anchor boxes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Recall that YOLOv3 was detecting objects for each cell and was assigning \n",
    "$$  \\begin{align}\n",
    "    y &= \\begin{pmatrix}\n",
    "           p \\\\\n",
    "           x \\\\\n",
    "           y \\\\\n",
    "           h \\\\\n",
    "           w \\\\\n",
    "           c_{1} \\\\\n",
    "           c_{2} \\\\\n",
    "           \\vdots \\\\\n",
    "           c_{n}\n",
    "         \\end{pmatrix}\n",
    "  \\end{align}\n",
    "$$\n",
    "<br>\n",
    "Let's imagine the scenario where two objects have the same center. Then our detection should decide by probability. But what if both of them are in the image?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Per cell we had a sigle vector:\n",
    "<img src=\"images/od/yolo_pred_1.jpeg\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For instance person and car have the same center (no the best place for center):\n",
    "<img src=\"images/od/anchor_1.jpeg\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Or the same here:\n",
    "<img src=\"images/od/anchor_2.jpg\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If we observe the shape of the objects, the car of them has bigger width and smaller height while the person has smaller width and bigger height\n",
    "<br>\n",
    "Generally different type if objects has different shape and size and they might be assigned to the detection algorithm\n",
    "<img src=\"images/od/anchor_3.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's use this information and add two predefined shapes to our algorithm, instead of one vector per cell, let's concatenate two of them, one pre-shape which we'll call anchor box:\n",
    "$$  \\begin{align}\n",
    "    y_1 = \\begin{pmatrix}\n",
    "           p \\\\\n",
    "           x \\\\\n",
    "           y \\\\\n",
    "           h \\\\\n",
    "           w \\\\\n",
    "           c_{1} \\\\\n",
    "           c_{2} \\\\\n",
    "           \\vdots \\\\\n",
    "           c_{n}\n",
    "         \\end{pmatrix}\n",
    "\\quad\n",
    "    y_2 = \\begin{pmatrix}\n",
    "           p \\\\\n",
    "           x \\\\\n",
    "           y \\\\\n",
    "           h \\\\\n",
    "           w \\\\\n",
    "           c_{1} \\\\\n",
    "           c_{2} \\\\\n",
    "           \\vdots \\\\\n",
    "           c_{n}\n",
    "         \\end{pmatrix}\n",
    "  \\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Concatenate them and we will get:\n",
    "$$ \\begin{align}\n",
    "    y = \\begin{pmatrix}\n",
    "           p \\\\\n",
    "           x \\\\\n",
    "           y \\\\\n",
    "           h \\\\\n",
    "           w \\\\\n",
    "           c_{1} \\\\\n",
    "           c_{2} \\\\\n",
    "           \\vdots \\\\\n",
    "           c_{n} \\\\\n",
    "           p \\\\\n",
    "           x \\\\\n",
    "           y \\\\\n",
    "           h \\\\\n",
    "           w \\\\\n",
    "           c_{1} \\\\\n",
    "           c_{2} \\\\\n",
    "           \\vdots \\\\\n",
    "           c_{n}\n",
    "         \\end{pmatrix}\n",
    "  \\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Or maybe we'll group probabilities, together or probabilities and bounding boxes, etc or put in other order, it does not matter:\n",
    "$$ \\begin{align}\n",
    "    y = \\begin{pmatrix}\n",
    "           x \\\\\n",
    "           y \\\\\n",
    "           h \\\\\n",
    "           w \\\\\n",
    "           c_{1} \\\\\n",
    "           c_{2} \\\\\n",
    "           \\vdots \\\\\n",
    "           c_{n} \\\\\n",
    "           x \\\\\n",
    "           y \\\\\n",
    "           h \\\\\n",
    "           w \\\\\n",
    "           c_{1} \\\\\n",
    "           c_{2} \\\\\n",
    "           \\vdots \\\\\n",
    "           c_{n} \\\\\n",
    "           p_1 \\\\\n",
    "           p_2\n",
    "         \\end{pmatrix}\n",
    "  \\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We will have $19 \\times 19 \\times 2 \\times {(1 + 2 + 4 + n)}$ dimensional output\n",
    "<br>\n",
    "Or as tensor $19 \\times 19 \\times {2 \\cdot (1 + 2 + 4 + n)} = 19 \\times 19 \\times {14 + 2 n}$\n",
    "<img src=\"images/od/yolo_2.gif\" height=\"600\" width=\"600\">\n",
    "This is the answer of the question from the previous presentation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In practice there are more than two anchor boxes for detection models:\n",
    "<img src=\"images/od/pred_anchor_1.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Actually anchor boxes are predefined and used as default during the training.\n",
    "During the training annotated object and it's bounding box is assigned to the concrete anchor box by the highest IoU value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "People might generate anchor boxes by hand or use KNN or other advanced techniques for this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The main flaw of anchor boxes is if there are more object with the same center that anchor boxes.\n",
    "<br>\n",
    "This is the case which anchor boxes approach can not deal with\n",
    "<br>\n",
    "But usually there are many enough anchor boxes and small enough cells in YOLO case to make this event almost impossible\n",
    "<br>\n",
    "For instance $19 \\times 19$ cells are small enough with $9$ anchor boxes to avoid $9$ different object with the same center"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now the anchor boxes are used in many detection algorithms. Just increase output value by the fixed number"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Non-max suppression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Generally detector models predict multiply bounding boxes per object (per image) for instance YOLO might consider several several cells as center of the instant object:\n",
    "<img src=\"images/od/non_max_1.jpg\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can filter them by confidence and IoU thresholds, but still there will be many of them:\n",
    "<img src=\"images/od/non_max_2.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Also, there might be many same category of object on a single image. How can we reduce number of boxes:\n",
    "<br>\n",
    "For a single object category:\n",
    "- First remove all boxes with lower confidence that threshold\n",
    "- Second we choose box with highest confidence\n",
    "- Then calculate IoU with other boxes and if IoU is higher than $0.5$ for instance, remove this box\n",
    "- Then do the same with the rest of the boxes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Remove low confidence, sort and then run non max suppression:\n",
    "<img src=\"images/od/non_max_det_1.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Leave only low IoU box scores others put $0$ values and filter\n",
    "<img src=\"images/od/non_max_det_2.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If we have a single object of the category:\n",
    "<img src=\"images/od/non_max_3.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "But why do we leave lower IoU images?\n",
    "<br>\n",
    "Because they might be the boxes for the different object of the same category:\n",
    "<img src=\"images/od/non_max_4.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "After we remove boxes with higher IoU we'll distinguish different objects of the same category:\n",
    "<img src=\"images/od/non_max_5.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Here's the other example of different of the same category:\n",
    "<img src=\"images/od/non_max_7.jpeg\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Multi-centers:\n",
    "<img src=\"images/od/non_max_8.jpg\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Questions\n",
    "<img src=\"images/od/questions_1.jpg\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## VGG-16"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Before we start describing SSD model let's have a quick look on VGG-16 architecture which is used as a backbone or feature extractor for SSD model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Model VGG (visual geometry group) was first presented at 2013 ImageNet competition and got high results (not winner ZENet was winner) but it was fast and had a simple architecture.\n",
    "VGG-16 is still used in\n",
    "- Backbone for many models\n",
    "- Perceptual loss for style transfer and auto-encoder (super-resolution, variational auto-encoder) models\n",
    "- From my experience, VGG-16 has a place, color and size invariant feature extraction property because of pooling layers and convolutional stack."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "VGG-16 architecture:\n",
    "<img src=\"images/od/vgg_16_1.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "One of the best features extractor for background jobs:\n",
    "- High dimension\n",
    "- Spark job for image clustering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## SSD (Single shot multi box detection)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "SSD is one of the single shot detection family members.\n",
    "<br>\n",
    "For object detection, as we remember from localization models, we need fixed number of bounding boxes.\n",
    "<br>\n",
    "SSD does this by assigning $n$ bounding boxes per pixel on feature map\n",
    "<br>\n",
    "Why on features map, because they have smaller number of pixels, smaller special (height and with) size\n",
    "<br>\n",
    "The number of bounding boxes might be different for the layer\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "General architecture with non max suppression:\n",
    "<img src=\"images/od/ssd_2.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Detector and classifier block:\n",
    "<img src=\"images/od/ssd_3.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We have a feature map with low spatial size for block, generate three bounding boxes per pixel:\n",
    "<img src=\"images/od/ssd_4.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "With corner correction:\n",
    "<img src=\"images/od/ssd_7.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Train with appropriated loss:\n",
    "<img src=\"images/od/ssd_10.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Different amount of bounding boxes per layer and correction at the end:\n",
    "<img src=\"images/od/ssd_12.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## ResNet and notion of the skip-connections"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "ResNet by the Microsoft was winner of the 2015 ImageNet competition and the first model which outperformed the human"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The main idea behind the ResNet architecture is so called skip-connections or residual blocks:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Residual block:\n",
    "<img src=\"images/od/resnet_1.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "With different interpretations:\n",
    "<img src=\"images/od/resnet_2.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "With the skip connections models models depth limit was increased with performance gain"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "ResNet vs other models\n",
    "<img src=\"images/od/resnet_3.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Almost all modern architectures for classification (which are used as backbones), or encoders use some kind of modification of residual connections:\n",
    "- Ineption-ResNet\n",
    "- ResNeXt\n",
    "- MobileNet\n",
    "- EfficientNet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Residual connections in other architectures:\n",
    "<img src=\"images/od/resnet_4.jpg\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Feature pyramid models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Lets combing down-sampling and up-sampling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "UNet\n",
    "<img src=\"images/od/unet.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's make it slightly faster"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/od/fpn_1.jpeg\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/od/fpn_2.jpeg\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## RetinaNet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Feature pyramids with ResNet50 as backbone:\n",
    "\n",
    "<img src=\"images/od/retina_1.jpg\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Feature pyramid extracts and preserves semantically and spatially, after some feature pyramid block (two) predictors are applied, bounding box regression with $H \\times W$ spacial size and $4A$ channels and classification networks with $H \\times W$ spacial size and $KA$ channels, class per anchor box:\n",
    "\n",
    "<img src=\"images/od/retina_2.jpg\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "And classification:\n",
    "<img src=\"images/od/retina_3.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Regression loss:\n",
    "$$\n",
    "\\begin{align}\n",
    "T^i_x &= (G^i_x - A^i_x) / A^i_w  \\\\\n",
    "T^i_y &= (G^i_y - A^i_y) / A^i_h \\\\\n",
    "T^i_w &= log(G^i_w / A^i_w) \\\\\n",
    "T^i_h &= log(G^i_h / A^i_h)\n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "And loss is applied as:\n",
    "$$\n",
    "L_{loc} = \\sum_{j \\in \\{x, y, w, h\\}}smooth_{L1}(P^i_j - T^i_j)\n",
    "$$\n",
    "<br>\n",
    "where:\n",
    "$$\n",
    "\\begin{equation}\n",
    "smooth_{L1}(x) = \n",
    "\\begin{cases}\n",
    "0.5x^2 &|x| < 1 \\\\\n",
    "|x| - 0.5 &|x| \\geq 1\n",
    "\\end{cases}\n",
    "\\end{equation}\n",
    "$$\n",
    "<br>\n",
    "and\n",
    "$$\n",
    "P^i = (P^i_x, P^i_y, P^i_w, P^i_h)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Focal loss:\n",
    "$$\n",
    "L_{cls} = -\\sum_{i=1}^{K}(y_ilog(p_i)(1-p_i)^\\gamma \\alpha_i + (1 - y_i)log(1 - p_i)p_i^\\gamma (1 - \\alpha_i))\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Here we have two dependent hyper-parameters:\n",
    "- Weighting parameter $\\alpha$ which is responsible for class imbalance $\\alpha_i \\in [0, 1]$\n",
    "- Focusing parameter $\\gamma$ for background foreground distinguish $\\gamma \\in (0, +\\infty)$\n",
    "- During the weight initialization, biases on the last layer are initialized bigger with some rules"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "This parameters are balancing for meaning hard negatives.\n",
    "<br>\n",
    "Because of big amount of background images, data might be classified as background easily"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Here's the example of training with and without focal loss:\n",
    "<img src=\"images/od/focal_1.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "At the end we'll add that each detector outputs $1$K boxes and then reduced by the non max suppression algorithm "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Multi-shot object detection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Problems with detecting small objects\n",
    "- Different size of sliding windows\n",
    "- Run classifier network per patch\n",
    "- Computationally expensive"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Region proposals"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "With classical CV (greedy) algorithm calculated regions of images - Selective Search\n",
    "<img src=\"images/od/selective_search.jpg\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### R-CNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We calculate 2000 region proposals and run them through the pre-trained cnn classifier (VGG-16 or ResNet-50)\n",
    "<img src=\"images/od/r_cnn_1.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We train SVM classifiers per extracted features and also BBox regression to improve boxes:\n",
    "<img src=\"images/od/r_cnn_2.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Improves accuracy\n",
    "- Computationally expensive for training\n",
    "- Computationally expensive for testing / inference\n",
    "- Slow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Fast-RCNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Instead of calculation of region proposals on input image, we first run image through CNN model and calculate region proposals on smaller feature map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Region proposals have high recall and many of them are classified as background"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Region proposals $N \\times 5$ image index and coordinates\n",
    "<img src=\"images/od/region_proposal_cat.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Then we use ROI pooling to adjust features and flatten pooled image parts for classification\n",
    "ROI poolint takes a section of the input feature map that corresponds to it and scales it to some pre-defined size\n",
    "<img src=\"images/od/roi_pooling_1.gif\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/od/fast_r_cnn.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Less computational resources\n",
    "- Faster training time\n",
    "- Faster testing / inference time\n",
    "- The same performance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Faster-RCNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Instead of using selective search, let's use separate convolutional neural network which calculates region proposals"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/od/faster_r_cnn_1.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/od/faster_r_cnn_2.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Less computationally expensive\n",
    "- Region proposals calculation is learnable\n",
    "- Faster for training\n",
    "- Faster for test / inference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Questions\n",
    "<img src=\"images/od/questions_2.jpg\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Thank you"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}