5.4_ResNet.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qi9L0veTKe7M"
   },
   "source": [
    "# Adapted from [Dive into Deep Learning](https://d2l.ai/index.html)\n",
    "\n",
    "Sourced from Chapter 7, Section 6. The authors are prominent Amazon data scientists, so they work primarily with the [MXNet](https://mxnet.apache.org/versions/1.7.0/) framework rather than [TensorFlow](https://www.tensorflow.org/) or [PyTorch](https://pytorch.org/). As such, this assignment will rely on MXNet as well. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3It3ictg77Bq"
   },
   "source": [
    "The following additional libraries are needed to run this\n",
    "notebook. Note that running on Colab is experimental, please report a Github\n",
    "issue if you have any problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tYaw1beRgYDV"
   },
   "source": [
    "## Instruction for running this notebook in Colab \n",
    "\n",
    "Make sure to navigate to Runtime > Change runtime type, select GPU in the Hardward accelerator drop-down menu, and click Save *before* running the following code cell! Otherwise, `mxnet-cu101` won't install."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4Q2SU8JE77Br"
   },
   "outputs": [],
   "source": [
    "!pip install d2l==0.15.1\n",
    "!pip install -U mxnet-cu101==1.7.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "i6RvPIsd77Bx",
    "origin_pos": 0
   },
   "source": [
    "# Residual Networks (ResNets)\n",
    ":label:`sec_resnet`\n",
    "\n",
    "As we design increasingly deeper networks it becomes imperative to understand how adding layers can increase the complexity and expressiveness of the network.\n",
    "Even more important is the ability to design networks where adding layers makes networks strictly more expressive rather than just different.\n",
    "To make some progress we need a bit of mathematics.\n",
    "\n",
    "\n",
    "## Function Classes\n",
    "\n",
    "Consider $\\mathcal{F}$, the class of functions that a specific network architecture (together with learning rates and other hyperparameter settings) can reach.\n",
    "That is, for all $f \\in \\mathcal{F}$ there exists some set of parameters (e.g., weights and biases) that can be obtained through training on a suitable dataset.\n",
    "Let us assume that $f^*$ is the \"truth\" function that we really would like to find.\n",
    "If it is in $\\mathcal{F}$, we are in good shape but typically we will not be quite so lucky.\n",
    "Instead, we will try to find some $f^*_\\mathcal{F}$ which is our best bet within $\\mathcal{F}$.\n",
    "For instance, \n",
    "given a dataset with features $\\mathbf{X}$\n",
    "and labels $\\mathbf{y}$,\n",
    "we might try finding it by solving the following optimization problem:\n",
    "\n",
    "$$f^*_\\mathcal{F} \\stackrel{\\mathrm{def}}{=} \\mathop{\\mathrm{argmin}}_f L(\\mathbf{X}, \\mathbf{y}, f) \\text{ subject to } f \\in \\mathcal{F}.$$\n",
    "\n",
    "It is only reasonable to assume that if we design a different and more powerful architecture $\\mathcal{F}'$ we should arrive at a better outcome. In other words, we would expect that $f^*_{\\mathcal{F}'}$ is \"better\" than $f^*_{\\mathcal{F}}$. However, if $\\mathcal{F} \\not\\subseteq \\mathcal{F}'$ there is no guarantee that this should even happen. In fact, $f^*_{\\mathcal{F}'}$ might well be worse. \n",
    "As illustrated by :numref:`fig_functionclasses`,\n",
    "for non-nested function classes, a larger function class does not always move closer to the \"truth\" function $f^*$. For instance,\n",
    "on the left of :numref:`fig_functionclasses`,\n",
    "though $\\mathcal{F}_3$ is closer to $f^*$ than $\\mathcal{F}_1$, $\\mathcal{F}_6$ moves away and there is no guarantee that further increasing the complexity can reduce the distance from $f^*$.\n",
    "With nested function classes\n",
    "where $\\mathcal{F}_1 \\subseteq \\ldots \\subseteq \\mathcal{F}_6$\n",
    "on the right of :numref:`fig_functionclasses`,\n",
    "we can avoid the aforementioned issue from the non-nested function classes.\n",
    "\n",
    "\n",
    "![For non-nested function classes, a larger (indicated by area) function class does not guarantee to get closer to the \"truth\" function ($f^*$). This does not happen in nested function classes.](http://d2l.ai/_images/functionclasses.svg)\n",
    ":label:`fig_functionclasses`\n",
    "\n",
    "Thus,\n",
    "only if larger function classes contain the smaller ones are we guaranteed that increasing them strictly increases the expressive power of the network.\n",
    "For deep neural networks,\n",
    "if we can \n",
    "train the newly-added layer into an identity function $f(\\mathbf{x}) = \\mathbf{x}$, the new model will be as effective as the original model. As the new model may get a better solution to fit the training dataset, the added layer might make it easier to reduce training errors.\n",
    "\n",
    "This is the question that He et al. considered when working on very deep computer vision models :cite:`He.Zhang.Ren.ea.2016`. \n",
    "At the heart of their proposed *residual network* (*ResNet*) is the idea that every additional layer should \n",
    "more easily\n",
    "contain the identity function as one of its elements. \n",
    "These considerations are rather profound but they led to a surprisingly simple\n",
    "solution, a *residual block*.\n",
    "With it, ResNet won the ImageNet Large Scale Visual Recognition Challenge in 2015. The design had a profound influence on how to\n",
    "build deep neural networks.\n",
    "\n",
    "\n",
    "\n",
    "## Residual Blocks\n",
    "\n",
    "Let us focus on a local part of a neural network, as depicted in :numref:`fig_residual_block`. Denote the input by $\\mathbf{x}$.\n",
    "We assume that the desired underlying mapping we want to obtain by learning is $f(\\mathbf{x})$, to be used as the input to the activation function on the top.\n",
    "On the left of :numref:`fig_residual_block`,\n",
    "the portion within the dotted-line box \n",
    "must directly learn the mapping $f(\\mathbf{x})$.\n",
    "On the right,\n",
    "the portion within the dotted-line box\n",
    "needs to\n",
    "learn the *residual mapping* $f(\\mathbf{x}) - \\mathbf{x}$,\n",
    "which is how the residual block derives its name.\n",
    "If the identity mapping $f(\\mathbf{x}) = \\mathbf{x}$ is the desired underlying mapping,\n",
    "the residual mapping is easier to learn:\n",
    "we only need to push the weights and biases\n",
    "of the\n",
    "upper weight layer (e.g., fully-connected layer and convolutional layer)\n",
    "within the dotted-line box\n",
    "to zero.\n",
    "The right figure in :numref:`fig_residual_block` illustrates the  *residual block* of ResNet,\n",
    "where the solid line carrying the layer input \n",
    "$\\mathbf{x}$ to the addition operator\n",
    "is called a *residual connection* (or *shortcut connection*).\n",
    "With residual blocks, inputs can \n",
    "forward propagate faster through the residual connections across layers.\n",
    "\n",
    "![A regular block (left) and a residual block (right).](https://github.com/d2l-ai/d2l-en-colab/blob/master/img/residual-block.svg?raw=1)\n",
    ":label:`fig_residual_block`\n",
    "\n",
    "\n",
    "ResNet follows VGG's full $3\\times 3$ convolutional layer design. The residual block has two $3\\times 3$ convolutional layers with the same number of output channels. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function. Then, we skip these two convolution operations and add the input directly before the final ReLU activation function.\n",
    "This kind of design requires that the output of the two convolutional layers has to be of the same shape as the input, so that they can be added together. If we want to change the number of channels, we need to introduce an additional $1\\times 1$ convolutional layer to transform the input into the desired shape for the addition operation. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7jtg4SUo77B1",
    "origin_pos": 4
   },
   "source": [
    "The code below generates two types of networks: one where we add the input to the output before applying the ReLU nonlinearity whenever `use_1x1conv=False`, and one where we adjust channels and resolution by means of a $1 \\times 1$ convolution before adding. :numref:`fig_resnet_block` illustrates this:\n",
    "\n",
    "![ResNet block with and without $1 \\times 1$ convolution.](https://github.com/d2l-ai/d2l-en-colab/blob/master/img/resnet-block.svg?raw=1)\n",
    ":label:`fig_resnet_block`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vMkpAKlSnYF9"
   },
   "source": [
    "### Exercise: \n",
    "Implement the figures above as the `Residual` class. \n",
    "\n",
    "Notes: \n",
    "1. The d2l module allows us to use utility functions developed for the Dive into Deep Learning book.\n",
    "2. MXNet's np and npx modules allow for Numpy-compatible coding, which MXNet's basic ndarray module doesn't, as [this](https://github.com/apache/incubator-mxnet/issues/14253) post illustrates. You might also find [this](https://stackoverflow.com/questions/58933738/mxnet-import-nd-or-np-to-use-arrays) Stack Overflow query helpful.\n",
    "3. [Gluon](https://mxnet.apache.org/versions/1.6/api/python/docs/api/gluon/index.html) is to MXNet as Keras is to TensorFlow.\n",
    "4. You might find the following functions useful: [`mxnet.gluon.nn.Conv2D()`](https://mxnet.apache.org/versions/1.7.0/api/python/docs/api/gluon/nn/index.html#mxnet.gluon.nn.Conv2D), [`mxnet.gluon.nn.BatchNorm()`](https://mxnet.apache.org/versions/1.7.0/api/python/docs/api/gluon/nn/index.html#mxnet.gluon.nn.BatchNorm), [`mxnet.gluon.nn.Activation()`](https://mxnet.apache.org/versions/1.7.0/api/python/docs/api/gluon/nn/index.html#mxnet.gluon.nn.Activation), [`npx.relu()`](https://mxnet.apache.org/versions/1.7.0/api/python/docs/api/ndarray/op/index.html#mxnet.ndarray.op.relu).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "-MCesdab77By",
    "origin_pos": 1,
    "tab": [
     "mxnet"
    ]
   },
   "outputs": [],
   "source": [
    "from d2l import mxnet as d2l\n",
    "from mxnet import np, npx\n",
    "from mxnet.gluon import nn\n",
    "npx.set_np()\n",
    "\n",
    "class Residual(nn.Block):  #@save\n",
    "    \"\"\"The Residual block of ResNet.\"\"\"\n",
    "    def __init__(self, num_channels, use_1x1conv=False, strides=1, **kwargs):\n",
    "\n",
    "        super().__init__(**kwargs)\n",
    "        # Convolutional layer with num_channels output channels, kernel size of 3, \n",
    "        # and padding of 1. Pass in the strides used to instantiate the class.\n",
    "        self.conv1 = nn.Conv2D(num_channels, kernel_size=3, padding=1, strides=strides)\n",
    "        # Convolutional layer like the first, but with the default stride of 1\n",
    "        self.conv2 = nn.Conv2D(num_channels, kernel_size=3, padding=1, strides=1)\n",
    "        # If we're using a 1 x 1 convolution on the residual branch...\n",
    "        if use_1x1conv:\n",
    "            # Convolutional layer with num_channels output channels and kernel size of 1. \n",
    "            # Pass in the strides used to instantiate the class.\n",
    "            self.conv3 = nn.Conv2D(num_channels, kernel_size=1, strides=strides)\n",
    "        else:\n",
    "            # DON'T replace the None on the next line!\n",
    "            self.conv3 = None\n",
    "        # Batch normalization. Use the default parameters. \n",
    "        self.bn1 = nn.BatchNorm()\n",
    "        # Batch normalization. Use the default parameters.\n",
    "        self.bn2 = nn.BatchNorm()\n",
    "\n",
    "    def forward(self, X):\n",
    "        # Pass the input layer through the first convolutional layer\n",
    "        Y = self.conv1(X)\n",
    "        # First batch normalization layer\n",
    "        Y = self.bn1(Y)\n",
    "        # ReLU activation\n",
    "        Y = npx.relu(Y)\n",
    "        # Second convolutional layer\n",
    "        Y = self.conv2(Y)\n",
    "        # Second batch normalization layer \n",
    "        Y = self.bn2(Y)\n",
    "        # If the third convolutional layer exists, pass the input layer through it\n",
    "        if self.conv3:\n",
    "            X = self.conv3(X)\n",
    "        # Add the two branches together, then apply a ReLU activation to the summed layer\n",
    "        return npx.relu(Y + X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-Mh0gv-x2oRo"
   },
   "source": [
    "Now let us look at a situation where the input and output are of the same shape."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "IScmkMsV77B1",
    "origin_pos": 5,
    "tab": [
     "mxnet"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4, 3, 6, 6)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "blk = Residual(3)\n",
    "blk.initialize()\n",
    "X = np.random.uniform(size=(4, 3, 6, 6))\n",
    "blk(X).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZSlypLnk77B4",
    "origin_pos": 8
   },
   "source": [
    "We also have the option to halve the output height and width while increasing the number of output channels.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "IJow9uXS77B5",
    "origin_pos": 9,
    "tab": [
     "mxnet"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4, 6, 3, 3)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "blk = Residual(6, use_1x1conv=True, strides=2)\n",
    "blk.initialize()\n",
    "blk(X).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "v9UMmyCE77B7",
    "origin_pos": 12
   },
   "source": [
    "## ResNet Model\n",
    "\n",
    "### Exercise: \n",
    "\n",
    "Instantiate the ResNet model with Gluon's [`nn.Sequential()`](https://mxnet.apache.org/versions/1.7.0/api/python/docs/api/gluon/nn/index.html#mxnet.gluon.nn.Sequential) API, then add to the model as follows: \n",
    "\n",
    "The first two layers of ResNet are the same as those of GoogLeNet: the $7\\times 7$ convolutional layer with 64 output channels and a stride of 2 is followed by the $3\\times 3$ maximum pooling layer with a stride of 2. The difference is the batch normalization layer added after each convolutional layer in ResNet.\n",
    "\n",
    "You may find Gluon's [`nn.Activation()`](https://mxnet.apache.org/versions/1.7.0/api/python/docs/api/gluon/nn/index.html#mxnet.gluon.nn.Activation) and [`nn.MaxPool2D()`](https://mxnet.apache.org/versions/1.7.0/api/python/docs/api/gluon/nn/index.html#mxnet.gluon.nn.MaxPool2D) useful.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "XSyCeayC77B8",
    "origin_pos": 13,
    "tab": [
     "mxnet"
    ]
   },
   "outputs": [],
   "source": [
    "# Instantiate the model\n",
    "net = nn.Sequential()\n",
    "# Start adding to the model\n",
    "net.add(# Convolutional layer\n",
    "        nn.Conv2D(64, kernel_size=7, strides=2, padding=3),\n",
    "        # Batch normalization\n",
    "        nn.BatchNorm(), \n",
    "        # ReLU activation\n",
    "        nn.Activation('relu'),\n",
    "        # Max pooling\n",
    "        nn.MaxPool2D(pool_size=3, strides=2, padding=1)\n",
    "       )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "InGdMxtV77B_",
    "origin_pos": 16
   },
   "source": [
    "GoogLeNet uses four modules made up of Inception blocks.\n",
    "However, ResNet uses four modules made up of residual blocks, each of which uses several residual blocks with the same number of output channels. \n",
    "The number of channels in the first module is the same as the number of input channels. Since a maximum pooling layer with a stride of 2 has already been used, it is not necessary to reduce the height and width. In the first residual block for each of the subsequent modules, the number of channels is doubled compared with that of the previous module, and the height and width are halved.\n",
    "\n",
    "Now, we implement this module. Note that special processing has been performed on the first module.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LX07kZHKNE1t"
   },
   "source": [
    "### Exercise: \n",
    "Build a function which wraps around a `Residual` block as defined earlier. Make use of the `nn.Sequential()` API and the `add()` method for MXNet models. Give the first block all possible default arguments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "r79QAAS277B_",
    "origin_pos": 17,
    "tab": [
     "mxnet"
    ]
   },
   "outputs": [],
   "source": [
    "def resnet_block(num_channels, num_residuals, first_block=False):\n",
    "    # Initialize the ResNet block\n",
    "    blk = nn.Sequential()\n",
    "    # Iterate through the residual sub-blocks\n",
    "    for i in range(num_residuals):\n",
    "        # If this is the first sub-block but NOT the first ResNet block of the whole model...\n",
    "        if i == 0 and not first_block:\n",
    "            # Add a residual sub-block with num_channels output channels, \n",
    "            # 1 x 1 convolutions, and a stride of 2\n",
    "            blk.add(Residual(num_channels, use_1x1conv=True, strides=2))\n",
    "        else:\n",
    "            # If not, add a residual sub-block with num_channels output channels \n",
    "            # and otherwise default arguments\n",
    "            blk.add(Residual(num_channels))\n",
    "    # Return the ResNet block\n",
    "    return blk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HdwX7M9d77CE",
    "origin_pos": 20
   },
   "source": [
    "Then, we add all the modules to ResNet. Here, two residual blocks are used for each module.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "o_onPSdRvWNR"
   },
   "source": [
    "### Exercise: \n",
    "Add 4 ResNet blocks to the model, with 64, 128, 256, and 512 respective output channels, and each with 2 sub-blocks. Don't forget about the `first_block` parameter!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "cLY4Gyv377CF",
    "origin_pos": 21,
    "tab": [
     "mxnet"
    ]
   },
   "outputs": [],
   "source": [
    "net.add(resnet_block(64, 2, first_block=True),\n",
    "        resnet_block(128, 2),\n",
    "        resnet_block(256, 2),\n",
    "        resnet_block(512, 2)\n",
    "       )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HvYuXVWd77CK",
    "origin_pos": 24
   },
   "source": [
    "### Exercise: \n",
    "Finally, just like GoogLeNet, we add a global average pooling layer, followed by the fully-connected layer output.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "dx2kUaB877CK",
    "origin_pos": 25,
    "tab": [
     "mxnet"
    ]
   },
   "outputs": [],
   "source": [
    "net.add(nn.GlobalAvgPool2D(),\n",
    "       nn.Dense(10)\n",
    "       )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4ibpfSd977CN",
    "origin_pos": 28
   },
   "source": [
    "There are 4 convolutional layers in each module (excluding the $1\\times 1$ convolutional layer). Together with the first $7\\times 7$ convolutional layer and the final fully-connected layer, there are 18 layers in total. Therefore, this model is commonly known as ResNet-18.\n",
    "By configuring different numbers of channels and residual blocks in the module, we can create different ResNet models, such as the deeper 152-layer ResNet-152. Although the main architecture of ResNet is similar to that of GoogLeNet, ResNet's structure is simpler and easier to modify. All these factors have resulted in the rapid and widespread use of ResNet. :numref:`fig_resnet18` depicts the full ResNet-18.\n",
    "\n",
    "![The ResNet-18 architecture.](http://d2l.ai/_images/resnet18.svg)\n",
    ":label:`fig_resnet18`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "v4sBmWZc1INk"
   },
   "source": [
    "### Exercise: \n",
    "Before training ResNet, let us observe how the input shape changes across different modules in ResNet. As in all the previous architectures, the resolution decreases while the number of channels increases up until the point where a global average pooling layer aggregates all features. Print the output shape of each module in the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "CbXajIRG77CN",
    "origin_pos": 29,
    "tab": [
     "mxnet"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "conv5 output shape:\t (1, 64, 112, 112)\n",
      "batchnorm4 output shape:\t (1, 64, 112, 112)\n",
      "relu0 output shape:\t (1, 64, 112, 112)\n",
      "pool0 output shape:\t (1, 64, 56, 56)\n",
      "sequential1 output shape:\t (1, 64, 56, 56)\n",
      "sequential2 output shape:\t (1, 128, 28, 28)\n",
      "sequential3 output shape:\t (1, 256, 14, 14)\n",
      "sequential4 output shape:\t (1, 512, 7, 7)\n",
      "pool1 output shape:\t (1, 512, 1, 1)\n",
      "dense0 output shape:\t (1, 10)\n"
     ]
    }
   ],
   "source": [
    "# Initialize the input with a random array\n",
    "X = np.random.uniform(size=(1, 1, 224, 224))\n",
    "# Initialize the model\n",
    "net.initialize()\n",
    "# Loop over the layers in the model\n",
    "for layer in net:\n",
    "    # Apply the current layer to its input or the output of the previous layer\n",
    "    X = layer(X)\n",
    "    # Print the current layer's name and the output's shape\n",
    "    print(layer.name, 'output shape:\\t', X.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6EY59nfUzrJe"
   },
   "source": [
    "### Expected Output: \n",
    "```\n",
    "conv5 output shape:\t (1, 64, 112, 112)\n",
    "batchnorm4 output shape:\t (1, 64, 112, 112)\n",
    "relu0 output shape:\t (1, 64, 112, 112)\n",
    "pool0 output shape:\t (1, 64, 56, 56)\n",
    "sequential1 output shape:\t (1, 64, 56, 56)\n",
    "sequential2 output shape:\t (1, 128, 28, 28)\n",
    "sequential3 output shape:\t (1, 256, 14, 14)\n",
    "sequential4 output shape:\t (1, 512, 7, 7)\n",
    "pool1 output shape:\t (1, 512, 1, 1)\n",
    "dense0 output shape:\t (1, 10)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5Mg6T7Tx2XQz"
   },
   "source": [
    "### Exercise: \n",
    "For a more detailed look, use the model's [`summary()`](https://mxnet.apache.org/versions/1.7.0/api/python/docs/api/gluon/block.html#mxnet.gluon.Block.summary) method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "#?net.initialize\n",
    "#?net.summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "Mab_Fgd3z0eg"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------------------------------------\n",
      "        Layer (type)                                Output Shape         Param #\n",
      "================================================================================\n",
      "               Input                            (1, 1, 224, 224)               0\n",
      "            Conv2D-1                           (1, 64, 112, 112)            3200\n",
      "         BatchNorm-2                           (1, 64, 112, 112)             256\n",
      "        Activation-3                           (1, 64, 112, 112)               0\n",
      "         MaxPool2D-4                             (1, 64, 56, 56)               0\n",
      "            Conv2D-5                             (1, 64, 56, 56)           36928\n",
      "         BatchNorm-6                             (1, 64, 56, 56)             256\n",
      "            Conv2D-7                             (1, 64, 56, 56)           36928\n",
      "         BatchNorm-8                             (1, 64, 56, 56)             256\n",
      "          Residual-9                             (1, 64, 56, 56)               0\n",
      "           Conv2D-10                             (1, 64, 56, 56)           36928\n",
      "        BatchNorm-11                             (1, 64, 56, 56)             256\n",
      "           Conv2D-12                             (1, 64, 56, 56)           36928\n",
      "        BatchNorm-13                             (1, 64, 56, 56)             256\n",
      "         Residual-14                             (1, 64, 56, 56)               0\n",
      "           Conv2D-15                            (1, 128, 28, 28)           73856\n",
      "        BatchNorm-16                            (1, 128, 28, 28)             512\n",
      "           Conv2D-17                            (1, 128, 28, 28)          147584\n",
      "        BatchNorm-18                            (1, 128, 28, 28)             512\n",
      "           Conv2D-19                            (1, 128, 28, 28)            8320\n",
      "         Residual-20                            (1, 128, 28, 28)               0\n",
      "           Conv2D-21                            (1, 128, 28, 28)          147584\n",
      "        BatchNorm-22                            (1, 128, 28, 28)             512\n",
      "           Conv2D-23                            (1, 128, 28, 28)          147584\n",
      "        BatchNorm-24                            (1, 128, 28, 28)             512\n",
      "         Residual-25                            (1, 128, 28, 28)               0\n",
      "           Conv2D-26                            (1, 256, 14, 14)          295168\n",
      "        BatchNorm-27                            (1, 256, 14, 14)            1024\n",
      "           Conv2D-28                            (1, 256, 14, 14)          590080\n",
      "        BatchNorm-29                            (1, 256, 14, 14)            1024\n",
      "           Conv2D-30                            (1, 256, 14, 14)           33024\n",
      "         Residual-31                            (1, 256, 14, 14)               0\n",
      "           Conv2D-32                            (1, 256, 14, 14)          590080\n",
      "        BatchNorm-33                            (1, 256, 14, 14)            1024\n",
      "           Conv2D-34                            (1, 256, 14, 14)          590080\n",
      "        BatchNorm-35                            (1, 256, 14, 14)            1024\n",
      "         Residual-36                            (1, 256, 14, 14)               0\n",
      "           Conv2D-37                              (1, 512, 7, 7)         1180160\n",
      "        BatchNorm-38                              (1, 512, 7, 7)            2048\n",
      "           Conv2D-39                              (1, 512, 7, 7)         2359808\n",
      "        BatchNorm-40                              (1, 512, 7, 7)            2048\n",
      "           Conv2D-41                              (1, 512, 7, 7)          131584\n",
      "         Residual-42                              (1, 512, 7, 7)               0\n",
      "           Conv2D-43                              (1, 512, 7, 7)         2359808\n",
      "        BatchNorm-44                              (1, 512, 7, 7)            2048\n",
      "           Conv2D-45                              (1, 512, 7, 7)         2359808\n",
      "        BatchNorm-46                              (1, 512, 7, 7)            2048\n",
      "         Residual-47                              (1, 512, 7, 7)               0\n",
      "  GlobalAvgPool2D-48                              (1, 512, 1, 1)               0\n",
      "            Dense-49                                     (1, 10)            5130\n",
      "================================================================================\n",
      "Parameters in forward computation graph, duplicate included\n",
      "   Total params: 11186186\n",
      "   Trainable params: 11178378\n",
      "   Non-trainable params: 7808\n",
      "Shared params in forward computation graph: 0\n",
      "Unique parameters in model: 11186186\n",
      "--------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Initialize the input, consisting of 1 example, 1 output channel, and a pixel height and width of 224\n",
    "X = np.random.uniform(size=(1, 1, 224, 224))\n",
    "# Reinitialize the model\n",
    "# HINT: Use the force_reinit parameter\n",
    "net.initialize(force_reinit=True)\n",
    "# Summarize the model\n",
    "net.summary(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "E18i4zci5DKD"
   },
   "source": [
    "### Expected Output: \n",
    "```\n",
    "--------------------------------------------------------------------------------\n",
    "        Layer (type)                                Output Shape         Param #\n",
    "================================================================================\n",
    "               Input                            (1, 1, 224, 224)               0\n",
    "            Conv2D-1                           (1, 64, 112, 112)            3200\n",
    "         BatchNorm-2                           (1, 64, 112, 112)             256\n",
    "        Activation-3                           (1, 64, 112, 112)               0\n",
    "         MaxPool2D-4                             (1, 64, 56, 56)               0\n",
    "            Conv2D-5                             (1, 64, 56, 56)           36928\n",
    "         BatchNorm-6                             (1, 64, 56, 56)             256\n",
    "            Conv2D-7                             (1, 64, 56, 56)           36928\n",
    "         BatchNorm-8                             (1, 64, 56, 56)             256\n",
    "          Residual-9                             (1, 64, 56, 56)               0\n",
    "           Conv2D-10                             (1, 64, 56, 56)           36928\n",
    "        BatchNorm-11                             (1, 64, 56, 56)             256\n",
    "           Conv2D-12                             (1, 64, 56, 56)           36928\n",
    "        BatchNorm-13                             (1, 64, 56, 56)             256\n",
    "         Residual-14                             (1, 64, 56, 56)               0\n",
    "           Conv2D-15                            (1, 128, 28, 28)           73856\n",
    "        BatchNorm-16                            (1, 128, 28, 28)             512\n",
    "           Conv2D-17                            (1, 128, 28, 28)          147584\n",
    "        BatchNorm-18                            (1, 128, 28, 28)             512\n",
    "           Conv2D-19                            (1, 128, 28, 28)            8320\n",
    "         Residual-20                            (1, 128, 28, 28)               0\n",
    "           Conv2D-21                            (1, 128, 28, 28)          147584\n",
    "        BatchNorm-22                            (1, 128, 28, 28)             512\n",
    "           Conv2D-23                            (1, 128, 28, 28)          147584\n",
    "        BatchNorm-24                            (1, 128, 28, 28)             512\n",
    "         Residual-25                            (1, 128, 28, 28)               0\n",
    "           Conv2D-26                            (1, 256, 14, 14)          295168\n",
    "        BatchNorm-27                            (1, 256, 14, 14)            1024\n",
    "           Conv2D-28                            (1, 256, 14, 14)          590080\n",
    "        BatchNorm-29                            (1, 256, 14, 14)            1024\n",
    "           Conv2D-30                            (1, 256, 14, 14)           33024\n",
    "         Residual-31                            (1, 256, 14, 14)               0\n",
    "           Conv2D-32                            (1, 256, 14, 14)          590080\n",
    "        BatchNorm-33                            (1, 256, 14, 14)            1024\n",
    "           Conv2D-34                            (1, 256, 14, 14)          590080\n",
    "        BatchNorm-35                            (1, 256, 14, 14)            1024\n",
    "         Residual-36                            (1, 256, 14, 14)               0\n",
    "           Conv2D-37                              (1, 512, 7, 7)         1180160\n",
    "        BatchNorm-38                              (1, 512, 7, 7)            2048\n",
    "           Conv2D-39                              (1, 512, 7, 7)         2359808\n",
    "        BatchNorm-40                              (1, 512, 7, 7)            2048\n",
    "           Conv2D-41                              (1, 512, 7, 7)          131584\n",
    "         Residual-42                              (1, 512, 7, 7)               0\n",
    "           Conv2D-43                              (1, 512, 7, 7)         2359808\n",
    "        BatchNorm-44                              (1, 512, 7, 7)            2048\n",
    "           Conv2D-45                              (1, 512, 7, 7)         2359808\n",
    "        BatchNorm-46                              (1, 512, 7, 7)            2048\n",
    "         Residual-47                              (1, 512, 7, 7)               0\n",
    "  GlobalAvgPool2D-48                              (1, 512, 1, 1)               0\n",
    "            Dense-49                                     (1, 10)            5130\n",
    "================================================================================\n",
    "Parameters in forward computation graph, duplicate included\n",
    "   Total params: 11186186\n",
    "   Trainable params: 11178378\n",
    "   Non-trainable params: 7808\n",
    "Shared params in forward computation graph: 0\n",
    "Unique parameters in model: 11186186\n",
    "--------------------------------------------------------------------------------\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "t4VsaGCD77CQ",
    "origin_pos": 32
   },
   "source": [
    "## Training\n",
    "\n",
    "### Train the ResNet on the Fashion-MNIST dataset. Notes: \n",
    "1. This process takes approximately 8 minutes when running a Colab on a single GPU \n",
    "2. The next cell uses utility functions developed for the Dive into Deep Learning (d2l) book\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "VBaUlpeS77CR",
    "origin_pos": 33,
    "tab": [
     "mxnet"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss 0.021, train acc 0.994, test acc 0.912\n",
      "1725.5 examples/sec on gpu(0)\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Created with matplotlib (https://matplotlib.org/) -->\n",
       "<svg height=\"180.65625pt\" version=\"1.1\" viewBox=\"0 0 238.965625 180.65625\" width=\"238.965625pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2021-06-19T10:35:55.427285</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.3.4, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linecap:butt;stroke-linejoin:round;}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 180.65625 \n",
       "L 238.965625 180.65625 \n",
       "L 238.965625 0 \n",
       "L 0 0 \n",
       "z\n",
       "\" style=\"fill:none;\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 30.103125 143.1 \n",
       "L 225.403125 143.1 \n",
       "L 225.403125 7.2 \n",
       "L 30.103125 7.2 \n",
       "z\n",
       "\" style=\"fill:#ffffff;\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <path clip-path=\"url(#p878e57563c)\" d=\"M 51.803125 143.1 \n",
       "L 51.803125 7.2 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" id=\"mdc5fb2c20c\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"51.803125\" xlink:href=\"#mdc5fb2c20c\" y=\"143.1\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 2 -->\n",
       "      <g transform=\"translate(48.621875 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 19.1875 8.296875 \n",
       "L 53.609375 8.296875 \n",
       "L 53.609375 0 \n",
       "L 7.328125 0 \n",
       "L 7.328125 8.296875 \n",
       "Q 12.9375 14.109375 22.625 23.890625 \n",
       "Q 32.328125 33.6875 34.8125 36.53125 \n",
       "Q 39.546875 41.84375 41.421875 45.53125 \n",
       "Q 43.3125 49.21875 43.3125 52.78125 \n",
       "Q 43.3125 58.59375 39.234375 62.25 \n",
       "Q 35.15625 65.921875 28.609375 65.921875 \n",
       "Q 23.96875 65.921875 18.8125 64.3125 \n",
       "Q 13.671875 62.703125 7.8125 59.421875 \n",
       "L 7.8125 69.390625 \n",
       "Q 13.765625 71.78125 18.9375 73 \n",
       "Q 24.125 74.21875 28.421875 74.21875 \n",
       "Q 39.75 74.21875 46.484375 68.546875 \n",
       "Q 53.21875 62.890625 53.21875 53.421875 \n",
       "Q 53.21875 48.921875 51.53125 44.890625 \n",
       "Q 49.859375 40.875 45.40625 35.40625 \n",
       "Q 44.1875 33.984375 37.640625 27.21875 \n",
       "Q 31.109375 20.453125 19.1875 8.296875 \n",
       "z\n",
       "\" id=\"DejaVuSans-50\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-50\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path clip-path=\"url(#p878e57563c)\" d=\"M 95.203125 143.1 \n",
       "L 95.203125 7.2 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"95.203125\" xlink:href=\"#mdc5fb2c20c\" y=\"143.1\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 4 -->\n",
       "      <g transform=\"translate(92.021875 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 37.796875 64.3125 \n",
       "L 12.890625 25.390625 \n",
       "L 37.796875 25.390625 \n",
       "z\n",
       "M 35.203125 72.90625 \n",
       "L 47.609375 72.90625 \n",
       "L 47.609375 25.390625 \n",
       "L 58.015625 25.390625 \n",
       "L 58.015625 17.1875 \n",
       "L 47.609375 17.1875 \n",
       "L 47.609375 0 \n",
       "L 37.796875 0 \n",
       "L 37.796875 17.1875 \n",
       "L 4.890625 17.1875 \n",
       "L 4.890625 26.703125 \n",
       "z\n",
       "\" id=\"DejaVuSans-52\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-52\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <path clip-path=\"url(#p878e57563c)\" d=\"M 138.603125 143.1 \n",
       "L 138.603125 7.2 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"138.603125\" xlink:href=\"#mdc5fb2c20c\" y=\"143.1\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 6 -->\n",
       "      <g transform=\"translate(135.421875 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 33.015625 40.375 \n",
       "Q 26.375 40.375 22.484375 35.828125 \n",
       "Q 18.609375 31.296875 18.609375 23.390625 \n",
       "Q 18.609375 15.53125 22.484375 10.953125 \n",
       "Q 26.375 6.390625 33.015625 6.390625 \n",
       "Q 39.65625 6.390625 43.53125 10.953125 \n",
       "Q 47.40625 15.53125 47.40625 23.390625 \n",
       "Q 47.40625 31.296875 43.53125 35.828125 \n",
       "Q 39.65625 40.375 33.015625 40.375 \n",
       "z\n",
       "M 52.59375 71.296875 \n",
       "L 52.59375 62.3125 \n",
       "Q 48.875 64.0625 45.09375 64.984375 \n",
       "Q 41.3125 65.921875 37.59375 65.921875 \n",
       "Q 27.828125 65.921875 22.671875 59.328125 \n",
       "Q 17.53125 52.734375 16.796875 39.40625 \n",
       "Q 19.671875 43.65625 24.015625 45.921875 \n",
       "Q 28.375 48.1875 33.59375 48.1875 \n",
       "Q 44.578125 48.1875 50.953125 41.515625 \n",
       "Q 57.328125 34.859375 57.328125 23.390625 \n",
       "Q 57.328125 12.15625 50.6875 5.359375 \n",
       "Q 44.046875 -1.421875 33.015625 -1.421875 \n",
       "Q 20.359375 -1.421875 13.671875 8.265625 \n",
       "Q 6.984375 17.96875 6.984375 36.375 \n",
       "Q 6.984375 53.65625 15.1875 63.9375 \n",
       "Q 23.390625 74.21875 37.203125 74.21875 \n",
       "Q 40.921875 74.21875 44.703125 73.484375 \n",
       "Q 48.484375 72.75 52.59375 71.296875 \n",
       "z\n",
       "\" id=\"DejaVuSans-54\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-54\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <path clip-path=\"url(#p878e57563c)\" d=\"M 182.003125 143.1 \n",
       "L 182.003125 7.2 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"182.003125\" xlink:href=\"#mdc5fb2c20c\" y=\"143.1\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 8 -->\n",
       "      <g transform=\"translate(178.821875 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 31.78125 34.625 \n",
       "Q 24.75 34.625 20.71875 30.859375 \n",
       "Q 16.703125 27.09375 16.703125 20.515625 \n",
       "Q 16.703125 13.921875 20.71875 10.15625 \n",
       "Q 24.75 6.390625 31.78125 6.390625 \n",
       "Q 38.8125 6.390625 42.859375 10.171875 \n",
       "Q 46.921875 13.96875 46.921875 20.515625 \n",
       "Q 46.921875 27.09375 42.890625 30.859375 \n",
       "Q 38.875 34.625 31.78125 34.625 \n",
       "z\n",
       "M 21.921875 38.8125 \n",
       "Q 15.578125 40.375 12.03125 44.71875 \n",
       "Q 8.5 49.078125 8.5 55.328125 \n",
       "Q 8.5 64.0625 14.71875 69.140625 \n",
       "Q 20.953125 74.21875 31.78125 74.21875 \n",
       "Q 42.671875 74.21875 48.875 69.140625 \n",
       "Q 55.078125 64.0625 55.078125 55.328125 \n",
       "Q 55.078125 49.078125 51.53125 44.71875 \n",
       "Q 48 40.375 41.703125 38.8125 \n",
       "Q 48.828125 37.15625 52.796875 32.3125 \n",
       "Q 56.78125 27.484375 56.78125 20.515625 \n",
       "Q 56.78125 9.90625 50.3125 4.234375 \n",
       "Q 43.84375 -1.421875 31.78125 -1.421875 \n",
       "Q 19.734375 -1.421875 13.25 4.234375 \n",
       "Q 6.78125 9.90625 6.78125 20.515625 \n",
       "Q 6.78125 27.484375 10.78125 32.3125 \n",
       "Q 14.796875 37.15625 21.921875 38.8125 \n",
       "z\n",
       "M 18.3125 54.390625 \n",
       "Q 18.3125 48.734375 21.84375 45.5625 \n",
       "Q 25.390625 42.390625 31.78125 42.390625 \n",
       "Q 38.140625 42.390625 41.71875 45.5625 \n",
       "Q 45.3125 48.734375 45.3125 54.390625 \n",
       "Q 45.3125 60.0625 41.71875 63.234375 \n",
       "Q 38.140625 66.40625 31.78125 66.40625 \n",
       "Q 25.390625 66.40625 21.84375 63.234375 \n",
       "Q 18.3125 60.0625 18.3125 54.390625 \n",
       "z\n",
       "\" id=\"DejaVuSans-56\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-56\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path clip-path=\"url(#p878e57563c)\" d=\"M 225.403125 143.1 \n",
       "L 225.403125 7.2 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"225.403125\" xlink:href=\"#mdc5fb2c20c\" y=\"143.1\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(219.040625 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 12.40625 8.296875 \n",
       "L 28.515625 8.296875 \n",
       "L 28.515625 63.921875 \n",
       "L 10.984375 60.40625 \n",
       "L 10.984375 69.390625 \n",
       "L 28.421875 72.90625 \n",
       "L 38.28125 72.90625 \n",
       "L 38.28125 8.296875 \n",
       "L 54.390625 8.296875 \n",
       "L 54.390625 0 \n",
       "L 12.40625 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-49\"/>\n",
       "        <path d=\"M 31.78125 66.40625 \n",
       "Q 24.171875 66.40625 20.328125 58.90625 \n",
       "Q 16.5 51.421875 16.5 36.375 \n",
       "Q 16.5 21.390625 20.328125 13.890625 \n",
       "Q 24.171875 6.390625 31.78125 6.390625 \n",
       "Q 39.453125 6.390625 43.28125 13.890625 \n",
       "Q 47.125 21.390625 47.125 36.375 \n",
       "Q 47.125 51.421875 43.28125 58.90625 \n",
       "Q 39.453125 66.40625 31.78125 66.40625 \n",
       "z\n",
       "M 31.78125 74.21875 \n",
       "Q 44.046875 74.21875 50.515625 64.515625 \n",
       "Q 56.984375 54.828125 56.984375 36.375 \n",
       "Q 56.984375 17.96875 50.515625 8.265625 \n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \n",
       "Q 6.59375 17.96875 6.59375 36.375 \n",
       "Q 6.59375 54.828125 13.0625 64.515625 \n",
       "Q 19.53125 74.21875 31.78125 74.21875 \n",
       "z\n",
       "\" id=\"DejaVuSans-48\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- epoch -->\n",
       "     <g transform=\"translate(112.525 171.376563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path d=\"M 56.203125 29.59375 \n",
       "L 56.203125 25.203125 \n",
       "L 14.890625 25.203125 \n",
       "Q 15.484375 15.921875 20.484375 11.0625 \n",
       "Q 25.484375 6.203125 34.421875 6.203125 \n",
       "Q 39.59375 6.203125 44.453125 7.46875 \n",
       "Q 49.3125 8.734375 54.109375 11.28125 \n",
       "L 54.109375 2.78125 \n",
       "Q 49.265625 0.734375 44.1875 -0.34375 \n",
       "Q 39.109375 -1.421875 33.890625 -1.421875 \n",
       "Q 20.796875 -1.421875 13.15625 6.1875 \n",
       "Q 5.515625 13.8125 5.515625 26.8125 \n",
       "Q 5.515625 40.234375 12.765625 48.109375 \n",
       "Q 20.015625 56 32.328125 56 \n",
       "Q 43.359375 56 49.78125 48.890625 \n",
       "Q 56.203125 41.796875 56.203125 29.59375 \n",
       "z\n",
       "M 47.21875 32.234375 \n",
       "Q 47.125 39.59375 43.09375 43.984375 \n",
       "Q 39.0625 48.390625 32.421875 48.390625 \n",
       "Q 24.90625 48.390625 20.390625 44.140625 \n",
       "Q 15.875 39.890625 15.1875 32.171875 \n",
       "z\n",
       "\" id=\"DejaVuSans-101\"/>\n",
       "       <path d=\"M 18.109375 8.203125 \n",
       "L 18.109375 -20.796875 \n",
       "L 9.078125 -20.796875 \n",
       "L 9.078125 54.6875 \n",
       "L 18.109375 54.6875 \n",
       "L 18.109375 46.390625 \n",
       "Q 20.953125 51.265625 25.265625 53.625 \n",
       "Q 29.59375 56 35.59375 56 \n",
       "Q 45.5625 56 51.78125 48.09375 \n",
       "Q 58.015625 40.1875 58.015625 27.296875 \n",
       "Q 58.015625 14.40625 51.78125 6.484375 \n",
       "Q 45.5625 -1.421875 35.59375 -1.421875 \n",
       "Q 29.59375 -1.421875 25.265625 0.953125 \n",
       "Q 20.953125 3.328125 18.109375 8.203125 \n",
       "z\n",
       "M 48.6875 27.296875 \n",
       "Q 48.6875 37.203125 44.609375 42.84375 \n",
       "Q 40.53125 48.484375 33.40625 48.484375 \n",
       "Q 26.265625 48.484375 22.1875 42.84375 \n",
       "Q 18.109375 37.203125 18.109375 27.296875 \n",
       "Q 18.109375 17.390625 22.1875 11.75 \n",
       "Q 26.265625 6.109375 33.40625 6.109375 \n",
       "Q 40.53125 6.109375 44.609375 11.75 \n",
       "Q 48.6875 17.390625 48.6875 27.296875 \n",
       "z\n",
       "\" id=\"DejaVuSans-112\"/>\n",
       "       <path d=\"M 30.609375 48.390625 \n",
       "Q 23.390625 48.390625 19.1875 42.75 \n",
       "Q 14.984375 37.109375 14.984375 27.296875 \n",
       "Q 14.984375 17.484375 19.15625 11.84375 \n",
       "Q 23.34375 6.203125 30.609375 6.203125 \n",
       "Q 37.796875 6.203125 41.984375 11.859375 \n",
       "Q 46.1875 17.53125 46.1875 27.296875 \n",
       "Q 46.1875 37.015625 41.984375 42.703125 \n",
       "Q 37.796875 48.390625 30.609375 48.390625 \n",
       "z\n",
       "M 30.609375 56 \n",
       "Q 42.328125 56 49.015625 48.375 \n",
       "Q 55.71875 40.765625 55.71875 27.296875 \n",
       "Q 55.71875 13.875 49.015625 6.21875 \n",
       "Q 42.328125 -1.421875 30.609375 -1.421875 \n",
       "Q 18.84375 -1.421875 12.171875 6.21875 \n",
       "Q 5.515625 13.875 5.515625 27.296875 \n",
       "Q 5.515625 40.765625 12.171875 48.375 \n",
       "Q 18.84375 56 30.609375 56 \n",
       "z\n",
       "\" id=\"DejaVuSans-111\"/>\n",
       "       <path d=\"M 48.78125 52.59375 \n",
       "L 48.78125 44.1875 \n",
       "Q 44.96875 46.296875 41.140625 47.34375 \n",
       "Q 37.3125 48.390625 33.40625 48.390625 \n",
       "Q 24.65625 48.390625 19.8125 42.84375 \n",
       "Q 14.984375 37.3125 14.984375 27.296875 \n",
       "Q 14.984375 17.28125 19.8125 11.734375 \n",
       "Q 24.65625 6.203125 33.40625 6.203125 \n",
       "Q 37.3125 6.203125 41.140625 7.25 \n",
       "Q 44.96875 8.296875 48.78125 10.40625 \n",
       "L 48.78125 2.09375 \n",
       "Q 45.015625 0.34375 40.984375 -0.53125 \n",
       "Q 36.96875 -1.421875 32.421875 -1.421875 \n",
       "Q 20.0625 -1.421875 12.78125 6.34375 \n",
       "Q 5.515625 14.109375 5.515625 27.296875 \n",
       "Q 5.515625 40.671875 12.859375 48.328125 \n",
       "Q 20.21875 56 33.015625 56 \n",
       "Q 37.15625 56 41.109375 55.140625 \n",
       "Q 45.0625 54.296875 48.78125 52.59375 \n",
       "z\n",
       "\" id=\"DejaVuSans-99\"/>\n",
       "       <path d=\"M 54.890625 33.015625 \n",
       "L 54.890625 0 \n",
       "L 45.90625 0 \n",
       "L 45.90625 32.71875 \n",
       "Q 45.90625 40.484375 42.875 44.328125 \n",
       "Q 39.84375 48.1875 33.796875 48.1875 \n",
       "Q 26.515625 48.1875 22.3125 43.546875 \n",
       "Q 18.109375 38.921875 18.109375 30.90625 \n",
       "L 18.109375 0 \n",
       "L 9.078125 0 \n",
       "L 9.078125 75.984375 \n",
       "L 18.109375 75.984375 \n",
       "L 18.109375 46.1875 \n",
       "Q 21.34375 51.125 25.703125 53.5625 \n",
       "Q 30.078125 56 35.796875 56 \n",
       "Q 45.21875 56 50.046875 50.171875 \n",
       "Q 54.890625 44.34375 54.890625 33.015625 \n",
       "z\n",
       "\" id=\"DejaVuSans-104\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-101\"/>\n",
       "      <use x=\"61.523438\" xlink:href=\"#DejaVuSans-112\"/>\n",
       "      <use x=\"125\" xlink:href=\"#DejaVuSans-111\"/>\n",
       "      <use x=\"186.181641\" xlink:href=\"#DejaVuSans-99\"/>\n",
       "      <use x=\"241.162109\" xlink:href=\"#DejaVuSans-104\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <path clip-path=\"url(#p878e57563c)\" d=\"M 30.103125 139.576255 \n",
       "L 225.403125 139.576255 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_12\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" id=\"mc277e90d8d\" style=\"stroke:#000000;stroke-width:0.8;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"30.103125\" xlink:href=\"#mc277e90d8d\" y=\"139.576255\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 0.0 -->\n",
       "      <g transform=\"translate(7.2 143.375474)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path d=\"M 10.6875 12.40625 \n",
       "L 21 12.40625 \n",
       "L 21 0 \n",
       "L 10.6875 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-46\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <path clip-path=\"url(#p878e57563c)\" d=\"M 30.103125 114.18666 \n",
       "L 225.403125 114.18666 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"30.103125\" xlink:href=\"#mc277e90d8d\" y=\"114.18666\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 0.2 -->\n",
       "      <g transform=\"translate(7.2 117.985879)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-50\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path clip-path=\"url(#p878e57563c)\" d=\"M 30.103125 88.797065 \n",
       "L 225.403125 88.797065 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"30.103125\" xlink:href=\"#mc277e90d8d\" y=\"88.797065\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 0.4 -->\n",
       "      <g transform=\"translate(7.2 92.596284)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-52\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_17\">\n",
       "      <path clip-path=\"url(#p878e57563c)\" d=\"M 30.103125 63.40747 \n",
       "L 225.403125 63.40747 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_18\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"30.103125\" xlink:href=\"#mc277e90d8d\" y=\"63.40747\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 0.6 -->\n",
       "      <g transform=\"translate(7.2 67.206689)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-54\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_19\">\n",
       "      <path clip-path=\"url(#p878e57563c)\" d=\"M 30.103125 38.017875 \n",
       "L 225.403125 38.017875 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_20\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"30.103125\" xlink:href=\"#mc277e90d8d\" y=\"38.017875\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 0.8 -->\n",
       "      <g transform=\"translate(7.2 41.817094)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-56\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_21\">\n",
       "      <path clip-path=\"url(#p878e57563c)\" d=\"M 30.103125 12.62828 \n",
       "L 225.403125 12.62828 \n",
       "\" style=\"fill:none;stroke:#b0b0b0;stroke-linecap:square;stroke-width:0.8;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_22\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.8;\" x=\"30.103125\" xlink:href=\"#mc277e90d8d\" y=\"12.62828\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 1.0 -->\n",
       "      <g transform=\"translate(7.2 16.427498)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-49\"/>\n",
       "       <use x=\"63.623047\" xlink:href=\"#DejaVuSans-46\"/>\n",
       "       <use x=\"95.410156\" xlink:href=\"#DejaVuSans-48\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"line2d_23\">\n",
       "    <path clip-path=\"url(#p878e57563c)\" d=\"M 12.743125 18.83494 \n",
       "L 17.083125 50.695632 \n",
       "L 21.423125 64.665602 \n",
       "L 25.763125 72.057056 \n",
       "L 30.103125 77.551173 \n",
       "L 34.443125 102.958174 \n",
       "L 38.783125 104.034152 \n",
       "L 43.123125 105.074415 \n",
       "L 47.463125 105.779776 \n",
       "L 51.803125 106.321622 \n",
       "L 56.143125 114.206253 \n",
       "L 60.483125 114.629873 \n",
       "L 64.823125 114.305896 \n",
       "L 69.163125 114.330286 \n",
       "L 73.503125 114.273407 \n",
       "L 77.843125 121.869904 \n",
       "L 82.183125 121.376148 \n",
       "L 86.523125 121.235398 \n",
       "L 90.863125 120.655612 \n",
       "L 95.203125 120.310955 \n",
       "L 99.543125 126.242683 \n",
       "L 103.883125 126.207903 \n",
       "L 108.223125 125.583223 \n",
       "L 112.563125 125.398833 \n",
       "L 116.903125 125.121268 \n",
       "L 121.243125 128.865135 \n",
       "L 125.583125 129.037931 \n",
       "L 129.923125 129.131179 \n",
       "L 134.263125 128.964907 \n",
       "L 138.603125 128.693175 \n",
       "L 142.943125 133.56064 \n",
       "L 147.283125 132.842638 \n",
       "L 151.623125 132.620844 \n",
       "L 155.963125 132.151688 \n",
       "L 160.303125 131.993919 \n",
       "L 164.643125 136.167429 \n",
       "L 168.983125 135.874891 \n",
       "L 173.323125 135.527781 \n",
       "L 177.663125 134.914903 \n",
       "L 182.003125 134.523913 \n",
       "L 186.343125 135.841214 \n",
       "L 190.683125 136.145934 \n",
       "L 195.023125 136.224853 \n",
       "L 199.363125 136.141127 \n",
       "L 203.703125 136.099427 \n",
       "L 208.043125 134.154494 \n",
       "L 212.383125 135.645535 \n",
       "L 216.723125 136.409717 \n",
       "L 221.063125 136.773198 \n",
       "L 225.403125 136.922727 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_24\">\n",
       "    <path clip-path=\"url(#p878e57563c)\" d=\"M 12.743125 50.896257 \n",
       "L 17.083125 42.223449 \n",
       "L 21.423125 38.126197 \n",
       "L 25.763125 35.837539 \n",
       "L 30.103125 34.211551 \n",
       "L 34.443125 26.122833 \n",
       "L 38.783125 25.584739 \n",
       "L 43.123125 25.120501 \n",
       "L 47.463125 24.906846 \n",
       "L 51.803125 24.715843 \n",
       "L 56.143125 22.092403 \n",
       "L 60.483125 21.733674 \n",
       "L 64.823125 21.80753 \n",
       "L 69.163125 21.868197 \n",
       "L 73.503125 21.903945 \n",
       "L 77.843125 19.074857 \n",
       "L 82.183125 19.280598 \n",
       "L 86.523125 19.352696 \n",
       "L 90.863125 19.557559 \n",
       "L 95.203125 19.711977 \n",
       "L 99.543125 17.376168 \n",
       "L 103.883125 17.297036 \n",
       "L 108.223125 17.520363 \n",
       "L 112.563125 17.647852 \n",
       "L 116.903125 17.723125 \n",
       "L 121.243125 16.458243 \n",
       "L 125.583125 16.352734 \n",
       "L 129.923125 16.268327 \n",
       "L 134.263125 16.307893 \n",
       "L 138.603125 16.419793 \n",
       "L 142.943125 14.474681 \n",
       "L 147.283125 14.822859 \n",
       "L 151.623125 14.924851 \n",
       "L 155.963125 15.139385 \n",
       "L 160.303125 15.173587 \n",
       "L 164.643125 13.577857 \n",
       "L 168.983125 13.709743 \n",
       "L 173.323125 13.876798 \n",
       "L 177.663125 14.123864 \n",
       "L 182.003125 14.301877 \n",
       "L 186.343125 13.715018 \n",
       "L 190.683125 13.625336 \n",
       "L 195.023125 13.627095 \n",
       "L 199.363125 13.654351 \n",
       "L 203.703125 13.688295 \n",
       "L 208.043125 14.337519 \n",
       "L 212.383125 13.804701 \n",
       "L 216.723125 13.553239 \n",
       "L 221.063125 13.440696 \n",
       "L 225.403125 13.377273 \n",
       "\" style=\"fill:none;stroke:#bf00bf;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_25\">\n",
       "    <path clip-path=\"url(#p878e57563c)\" d=\"M 30.103125 33.485832 \n",
       "L 51.803125 26.160934 \n",
       "L 73.503125 34.56489 \n",
       "L 95.203125 27.214602 \n",
       "L 116.903125 26.782979 \n",
       "L 138.603125 27.189212 \n",
       "L 160.303125 23.114182 \n",
       "L 182.003125 23.964734 \n",
       "L 203.703125 28.725283 \n",
       "L 225.403125 23.736228 \n",
       "\" style=\"fill:none;stroke:#008000;stroke-dasharray:9.6,2.4,1.5,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 30.103125 143.1 \n",
       "L 30.103125 7.2 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 225.403125 143.1 \n",
       "L 225.403125 7.2 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 30.103125 143.1 \n",
       "L 225.403125 143.1 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 30.103125 7.2 \n",
       "L 225.403125 7.2 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;stroke-width:0.8;\"/>\n",
       "   </g>\n",
       "   <g id=\"legend_1\">\n",
       "    <g id=\"patch_7\">\n",
       "     <path d=\"M 140.634375 98.667187 \n",
       "L 218.403125 98.667187 \n",
       "Q 220.403125 98.667187 220.403125 96.667187 \n",
       "L 220.403125 53.632812 \n",
       "Q 220.403125 51.632812 218.403125 51.632812 \n",
       "L 140.634375 51.632812 \n",
       "Q 138.634375 51.632812 138.634375 53.632812 \n",
       "L 138.634375 96.667187 \n",
       "Q 138.634375 98.667187 140.634375 98.667187 \n",
       "z\n",
       "\" style=\"fill:#ffffff;opacity:0.8;stroke:#cccccc;stroke-linejoin:miter;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_26\">\n",
       "     <path d=\"M 142.634375 59.73125 \n",
       "L 162.634375 59.73125 \n",
       "\" style=\"fill:none;stroke:#1f77b4;stroke-linecap:square;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_27\"/>\n",
       "    <g id=\"text_13\">\n",
       "     <!-- train loss -->\n",
       "     <g transform=\"translate(170.634375 63.23125)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path d=\"M 18.3125 70.21875 \n",
       "L 18.3125 54.6875 \n",
       "L 36.8125 54.6875 \n",
       "L 36.8125 47.703125 \n",
       "L 18.3125 47.703125 \n",
       "L 18.3125 18.015625 \n",
       "Q 18.3125 11.328125 20.140625 9.421875 \n",
       "Q 21.96875 7.515625 27.59375 7.515625 \n",
       "L 36.8125 7.515625 \n",
       "L 36.8125 0 \n",
       "L 27.59375 0 \n",
       "Q 17.1875 0 13.234375 3.875 \n",
       "Q 9.28125 7.765625 9.28125 18.015625 \n",
       "L 9.28125 47.703125 \n",
       "L 2.6875 47.703125 \n",
       "L 2.6875 54.6875 \n",
       "L 9.28125 54.6875 \n",
       "L 9.28125 70.21875 \n",
       "z\n",
       "\" id=\"DejaVuSans-116\"/>\n",
       "       <path d=\"M 41.109375 46.296875 \n",
       "Q 39.59375 47.171875 37.8125 47.578125 \n",
       "Q 36.03125 48 33.890625 48 \n",
       "Q 26.265625 48 22.1875 43.046875 \n",
       "Q 18.109375 38.09375 18.109375 28.8125 \n",
       "L 18.109375 0 \n",
       "L 9.078125 0 \n",
       "L 9.078125 54.6875 \n",
       "L 18.109375 54.6875 \n",
       "L 18.109375 46.1875 \n",
       "Q 20.953125 51.171875 25.484375 53.578125 \n",
       "Q 30.03125 56 36.53125 56 \n",
       "Q 37.453125 56 38.578125 55.875 \n",
       "Q 39.703125 55.765625 41.0625 55.515625 \n",
       "z\n",
       "\" id=\"DejaVuSans-114\"/>\n",
       "       <path d=\"M 34.28125 27.484375 \n",
       "Q 23.390625 27.484375 19.1875 25 \n",
       "Q 14.984375 22.515625 14.984375 16.5 \n",
       "Q 14.984375 11.71875 18.140625 8.90625 \n",
       "Q 21.296875 6.109375 26.703125 6.109375 \n",
       "Q 34.1875 6.109375 38.703125 11.40625 \n",
       "Q 43.21875 16.703125 43.21875 25.484375 \n",
       "L 43.21875 27.484375 \n",
       "z\n",
       "M 52.203125 31.203125 \n",
       "L 52.203125 0 \n",
       "L 43.21875 0 \n",
       "L 43.21875 8.296875 \n",
       "Q 40.140625 3.328125 35.546875 0.953125 \n",
       "Q 30.953125 -1.421875 24.3125 -1.421875 \n",
       "Q 15.921875 -1.421875 10.953125 3.296875 \n",
       "Q 6 8.015625 6 15.921875 \n",
       "Q 6 25.140625 12.171875 29.828125 \n",
       "Q 18.359375 34.515625 30.609375 34.515625 \n",
       "L 43.21875 34.515625 \n",
       "L 43.21875 35.40625 \n",
       "Q 43.21875 41.609375 39.140625 45 \n",
       "Q 35.0625 48.390625 27.6875 48.390625 \n",
       "Q 23 48.390625 18.546875 47.265625 \n",
       "Q 14.109375 46.140625 10.015625 43.890625 \n",
       "L 10.015625 52.203125 \n",
       "Q 14.9375 54.109375 19.578125 55.046875 \n",
       "Q 24.21875 56 28.609375 56 \n",
       "Q 40.484375 56 46.34375 49.84375 \n",
       "Q 52.203125 43.703125 52.203125 31.203125 \n",
       "z\n",
       "\" id=\"DejaVuSans-97\"/>\n",
       "       <path d=\"M 9.421875 54.6875 \n",
       "L 18.40625 54.6875 \n",
       "L 18.40625 0 \n",
       "L 9.421875 0 \n",
       "z\n",
       "M 9.421875 75.984375 \n",
       "L 18.40625 75.984375 \n",
       "L 18.40625 64.59375 \n",
       "L 9.421875 64.59375 \n",
       "z\n",
       "\" id=\"DejaVuSans-105\"/>\n",
       "       <path d=\"M 54.890625 33.015625 \n",
       "L 54.890625 0 \n",
       "L 45.90625 0 \n",
       "L 45.90625 32.71875 \n",
       "Q 45.90625 40.484375 42.875 44.328125 \n",
       "Q 39.84375 48.1875 33.796875 48.1875 \n",
       "Q 26.515625 48.1875 22.3125 43.546875 \n",
       "Q 18.109375 38.921875 18.109375 30.90625 \n",
       "L 18.109375 0 \n",
       "L 9.078125 0 \n",
       "L 9.078125 54.6875 \n",
       "L 18.109375 54.6875 \n",
       "L 18.109375 46.1875 \n",
       "Q 21.34375 51.125 25.703125 53.5625 \n",
       "Q 30.078125 56 35.796875 56 \n",
       "Q 45.21875 56 50.046875 50.171875 \n",
       "Q 54.890625 44.34375 54.890625 33.015625 \n",
       "z\n",
       "\" id=\"DejaVuSans-110\"/>\n",
       "       <path id=\"DejaVuSans-32\"/>\n",
       "       <path d=\"M 9.421875 75.984375 \n",
       "L 18.40625 75.984375 \n",
       "L 18.40625 0 \n",
       "L 9.421875 0 \n",
       "z\n",
       "\" id=\"DejaVuSans-108\"/>\n",
       "       <path d=\"M 44.28125 53.078125 \n",
       "L 44.28125 44.578125 \n",
       "Q 40.484375 46.53125 36.375 47.5 \n",
       "Q 32.28125 48.484375 27.875 48.484375 \n",
       "Q 21.1875 48.484375 17.84375 46.4375 \n",
       "Q 14.5 44.390625 14.5 40.28125 \n",
       "Q 14.5 37.15625 16.890625 35.375 \n",
       "Q 19.28125 33.59375 26.515625 31.984375 \n",
       "L 29.59375 31.296875 \n",
       "Q 39.15625 29.25 43.1875 25.515625 \n",
       "Q 47.21875 21.78125 47.21875 15.09375 \n",
       "Q 47.21875 7.46875 41.1875 3.015625 \n",
       "Q 35.15625 -1.421875 24.609375 -1.421875 \n",
       "Q 20.21875 -1.421875 15.453125 -0.5625 \n",
       "Q 10.6875 0.296875 5.421875 2 \n",
       "L 5.421875 11.28125 \n",
       "Q 10.40625 8.6875 15.234375 7.390625 \n",
       "Q 20.0625 6.109375 24.8125 6.109375 \n",
       "Q 31.15625 6.109375 34.5625 8.28125 \n",
       "Q 37.984375 10.453125 37.984375 14.40625 \n",
       "Q 37.984375 18.0625 35.515625 20.015625 \n",
       "Q 33.0625 21.96875 24.703125 23.78125 \n",
       "L 21.578125 24.515625 \n",
       "Q 13.234375 26.265625 9.515625 29.90625 \n",
       "Q 5.8125 33.546875 5.8125 39.890625 \n",
       "Q 5.8125 47.609375 11.28125 51.796875 \n",
       "Q 16.75 56 26.8125 56 \n",
       "Q 31.78125 56 36.171875 55.265625 \n",
       "Q 40.578125 54.546875 44.28125 53.078125 \n",
       "z\n",
       "\" id=\"DejaVuSans-115\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-116\"/>\n",
       "      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"80.322266\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"141.601562\" xlink:href=\"#DejaVuSans-105\"/>\n",
       "      <use x=\"169.384766\" xlink:href=\"#DejaVuSans-110\"/>\n",
       "      <use x=\"232.763672\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "      <use x=\"264.550781\" xlink:href=\"#DejaVuSans-108\"/>\n",
       "      <use x=\"292.333984\" xlink:href=\"#DejaVuSans-111\"/>\n",
       "      <use x=\"353.515625\" xlink:href=\"#DejaVuSans-115\"/>\n",
       "      <use x=\"405.615234\" xlink:href=\"#DejaVuSans-115\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_28\">\n",
       "     <path d=\"M 142.634375 74.409375 \n",
       "L 162.634375 74.409375 \n",
       "\" style=\"fill:none;stroke:#bf00bf;stroke-dasharray:5.55,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_29\"/>\n",
       "    <g id=\"text_14\">\n",
       "     <!-- train acc -->\n",
       "     <g transform=\"translate(170.634375 77.909375)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-116\"/>\n",
       "      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-114\"/>\n",
       "      <use x=\"80.322266\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"141.601562\" xlink:href=\"#DejaVuSans-105\"/>\n",
       "      <use x=\"169.384766\" xlink:href=\"#DejaVuSans-110\"/>\n",
       "      <use x=\"232.763672\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "      <use x=\"264.550781\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"325.830078\" xlink:href=\"#DejaVuSans-99\"/>\n",
       "      <use x=\"380.810547\" xlink:href=\"#DejaVuSans-99\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"line2d_30\">\n",
       "     <path d=\"M 142.634375 89.0875 \n",
       "L 162.634375 89.0875 \n",
       "\" style=\"fill:none;stroke:#008000;stroke-dasharray:9.6,2.4,1.5,2.4;stroke-dashoffset:0;stroke-width:1.5;\"/>\n",
       "    </g>\n",
       "    <g id=\"line2d_31\"/>\n",
       "    <g id=\"text_15\">\n",
       "     <!-- test acc -->\n",
       "     <g transform=\"translate(170.634375 92.5875)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#DejaVuSans-116\"/>\n",
       "      <use x=\"39.208984\" xlink:href=\"#DejaVuSans-101\"/>\n",
       "      <use x=\"100.732422\" xlink:href=\"#DejaVuSans-115\"/>\n",
       "      <use x=\"152.832031\" xlink:href=\"#DejaVuSans-116\"/>\n",
       "      <use x=\"192.041016\" xlink:href=\"#DejaVuSans-32\"/>\n",
       "      <use x=\"223.828125\" xlink:href=\"#DejaVuSans-97\"/>\n",
       "      <use x=\"285.107422\" xlink:href=\"#DejaVuSans-99\"/>\n",
       "      <use x=\"340.087891\" xlink:href=\"#DejaVuSans-99\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p878e57563c\">\n",
       "   <rect height=\"135.9\" width=\"195.3\" x=\"30.103125\" y=\"7.2\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 252x180 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "lr, num_epochs, batch_size = 0.05, 10, 256\n",
    "train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size, resize=96)\n",
    "d2l.train_ch6(net, train_iter, test_iter, num_epochs, lr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NGesxlGC77CT",
    "origin_pos": 34
   },
   "source": [
    "## Summary\n",
    "\n",
    "* Nested function classes are desirable. Learning an additional layer in deep neural networks as an identity function (though this is an extreme case) should be made easy.\n",
    "* The residual mapping can learn the identity function more easily, such as pushing parameters in the weight layer to zero.\n",
    "* We can train an effective deep neural network by having residual blocks. Inputs can forward propagate faster through the residual connections across layers.\n",
    "* ResNet had a major influence on the design of subsequent deep neural networks, both for convolutional and sequential nature.\n"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "5.4_ResNet.ipynb",
   "provenance": [
    {
     "file_id": "1vFtPWM6QGkRz28tEIv9WA3GwsgLFabki",
     "timestamp": 1605144154220
    },
    {
     "file_id": "https://github.com/d2l-ai/d2l-en-colab/blob/master/chapter_convolutional-modern/resnet.ipynb",
     "timestamp": 1605075199105
    }
   ]
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}