3b. CNN. Practice.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CNN. Practice\n",
    "# Keras API\n",
    "\n",
    "Source: \n",
    "- https://github.com/Hvass-Labs/TensorFlow-Tutorials\n",
    "- https://www.youtube.com/playlist?list=PL9Hr9sNUjfsmEu1ZniY0XpHSzl5uihcXZ"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "The previous tutorial showed that a simple neural net had about 92% classification accuracy for recognizing hand-written digits in the MNIST data-set.\n",
    "\n",
    "Convolutional Networks work by moving small filters across the input image. This means the filters are re-used for recognizing patterns throughout the entire input image. This makes the Convolutional Networks much more powerful than Fully-Connected networks with the same number of variables. This in turn makes the Convolutional Networks faster to train."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Flowchart"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following chart shows roughly how the data flows in the Convolutional Neural Network that is implemented below.\n",
    "\n",
    "![Flowchart](images/02_network_flowchart.png)\n",
    "\n",
    "The input image is processed in the first convolutional layer using the filter-weights. This results in 16 new images, one for each filter in the convolutional layer. The images are also down-sampled so the image resolution is decreased from 28x28 to 14x14.\n",
    "\n",
    "These 16 smaller images are then processed in the second convolutional layer. We need filter-weights for each of these 16 channels, and we need filter-weights for each output channel of this layer. There are 36 output channels so there are a total of 16 x 36 = 576 filters in the second convolutional layer. The resulting images are down-sampled again to 7x7 pixels.\n",
    "\n",
    "The output of the second convolutional layer is 36 images of 7x7 pixels each. These are then flattened to a single vector of length 7 x 7 x 36 = 1764, which is used as the input to a fully-connected layer with 128 neurons (or elements). This feeds into another fully-connected layer with 10 neurons, one for each of the classes, which is used to determine the class of the image, that is, which number is depicted in the image.\n",
    "\n",
    "The convolutional filters are initially chosen at random, so the classification is done randomly. The error between the predicted and true class of the input image is measured as the so-called cross-entropy. The optimizer then automatically propagates this error back through the Convolutional Network using the chain-rule of differentiation and updates the filter-weights so as to improve the classification error. This is done iteratively thousands of times until the classification error is sufficiently low.\n",
    "\n",
    "These particular filter-weights and intermediate images are the results of one optimization run and may look different if you re-run this Notebook.\n",
    "\n",
    "Note that the computation in TensorFlow is actually done on a batch of images instead of a single image, which makes the computation more efficient. This means the flowchart actually has one more data-dimension when implemented in TensorFlow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convolutional layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following chart shows the basic idea of processing an image in the first convolutional layer. The input image depicts the number 7 and four copies of the image are shown here, so we can see more clearly how the filter is being moved to different positions of the image. For each position of the filter, the dot-product is being calculated between the filter and the image pixels under the filter, which results in a single pixel in the output image. So moving the filter across the entire input image results in a new image being generated.\n",
    "\n",
    "The red filter-weights means that the filter has a positive reaction to black pixels in the input image, while blue pixels means the filter has a negative reaction to black pixels.\n",
    "\n",
    "In this case it appears that the filter recognizes the horizontal line of the 7-digit, as can be seen from its stronger reaction to that line in the output image.\n",
    "\n",
    "![Convolution example](images/02_convolution.png)\n",
    "\n",
    "The step-size for moving the filter across the input is called the stride. There is a stride for moving the filter horizontally (x-axis) and another stride for moving vertically (y-axis).\n",
    "\n",
    "In the source-code below, the stride is set to 1 in both directions, which means the filter starts in the upper left corner of the input image and is being moved 1 pixel to the right in each step. When the filter reaches the end of the image to the right, then the filter is moved back to the left side and 1 pixel down the image. This continues until the filter has reached the lower right corner of the input image and the entire output image has been generated.\n",
    "\n",
    "When the filter reaches the end of the right-side as well as the bottom of the input image, then it can be padded with zeroes (white pixels). This causes the output image to be of the exact same dimension as the input image.\n",
    "\n",
    "Furthermore, the output of the convolution may be passed through a so-called Rectified Linear Unit (ReLU), which merely ensures that the output is positive because negative values are set to zero. The output may also be down-sampled by so-called max-pooling, which considers small windows of 2x2 pixels and only keeps the largest of those pixels. This halves the resolution of the input image e.g. from 28x28 to 14x14 pixels.\n",
    "\n",
    "Note that the second convolutional layer is more complicated because it takes 16 input channels. We want a separate filter for each input channel, so we need 16 filters instead of just one. Furthermore, we want 36 output channels from the second convolutional layer, so in total we need 16 x 36 = 576 filters for the second convolutional layer. It can be a bit challenging to understand how this works."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import math\n",
    "from helpers import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to import several things from Keras. Note the long import-statements. This might be a bug. Hopefully it will be possible to write shorter and more elegant lines in the future."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# from tf.keras.models import Sequential  # This does not work!\n",
    "from tensorflow.python.keras.models import Sequential\n",
    "from tensorflow.python.keras.layers import InputLayer, Input\n",
    "from tensorflow.python.keras.layers import Reshape, MaxPooling2D\n",
    "from tensorflow.python.keras.layers import Conv2D, Dense, Flatten"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This was developed using Python 3.6 (Anaconda) and TensorFlow version:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1.4.0'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2.0.8-tf'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.keras.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The MNIST data-set is about 12 MB and will be downloaded automatically if it is not located in the given path."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting data/MNIST/train-images-idx3-ubyte.gz\n",
      "Extracting data/MNIST/train-labels-idx1-ubyte.gz\n",
      "Extracting data/MNIST/t10k-images-idx3-ubyte.gz\n",
      "Extracting data/MNIST/t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "data = input_data.read_data_sets('data/MNIST/', one_hot=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The MNIST data-set has now been loaded and consists of 70,000 images and associated labels (i.e. classifications of the images). The data-set is split into 3 mutually exclusive sub-sets. We will only use the training and test-sets in this tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size of:\n",
      "- Training-set:\t\t55000\n",
      "- Test-set:\t\t10000\n",
      "- Validation-set:\t5000\n"
     ]
    }
   ],
   "source": [
    "print(\"Size of:\")\n",
    "print(\"- Training-set:\\t\\t{}\".format(len(data.train.labels)))\n",
    "print(\"- Test-set:\\t\\t{}\".format(len(data.test.labels)))\n",
    "print(\"- Validation-set:\\t{}\".format(len(data.validation.labels)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The class-labels are One-Hot encoded, which means that each label is a vector with 10 elements, all of which are zero except for one element. The index of this one element is the class-number, that is, the digit shown in the associated image. We also need the class-numbers as integers for the test-set, so we calculate it now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data.test.cls = np.argmax(data.test.labels, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Dimensions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data dimensions are used in several places in the source-code below. They are defined once so we can use these variables instead of numbers throughout the source-code below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# We know that MNIST images are 28 pixels in each dimension.\n",
    "img_size = 28\n",
    "\n",
    "# Images are stored in one-dimensional arrays of this length.\n",
    "img_size_flat = img_size * img_size\n",
    "\n",
    "# Tuple with height and width of images used to reshape arrays.\n",
    "# This is used for plotting the images.\n",
    "img_shape = (img_size, img_size)\n",
    "\n",
    "# Tuple with height, width and depth used to reshape arrays.\n",
    "# This is used for reshaping in Keras.\n",
    "img_shape_full = (img_size, img_size, 1)\n",
    "\n",
    "# Number of colour channels for the images: 1 channel for gray-scale.\n",
    "num_channels = 1\n",
    "\n",
    "# Number of classes, one class for each of 10 digits.\n",
    "num_classes = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot a few images to see if data is correct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUMAAAD5CAYAAAC9FVegAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHihJREFUeJzt3XmUFNXZx/HvA0LYVQQFFWdOwAVCFBWDu0aBKCogccG4\nEGM0osEtAaNx1xglKBzRE7YD4QQNigKCUVFAEV8EJIIi4wYiCsRlhLggIsJ9/5i5XdUzPXtXVU/7\n+5zjmequ6qpnvPSdp27dxZxziIj80DVIOgARkVygylBEBFWGIiKAKkMREUCVoYgIoMpQRARQZSgi\nAqgyFBEBVBmKiACwS00ObtOmjSssLIwolNzzwQcfUFxcbEnHESeVcf5TGWdWo8qwsLCQZcuW1T6q\neqZ79+5JhxA7lXH+UxlnpttkERFUGYqIAKoMRUQAVYYiIoAqQxERoIZPk0Vqa8SIEQBs3boVgDfe\neAOAxx9/vNyxgwcPBuCoo44C4MILL4wjRPmBU2YoIoIyQ4nYueeeC8C0adMy7jcr3xd2zJgxAMyd\nOxeAE044AYD99tsvihAlQe+++y4ABx54IAAPPPAAAEOGDIk9FmWGIiIoM5QI+GwQKs4IDzroIABO\nOeUUAN5///3UvlmzZgGwevVqAKZMmQLAjTfemP1gJVHLly8HoEGDkrxsn332SSwWZYYiIigzlCzy\n411nzJhRbl/Xrl2BIOtr06YNAC1atADgu+++Sx3bo0cPAF5//XUAPv/884gilqStWLECCP4dDBgw\nILFYlBmKiBBDZuj7kY0fPx6AvffeO7WvSZMmAJx//vkAtGvXDoBOnTpFHZZE4L///S8AzrnUez4j\nnDNnDgDt27fP+FnfDxHgrbfeStt3+umnZzVOSd7KlSsBGD16NAAXXXRRkuEAygxFRIAYMsOhQ4cC\nJRMsVsT3K2vVqhUAXbp0ycq1O3ToAMCwYcOAH+bcdXE644wzgOApMEDLli0BaN26daWfffTRR1Pb\n4fZDyU/vvPMOAFu2bAHSeyAkRZmhiAiqDEVEgBhukydMmAAE3STCt8BFRUVA0PHyxRdfBGDx4sVA\nMPzqww8/rPD8jRo1AoKuGr4RP3wef7us2+R4FBQUVPvYv/3tb0AwLCvMd7HxPyV/DB8+HChZggBy\n47upzFBEhBgyw5NPPjntZ5gfiuVt3rwZCDJF/9fi1VdfrfD8P/rRj4BgoLcf5gWwadMmADp27Fir\n2CU6Tz31FAC33HILANu2bUvt22uvvQC45557AGjWrFnM0UkUwg9R/Xfaf2+bN2+eREhplBmKiJBj\nw/F23313AE466aS09zNllWU98cQTQJBdAhx88MEADBw4MFshSpb4oXvhjNDz3Sz81F2SHxYsWFDu\nvbZt2yYQSWbKDEVEyLHMsDY+/fRTAK644gogfSiYb4+qqsOvxKd///5AMDzPGzRoUGr7rrvuijUm\niYdf6iHMD4jIBcoMRUTIg8zwoYceAoIMcbfddkvt80+qJHm+/+eiRYuAoK3QtxnddNNNqWP9dE6S\nH1555RUAJk2alHrv0EMPBaBXr16JxJSJMkMREepxZvjyyy8DQV8078knn0xt++mjJHl+0s7i4uK0\n9/30beoLmr/mzZsHpPf08H2M/TR+uUCZoYgIqgxFRIB6fJv89NNPA8Hcdz179gTgqKOOSiwmKc+v\neeKHWHonnngiAHfccUfcIUnM/CQtYWeffXYCkVROmaGICPUwM9y6dSsAzz77LBBM1HD77bcDwZRe\nkpzwanZ33303UH726m7dugHqRpPPPv74YwAWLlwIpE+icuaZZyYSU2WUGYqIUA8zQz8ZqG+DOvXU\nUwE4+uijE4tJ0t13332p7aVLl6bt88Px1FaY//7xj38A8MknnwDBdzVXKTMUEaGeZIZ+IlCAO++8\nE4Bdd90VgJtvvjmRmKRi999/f4X7/PBJtRXmv3Xr1qW99lP05SplhiIi5Hhm6J9KXnXVVan3vv/+\newD69OkDqF9hfePLtDpP/X3274/dvn07AF988UW5Y/1Qr5EjR2Y8V8OGDVPb9957L6DlBKI2e/bs\ntNenn356QpFUjzJDERFUGYqIADl6m7xjxw4gmNli7dq1qX2dOnUCggcpUr/4dWmq45xzzgGgffv2\nQNBFY+rUqXWKwa++F55DUbLHd7L25VVfKDMUESFHM8M1a9YAwQpqYb7bhua/y13+4RbAzJkza32e\nxx57rMpj/MOVBg3S/6737dsXCNbeDjv22GNrHZNUbcaMGUDwsNPPap3rqx0qMxQRIccyQ99Js3fv\n3mnvjxgxIrWd64/nBaZPn57aHj58OFB+ogavqKgIqLwd8JJLLgGgoKCg3L5f/vKXAHTu3Ll2wUrW\nfPPNNwA888wzae/76brC3ZtykTJDERFyLDMcO3YsUH4YT7itwcxijUnqprrr4j7yyCMRRyJR8+23\nfoXKfv36AXD11VcnFlNNKDMUESFHMkPfL+nBBx9MOBIRqS2fGfp1kusbZYYiIuRIZujXQP7qq6/S\n3vejTTTdk4hETZmhiAiqDEVEgBy5TS7Lr5w2b948AFq3bp1kOCLyA6DMUESEHMkMb7jhhrSfIiJx\nU2YoIgKYc676B5t9Bqyr8sD8UeCca5t0EHFSGec/lXFmNaoMRUTylW6TRURQZSgiAkT8NNnM9gDm\nlb5sB+wAPit9/TPnXOYZP+t2zS5AeD6ojsANzjnNAhGBhMq4AJgM7Ak44O8q3+gkUcal150M9AE2\nOOe6RXGNtOvF1WZoZrcBXzvnRpR530rj2BnBNRsBG4DDnHPrs31+SRdXGZvZ3sCezrkVZtYKWA6c\n6px7Nxvnl4rF+T02sxOArcC4OCrDRG6TzayTmRWZ2cPAKqCDmf0vtH+gmU0o3d7LzKab2TIzW2pm\nR9bgUr2At1QRxi/KMnbObXTOrSjd/hJ4G9gnut9GMon6e+ycWwBsiuwXKCPJNsODgJHOuS6UZG8V\neQAY7pzrDpwD+P+5PcxsTBXXGAj8KxvBSq1EXsZm9mOgK/BqdkKWGorjexyLJEegrHHOlV8LtLye\nwIGh6f53N7OmzrklwJKKPmRmTYDTgOvqHKnUVtRl3Ap4AhjinPu6ztFKbURaxnFKsjLcEtreCYQX\nN2kS2jZq10h7GrDEOVdcy/ik7iIrYzNrDEwHJjnnZtUpSqmLqL/HscmJrjWlja6bzWx/M2sAnBna\nPRe40r8ws+o2pJ6HbpFzRjbLuLSx/h/ACufcAxGEK7UQ0fc4NjlRGZa6HpgDLALCDzyuBI4xszfM\nrAi4FCpvazCzlsDPgZnRhiw1lK0yPoGSP3a9zGxF6X+/iDh2qZ5sfo+nAQuBLma23sx+HWXgGo4n\nIkJuZYYiIolRZSgigipDERFAlaGICKDKUEQEqGGn6zZt2rjCwsKIQsk9H3zwAcXFxVb1kflDZZz/\nVMaZ1agyLCwsZNmy6oy8yQ/du3dPOoTYqYzzn8o4M90mi4igylBEBFBlKCICqDIUEQFUGYqIAKoM\nRUSAZCd3rdCWLSXzRQ4dOhSAMWOCGX78Y/Jp06YBUFBQEHN0IpKPlBmKiJCjmeHGjRsBGD9+PAAN\nGzZM7fOdRWfPng3A73//+5ijk9p47bXXABgwYABQMiqgtp577rnUdufOnQHo0KFD7YOTxPjvcd++\nfQEYPXo0AIMHD04dE/7+R0mZoYgIOZYZfvbZZwAMGjQo4Ugk2+bMmQPAtm3b6nyuWbOC9Z8mTpwI\nwNSpU+t8XonP559/DqRngABDhgwB4JJLLkm917Rp01hiUmYoIkKOZIYPPFCywNnMmSXrN736atXr\ngS9cuBAAv4bLIYccAsDxxx8fRYhSS99//z0ATz/9dNbOGR54f//99wNBD4TmzZtn7ToSnZdeegmA\nDRvS150/77zzAGjSpEm5z0RNmaGICDmSGV5zzTVAzZ4aTZ8+Pe3nfvvtB8Bjjz2WOubwww/PVohS\nSy+88AIAixYtAuD666+v8zk3bdqU2l61ahUA33zzDaDMMJeF24vvuuuujMdceOGFAJQsjR0vZYYi\nIqgyFBEBEr5N7tOnDxA8BNmxY0eVn2nTpg0Q3A6tW7cOgLVr1wJwxBFHpI7duXNn9oKValu5cmVq\ne+DAgQB06tQJgBtvvLHO5w93rZH644033kht+0743i67lFRFp556aqwxhSkzFBEhgcxwwYIFqe23\n334bCBpLK3qAcvnll6e2e/fuDcCuu+4KwPz58wH4y1/+Uu5zf//734HyHTslWuGy8A82pkyZAkCL\nFi1qfV7/4CT8byiJhnapHf+wM5NevXrFGElmygxFRIgxM/QD830bEkBxcXHGY303mbPOOguAW2+9\nNbWvWbNmacf6KbzGjh1b7pzDhg0D4NtvvwWCSR0aNWpUu19CKvX4448D6R2sfVthuC23tnx3jHA2\neOKJJwKw22671fn8Eq1wRu81btwYgLvvvjvucMpRZigiQoyZ4fbt24GKs0EIhtI9+uijQPDkuDI+\nM/RPKa+77rrUPj9Ey2eIfpqgjh071ih2qR4/4a7//w7Zaa/1dxWPPPIIEDx5BLjpppsAZfu5zHe4\nf+WVV8rt83d63bp1izWmTJQZioiQI8PxfHvSpEmTgOplhGX5rO/hhx9Ovbd06dIsRCdV+eKLLwBY\nvHhxuX1XXHFFnc8/btw4IJjirUuXLql9J510Up3PL9GqbOKVXOrpocxQRIQEMsNMo0yWLFlS5/P6\nUSzhUSdlR7b4p9K+z5tkhx+Av379eiCYhilb1qxZk/a6a9euWT2/RCtTZuif/mfjziFblBmKiKDK\nUEQEiPE22a99HNVKV36VreXLl6feKzvM7/bbb4/k2j90LVu2BILuEeGJGvwQutatW9f4vJ9++ikQ\ndNnxjjnmmFrFKfF6+eWXgaBLVJgfTrvvvvvGGlNllBmKiBBjZvjUU09l9Xy+m0VRURFQ+XAe31VH\nHXOj4Vcv80Pv/LA8gNNOOw1I7wyfyZtvvpna9g9M/PRsZSdjaNBAf8PrA78Cnn+QGZYLEzOUpX9V\nIiLkSKfr2vDTRD300EMVHlNYWAjA5MmTgWACCInGbbfdBqRnAv6OIDxBRyZt27ZNbftMsKKhmxdf\nfHFdwpSYlG3rDU+mcdlll8UdTpWUGYqIUA8zQ79UgJ8YtjJ+2NZxxx0XaUxSonPnzkD6CoX+6X7Z\njtNl+enawgYNGgSU7yTv2yglN/nO92WfIoefHGdjSrdsU2YoIkKMmWFliz4988wzaa8vvfRSADZu\n3Fjheaoz3Xu2n2BLzR166KFpP2vixz/+ccb3w/0Yf/rTn9YuMImMn7Kr7FPkfv36JRFOtSkzFBFB\nlaGICBDjbbKft8zPOh3mO+aWHaqXaeiev82uzkp6Ur/526yyt1u6Nc5tvrO15wc9XHPNNUmEU23K\nDEVEiDEzHDBgAADDhw9PvVfZeihV8X9tfHeO8ePHA9C+fftan1Nyi39IprWR65c5c+akve7QoQMQ\nTM6Qq5QZiogQY2boV7HzK98BzJw5E4BRo0bV+Hx//vOfgWAtZMk/fr1rT52tc5tfAXP16tVp7zdp\n0gTI/YlSlBmKiJDAcDy/NnJ4u3fv3kCwCpqfqPWMM84A4He/+13qM/7JYniFNMlPfrVEP8D/lltu\nSTIcqYKfWs0PtVu1ahUA+++/f2Ix1YQyQxERcmSihlNOOSXtpwgEGca1114LaI3kXOf7/vrp9Xwv\ngMMOOyyxmGpCmaGICDmSGYpk4tuOpX7Ze++9AZg4cWLCkdSMMkMREVQZiogAqgxFRABVhiIigCpD\nERFAlaGICACWabX7Cg82+wxYF104OafAOde26sPyh8o4/6mMM6tRZSgikq90mywigipDERFAlaGI\nCBDx2GQz2wOYV/qyHbAD+Kz09c+cc99FdN0+wEigITDWOfe3KK4jyZVx6bV3AV4D3nfO9Y/qOj90\nCX6PJwN9gA3OuW5RXCPtenE9QDGz24CvnXMjyrxvpXHszNJ1GgHvAD8HPgaWAb90zr2bjfNLxeIq\n49B5hwHdgGaqDOMRZxmb2QnAVmBcHJVhIrfJZtbJzIrM7GFgFdDBzP4X2j/QzCaUbu9lZtPNbJmZ\nLTWzI6s4/ZHAW865dc65bcBjQL+ofhfJLOIyxswKgF7ApKh+B6lc1GXsnFsAbIrsFygjyTbDg4CR\nzrkuwIZKjnsAGO6c6w6cA/j/uT3MbEyG4/cBPgq9Xl/6nsQvqjIGGAUMBdQ3LFlRlnGskpzPcI1z\nblk1jusJHBhaO3d3M2vqnFsCLIksOsmGSMrYzPoDHznnVphZz+yFK7WQN9/jJCvDLaHtnUB4pfAm\noW2jZo20G4AOodf7UvlfLIlOVGV8NDDAzPqWnqeVmU12zg2qU7RSG1GVcexyomtNaaPrZjPb38wa\nAGeGds8FrvQvzKyqhtTFQBczKzCzH1GSks/KdsxSM9ksY+fcMOfcvs65QuAC4DlVhMnL8vc4djlR\nGZa6HpgDLKKknc+7EjjGzN4wsyLgUqi4rcE5tx24CngeKAKmOOfeiTp4qZaslLHktKyVsZlNAxZS\nktysN7NfRxm4xiaLiJBbmaGISGJUGYqIoMpQRARQZSgiAtSwn2GbNm1cYWFhRKHkng8++IDi4mKr\n+sj8oTLOfyrjzGpUGRYWFrJsWXU6m+eH7t27Jx1C7FTG+U9lnJluk0VEUGUoIgKoMhQRAVQZiogA\nqgxFRABVhiIigCpDEREg2cldRUQA2Lx5MwAffvhhhccUFBQAMHLkSAC6du0KwAEHHADAIYccUqcY\nlBmKiJBwZvjpp58CcM455wBw9NFHA3DZZZcBJT3ls+GLL74A4KWXXgLglFNOAaBRo0ZZOb+I1MxT\nTz0FwOzZswF48cUXAXjvvfcq/MyBBx4IlAyvA9i2bVva/p0767ZKqTJDERESyAx92wDAT37yEyDI\n3Pbaay8g+xnhYYcdBkBxcTFAalzm/vvvn5XrSPV9+eWXAPzpT38CYNWqVQDMnTs3dYwy9vywZs0a\nAB566CEAxo0bl9q3detWAGoy0/4770S7eocyQxERYswMfVbm2wcBPv/8cwCuvLJk0azRo0dn9Zp3\n3XUXAGvXrgWCv0zKCOM3ZcoUAG666Sag/FNDnzEC7LHHHvEFJpFZv75kPahRo0bV6TwHHXQQEDw9\njooyQxERYswMX3vtNSB4ahR2yy23ZO06b775Zmp7xIgRAJx5Zsnyreeee27WriPV47ODa6+9Fgju\nEMzS59ocMmRIavvBBx8EoHXr1nGEKLXgyxGCzO/YY48Fgt4ajRs3BmDXXXcFoEWLFqnPfP311wD8\n4he/AIKsr0ePHgAceuihqWObNm0KQPPmzbP8W6RTZigigipDEREghttk37H6iSeeKLdv4sSJALRt\n27bO1/G3x7169Sq3b8CAAQC0bNmyzteRmvFNFf5hWUWmTp2a2n7mmWeA4GGLv4X2t12SnC1btgDp\n37PXX38dgJkzZ6Yde9RRRwGwfPlyIL3LnH+Atu+++wLQoEHyeVnyEYiI5IDIM8M//OEPQNC1wneA\nBjj77LOzdp2XX34ZgI8//jj13sUXXwzABRdckLXrSNXWrVuX2p40aVLaPj+Y3newf/7558t93neW\n91nl+eefD0C7du2yH6xUy3fffQfAr371KyDIBgFuvPFGAHr27Jnxs5kGUey3335ZjrDulBmKiBBD\nZui7UPif++yzT2pfXdqA/HCeu+++GwiG/IS7bPg2SYnXihUrUtu+M/Xxxx8PwIIFCwD49ttvAXjk\nkUcA+Otf/5r6zOrVq4Egy+/Xrx8QtCWqy018fBcY/z3zEyuE2/mHDh0KQLNmzWKOLruUGYqIkMBE\nDX7qHoDevXsDsNtuuwEwePDgKj/vO237n4sXL07bn812SKmd8NRKPlP3na69Jk2aAPCb3/wGgMcf\nfzy1zw/w94P4fcahp8nx80+I77nnHiCYYHXhwoWpY3yn6vpOmaGICDFkhldffTUA8+fPB2Djxo2p\nfb79yGcATz75ZJXn88eWHc7VsWNHIGjbkOT861//Kvfev//9bwD69++f8TN+WrVMjjzySCB9OJfE\nY9GiRWmv/TA53z8wnygzFBEhhszw8MMPB2DlypVA+pPGZ599FoDhw4cDsOeeewIwaNCgCs934YUX\nAnDwwQenve+XDPAZoiTnvPPOS237bP/VV18F4O233waCfw8zZswA0if99W3I/j0/9Zov+y5dukQW\nu6QLt+VC8ET/9ttvT73Xt29fIH1yhfpImaGICKoMRUQAsJqsQdC9e3dXWUN3HN5//30guB3u1q0b\nAM899xyQnUkfvO7du7Ns2TKr+sj8kY0y3rRpU2rbl5MfYlfRA7DwwH/fgf70008H4N133wWCVRPH\njBlTp/jCVMaVKztoIpOGDRsCcPnllwPBnIQfffQRAJ06dQKCNY/C/Bo4flKHKB7MVLeMlRmKiJDw\nusm1cccddwDBXyr/8CWbGaHUTXi43LRp0wA466yzgPIZ4lVXXQXAvffem/qM75Dtp17zQ/XmzJkD\nBJ2yQQ/MovbHP/4RgPvuu6/CY3bs2AEEGb3/WRP+4emJJ54IpE/pFhdlhiIi1JPM0GcXAJMnTwag\nVatWgFZSy3V+WiffRcNPzOC7z/hM32eDYTfffDMAb731FhB00/GfgeDfg0TDD8Pzq1r66dS2b9+e\nOsavc+MzxNrwk0D773p4JTw/yW/UlBmKiFBPMkPf0TPstNNOA9Ini5Xc5TPEiiYAzcSviuZXNfSZ\n4QsvvJA6xj+51rRe0fBPio844gggeLIfNm/ePCDIFm+77TYAli5dWuPr+bbk//znPzX+bF0pMxQR\noR5mhn7tVP+US/Kfb6+aNWsWkP6k0a+xnM21t6VmTj755LTXfsitzwwbNWoEBMtwAFx66aUAjBw5\nEgjakpOkzFBEBFWGIiJAjt8m+2FX4RXv/KpqenDyw+HX1B02bBiQvj6vb6wfOHAgAAcccEC8wUk5\nfgZ7v2qef7DiZx8CeO+994BgxvqywmslxUWZoYgI9SQzDA8S79OnT9oxX331FRDMfZeL67FKdvhJ\nOe68887Ue/5B2g033AAE63P7bjkSv86dOwNBl6hHH3203DHh7lEAu+xSUhX5LnPh4ZlxUWYoIkKO\nZ4aZ+L8gPgPwj+b98B0Nz8p/F110UWp77NixAEyfPh0I2qLKzoQu8fFZ+ahRo4Dg7i3ckfqTTz4B\noLCwEAjK1LcBJ0GZoYgI9TAzHD9+PAATJkwA4Le//S0QDOqX/Beerm3u3LlAsJ6vn1ggFzrx/tD5\nnh9+rfR//vOfqX2vvPIKEGSCfgqvJCkzFBEhxzPD0aNHA3Drrbem3jv++OMBGDx4MAC77747AI0b\nN445OskFvveAXzbAD9krKioCtJJeLvGrG5bdzhXKDEVEyPHM8LjjjgNg/vz5CUciuc5PHnvIIYcA\nsHr1akCZoVSfMkMREVQZiogAOX6bLFJdfk2ctWvXJhyJ1FfKDEVEUGUoIgKoMhQRAcD8alTVOtjs\nM2BddOHknALnXNuqD8sfKuP8pzLOrEaVoYhIvtJtsogIqgxFRICI+xma2R7AvNKX7YAdwGelr3/m\nnPsuwmvvArwGvO+c6x/VdX7okipjM7sOuKT05Rjn3OgoriOJlvF6YHPp9bY553pEcZ3U9eJqMzSz\n24CvnXMjyrxvpXHszPL1hgHdgGaqDOMRVxmbWTdgMnAk8D3wHPAb55x6XEcszu9xaWXY1Tn3v2yd\nszKJ3CabWSczKzKzh4FVQAcz+19o/0Azm1C6vZeZTTezZWa21MyOrMb5C4BewKSofgepXMRl3BlY\n7Jzb6pzbDrwEnBnV7yKZRf09jluSbYYHASOdc12ADZUc9wAw3DnXHTgH8P9ze5jZmAo+MwoYCuhR\nebKiKuOVwAlm1trMmgOnAh2yG7pUU5TfYwfMN7P/mNklFRyTNUmOTV7jnFtWjeN6AgeGlgvd3cya\nOueWAEvKHmxm/YGPnHMrzKxn9sKVWoikjJ1zb5rZ/cBc4GtgOSXtShK/SMq41JHOuQ1m1g543sze\ncs4tykLMGSVZGW4Jbe8ELPS6SWjbqFkj7dHAADPrW3qeVmY22Tk3qE7RSm1EVcY458YB4wDMbDiw\nug5xSu1FWcYbSn9+bGZPAj8DIqsMc6JrTWmj62Yz29/MGpDe/jMXuNK/KG08r+xcw5xz+zrnCoEL\ngOdUESYvm2VcesyepT8Lgb7A1GzGKzWXzTI2sxZm1qJ0uzklzwDezH7UgZyoDEtdD8yhpOZfH3r/\nSuAYM3vDzIqAS6HKtgbJTdks45mlx84ELnfOfRlh3FJ92Srj9sD/mdnrwFJghnNubpSBazieiAi5\nlRmKiCRGlaGICKoMRUQAVYYiIoAqQxERQJWhiAigylBEBFBlKCICwP8D3P5bzM0W5d8AAAAASUVO\nRK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f024d4e2668>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Get the first images from the test-set.\n",
    "images = data.test.images[0:9]\n",
    "\n",
    "# Get the true classes for those images.\n",
    "cls_true = data.test.cls[0:9]\n",
    "\n",
    "# Plot the images and labels using our helper-function above.\n",
    "plot_images(images=images, cls_true=cls_true, img_shape=img_shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sequential Model\n",
    "\n",
    "The Keras API has two modes of constructing Neural Networks. The simplest is the Sequential Model which only allows for the layers to be added in sequence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Start construction of the Keras Sequential model.\n",
    "model = Sequential()\n",
    "\n",
    "# Add an input layer which is similar to a feed_dict in TensorFlow.\n",
    "# Note that the input-shape must be a tuple containing the image-size.\n",
    "model.add(InputLayer(input_shape=(img_size_flat,)))\n",
    "\n",
    "# The input is a flattened array with 784 elements,\n",
    "# but the convolutional layers expect images with shape (28, 28, 1)\n",
    "model.add(Reshape(img_shape_full))\n",
    "\n",
    "# First convolutional layer with ReLU-activation and max-pooling.\n",
    "model.add(Conv2D(kernel_size=5, strides=1, filters=16, padding='same',\n",
    "                 activation='relu', name='layer_conv1'))\n",
    "model.add(MaxPooling2D(pool_size=2, strides=2))\n",
    "\n",
    "# Second convolutional layer with ReLU-activation and max-pooling.\n",
    "model.add(Conv2D(kernel_size=5, strides=1, filters=36, padding='same',\n",
    "                 activation='relu', name='layer_conv2'))\n",
    "model.add(MaxPooling2D(pool_size=2, strides=2))\n",
    "\n",
    "# Flatten the 4-rank output of the convolutional layers\n",
    "# to 2-rank that can be input to a fully-connected / dense layer.\n",
    "model.add(Flatten())\n",
    "\n",
    "# First fully-connected / dense layer with ReLU-activation.\n",
    "model.add(Dense(128, activation='relu'))\n",
    "\n",
    "# Last fully-connected / dense layer with softmax-activation\n",
    "# for use in classification.\n",
    "model.add(Dense(num_classes, activation='softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Compilation\n",
    "\n",
    "The Neural Network has now been defined and must be finalized by adding a loss-function, optimizer and performance metrics. This is called model \"compilation\" in Keras.\n",
    "\n",
    "We can either define the optimizer using a string, or if we want more control of its parameters then we need to instantiate an object. For example, we can set the learning-rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from tensorflow.python.keras.optimizers import Adam\n",
    "\n",
    "optimizer = Adam(lr=1e-3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For a classification-problem such as MNIST which has 10 possible classes, we need to use the loss-function called `categorical_crossentropy`. The performance metric we are interested in is the classification accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.compile(optimizer=optimizer,\n",
    "              loss='categorical_crossentropy',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training\n",
    "\n",
    "Now that the model has been fully defined with loss-function and optimizer, we can train it. This function takes numpy-arrays and performs the given number of training epochs using the given batch-size. An epoch is one full use of the entire training-set. So for 10 epochs we would iterate randomly over the entire training-set 10 times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 52250 samples, validate on 2750 samples\n",
      "Epoch 1/2\n",
      "52250/52250 [==============================] - 33s - loss: 0.2241 - acc: 0.9346 - val_loss: 0.0627 - val_acc: 0.9829\n",
      "Epoch 2/2\n",
      "52250/52250 [==============================] - 33s - loss: 0.0642 - acc: 0.9800 - val_loss: 0.0490 - val_acc: 0.9884\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(x=data.train.images,\n",
    "          y=data.train.labels, validation_split=0.05,\n",
    "          epochs=2, batch_size=128)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation\n",
    "\n",
    "Now that the model has been trained we can test its performance on the test-set. This also uses numpy-arrays as input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 9952/10000 [============================>.] - ETA: 0s"
     ]
    }
   ],
   "source": [
    "result = model.evaluate(x=data.test.images,\n",
    "                        y=data.test.labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can print all the performance metrics for the test-set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss 0.0396336512798\n",
      "acc 0.9877\n"
     ]
    }
   ],
   "source": [
    "for name, value in zip(model.metrics_names, result):\n",
    "    print(name, value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or we can just print the classification accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "acc: 98.77%\n"
     ]
    }
   ],
   "source": [
    "print(\"{0}: {1:.2%}\".format(model.metrics_names[1], result[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['val_acc', 'val_loss', 'loss', 'acc'])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "history_dict = history.history\n",
    "history_dict.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "loss_values = history_dict['loss']\n",
    "val_loss_values = history_dict['val_loss']\n",
    "acc_values = history_dict['acc']\n",
    "val_acc_values = history_dict['val_acc']\n",
    "epochs = range(1, len(loss_values) + 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x7f024ceb5470>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEKCAYAAAA4t9PUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGAxJREFUeJzt3X+wX3V95/Hni8SwMAqCue3QhJC0pmvjqrF+ie62Za1a\nDd0doFtUWBRwGTPWUt1x64DLbKFRZorOlg4ttcQfRNwoKv5Kt1JkFW1nNDY3iITAoJfwK5EOEfBX\nY8HIe//4nuiX603uNzn33Jt783zMnPme8zmfc/L+zM3c1z0/vuekqpAk6WAdMdMFSJJmN4NEktSK\nQSJJasUgkSS1YpBIkloxSCRJrRgkkqRWDBJJUisGiSSplfkzXcB0WLhwYS1dunSmy5CkWWXLli3f\nqaqRyfodFkGydOlSRkdHZ7oMSZpVktw/TD9PbUmSWjFIJEmtGCSSpFYMEklSK50GSZLVSe5OMpbk\n4gnWvy3JnUluT/KFJCc17SuTfDXJtmbdawe2WZ/k3iS3NdPKLscgSdq/zoIkyTzgauBUYAVwdpIV\n47p9HehV1fOBG4B3N+27gXOr6rnAauAvkjxzYLu3V9XKZrqti/o3bIClS+GII/qfGzZ08a9I0uzX\n5RHJKmCsqrZX1RPA9cDpgx2q6paq2t0sbgIWN+3frKpvNfPfBh4GJr2Xeaps2ABr1sD990NV/3PN\nGsNEkibSZZAsAh4cWN7RtO3LBcCN4xuTrAIWAPcMNF/enPK6MsmRE+0syZoko0lGd+3adUCFX3IJ\n7N791Lbdu/vtkqSnOiQutid5HdAD3jOu/QTgw8AbqurJpvkdwHOAk4HjgYsm2mdVrauqXlX1RkYO\n7GDmgQcOrF2SDmddBslO4MSB5cVN21MkeQVwCXBaVT0+0H4M8HfAJVW1aW97VT1UfY8D19I/hTal\nliw5sHZJOpx1GSSbgeVJliVZAJwFbBzskOSFwDX0Q+ThgfYFwKeB66rqhnHbnNB8BjgDuGOqC7/8\ncjj66Ke2HX10v12S9FSdBUlV7QEuBG4C7gI+XlXbkqxNclrT7T3A04FPNLfy7g2a1wCnAOdPcJvv\nhiRbga3AQuBdU137OefAunVw0kmQ9D/Xreu3S5KeKlU10zV0rtfrlQ9tlKQDk2RLVfUm63dIXGyX\nJM1eBokkqRWDRJLUikEiSWrFIJEktWKQSJJaMUgkSa0YJJKkVgwSSVIrBokkqRWDRJLUikEiSWrF\nIJEktWKQSJJaMUgkSa0YJJKkVgwSSVIrBokkqRWDRJLUSqdBkmR1kruTjCW5eIL1b0tyZ5Lbk3wh\nyUkD685L8q1mOm+g/UVJtjb7vCpJuhyDJGn/OguSJPOAq4FTgRXA2UlWjOv2daBXVc8HbgDe3Wx7\nPHAp8GJgFXBpkuOabd4LvBFY3kyruxqDJGlyXR6RrALGqmp7VT0BXA+cPtihqm6pqt3N4iZgcTP/\nKuDmqnq0qh4DbgZWJzkBOKaqNlVVAdcBZ3Q4BknSJLoMkkXAgwPLO5q2fbkAuHGSbRc185PuM8ma\nJKNJRnft2nWApUuShnVIXGxP8jqgB7xnqvZZVeuqqldVvZGRkanarSRpnC6DZCdw4sDy4qbtKZK8\nArgEOK2qHp9k25387PTXPvcpSZo+XQbJZmB5kmVJFgBnARsHOyR5IXAN/RB5eGDVTcArkxzXXGR/\nJXBTVT0EfD/JS5q7tc4FPtvhGCRJk5jf1Y6rak+SC+mHwjzgg1W1LclaYLSqNtI/lfV04BPNXbwP\nVNVpVfVoknfSDyOAtVX1aDP/ZmA9cBT9ayo3IkmaMenf/DS39Xq9Gh0dnekyJGlWSbKlqnqT9Tsk\nLrZLkmYvg0SS1IpBIklqxSCRJLVikEiSWjFIJEmtGCSSpFYMEklSKwaJJKkVg0SS1IpBIklqxSCR\nJLVikEiSWjFIJEmtGCSSpFYMEklSKwaJJKkVg0SS1IpBIklqpdMgSbI6yd1JxpJcPMH6U5LcmmRP\nkjMH2n87yW0D078mOaNZtz7JvQPrVnY5BknS/s3vasdJ5gFXA78D7AA2J9lYVXcOdHsAOB/448Ft\nq+oWYGWzn+OBMeDzA13eXlU3dFW7JGl4nQUJsAoYq6rtAEmuB04HfhokVXVfs+7J/eznTODGqtrd\nXamSpIPV5amtRcCDA8s7mrYDdRbw0XFtlye5PcmVSY482AIlSe0d0hfbk5wAPA+4aaD5HcBzgJOB\n44GL9rHtmiSjSUZ37drVea2SdLjqMkh2AicOLC9u2g7Ea4BPV9WP9zZU1UPV9zhwLf1TaD+nqtZV\nVa+qeiMjIwf4z0qShtVlkGwGlidZlmQB/VNUGw9wH2cz7rRWc5RCkgBnAHdMQa2SpIPUWZBU1R7g\nQvqnpe4CPl5V25KsTXIaQJKTk+wAXg1ck2Tb3u2TLKV/RPPlcbvekGQrsBVYCLyrqzFIkiaXqprp\nGjrX6/VqdHR0psuQpFklyZaq6k3W75C+2C5JOvQZJJKkVgwSSVIrBokkqRWDRJLUikEiSWrFIJEk\ntWKQSJJaMUgkSa0YJJKkVgwSSVIrBokkqRWDRJLUikEiSWrFIJEktWKQSJJaMUgkSa0YJJKkVgwS\nSVIrnQZJktVJ7k4yluTiCdafkuTWJHuSnDlu3U+S3NZMGwfalyX5WrPPjyVZ0OUYJEn711mQJJkH\nXA2cCqwAzk6yYly3B4DzgY9MsIsfVdXKZjptoP0K4MqqejbwGHDBlBcvSRpal0ckq4CxqtpeVU8A\n1wOnD3aoqvuq6nbgyWF2mCTAy4AbmqYPAWdMXcmSpAPVZZAsAh4cWN7RtA3r3yQZTbIpyd6weBbw\n3arac5D7lCRNsfkzXcB+nFRVO5P8MvDFJFuB7w27cZI1wBqAJUuWdFSiJKnLI5KdwIkDy4ubtqFU\n1c7mczvwJeCFwCPAM5PsDcB97rOq1lVVr6p6IyMjB169JGkoXQbJZmB5c5fVAuAsYOMk2wCQ5Lgk\nRzbzC4HfAO6sqgJuAfbe4XUe8Nkpr1ySNLTOgqS5jnEhcBNwF/DxqtqWZG2S0wCSnJxkB/Bq4Jok\n25rNfw0YTfIN+sHxZ1V1Z7PuIuBtScboXzP5QFdjkCRNLv0/8ue2Xq9Xo6OjM12GJM0qSbZUVW+y\nfn6zXZLUylBBkuRXBq5ZvDTJW5I8s9vSJEmzwbBHJJ8EfpLk2cA6+ndjTfRtdEnSYWbYIHmyuXj+\ne8BfVtXbgRO6K0uSNFsMGyQ/TnI2/dtt/2/T9rRuSpIkzSbDBskbgH8PXF5V9yZZBny4u7IkSbPF\nUI9Iab7D8Rbof1kQeEZVXdFlYZKk2WHYu7a+lOSYJMcDtwLvS/Ln3ZYmSZoNhj21dWxVfR/4L8B1\nVfVi4BXdlSVJmi2GDZL5SU4AXsPPLrZLkjR0kKyl/8yse6pqc/No9291V5YkabYY9mL7J4BPDCxv\nB36/q6IkSbPHsBfbFyf5dJKHm+mTSRZ3XZwk6dA37Kmta+m/S+SXmulvmzZJ0mFu2CAZqaprq2pP\nM60HfO2gJGnoIHkkyeuSzGum19F/7a0k6TA3bJD8N/q3/v4z8BD9V92e31FNkqRZZKggqar7q+q0\nqhqpql+oqjPwri1JEu3ekPi2KatCkjRrtQmSTNohWZ3k7iRjSS6eYP0pSW5NsifJmQPtK5N8Ncm2\nJLcnee3AuvVJ7k1yWzOtbDEGSVJLQ30hcR9qfyuTzAOuBn4H2AFsTrKxeZLwXg/Qv9byx+M23w2c\nW1XfSvJLwJYkN1XVd5v1b6+qG1rULkmaIvsNkiQ/YOLACHDUJPteBYw134InyfXA6cBPg6Sq7mvW\nPTm4YVV9c2D+20kepn+78XeRJB1S9ntqq6qeUVXHTDA9o6omO5pZBDw4sLyjaTsgSVYBC4B7Bpov\nb055XZnkyAPdpyRp6rS5RtK55onDHwbeUFV7j1reATwHOBk4HrhoH9uuSTKaZHTXrl3TUq8kHY66\nDJKdwIkDy4ubtqEkOQb4O+CSqtq0t72qHqq+x+k/pmXVRNtX1bqq6lVVb2TEL+FLUle6DJLNwPIk\ny5IsAM6i/7yuSTX9P03/JVo3jFt3QvMZ4AzgjimtWpJ0QDoLkqraA1xI/z0mdwEfr6ptSdYmOQ0g\nyclJdgCvBq5Jsq3Z/DXAKcD5E9zmuyHJVmArsBB4V1djkCRNLlX7vYt3Tuj1ejU6OjrTZUjSrJJk\nS1X1Jut3SF9slyQd+gwSSVIrBokkqRWDRJLUikEiSWrFIJEktWKQSJJaMUgkSa0YJJKkVgwSSVIr\nBokkqRWDRJLUikEiSWrFIJEktWKQSJJaMUgkSa0YJJKkVgwSSVIrBokkqRWDRJLUSqdBkmR1kruT\njCW5eIL1pyS5NcmeJGeOW3dekm8103kD7S9KsrXZ51VJ0uUYJEn711mQJJkHXA2cCqwAzk6yYly3\nB4DzgY+M2/Z44FLgxcAq4NIkxzWr3wu8EVjeTKs7GoIkaQhdHpGsAsaqantVPQFcD5w+2KGq7quq\n24Enx237KuDmqnq0qh4DbgZWJzkBOKaqNlVVAdcBZ3Q4BknSJLoMkkXAgwPLO5q2NtsuauYn3WeS\nNUlGk4zu2rVr6KIlSQdmzl5sr6p1VdWrqt7IyMhMlyNJc1aXQbITOHFgeXHT1mbbnc38wexTktSB\nLoNkM7A8ybIkC4CzgI1DbnsT8MokxzUX2V8J3FRVDwHfT/KS5m6tc4HPdlG8JGk4nQVJVe0BLqQf\nCncBH6+qbUnWJjkNIMnJSXYArwauSbKt2fZR4J30w2gzsLZpA3gz8H5gDLgHuLGrMUiSJpf+zU9z\nW6/Xq9HR0ZkuQ5JmlSRbqqo3Wb85e7FdkjQ9DBJJUisGiSSpFYNEktSKQSJJasUgkSS1YpBIklox\nSCRJrRgkkqRWDBJJUisGiSSpFYNEktSKQSJJasUgkSS1YpBIkloxSCRJrRgkkqRWDBJJUisGiSSp\nlU6DJMnqJHcnGUty8QTrj0zysWb915IsbdrPSXLbwPRkkpXNui81+9y77he6HIMkaf86C5Ik84Cr\ngVOBFcDZSVaM63YB8FhVPRu4ErgCoKo2VNXKqloJvB64t6puG9junL3rq+rhrsYgSZpcl0ckq4Cx\nqtpeVU8A1wOnj+tzOvChZv4G4OVJMq7P2c22kqRDUJdBsgh4cGB5R9M2YZ+q2gN8D3jWuD6vBT46\nru3a5rTW/5ogeCRJ0+iQvtie5MXA7qq6Y6D5nKp6HvBbzfT6fWy7JsloktFdu3ZNQ7WSdHjqMkh2\nAicOLC9u2ibsk2Q+cCzwyMD6sxh3NFJVO5vPHwAfoX8K7edU1bqq6lVVb2RkpMUwJEn702WQbAaW\nJ1mWZAH9UNg4rs9G4Lxm/kzgi1VVAEmOAF7DwPWRJPOTLGzmnwb8Z+AOJEkzZn5XO66qPUkuBG4C\n5gEfrKptSdYCo1W1EfgA8OEkY8Cj9MNmr1OAB6tq+0DbkcBNTYjMA/4f8L6uxiBJmlyaA4A5rdfr\n1ejo6EyXIUmzSpItVdWbrN8hfbFdknToM0gkSa0YJJKkVgwSSVIrBokkzTEbNsDSpXDEEf3PDRu6\n/fcMkklcdtlMVyBJw9uwAdasgfvvh6r+55o13YaJt/9OIun/MCRpNli6tB8e4510Etx334Hty9t/\nJekw9MADB9Y+FQySCVx2Wf9IZO9zhffOe5pL0qFuyZIDa58KBskELrusfzpr7ymtvfMGiaRD3eWX\nw9FHP7Xt6KP77V0xSCRpDjnnHFi3rn9NJOl/rlvXb+9KZw9tnCsuvXSmK5CkA3POOd0Gx3gekUzC\n01mStH8GiSSpFYNEktSKQSJJasUgkSS1YpBIkloxSCRJrXQaJElWJ7k7yViSiydYf2SSjzXrv5Zk\nadO+NMmPktzWTH8zsM2Lkmxttrkq2fsgE0nSTOgsSJLMA64GTgVWAGcnWTGu2wXAY1X1bOBK4IqB\ndfdU1cpmetNA+3uBNwLLm2l1V2OQJE2uyyOSVcBYVW2vqieA64HTx/U5HfhQM38D8PL9HWEkOQE4\npqo2Vf/599cBZ0x96ZI0+03XF6q7DJJFwIMDyzuatgn7VNUe4HvAs5p1y5J8PcmXk/zWQP8dk+xT\nkgT86Z9Oz79zqD5r6yFgSVU9kuRFwGeSPPdAdpBkDbAGYEmXz0+WpMNcl0ckO4ETB5YXN20T9kky\nHzgWeKSqHq+qRwCqagtwD/CrTf/Fk+yTZrt1VdWrqt7IyMgUDEeSDn0z8T6lLoNkM7A8ybIkC4Cz\ngI3j+mwEzmvmzwS+WFWVZKS5WE+SX6Z/UX17VT0EfD/JS5prKecCn+1wDJI0q8zE+5Q6O7VVVXuS\nXAjcBMwDPlhV25KsBUaraiPwAeDDScaAR+mHDcApwNokPwaeBN5UVY82694MrAeOAm5sJknSDOn0\nGklVfQ743Li2PxmY/1fg1RNs90ngk/vY5yjw76a2Ukmae6brfUp+s12S5qi5cPuvJOkwYJBIklox\nSCRJrRgkkqRWDBJJUiupvd9amcOS7ALuP8jNFwLfmcJyZgPHfHhwzHNf2/GeVFWTPhrksAiSNpKM\nVlVvpuuYTo758OCY577pGq+ntiRJrRgkkqRWDJLJrZvpAmaAYz48OOa5b1rG6zUSSVIrHpFIklox\nSIAkH0zycJI79rE+Sa5KMpbk9iS/Pt01TrUhxnxOM9atSb6S5AXTXeNUm2zMA/1OTrInyZnTVVtX\nhhlzkpcmuS3JtiRfns76ujDE/+1jk/xtkm80Y37DdNc4lZKcmOSWJHc243nrBH06/R1mkPStB1bv\nZ/2p9F+utZz+63vfOw01dW09+x/zvcB/rKrnAe9kbpxbXs/+x0zzQrUrgM9PR0HTYD37GXOSZwJ/\nDZxWVc9lgtc6zELr2f/P+Q+BO6vqBcBLgf/dvHxvttoD/I+qWgG8BPjDJCvG9en0d5hBAlTVP9B/\nsda+nA5cV32bgGcmOWF6quvGZGOuqq9U1WPN4iae+orjWWmInzPAH9F/F87D3VfUvSHG/F+BT1XV\nA03/WT/uIcZcwDOat6w+vem7Zzpq60JVPVRVtzbzPwDuAhaN69bp7zCDZDiLgAcHlnfw8z+ouewC\nDoM3USZZBPwec+OIc1i/ChyX5EtJtiQ5d6YLmgZ/Bfwa8G1gK/DWqnpyZkuaGkmWAi8EvjZuVae/\nwzp9Q6JmvyS/TT9IfnOma5kGfwFcVFVP9v9YPSzMB14EvJz+66u/mmRTVX1zZsvq1KuA24CXAb8C\n3JzkH6vq+zNbVjtJnk7/aPq/T/dYDJLh7AROHFhe3LTNaUmeD7wfOLWqHpnpeqZBD7i+CZGFwO8m\n2VNVn5nZsjq1A3ikqv4F+Jck/wC8AJjLQfIG4M+q/92HsST3As8B/mlmyzp4SZ5GP0Q2VNWnJujS\n6e8wT20NZyNwbnPnw0uA71XVQzNdVJeSLAE+Bbx+jv91+lNVtayqllbVUuAG4M1zPEQAPgv8ZpL5\nSY4GXkz/HPtc9gD9IzCS/CLwb4HtM1pRC821ng8Ad1XVn++jW6e/wzwiAZJ8lP7dGwuT7AAuBZ4G\nUFV/A3wO+F1gDNhN/y+aWW2IMf8J8Czgr5u/0PfM9ofdDTHmOWeyMVfVXUn+HrgdeBJ4f1Xt9/bo\nQ90QP+d3AuuTbAVC/3TmbH4i8G8Arwe2JrmtafufwBKYnt9hfrNdktSKp7YkSa0YJJKkVgwSSVIr\nBokkqRWDRJLUikEiHaQkP2memrt3ungK9710sqcUS4cKv0ciHbwfVdXKmS5CmmkekUhTLMl9Sd7d\nvMvln5I8u2lfmuSLzfsgvtA8PYAkv5jk0837Mb6R5D80u5qX5H3NOyY+n+Sopv9bmndP3J7k+hka\npvRTBol08I4ad2rrtQPrvte8y+Wv6D8MEuAvgQ9V1fOBDcBVTftVwJeb92P8OrCtaV8OXN28J+S7\nwO837RcDL2z286auBicNy2+2SwcpyQ+r6ukTtN8HvKyqtjcP0/vnqnpWku8AJ1TVj5v2h6pqYZJd\nwOKqenxgH0uBm6tqebN8EfC0qnpX80iTHwKfAT5TVT/seKjSfnlEInWj9jF/IB4fmP8JP7um+Z+A\nq+kfvWxO4rVOzSiDROrGawc+v9rMfwU4q5k/B/jHZv4LwB9A/1W/SY7d106THAGcWFW3ABcBx9J/\ny580Y/xLRjp4Rw08bRXg76tq7y3AxyW5nf5RxdlN2x8B1yZ5O7CLnz2B9a3AuiQX0D/y+ANgX4/4\nngf8nyZsAlxVVd+dshFJB8FrJNIUa66R9Gb5o8mloXlqS5LUikckkqRWPCKRJLVikEiSWjFIJEmt\nGCSSpFYMEklSKwaJJKmV/w9VAX9jGOWOBgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f024cf3bd68>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(epochs, loss_values, 'bo')\n",
    "plt.plot(epochs, val_loss_values, 'b+')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Loss')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFfVJREFUeJzt3X/wXXV95/HniwAKAwolKeuQX6zGrWmLYL+i1TpBXLeg\nXSixXcGsDl1nMlaxdGfpCktbKTZFHbrTslB3olJgNytlWcviDBYoBHFG2ebLQsJCNjSmAgm0hipt\nQ9piwnv/uOcLl+8m33Mh3/O932/u8zFz557zOZ977/szydzX95zPPeekqpAkaSqHDLsASdLsZ1hI\nkloZFpKkVoaFJKmVYSFJamVYSJJaGRaSpFaGhSSplWEhSWp16LALmC7z58+vpUuXDrsMSZpT7r//\n/qerakFbv07DIskZwO8D84AvVdVnJ21fAlwLLAC+D/zrqtrebPsc8P6m62eq6o+m+qylS5cyPj4+\nzSOQpINbkscG6dfZYagk84BrgDOB5cB5SZZP6nYlcENVnQRcDlzRvPb9wFuAk4G3ARcleU1XtUqS\nptblnMWpwNaq2lZVzwE3AmdP6rMcuLtZXt+3fTlwb1XtqapngU3AGR3WKkmaQpdhcQLwRN/69qat\n30ZgZbN8DnB0kuOa9jOSHJlkPvBuYNHkD0iyOsl4kvGdO3dO+wAkST3D/jXURcCKJA8AK4AdwN6q\nugO4DfgW8BXg28DeyS+uqrVVNVZVYwsWtM7PSJJeoS7DYgcv3RtY2LS9oKqerKqVVXUKcGnT9kzz\nvKaqTq6q9wIBHu2wVknSFLoMiw3AsiQnJjkcOBe4tb9DkvlJJmq4hN4vo0gyrzkcRZKTgJOAOzqs\nVZLmrMsu6/4zOguLqtoDXADcDmwGbqqqh5NcnuSspttpwJYkjwLHA2ua9sOAbyZ5BFhL7ye1e7qq\nVZLmst/6re4/IwfLbVXHxsbK8ywkjaIEXulXeZL7q2qsrd+wJ7glSa/AZZf1QiLprU8sd3VIyj0L\nSZrj3LOQJM0KhoUkzXGf/nT3n2FYSNIcN6d/OitJOngYFpKkVoaFJKmVYSFJamVYSJJaGRaSpFaG\nhSSplWEhSWplWEiSWhkWkqRWhoUkqZVhIUlqZVg0ZuJCXJI0VxkWjZm4h60kzVWGhSSp1UiHxUzf\nw1aS5irvwd04kHvYStJc5T24JUnTxrBozMQ9bCVprjIsGs5TSNL+GRaSpFaGhSSpVadhkeSMJFuS\nbE1y8T62L0lyV5JNSe5JsrBv2+eTPJxkc5KrkokfuEqSZlpnYZFkHnANcCawHDgvyfJJ3a4Ebqiq\nk4DLgSua174DeCdwEvATwFuBFV3VKklz0bp1sHQpHHJI73nduu4+q8s9i1OBrVW1raqeA24Ezp7U\nZzlwd7O8vm97Aa8GDgdeBRwG/FWHtUrSnLJuHaxeDY891jtH7LHHeutdBUaXYXEC8ETf+vamrd9G\nYGWzfA5wdJLjqurb9MLjqeZxe1Vt7rBWSZpTLr0Udu9+advu3b32Lgx7gvsiYEWSB+gdZtoB7E3y\nBuBNwEJ6AXN6kndNfnGS1UnGk4zv3LlzJuuWpKF6/PGX136gugyLHcCivvWFTdsLqurJqlpZVacA\nlzZtz9Dby7ivqnZV1S7g68BPT/6AqlpbVWNVNbZgwYKuxiFJs87ixS+v/UB1GRYbgGVJTkxyOHAu\ncGt/hyTzk0zUcAlwbbP8OL09jkOTHEZvr8PDUJLUWLMGjjzypW1HHtlr70JnYVFVe4ALgNvpfdHf\nVFUPJ7k8yVlNt9OALUkeBY4HJoZ5M/Ad4CF68xobq+prXdUqSXPNqlWwdi0sWdK7EOqSJb31Vau6\n+TyvOitJI8yrzkqSpo1hIUlqZVhIkloZFpKkVoaFJKmVYSFJamVYSJJaGRaSpFaGhSSplWEhSWpl\nWEiSWhkWkqRWhoUkqZVhIUlqZVhIkloZFpKkVoaFJKmVYSFJamVYSJJaGRaSpFaGhSSplWEhSWpl\nWEiSWhkWkqRWhoUkqZVhIUlqZVhIkloZFpKkVp2GRZIzkmxJsjXJxfvYviTJXUk2JbknycKm/d1J\nHux7/EOSn++yVknS/nUWFknmAdcAZwLLgfOSLJ/U7Urghqo6CbgcuAKgqtZX1clVdTJwOrAbuKOr\nWiVJU+tyz+JUYGtVbauq54AbgbMn9VkO3N0sr9/HdoBfAL5eVbs7q1SSNKUuw+IE4Im+9e1NW7+N\nwMpm+Rzg6CTHTepzLvCVTiqUJA1k2BPcFwErkjwArAB2AHsnNiZ5HfCTwO37enGS1UnGk4zv3Llz\nJuqVpJHUZVjsABb1rS9s2l5QVU9W1cqqOgW4tGl7pq/LvwL+uKp+uK8PqKq1VTVWVWMLFiyY3uol\nSS/oMiw2AMuSnJjkcHqHk27t75BkfpKJGi4Brp30HufhIShJGrrOwqKq9gAX0DuEtBm4qaoeTnJ5\nkrOabqcBW5I8ChwPrJl4fZKl9PZMvtFVjZKkwaSqhl3DtBgbG6vx8fFhlyFJc0qS+6tqrK3fsCe4\nJUlzgGEhSWplWEiSWhkWkqRWhoUkqVVrWCT5ZJJjZ6IYSdLsNMiexfHAhiQ3NZccT9dFSZJml9aw\nqKpfB5YBXwbOB/48ye8keX3HtUmSZomB5iyqd+beXzaPPcCxwM1JPt9hbZKkWeLQtg5JLgQ+AjwN\nfAn4tar6YXNNpz8H/n23JUqShq01LIAfAVZW1WP9jVX1fJKf66YsSdJsMshhqK8D359YSfKaJG8D\nqKrNXRUmSZo9BgmLLwC7+tZ3NW2SpBExSFik+i5NW1XPM9jhK0nSQWKQsNiW5FeSHNY8LgS2dV2Y\nJGn2GCQsPga8g94tUbcDbwNWd1mUJGl2aT2cVFXfo3dLVEnSiBrkPItXAx8Ffhx49UR7Vf2bDuuS\nJM0igxyG+i/APwF+lt79sBcCf9dlUZKk2WWQsHhDVf0G8GxVXQ+8n968hSRpRAwSFj9snp9J8hPA\na4Ef7a4kSdJsM8j5Emub+1n8OnArcBTwG51WJUmaVaYMi+ZigX9bVT8A7gX+6YxUJUmaVaY8DNWc\nre1VZSVpxA0yZ/GnSS5KsijJj0w8Oq9MkjRrDDJn8cHm+RN9bYWHpCRpZAxyBveJM1GIJGn2GuQM\n7o/sq72qbpj+ciRJs9EgcxZv7Xu8C7gMOGuQN09yRpItSbYmuXgf25ckuSvJpiT3JFnYt21xkjuS\nbE7ySJKlg3ymJGn6DXIY6pP960mOAW5se12SecA1wHvpXa12Q5Jbq+qRvm5XAjdU1fVJTgeuAD7c\nbLsBWFNVdyY5Cnh+kAFJkqbfIHsWkz0LDDKPcSqwtaq2VdVz9ALm7El9lgN3N8vrJ7YnWQ4cWlV3\nAlTVrqra/QpqlSRNg0HmLL5G79dP0AuX5cBNA7z3CcATfesT98LotxFYCfw+cA5wdJLjgDfSu7zI\nV+kF058CF1fV3gE+V5I0zQb56eyVfct7gMeqavs0ff5FwNVJzqd3hvgOYG9T17uAU4DHgT8Czge+\n3P/iJKtpbsS0ePHiaSpJkjTZIGHxOPBUVf0DQJIjkiytqu+2vG4HsKhvfWHT9oKqepLengXNvMQH\nquqZJNuBB6tqW7PtFuDtTAqLqloLrAUYGxsrJEmdGGTO4r/z0snlvU1bmw3AsiQnJjmc3t32bu3v\nkGR+c/0pgEuAa/tee0ySBc366UD/xLgkaQYNEhaHNhPUADTLh7e9qKr2ABcAtwObgZuq6uEklyeZ\n+OntacCWJI8CxwNrmtfupXeI6q4kDwEBvjjwqCRJ02qQw1A7k5xVVbcCJDkbeHqQN6+q24DbJrX9\nZt/yzcDN+3ntncBJg3yOJKlbg4TFx4B1Sa5u1rcD+zyrW5J0cBrkpLzvAG9vJqCpql2dVyVJmlVa\n5yyS/E6SY5oT43YlOTbJb89EcZKk2WGQCe4zq+qZiZXmrnnv664kSdJsM0hYzEvyqomVJEcAr5qi\nvyTpIDPIBPc6ej9h/UN6P2E9H7i+y6IkSbPLIBPcn0uyEfjn9K4RdTuwpOvCJEmzx6BXnf0rekHx\ni/TOpt7cWUWSpFlnv3sWSd4InNc8nqZ3Mb9U1btnqDZJ0iwx1WGo/wt8E/i5qtoKkOTfzkhVkqRZ\nZarDUCuBp4D1Sb6Y5D30JrglSSNmv2FRVbdU1bnAj9G7i92vAj+a5AtJ/sVMFShJGr7WCe6qeraq\n/ltV/Ut696R4APhU55VJkmaNl3UP7qr6QVWtrar3dFWQJGn2eVlhIUkaTYaFJKmVYSFJamVYSJJa\nGRaSpFaGhSSplWEhSWplWEiSWhkWkqRWhoUkqZVhIUlqZVhIkloZFpKkVoaFJKlVp2GR5IwkW5Js\nTXLxPrYvSXJXkk1J7kmysG/b3iQPNo9bu6xTkjS1qe7BfUCSzAOuAd4LbAc2JLm1qh7p63YlcENV\nXZ/kdOAK4MPNtr+vqpO7qk+SNLgu9yxOBbZW1baqeg64ETh7Up/lwN3N8vp9bJckzQJdhsUJwBN9\n69ubtn4bgZXN8jnA0UmOa9ZfnWQ8yX1Jfn5fH5BkddNnfOfOndNZuySpz7AnuC8CViR5AFgB7AD2\nNtuWVNUY8CHg95K8fvKLm1u8jlXV2IIFC2asaEkaNZ3NWdD74l/Ut76waXtBVT1Js2eR5CjgA1X1\nTLNtR/O8Lck9wCnAdzqsV5K0H13uWWwAliU5McnhwLnAS37VlGR+kokaLgGubdqPTfKqiT7AO4H+\niXFJ0gzqLCyqag9wAXA7sBm4qaoeTnJ5krOabqcBW5I8ChwPrGna3wSMJ9lIb+L7s5N+RSVJmkGp\nqmHXMC3GxsZqfHx82GVI0pyS5P5mfnhKw57gliTNAYaFJKmVYSFJamVYSJJaGRaSpFaGhSSplWEh\nSWplWEiSWhkWkqRWhoUkqZVhIUlqZVhIkloZFpKkVoaFJKmVYSFJamVYSJJaGRaSpFaGhSSplWEh\nSWplWEiSWhkWkqRWhoUkqZVhIUlqZVhIkloZFpKkVoaFJKmVYSFJatVpWCQ5I8mWJFuTXLyP7UuS\n3JVkU5J7kiyctP01SbYnubrLOiVJU+ssLJLMA64BzgSWA+clWT6p25XADVV1EnA5cMWk7Z8B7u2q\nRknSYLrcszgV2FpV26rqOeBG4OxJfZYDdzfL6/u3J/kp4Hjgjg5rlCQNoMuwOAF4om99e9PWbyOw\nslk+Bzg6yXFJDgF+F7iow/okSQMa9gT3RcCKJA8AK4AdwF7g48BtVbV9qhcnWZ1kPMn4zp07u69W\nkkbUoR2+9w5gUd/6wqbtBVX1JM2eRZKjgA9U1TNJfhp4V5KPA0cBhyfZVVUXT3r9WmAtwNjYWHU2\nEkkacV2GxQZgWZIT6YXEucCH+jskmQ98v6qeBy4BrgWoqlV9fc4HxiYHhSRp5nR2GKqq9gAXALcD\nm4GbqurhJJcnOavpdhqwJcmj9Caz13RVjyTplUvVwXH0ZmxsrMbHx4ddhiTNKUnur6qxtn7DnuCW\nJM0BhoUkqZVhIUlqZVhIkloZFpKkVoaFJKmVYSFJamVYSJJaGRaSpFaGhSSplWEhSWplWEiSWo18\nWKxbB0uXwiGH9J7XrRt2RZI0+3R5P4tZb906WL0adu/urT/2WG8dYNWq/b9OkkbNSO9ZXHrpi0Ex\nYffuXrsk6UUjHRaPP/7y2iVpVI10WCxe/PLaJWlUjXRYrFkDRx750rYjj+y1S5JeNNJhsWoVrF0L\nS5ZA0nteu9bJbUmabKR/DQW9YDAcJGlqI71nIUkajGEhSWplWEiSWhkWkqRWhoUkqVWqatg1TIsk\nO4HHDuAt5gNPT1M5c8WojXnUxguOeVQcyJiXVNWCtk4HTVgcqCTjVTU27Dpm0qiNedTGC455VMzE\nmD0MJUlqZVhIkloZFi9aO+wChmDUxjxq4wXHPCo6H7NzFpKkVu5ZSJJajVRYJLk2yfeS/J/9bE+S\nq5JsTbIpyVtmusbpNsCYVzVjfSjJt5K8eaZrnG5tY+7r99Yke5L8wkzV1oVBxpvktCQPJnk4yTdm\nsr4uDPD/+rVJvpZkYzPmX5rpGqdbkkVJ1id5pBnThfvo09l32EiFBXAdcMYU288EljWP1cAXZqCm\nrl3H1GP+C2BFVf0k8BkOjuO91zH1mEkyD/gccMdMFNSx65hivEmOAf4AOKuqfhz4xRmqq0vXMfW/\n8SeAR6rqzcBpwO8mOXwG6urSHuDfVdVy4O3AJ5Isn9Sns++wkQqLqroX+P4UXc4Gbqie+4Bjkrxu\nZqrrRtuYq+pbVfWDZvU+YOGMFNahAf6dAT4J/A/ge91X1K0Bxvsh4KtV9XjTfxTGXMDRSQIc1fTd\nMxO1daWqnqqq/90s/x2wGThhUrfOvsNGKiwGcALwRN/6dv7/f4yD2UeBrw+7iK4lOQE4h4Njz3EQ\nbwSOTXJPkvuTfGTYBc2Aq4E3AU8CDwEXVtXzwy1p+iRZCpwC/K9Jmzr7Dhv5mx+pJ8m76YXFzwy7\nlhnwe8Cnqur53h+eB71DgZ8C3gMcAXw7yX1V9ehwy+rUzwIPAqcDrwfuTPLNqvrb4ZZ14JIcRW+v\n+FdncjyGxUvtABb1rS9s2g5qSU4CvgScWVV/Pex6ZsAYcGMTFPOB9yXZU1W3DLeszmwH/rqqngWe\nTXIv8GbgYA6LXwI+W71zA7Ym+Qvgx4A/G25ZBybJYfSCYl1VfXUfXTr7DvMw1EvdCnyk+UXB24G/\nqaqnhl1Ul5IsBr4KfPgg/0vzBVV1YlUtraqlwM3Axw/ioAD4n8DPJDk0yZHA2+gd7z6YPU5vT4ok\nxwP/DNg21IoOUDP/8mVgc1X9x/106+w7bKT2LJJ8hd4vI+Yn2Q58GjgMoKr+M3Ab8D5gK7Cb3l8n\nc9oAY/5N4DjgD5q/tPfM9YuwDTDmg0rbeKtqc5I/ATYBzwNfqqopf1Y82w3wb/wZ4LokDwGhd9hx\nrl+J9p3Ah4GHkjzYtP0HYDF0/x3mGdySpFYehpIktTIsJEmtDAtJUivDQpLUyrCQJLUyLKQWSfY2\nV2ydeFw8je+9tO3quNJsMFLnWUiv0N9X1cnDLkIaJvcspFcoyXeTfL65F8ifJXlD0740yd3N/QTu\nas6SJ8nxSf64ucfCxiTvaN5qXpIvNvcouCPJEU3/X2nuXbApyY1DGqYEGBbSII6YdBjqg33b/qa5\nF8jV9C5QCPCfgOur6iRgHXBV034V8I3mHgtvAR5u2pcB1zT3mngG+EDTfjFwSvM+H+tqcNIgPINb\napFkV1UdtY/27wKnV9W25gJvf1lVxyV5GnhdVf2waX+qquYn2QksrKp/7HuPpcCdVbWsWf8UcFhV\n/XZziY5dwC3ALVW1q+OhSvvlnoV0YGo/yy/HP/Yt7+XFucT3A9fQ2wvZkMQ5Rg2NYSEdmA/2PX+7\nWf4WcG6zvAr4ZrN8F/DL0Luta5LX7u9NkxwCLKqq9cCngNfSu+ObNBT+pSK1O6LvKp8Af1JVEz+f\nPTbJJnp7B+c1bZ8E/jDJrwE7efHKnxcCa5N8lN4exC8D+7t89DzgvzaBEuCqqnpm2kYkvUzOWUiv\nUDNnMXYQXPpaauVhKElSK/csJEmt3LOQJLUyLCRJrQwLSVIrw0KS1MqwkCS1MiwkSa3+HzU4Tly9\nCphCAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f024ceed6a0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(epochs, acc_values, 'bo')\n",
    "plt.plot(epochs, val_acc_values, 'b+')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Accuracy')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prediction\n",
    "\n",
    "We can also predict the classification for new images. We will just use some images from the test-set but you could load your own images into numpy arrays and use those instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "images = data.test.images[0:9]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are the true class-number for those images. This is only used when plotting the images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cls_true = data.test.cls[0:9]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the predicted classes as One-Hot encoded arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_pred = model.predict(x=images)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the predicted classes as integers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cls_pred = np.argmax(y_pred,axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAU0AAAD5CAYAAACj3GcTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xe8VNW5//HPg2IQVBCxU86NWECjmGCvV5EoFpQYxViI\nsUQxlngDSYwSTJAYbPwsV1FfoglWEBGJimJXQISASrGAokEvIqJGiWJ7fn/MXjN7Tp19ph++79eL\n19kzs8uj68w6z1p77bXM3RERkdy0KncAIiLVRJWmiEgCqjRFRBJQpSkikoAqTRGRBFRpiogkoEpT\nRCQBVZoiIgmo0hQRSWDdfA7u1KmT19TUFCiU6jBnzpyV7r5pueMoFZVxy6cyTiavSrOmpobZs2fn\nc4qqY2bvlDuGUlIZt3wq42TUPBcRSUCVpohIAqo0RUQSUKUpIpKAKk0RkQTyunsu0lxXXnklAF98\n8QUAr7zyCgATJkyos+/ZZ58NwF577QXAySefXIoQReqlTFNEJAFlmlJSxx9/PADjx4+v93Mzq/Pe\nTTfdBMC0adMAOOCAAwDo2rVrMUKUMnrjjTcA2H777QG49tprATj33HPLFlNtyjRFRBJQpilFF7JL\naDjD3GGHHQA49NBDAXjrrbfSn02ePBmAxYsXAzBu3DgALrroosIHK2U1d+5cAFq1SuVzW2+9dTnD\nqZcyTRGRBJRpStGE55kfeOCBOp/ttNNOQCaL7NSpEwAbbLABAF999VV63z322AOAl19+GYCPPvqo\nSBFLuc2bNw/I/B4MGDCgnOHUS5mmiEgCJc80wzi8W265BYCtttoq/VmbNm0AOPHEEwHYYostAOje\nvXspQ5QC+b//+z8A3D39Xsgwp06dCsCWW25Z77FhHCfAokWLsj474ogjChqnlN+rr74KwHXXXQfA\nKaecUs5wGqVMU0QkgZJnmkOGDAFg6dKlDe4TxuVttNFGAPTs2bMg1+7SpQsAQ4cOBaB3794FOa/U\n78gjjwQyd70BNtxwQwA6duzY6LH33ntvejvevykt0+uvvw7A6tWrgewRF5VGmaaISAKqNEVEEih5\n8/zWW28FMsNH4k3vhQsXApkBrk8//TQAM2fOBDKPzb377rsNnr9169ZAZghLuBkRP09opqt5Xhrd\nunXLed8rrrgCyDxOFxeGHoWf0nKMGjUKSC29AZX93VSmKSKSQMkzzYMPPjjrZ1x4hC74+OOPgUzm\nGf76vPTSSw2e/3vf+x6QeeA/PJ4HsGrVKgC22WabZsUuxTNlyhQAhg0bBsCaNWvSn22++eYAXH75\n5QC0bdu2xNFJMcRvBofvdPjetmvXrhwh5USZpohIAhX9GOXGG28MwEEHHZT1fn1Zam33338/kMlW\nAXbeeWcABg4cWKgQpUDCI5fxDDMIw0/ClHDSMjzzzDN13tt008pfbl6ZpohIAhWdaTbHihUrABg8\neDCQ/Qhf6C9ramC1lM7RRx8NZB6rDAYNGpTeHjFiREljktIIS5zEhQdPKpkyTRGRBFpcpnnDDTcA\nmYyzQ4cO6c/CnTkpvzB+dvr06UCmLzP0aV188cXpfcM0YdIyzJgxA4CxY8em39t1110BOOSQQ8oS\nUxLKNEVEEmgxmebzzz8PZMbyBQ8++GB6O0xLJuUXJpdduXJl1vthWkCNpW25nnjiCSB7ZEsYox2m\nh6xkyjRFRBJQpSkikkCLaZ4//PDDQGbuxT59+gCw1157lS0mqSusCRQejQ0OPPBAAP70pz+VOiQp\nsTBZT9xPf/rTMkTSPMo0RUQSqPpM84svvgDg0UcfBTITdlx66aVAZqo4KZ/46pEjR44E6s7G3qtX\nL0DDi1qy5cuXA/Dcc88B2ZPpHHPMMWWJqTmUaYqIJFD1mWaYtDb0kR122GEA7L333mWLSbJdddVV\n6e1Zs2ZlfRYeo1RfZst3++23A/DBBx8Ame9qtVGmKSKSQFVmmmHCWoA///nPALRv3x6ASy65pCwx\nScOuvvrqBj8Lj72qL7Ple+edd7Jeh6kfq40yTRGRBKoq0wx3Yc8777z0e9988w0A/fr1AzQus9qE\nMs1llENoTYR9v/76awA+/fTTOvuGR/Suueaaes+1zjrrpLf/+te/AlpGo9geeuihrNdHHHFEmSLJ\njzJNEZEEVGmKiCRQFc3zb7/9FsjMhPL222+nP+vevTuQuSEk1SWs25SL4447DoAtt9wSyAxdueee\ne/KKIax2GZ/DUwonDGYP5VXtlGmKiCRQFZnmkiVLgMyKhXFhOIvmX6xc4SYdwKRJk5p9nvvuu6/J\nfcJNolatsvOBo446CoDevXvXOWbfffdtdkzStAceeADI3LQNs7RX6+qiyjRFRBKo6EwzDIbt27dv\n1vtXXnllertahy2sTSZOnJjeHjVqFFB3wo5g4cKFQOP9lKeddhoA3bp1q/PZT37yEwB69OjRvGCl\nYP7zn/8A8Mgjj2S9H6aBiw/7qibKNEVEEqjoTHPMmDFA3cev4n0hZlbSmCQ/ua5rfddddxU5Eim2\n0L8cVoTt378/AOeff37ZYioEZZoiIglUZKYZxnVdf/31ZY5ERJorZJphnfOWQpmmiEgCFZlphjXM\nP/vss6z3w9M/mkZMRMpFmaaISAKqNEVEEqjI5nltYaXCJ554AoCOHTuWMxwRWYsp0xQRSaAiM83f\n//73WT9FRCqFMk0RkQTM3Zt/sNmHwDtN7tiydHP3TcsdRKmojFs+lXEyeVWaIiJrGzXPRUQSUKUp\nIpJAo5WmmW1iZvOif8vN7L3Y6/WKEZCZ9YxdY56ZfWZmv2rimNPN7MNo/0Vm9os8YxhnZkc3sc/v\nYjEuMLNvzKx9PtcthzKVcTcze9rMFkb/7xot3+iYcpTxKWb2qpm9YmYvmNkP8rlmuZSjjKPr3hHK\nLMf9y1HGO5rZDDNbY2YX5HRid8/pHzAc+E097xvQKtfzJPkHtAZWAJ2b2O90YHS0vQWwEuhUa591\nE1x3HHB0gv2PAR4rxv+DUv4rVRkDWwG9ou2NgCXAdpVWxsA+QIdo+0jghXKXUbWUcXTOA4DdgXk5\n7l+OMt4c6A1cDlyQy3mb1Tw3s+5RlnAnsADoYmafxD4faGa3Rtubm9lEM5ttZrPMbM8ElzoEWOTu\ny3I9wN2XA0uBrmY2wsz+ZmYvALeb2bpmdnUUxytmdnoUYysz+18ze83MHgc6JYgR4ATg7oTHVLRi\nlrG7v+/u86LtfwOvAVvnGlupytjdX3D38N88E+ica4zVoNjfY3d/BljVnNhKWMYfuPts4JtcY8tn\ncPsOwCnuPtvMGjvPtcAod59pZjXAFGAnM9sDONXdz2rk2IEkrIzMrDvQDXgrFuf+7v6lmQ0GVrj7\n7mb2PWCmmT0G7An8F9CTVBa0ELgpOt9lpDKMhxu43gZAH+CMJHFWiaKXsZl9H9gJeCnXoEpdxpHT\ngEca+bxaleJ7nFiZyjgn+VSaS6Iauil9gO0tsyzFxma2vru/CLzY0EFm1gY4HLgwx3hONLMDgTXA\n6e7+SXTNB939y2ifvkAPMxsYvW4PbAvsD9zt7t8By8zs6XBSd/9DE9ftDzzj7p/mGGc1KXYZbwTc\nD5zr7p/ncJ2ylLGZ9QFOBlriWr9FLeNmKNf3OGf5VJqrY9vfkeoTCdrEtg3Y3d3rX36wYYcDL7r7\nyhz3v9Pd6+vIjcdpwGB3fyK+g5kdkzC2uIHA3/M4vpIVrYwtdQNiIjDW3SfneFjJy9jMegFjgB+7\n+8fNOUeFK/b3OKlyfY9zVpAhR1HN/rGZbWtmrUjdGAmmAeeEF9EvYS7q9BOa2flmlk8zYCowODRD\nzGx7M1sfeBY4PuoT2ZpUB3aTzGxjYG/goTxiqgqFLGNLpQ63k7pBcG2tzyqmjKNm6ATgZ+6+OI+Y\nqkKRvsd1VFIZN0chx2n+ltR/zHQgfuPmHGCfqMN2IVHfn5ntYWY31XciM9sQ+G9gUq2PegAf5RHj\nGOBNYJ6ZzQduJJVtTwDeJdUHMhZIL2piZpeZWb8GzvcT4BF3/yKPmKpJocr4AFJ/FA+xzNCXH0ef\nVVIZDwc6AmOiGAvZDK1UhfwejweeA3qa2TIz+3n0UcWUsZl1NrNlwHnA8CjOto1dvKoeozSzfwD9\n3T3nO11SXVTGLV+1l3FVVZoiIuWmxyhFRBJQpSkikoAqTRGRBFRpiogkkNcaQZ06dfKampoChVId\n5syZs9LXolm9VcYtn8o4mbwqzZqaGmbPzuUJrJbDzNaqZQFUxi2fyjgZNc9FRBJQpSkikoAqTRGR\nBFRpiogkoEpTRCQBVZoiIgnkNeSoVFavTs0/OmTIEABuuikzE1Xv3r0BGD9+PADdunUrcXQisjZR\npikikkBVZJrvv/8+ALfccgsA66yzTvqzMCj3oYdSk6f/6ldNLqEtFeCf//wnAAMGDABg6dKlzT7X\nY489lt7u0aMHAF26dGl+cFI24Xt81FFHAXDdddcBcPbZZ6f3iX//y0GZpohIAhWdaX744YcADBo0\nqMyRSKFNnToVgDVr1uR9rsmTM+uy3XbbbQDcc889eZ9XSuejj1KrX8QzSoBzzz0XgNNOOy393vrr\nr1+6wOqhTFNEJIGKzDSvvTa1QOGkSal11V566aUmj3nuuecACMt37LLLLgDsv//+xQhRmumbb1LL\nwjz88MMFO2cYQQFw9dVXA5kRF+3atSvYdaR4nn32WQDee++9rPdPOOEEANq0aVPnmHJRpikikkBF\nZpoXXJBaKz7JXbKJEydm/ezatSsA9913X3qfH/3oR4UKUZrpqaeeAmD69OkA/Pa3v837nKtWrUpv\nL1iwAID//Oc/gDLNShbvzx4xYkS9+5x88skAmFlJYsqFMk0RkQRUaYqIJFBRzfN+/foBmZs53377\nbZPHdOrUCcg0w955JzUh89tvvw3Abrvtlt73u+++K1ywkrNXX301vT1w4EAAunfvDsBFF12U9/nj\nQ46kerzyyivp7fCwQ7Duuqmq6bDDDitpTLlQpikikkDZM81nnnkmvf3aa68BmU7fhm4EnXXWWent\nvn37AtC+fXsAnnzySQAuu+yyOsfdeOONQN0BtFJc8bIIN2jGjRsHwAYbbNDs84YbQPHfoUq6YSCN\nCzdt63PIIYeUMJJklGmKiCRQtkwzTNAQ+rgAVq5cWe++YfjQscceC8Af//jH9Gdt27bN2jdMDTdm\nzJg65xw6dCgAX375JZCZ3KN169bN+4+QRk2YMAHIHsge+jLjfc3NFYapxLPLAw88EIAOHTrkfX4p\nrngLIVhvvfUAGDlyZKnDyZkyTRGRBMqWaX799ddAw9klZB6BvPfee4HMnfLGhEwz3JW98MIL05+F\nR+tCxhmmn9pmm20SxS65CRNDh//vUJj+5NBKueuuu4DMnVaAiy++GFDroZKFBxtmzJhR57PQcuzV\nq1dJY0pCmaaISAJlv3ten9DfNXbsWCC3DLO2kEXeeeed6fdmzZpVgOikKZ9++ikAM2fOrPPZ4MGD\n8z7/zTffDGSmDuzZs2f6s4MOOijv80txNTYBTzWMbFGmKSKSQNkzzfqe+nnxxRfzPm94qij+FFDt\nJ43CXfgwZlAKI0zEsGzZMiAzvVehLFmyJOv1TjvtVNDzS3HVl2mG0Q6FaIkUmzJNEZEEVGmKiCRQ\ntuZ5WLu8WCvLhVXt5s6dm36v9uOZl156aVGuvbbbcMMNgcywkfiEHeHRx44dOyY+74oVK4DMUKZg\nn332aVacUlrPP/88kBkqFhceg+7cuXNJY2oOZZoiIgmULdOcMmVKQc8Xhp8sXLgQaPwxrDCESQOg\niyOsFhgemQyPUwIcfvjhQPZDB/WZP39+ejvc+AnT/tWelKNVK/3trwZhxclwQzaukifoqE2/bSIi\nCZR9yFGhhOnHbrjhhgb3qampAeCOO+4AMhOBSHEMHz4cyM4sQgsjPlFLfTbddNP0dsgsG3rk9tRT\nT80nTCmR2n3R8UlVzjzzzFKH02zKNEVEEqj6TDMskREmMG5MeNxuv/32K2pMktKjRw8ge0XQMJqh\n9gD12sI0gHGDBg0C6j6MEPpQpTKFhxxq3zWP3ykvxFSBpaJMU0QkgbJlmo0tnvbII49kvT7jjDMA\neP/99xs8Ty7LHBT6jr0kt+uuu2b9TOL73/9+ve/Hx4H+4Ac/aF5gUjRhKrjad8379+9fjnDypkxT\nRCQBVZoiIgmUrXke5s0Ls6jHhQHQtR+xrO+Ry9C8z2XlSqluoXlXu5mnJnllC4Pag/BwyQUXXFCO\ncPKmTFNEJIGyZZoDBgwAYNSoUen3GlsvqCnhr1cY5nLLLbcAsOWWWzb7nFJZws0+rW1eXaZOnZr1\nukuXLkBmko5qo0xTRCSBsmWaYdXIsNIkwKRJkwAYPXp04vP94Q9/ADJrmUvLE9arDzSovbKFFWcX\nL16c9X6bNm2A6p0wR5mmiEgCZX+MMqxtHt/u27cvkFl1MEwofOSRRwLwy1/+Mn1MuJMaX5FQWqaw\nOmmY6GHYsGHlDEeaEKbsC49ILliwAIBtt922bDEVgjJNEZEEyp5p1ufQQw/N+ikCmYzl17/+NaA1\nzitdGDsdpm0Mox5++MMfli2mQlCmKSKSQEVmmiL1CX3bUl222morAG677bYyR1IYyjRFRBJQpSki\nkoAqTRGRBFRpiogkoEpTRCQBVZoiIglY7QldEx1s9iHwTuHCqQrd3H3TpndrGVTGLZ/KOJm8Kk0R\nkbWNmuciIgmo0hQRSUCVpohIAo1Wmma2iZnNi/4tN7P3Yq/XK1ZQZtbPzF43s8VmNiSH/UfEYnvV\nzA7P8/rPm1mvJvZpY2YTohhnmFnXfK5ZLuUq4+ja65rZK2Y2KYd9S17GsX2PNzPPdf9KU8bv8R1m\n9qGZzctx/9PD/ma2yMx+kef1x5nZ0U3ss2P0/V1jZjktj9nohB3u/hHQKzr5cOBzd7+y1kWN1A2l\n73K5YFPMrDVwPfDfwHJgtpk96O5vNHHoFe4+2sx2Ap4ys808dpfLzNZ1928KEWPkTGC5u3c3s5OA\nvwAnFvD8JVGOMo65EJgPtM1x/1KXMWa2ETAYmF3I85ZSGcv4NuAG4OYEx9zp7heY2RbAfDOb7O7p\nFReLUMYrgXOBY3M9oFnNczPrbmYLzexOYAHQxcw+iX0+0MxujbY3N7OJZjbbzGaZ2Z5NnH5PYJG7\nv+Pua4D7gP65xubu8wEDNo7+0txoZrOAkWa2gZndHsUx18yOjGJsa2bjo79u9wNtcrhUf+COaPs+\n4Me5xlgNilzGmFk34BBgbNLYSljGACOjf2uSxlnpil3G7v4MsKo5sbn7cmAp0DVqZfzNzF4Abo9a\nKFdHcbxiZqdHMbYys/81s9fM7HGgUw7X+cDdZwM5V8T59GnuAFzj7j2B9xrZ71pglLv3Bo4DQiHs\nYWY31bP/1sC/Yq+XRe/lxMz2Br5091BYWwJ7uvtQYBjwqLvvDhwEXGVmbYBfAR+7ew9gBLBr7Hxj\nG2iWpeN096+A1WbWIdc4q0SxyhhgNDAESDzmrVRlbGa7AZu5+9Tan7UgxSzjZjOz7kA34K1YnAe7\n+0mkWnkrojLeDTjHUt1jxwL/BfQETgX2jp3vMjPrV4jY8plPc0lUQzelD7C9Zdaq3tjM1nf3F4EX\n87h+bUPM7OfAZ8DxsffHx5ocfYHDzOx30es2QFdgf2AUgLvPNbMF4WB3P7WAMVabopRx1M/0L3ef\nZ2Z9EsRTsjI2s1bAVVRhl0tClfY9PtHMDiSV2Z/u7p9E13zQ3cNypH2BHmY2MHrdHtiWVBnfHf0u\nLDOzp8NJ3f0PhQown0pzdWz7O1LNpSDe9DFg9ygby8V7QJfY6840/hcwuMLd61v7Nx6nAUe7+5L4\nDrFfhCRCnMst1Znezt0/aeKYalOsMt4bGGBmR0Xn2cjM7nD3QU0cV8oy7kAqY3kuOnYL4GEzO9zd\n5yY9WQUrVhk3153uXt8NmdplPNjdn4jvYGbHFDWySEGGHEU1+8dmtm30Fzoe/DTgnPCigaZu3Eyg\np5l1M7PvkWoKTI6OHRX6qJppKqlO3xBLaKI9C/wsem8XYMcczjUZCF/y44DH8oir4hWyjN19qLt3\ndvca4CTgsVBhVkoZu/sqd+/k7jVRnLOBfi2swsxS4O9xg8zsfDM7q/mRMhUYbGbrRufb3szWJ1XG\nx0d9m1sDB+RxjQYVcpzmb0n9x0wn1Q8ZnAPsE3XYLgTOgIb7Qtz9a+A84HFgITDO3V+PPt6Z1B31\n5roUaGepISsLgOHR+9cDm5jZIuASIP3FaKRP82ZgSzNbTKq/7KI84qoWBSnjJlRSGa+NClbGZjYe\neI5UErQs6loB6AF8lEeMY4A3gXlmNh+4kVSreQLwLql6YywwIxZLvX2aZtbZzJaRqnOGR3E2Opqj\nap49t1Qb6RF31xKVLZTKeO1gZv8A+hd6eFipVE2lKSJSCfQYpYhIAqo0RUQSUKUpIpJAPuM06dSp\nk9fU1BQolOowZ86clWvTrN4q45ZPZZxMXpVmTU0Ns2dX7TwGzWJma9WyACrjlk9lnIya5yIiCajS\nFBFJQJWmiEgCqjRFRBJQpSkikoAqTRGRBFRpiogkkNc4TRGRYvj4448BePfddxvcp1u3bgBcc801\nAOy0004AbLfddgDssssuRYlNmaaISAIVlWmuWLECgOOOOw6AvfdOrYt05plnAqknFwrh008/BeDZ\nZ58F4NBDU9M3tm7duiDnF5FkpkyZAsBDDz0EwNNPPw3Am2++2eAx22+/PQBLly4FYM2a7AVDv/uu\n0CtOpyjTFBFJoOyZZui7ANhxx9SyLSET3HzzzYHCZ5g//OEPAVi5MrUGfXjudtttty3IdSR3//73\nvwH43e9Si0cuWJBaJHLatGnpfdQCaBmWLEmtdXfDDTcAcPPNN6c/++KLLwBIMin666+/3vRORaBM\nU0QkgbJlmiHLC/2XAB99lFpr6ZxzUoveXXfddQW95ogRIwB4++23gcxfOmWYpTdu3DgALr74YqDu\nXdKQgQJssskmpQtMimbZstQ6baNH17cKc+522GEHIHO3vNSUaYqIJFC2TPOf//wnkLlLFjds2LCC\nXWf+/Pnp7SuvvBKAY45JLed8/PHHF+w6kpuQbfz6178GMi2O1EKUGeeem166nOuvvx6Ajh07liJE\naYZQjpDJJPfdd18gMzplvfXWA6B9+/YAbLDBBuljPv/8cwB+/OMfA5ksco899gBg1113Te+7/vrr\nA9CuXbsC/1fkRpmmiEgCqjRFRBIoefM8DGC///7763x22223AbDppvkvzxKa5YccckidzwYMGADA\nhhtumPd1JJnQRRJu+jXknnvuSW8/8sgjQOamUWi6h+aelM/q1auB7O/Zyy+/DMCkSZOy9t1rr70A\nmDt3LpA9lDDcCOzcuTMArVpVbj5XuZGJiFSgkmea//M//wNkhpyEgeYAP/3pTwt2neeffx6A5cuX\np9879dRTATjppJMKdh1p2jvvZNawGjt2bNZnYVKF8CDD448/Xuf48FBCyFJPPPFEALbYYovCBys5\n+eqrrwD42c9+BmSyS4CLLroIgD59+tR7bH0Pq3Tt2rXAERaPMk0RkQRKnmmGoSXh59Zbb53+LJ8+\nqvAY1siRI4HMo1rxoSyhz1RKa968eentMGh9//33B+CZZ54B4MsvvwTgrrvuAuAvf/lL+pjFixcD\nmVZD//79gUxfp4YilU4YGhS+Z2GCjfh9iCFDhgDQtm3bEkdXGso0RUQSKPuEHWFKKIC+ffsC0KFD\nBwDOPvvsJo8Pg+PDz5kzZ2Z9Xsh+Umme+JRdIfMPg9uDNm3aAPCLX/wCgAkTJqQ/CxM9hMkcQgaj\nu+elF+6IX3755UBmIuDnnnsuvU8YvN5SKdMUEUmg5Jnm+eefD8CTTz4JwPvvv5/+LPRvhYziwQcf\nbPJ8Yd/aj+Fts802QKbvRcrn7rvvrvPeP/7xDwCOPvroeo8J0/XVZ8899wSyH8OT0pg+fXrW6/B4\nYxhfuTZQpikikkDJM80f/ehHALz66qtA9p3VRx99FIBRo0YBsNlmmwEwaNCgBs938sknA7Dzzjtn\nvR+WyggZp5TPCSeckN4OrYeXXnoJgNdeew3I/D488MADQPbk1KGPO7wXpvQLZd+zZ8+ixS7Z4n3N\nkBnBcOmll6bfO+qoo4DsSTZaEmWaIiIJqNIUEUnAkqzJUVvv3r29sQ77UnjrrbeATDO8V69eADz2\n2GNAYSb/iDOzOe7eu6AnrWCFKONVq1alt0M5hUcjG7qRF58AIjyocMQRRwDwxhtvAJlVSm+66aa8\n4qtNZdyw2g+n1GedddYB4KyzzgIyc2L+61//AqB79+5AZk2wuLBGVJjco1g3mPIpY2WaIiIJlH1w\ne77+9Kc/AZm/fOEmUqEzTGm++GOO48ePB+DYY48F6mac5513HgB//etf08eEge9hSr/wiOXUqVOB\nzOB30I2/YvvNb34DwFVXXdXgPt9++y2QaSGEn0mEm8AHHnggkD1VYLkp0xQRSaAqM82QrQDccccd\nAGy00UaAVi6sdGG6sDB0JUzQEYYVhZZDyC7jLrnkEgAWLVoEZIYvhWMg8/sgxREenwyryIZp+r7+\n+uv0PmEdqJBxNkeYrDx81+MrT4bJqMtFmaaISAJVmWmGAbVxhx9+OJA9qbFUrpBxNjRRbX3CKoRh\nFdGQaT711FPpfcKdek0XVxzhzvhuu+0GZEYyxD3xxBNAJvscPnw4ALNmzUp8vdDXPWfOnMTHFosy\nTRGRBKo+0wxrH4e7etLyhf60yZMnA9l3VsMa6cOGDSt9YALAwQcfnPU6PCodMs3WrVsDmeVnAM44\n4wwArrnmGiDT112JlGmKiCSgSlNEJIGqap6Hx+XiK0yGVQx1A2jtEdbEHjp0KJC9vna46TBw4EAA\ntttuu9IGJ3WEFRnCKpXhBlGYrQrgzTffBDIrMNQWX0us3JRpiogkUJWZZnyygH79+mXt89lnnwGZ\nuReraT1lSSZMzvLnP/85/V64Ifj73/8egHHjxgGZ4UpSej169AAyQ8XuvffeOvvEh40BrLtuqmoK\nQwnjj9XY76NRAAAG3klEQVSWmzJNEZEEqirTrE/4ixQyijBkITx2pcfqWr5TTjklvT1mzBgAJk6c\nCGT6ymrP7C+lE7L80aNHA5nWYHzA+gcffABATU0NkCnT0EddSZRpiogkUPWZ5i233ALArbfeCsDp\np58OZCZ3kJYvPg3gtGnTgMx63GGCiUoeLL22CCNdpkyZAsDf//739GczZswAMpllmBquEinTFBFJ\noKoyzeuuuw6AP/7xj+n39t9/fwDOPvtsADbeeGMA1ltvvRJHJ5UgjJYIy2WERy0XLlwIaOXKShJW\nE629XemUaYqIJFBVmeZ+++0HwJNPPlnmSKTShUmOd9llFwAWL14MKNOU/CnTFBFJQJWmiEgCVdU8\nF8lVWDPq7bffLnMk0tIo0xQRSUCVpohIAqo0RUQSsLDaW7MONvsQeKdw4VSFbu6+adO7tQwq45ZP\nZZxMXpWmiMjaRs1zEZEEVGmKiCTQaKVpZpuY2bzo33Izey/2uqgzYpjZumb2iplNymHfEbHYXjWz\nw/O89vNm1ivHfY83M891/0pTrjI2swvNbEH079wc9j/dzD6M4lpkZr/I8/rjzOzoJvbpaGaTo9/D\nF82sKp/BLGMZL4u+j/PM7MUc9i9HGfcxs09j/z/+0NR5Gx3c7u4fAb2ikw8HPnf3K2td1Ej1jX7X\n1MUSuhCYD7TNcf8r3H20me0EPGVmm3msw9bM1nX3bwoZoJltBAwGZhfyvKVUjjKO/sAMAnoD3wCP\nmdkUd29qJPqd7n6BmW0BzDezye6+MnbeQpfxJcCL7n6Ume0I/D/gkAKevyTK/D3ez90/SbB/qcsY\n4Cl3b7RyjWtW89zMupvZQjO7E1gAdDGzT2KfDzSzW6Ptzc1sopnNNrNZZrZnDufvRuqXc2zS2Nx9\nPmDAxtFfmhvNbBYw0sw2MLPbozjmmtmR0fXamtn46K/b/UCbHC83Mvq3Jmmcla7IZdwDmOnuX7j7\n18CzwDG5xubuy4GlQNeolfE3M3sBuD1qoVwdxfGKmZ0exdjKzP7XzF4zs8eBTjlcqifwZHTNBcB2\nZrZJrnFWumJ/j/NRwjJOLJ8+zR2Aa9y9J/BeI/tdC4xy997AcUAohD3M7KYGjhkNDAES39o3s72B\nL919VfTWlsCe7j4UGAY86u67AwcBV5lZG+BXwMfu3gMYAewaO99Yq6fpbWa7AZu5+9SkMVaRYpXx\nq8ABlmr+tgMOA7rkGpSZdQe6AW/F4jzY3U8CzgRWRGW8G3COmXUFjgX+i1RFeCqwd+x8l5lZ9rKm\nKS8DA6J99gI6R/9akmJ+jx140szmmNlpSYIqYRkD7GtmL5vZw5ZDF0w+z54vcfdcmqV9gO0ts+zu\nxma2vru/CNTp54j6IP7l7vPMrE+CeIaY2c+Bz4DjY++PjzU5+gKHmdnvotdtgK7A/sAoAHefa2YL\nwsHufmo9MbYCrgJOTBBfNSpKGbv7fDO7GpgGfA7MBb7N4TonmtmBpDL70939k+iaD7r7l9E+fYEe\nZjYwet0e2JZUGd8d/S4sM7OnY/E01I91GXCtmc0jVYG+nGOc1aQoZRzZ093fi5raj5vZInef3sR1\nSl3GLwE17v551PKcSKqCblA+lebq2PZ3pJrEQbx5a8Du7v5VjufdGxhgZkdF59nIzO5w90FNHHeF\nu49uIk4Djnb3JfEdYr8IuepA6q/Zc9GxWwAPm9nh7j436ckqWLHKGHe/GbgZwMxGAYtzOOxOd7+g\niTgNGOzuT8R3MLOcm/+xGD8l1fca/lAuBVraDCDFLOP3op/LzexBYHegqUqzHGUcth+KuvM6NNYP\nW5AhR1HN/rGZbRv9csWDnwacE17U19Stda6h7t7Z3WuAk4DHQoVpZqNCP2QzTQXSd2rNLDTDnwV+\nFr23C7BjEzGucvdO7l4TxTkb6NfCKswshSzjaJ/Nop81wFHAPdHr883srDxCnQoMNrN1o/Ntb2br\nkyrj46N+r62BA3KIsYOZtY5e/hKY5u6rGzummhWyjC11/2CDaLsdqXsU86PXlVTGW8S29wS+aerG\nVSHHaf6W1H/MdGBZ7P1zgH2iDtuFwBlRgI31hTRkZ2B5HjFeCrSz1DCIBcDw6P3rgU3MbBGpO6bp\nyq+hPs21VCHLeFK07yTgLHf/d/R+D+CjPGIcA7wJzDOz+cCNpFpUE4B3gYWkbjDOCAc00t/1A2Ch\nmb0OHExqREdLV6gy3hJ4wcxeBmYBD7j7tOizSirjgZYa9jYPuIbsrr16Vc1jlJZqBz/i7oeWOxYp\nHjP7B9C/CMNKpEJUexlXTaUpIlIJ9BiliEgCqjRFRBJQpSkikoAqTRGRBFRpiogkoEpTRCQBVZoi\nIgn8f6S1hmwDdPmXAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f024ce82d30>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_images(images=images,\n",
    "            img_shape=img_shape,\n",
    "            cls_true=cls_true,\n",
    "            cls_pred=cls_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Examples of Mis-Classified Images\n",
    "\n",
    "We can plot some examples of mis-classified images from the test-set.\n",
    "\n",
    "First we get the predicted classes for all the images in the test-set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_pred = model.predict(x=data.test.images)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we convert the predicted class-numbers from One-Hot encoded arrays to integers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cls_pred = np.argmax(y_pred,axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot some of the mis-classified images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAU0AAAD5CAYAAACj3GcTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmcFNW5//HPQ4iC4IKggogOht3luiAioqgYEjdwl4sL\nGIga9LoH1+RqFBdcg7vXXVCjIqLyE6KioqgoKLK6GxUUAQMqGnHh/P6oOl3Vw8x01/Q+fN+v17ym\nprq66pk5U6efOnXqHHPOISIi2WlU6gBERCqJKk0RkQRUaYqIJKBKU0QkAVWaIiIJqNIUEUlAlaaI\nSAKqNEVEElClKSKSQONc3tyqVStXVVWVp1Aqw8yZM5c55zYpdRzFojJu+FTGyeRUaVZVVTFjxoxc\ndlFxzOyTUsdQTCrjhk9lnIwuz0VEElClKSKSgCpNEZEEVGmKiCSgSlNEJIGc7p4X2pw5cwDYZ599\nAFi2bBkAb7zxRmqb7t27Fz8wKaqZM2cCsO+++wKw0UYbATB58mQAOnXqVJrAZK2kTFNEJIGyzDSH\nDh0KwP333w/Azz//DEQZRevWrUsTmBTc999/D8CJJ56YWvfUU08B8M0336R9P+KIIwB4++23ixmi\n5ImZpZYPPfRQAPz0O9tssw0Al1xySfEDy0CZpohIAmWZafq2quoZ5qRJkwDYYostShOYFIxvv/7D\nH/4AwJtvvpl6zWcf8cwEYK+99ipOcFIQ8fJ8/PHHgaisJ0yYAMCOO+4IRJloOVCmKSKSQFllmsOH\nDwfgyy+/BKBz584APP3000DwjKw0LJ9//jkA119/PZCeYWZy9913A7DLLruk1h1zzDF5jE4K6dZb\nb11j3YUXXghEPWUuv/xyQJmmiEjFKqtM85FHHgHgl19+AeDhhx8GlGE2ZFdccQUA99xzT+L3fvfd\ndwAMHjw4tc5nqjvssAMAxx13XI4RSqGccMIJa6zz5fd///d/xQ4na8o0RUQSKHmmedddd6WWV6xY\nAcBRRx0FQNeuXWt8j28HA5gyZUraa/7poc033zyvcUp++ad8fF9cf9e0JnW9Vt11110HRP9DyjQr\nky/zPfbYo8SRrEmZpohIAqo0RUQSKPnluX8kDqIbQLvuuisAjRsH4fkuR1deeSUAH330Ueo9Cxcu\nTNuf7/jerFkzAFq1apV67dRTTwWiQT7at2+fp99CkrrxxhuBqPyrd1z3N3IAnnjiCSB6XNJ3T3ru\nuedq3f/EiROBqFvS8ccfn4+wpcDGjx8PRP8PhxxySCnDqZEyTRGRBEqead50001rrPON+H6gBj8w\nw6pVqzLur3rm+e6776aWp02bBkC3bt3S9q8uTcV37733AmtmmD169ADgscceS61r06YNAG3btgWg\nRYsWQN2Z5nrrrQfAZpttlqeIpRh8Znn77bcDuhEkIlLxSpZp+kzjX//61xqvXX311UDUvuEzzN69\newNw9tlnp7b12Udt/vGPf6SWH3jgAQDmz58PwA033ADANddckzh+KYxhw4YB0LRp09S6b7/9Foi6\npN1xxx0Z9+O7nu2///75DlHyZOnSpall/7ikP+f91WA5UqYpIpJAyTJNPyiHv2Me5zsoe75dyj9q\nt/XWW2d9nJ133jm1fMABBwCw9957A9Ed3J49ewJR26mUjr+KiA/m0Lx5cwCmTp2a9X769++f38Ak\nZ5988gkAm2yyCQBjxoxJveZ7RPi26BdffLHI0WVPmaaISAIlv3tel0033RSAhx56CEiWYdakY8eO\nQJS5+mz3tddeA5RpFsOZZ54J1P5o5Ndffw1Ej1nGt61+p70mvk1swIABOcUp+ed7Rvh7CH6wFojK\n9vzzzwegS5cuRY4ue8o0RUQSKMtM07d5nHTSSQD06dMnL/v1g3j4zNW3bd55552A7qIXyp/+9KfU\n8oMPPghEmUU22aOXzbbKMMuP73O7ZMkSAC677LK0nyEanMdnmuVMmaaISAKqNEVEEijLy/Pql8/5\n1qFDh4LsV9L5GSbHjRuXWuc7qle3wQYbANGgLPGHHuI3DDI577zzALj44osBWGeddbIPWPJiwYIF\nQFTuvkx988rhhx8OwLx581Lv8bNRXnrppUA0V1A5UqYpIpJAWWaahe5u4Ge6k8K65ZZbAPjqq69q\n3aZXr14A/O1vfwNqvrpYuXIlALfddhsAP//8c63781nN+uuvD1TGjYWGwHdcB7jggguAqPuXv5Hr\nrx4GDRoERHM8QfTY5F/+8hcgGkSnHGcXVaYpIpJAWWaahfLTTz8BUZcH77DDDitFOA3WrFmzAHjy\nySczbuvnuq+r/Xr06NEATJgwAVhz+L+a+PZUKY74XEwvv/wyED2ccu211wKw5ZZbAtHA4N9//33q\nPb7LkR8abuTIkUD0WKXmPRcRqVAlyzT3228/AP73f/83tc4PATd27FgAhg4dCkSDzubKDwnn51f3\n7V5nnXVWXvYvgffeew+ARYsW1brNjjvuCKw5dJu/ux4f/s0/eplN5/ZynsWwIfLDu8UHU/FtmC+8\n8EKd7/VZZNxOO+0ERHfefbYaHyjcb1MqyjRFRBIoWaa53XbbAentiX6Q4BEjRgBRtnHKKacA0WOV\nfsK1uvg7rB988EFqnZ9Gw/P9xcp5wNNKlM0jkr5c/N1S317phwr84osvEu3P++yzzwBo3bp10rCl\nHqpPhAb5nQzt/vvvB6KrRFCmKSJSUVRpiogkUPIuR/GbMD7F943A/oaCn698+vTpQPpc5tX5uWEe\nfvhhILqpBNCyZUsgGiW6UI9pSmb+hk9Ns5Fma9111wXgoosuSq3LNGeU5Jc/F+PnpH8IoV27dkD9\nugv5kZF881388r/UHd6VaYqIJFDyTNN3PYGo0dffCBo1ahRQc9ZYm7///e9pP8dvCPiM5IQTTqh/\nwJLRbrvtBkCnTp2A6IqhvnzG0qhR+me8L0f//yLF57PITz/9NLXO38AdPHgwAO+88w6Q3SOtfsCO\n6oN8lNMAHso0RUQSKHmmWRPfHclnnn64Lz83cl0Z5xZbbAFEWUi8S5N/VEsKy5fBkCFDgGSDZvjs\nZPvtt0+tO/300/MXnBREvIx+97vfAdEDLNlc2R177LFAlJX6NtJ7770X0GOUIiIVy2qbFTAb3bt3\ndzNmzMhjOOXPzGY657qXOo5iURk3fIUq45deegmIrvCq93rxgxVDNFOlv6r02WldPWVykUsZK9MU\nEUmgLNs0RaTyZRo0JX6PobZpUMqRMk0RkQRUaYqIJKBKU0QkAVWaIiIJqNIUEUlAlaaISAKqNEVE\nElClKSKSQE6PUZrZUuCT/IVTEbZyzm1S6iCKRWXc8KmMk8mp0hQRWdvo8lxEJAFVmiIiCdRZaZpZ\nSzObFX4tNrNFsZ/XKVRQZraxmT1mZu+Y2QIz65Fh+2FmtjSMa4GZ/SHH448xs4MzbGNmdrOZfWBm\ns81sh1yOWSolLOOzzWyemc01s7Fmtm6G7S+NxTbHzA7I8fgvZ1NmZvbfZjY/jPW+XI5ZKirjOrcZ\nEdYZb5vZM2bWLuOOnXNZfQEXAWfXsN6ARtnuJ8tjjQWGhMvrABtm2H4YcH243BpYBrSqtk3jBMcf\nAxycYZv+wJPhcm9gWj7/BqX4KlYZA1sBHwBNwn2PA47J8J5LgdPD5W2BpYRt8vUs45eBHTJs0wWY\nCWwU/rxpqctIZZz3Mt4HaBou/w8wNtN+63V5bmYdwk/fscA8oJ2ZrYi9PtDM7giXNwuzxhlm9rqZ\n9cyw742BXZ1z9wA45350zn2dbWzOucXAv4Atw0+u+8xsGnCPmTU2s2vDOGab2bDwmI3CrPEdM3sG\nyGbk0wHAfeExXwZam1mDueNayDIO/ZrghGoMrAd8nm1szrm5BCdii/Cq4BYzex24zMyam9k9YRxv\nmdlBYYzrmdkjYVYxLjx2JicANzjnVoTHXZJtjJVAZQzOuSnOuf+EP74GbJHpPbm0aXYBrnPOdQMW\n1bHdaGCUC0ZJPhLwhbCrmd1aw/ZbA0vDyu4tM7vdzNbLNigz60DwKfdRLM6+zrljCE6CJc65HsAu\nwMlmtiVwONAe6AYcD/SK7W+kme1fw6HaAp/Ffl4YrmtIClLGzrlPgL8T/P2+ICiTKdkGZWa9gB+c\nc/8OV7UBejrnRgB/BSaFZbwPcI2ZNQFOAZY757oSZDQ7xvZ3dy2XcZ2ArmY2zcxeNbN+2cZYQdb2\nMo4bCjydKbZcBiH+0DmXzRj5+wKdLZrsvYWZNXXOTQem1xJTd4JUeSZwA/Bn4OIMxznazPYCVgHD\nnHMrwmNOcM79EG7Tj+AkGBj+vCHQEdgTeNA5txpYaGYv+J065y7I4ndsqApSxmbWEjiQ4IPqG2Cc\nmQ10zj2U4Th/NrMhwLfAUbH1j4RlB0EZ72dm54Y/NwG2JCjjUQDOubfMbJ5/s3Pu+FqO15jgQ7wP\nwQfxi2bWzTn3TYY4K8naXsY+3iHAdsCpGeLLqdL8Lra8miCV9uJpsQE9nHM/ZrnfhcCnviDDNDub\n6QjHOudq2i4epwHDnXPPxTcws0OyjC1uEdCOIKWHIK2v65O6EhWqjPsB7zvnlgGY2XiC7D7TCXWV\nc+76DHEaQXv0h/ENYid7EguBF51zPwMfmtmHwG+At+qzszK1tpcxZvZ7gsSsTza/X166HIWfAMvN\nrKOZNQLildCzwMmxAOtMkZ1zC4Evw8tsgL7A/PC9p5nZSTmEOhkYbmaNw/11NrOmwFTgqLBtsy1B\nZpHJE8Bx4X56A18655bmEFtZy2cZA58Cu5lZUwv+0/sCC8L3jvJtVPU0meAqxcfiL9GmAoPCdf8F\nbJPFvh4H9grfsylBhflxDrGVtbWxjM2sO3AT0N9X8Jnks5/mOQS/zCsEn9DeycDuFtx4mQ/8MQy2\ntjZNCP4g/zCz2QS/+BXh+q7AVznEeBvwPjDLzOYCtxBk248SFPJ84G7gVf+GOto0nwQWhdnHLcT+\noRqwvJSxc24awYfOW8Ac4GfgzvDl7YHFOcR4MdDMgi4r8wjuFgPcCLQ0swXAX4hli3W0d00EVoa/\n07PAGf6mUAO2tpXx1UAzguaDWWFGXKeKeozSzCYCA8LLJWlgwozkaefc70sdixRGQyjjiqo0RURK\nTY9RiogkoEpTRCQBVZoiIgmo0hQRSSCXzu20atXKVVVV5SmUyjBz5sxlbi0a1Vtl3PCpjJPJqdKs\nqqpixoxsnsBqOMxsrZoWQGXc8KmMk9HluYhIAqo0RUQSUKUpIpKAKk0RkQRUaYqIJKBKU0QkgZy6\nHOXDmDFjUsuDBw9Oe+3+++8HYNCgQUWNSUSyM3XqVABOPPHE1Lp3330XgD322AOArl27pr1nwYIF\nae+FaABhP4CQP/ePOeaYQoSdE2WaIiIJlDzTjGeXv/rVr9JeGzJkCADffvstAN26dQOiTzAprZ9+\n+gmAr74KxoWeP38+AMuWBQNgv/HGGwA8/XQ0V9V33wWzFhxxxBFp+zrzzDMB2HDDDQFo2rRpocKW\nPHrnnXeAKLuEKGt86aWXAHj55ZeBKIv0r8enpzjssMMA6NKlCwCHHnpoIcPOiTJNEZEESp5pZmP4\n8OEAbLNNMOXHzTffnHqtd+/eJYlpbfX559HU1aNHjwbgqquuqnHb6plF3DXXXJP289VXXw1E5Xnx\nxdHko3vvvXcOEUsh+au+mgYzv/XW2mazSX8vrNnuWc6UaYqIJFDyTNPfJYOoDbM2vv3EfwdlmsV2\n/fXR7Ko+W2zVqhUAO+20U9q2PvtYuXJlat2rr75KXaZNmwbAOeeck1r33HPBjMvrr79+fcOWAvEZ\nYvxqwi/7/4tybp+sD2WaIiIJlDzT7NSpU2r5l19+qXGb1atXp/0c7xO23nrrAerLWSxnnHFGavnY\nY48FoHnz5gC0b9++xvf85z//SS0/++yzQNQO6jPL6mbOnJlanjhxIgADBw6sb9hSYPH2SX/X/PLL\nLweUaYqIrNVUaYqIJFDyy/NNNolGnO/Tpw8QdYatrnrnd4huHunyvDjatGlT43Jd4h3VDzroIAD2\n3XdfAI488kggugSvybBhw4Co4/t+++2XIGIphvPPPz+1vP/++wNr3rj1HdcrnTJNEZEESp5pbrXV\nVqll32ndd2avLeOUyuezzyeffBKIspNJkyatse33338PwAEHHABEj9U2a9as4HFKdn73u9+lln3X\nMz/v0CefBNPxKNMUEVkLlTzTjPOfRJ07dwaUaa5N7rrrLgDatm2bcVvfLjplypSCxiT14zu8+25j\nvutRPButZMo0RUQSKKtM07vllluAqE0kmzmZ/aebbxOLt5VK+WvRogUAffv2BaJHJ2viMxg/FJ0f\nMlDKw3nnnQdEj0j7wYb99z333LPW9/rz9/HHHwfgtttuA9If0/SPZ/ohB3feeee8xZ4NZZoiIgmU\nZabp+cev3nzzTaDmfpree++9B8CVV14JpA8fJ+Vv3XXXBWCfffYB6s40GzUKPuubNGlS+MAkseqD\nePjvvr+m75s9bty41HsmTJgARFcR1d8bzzT9INe+N8Xzzz+fdtxCU6YpIpJAWWeavm3kwgsvLHEk\nUixnnXUWAMuXL0+t8wMUe19//TUAxx13HKBeFuXKD+LhB/DwA+1Un0StpnWHHHIIAEcffTSQnkX6\nK1A/xYZ/ktDf+9hyyy3z/aukUaYpIpKAKk0RkQTK+vK8PsaPHw9Av379ADj44INLGY5kafHixUDU\njeTjjz9OvVZ9/hn/s79MX7VqVeo1f0NJSs8P4uEfkfX8pXh8sB5/yf3HP/4RiB508ePlxvmbRX48\nV3/O+7Fe4zeYCkGZpohIAhWRaVYfub2ubZYsWQJE3RKkvHz00UcATJ8+HYg6M/uO0DXNXFnTOog6\nt/vHKgGuuOIKYM35iqT4fPn4KwP//YILLgDgkksuqdd+ffbpM0r//+EzzqVLlwLpmWw+KdMUEUmg\nrDPNxx57DIg6M9fVud3z2/huKPE2Tf/4lRSXH8oN4KijjgKiBxbyId4RfsCAAQDMnj0biB7PlOLz\nXYKqt2H6dst8qd4B3mecJ5xwQl6P4ynTFBFJoKwzTd8+VR8PPPAAAKeddlpqnTLN0oh3VN94441r\n3MYPurDNNtus8ZofqHjFihU1vnfzzTdPLfv50rO5KpHC8j1YJk+eDMDQoUOB/HU+9wOAVO9dUagM\n01OmKSKSQFlnmvfddx9Qc/YhlSOeWTzxxBNA1MfS81NX1DSFhR8ezE+DUt22226bWj7llFNyC1bq\nZcGCBall/3jrG2+8AURDweX7Ss8PbuzbMos1RKAyTRGRBFRpiogkUNaX5/5Rqk6dOgHRmJk1yaYD\nvJSef8xx0003zfo9u+++OwDNmzcH4Jtvvkl73Y/TCPDFF18A2c/JLvkRn0XUP+aYzUjt9eFHPfM3\nmPyNoHPPPTevx6mNMk0RkQTKOtP0xowZA8Buu+2WcVvf1aR3796Auhk1BP5GT1VVFQBz5swBohsA\n8S5NK1euLG5wAkQd2SEqFz82am2d2ePzoPurBb8ufvUAUYd1iDJMfxw/9qYf9KPQlGmKiCRQEZmm\nn1ly0KBBQNRxvS5+23bt2hUuMCkqPxTYiBEj0tb7Nm+IHp/t2LFj8QKTtCs638boR1L3bZx+fTYj\nt9f2M0QDdvih5/z3YlGmKSKSQEVkmv5TrFevXkB2maY0PIMHDwbg3nvvBaKhx3wmA3D22WcD0L9/\nfwBatmxZzBDXWvFsz2eF1YdnjHeAh/R5f/xrfmhH/0BLTfck/KPR8TbRYlKmKSKSQEVkmp5/EL/Q\nD+RLefJDi/nH5/zQgd9//31qG5+paNqL4opPS1HfwYUrhTJNEZEEKirTFAE48MAD076LFJMyTRGR\nBFRpiogkoEpTRCQBVZoiIgmo0hQRSUCVpohIAlZ9JrdEbzZbCnySv3AqwlbOuU1KHUSxqIwbPpVx\nMjlVmiIiaxtdnouIJKBKU0QkAVWaIiIJ1FlpmllLM5sVfi02s0Wxn9cpVFBmdq+ZLTWzWVluP8xv\nb2YLzOwPOR5/jJkdnGGbFmY20czeNrN5ZnZcLscslVKUsZl1ix1jlpl9a2anZHhPKcr4UDObHR7z\nDTPrlcsxS6WE5/FCM5sTHmd6FttXxnnsnMvqC7gIOLuG9QY0ynY/WR6rD9ADmJXl9sOA68Pl1sAy\noFW1bRonOP4Y4OAM2/wVGBkubwYsT3KMcvwqZhnH9v1rYAmwRRmWcXOim6U7AXNLXUaVVMbAQmCj\nBNtXxHlcr8tzM+tgZvPNbCwwD2hnZitirw80szvC5c3M7DEzm2Fmr5tZz0z7d869CPy7PrE55xYD\n/wK2NLNLzew+M5sG3GNmjc3s2jCO2WY2LIyxkZndbGbvmNkzQDZTWDpg/XC5OUEB/1KfmMtRocs4\n5rfAAufcwmzfUKwyds6tdOHZBDQjKPMGo4hlnFg5n8e5DA3XBTjOOTfDzOraz2hglHPuNTOrAp4C\ntjWzXYHjnXMn5RDDGsysA7AV8FEszj2dcz+Y2XBgiXOuh5mtC7xmZv8EegLtgW7A5sB84NZwfyOB\nac65/1ftUH8HnjKzz4ENgMNjJ1hDUYwyHgg8mCSoIpYxZnY4MJLgBNw/SZwVopBl7IApZuaAm51z\nd2YbVDmfx7lUmh8652Zksd2+QGeLZpNrYWZNnXPTgYztHAkcbWZ7AauAYc65FeExJzjnfgi36Qd0\nNbOB4c8bAh2BPYEHnXOrgYVm9oLfqXPuglqOtz/wOkFTQidgkplt55xrSBNvF7SMzawJcABwZpbx\nFLuMcc49CjxqZnsDl4T7b0gKWcY9nXOLzKw18IyZLXDOvZLhOGV/HudSaX4XW15N0CbiNYktG9DD\nOfdjDsfKxljn3Ok1rI/HacBw59xz8Q3M7JB6HO944KLwU+ldM/uM4I/+Zj32Va4KXcYHANOdc8sy\nbhkodhmnOOeet+AG5UbOuRWZ31ExClbGzrlF4ffFZjaB4D5Fpkqz7M/jvHQ5Cmv25WbW0cwaAfHg\nnwVO9j+Y2Q71PY6ZnWZmuVzOTwaG+8sQM+tsZk2BqcBRYZtIW4JPnUw+BfqG+2kDdAA+ziG2slag\nMv5vql2al1MZh21+Fi53J7gp1JAqzDT5LGMza25mzcPlZgRt13PDn8umjKnHeZzPfprnEPwyrxDc\nNfNOBnYPG2znA38MA9zVzG6taUdm9gjwEtDNgm4LQ8KXugJf5RDjbcD7wCwzmwvcQpBtP0rwx5sP\n3A28GotlpJnV1JZ1EdDHzGYDzxDckVyeQ2yVIJ9lvD6wN/B4tZfKqYyPBOZa0PVtNHBUDnFVinyV\ncRtgmpm9TXD5O94592z4WjmV8UUkPI8r6tlzM5sIDHDO/VzqWKQwVMYNX6WXcUVVmiIipabHKEVE\nElClKSKSgCpNEZEEcumnSatWrVxVVVWeQqkMM2fOXObWolG9VcYNn8o4mZwqzaqqKmbMyOZhgobD\nzNaqaQFUxg2fyjgZXZ6LiCSgSlNEJAFVmiIiCajSFBFJQJWmiEgCqjRFRBJQpSkikoAqTRGRBFRp\niogkkNMTQSIimXz77bcA3HjjjWnr//nPf6aWX3vtNQDOOOMMAM466ywAWrZsWYwQE1GmKSKSQEVl\nml9//TUABx544BqvDRkyBIChQ4cWMyTJ0UcfBTO0Pvvss2nrZ8+enVr2MyBut912AAwaNAiA5s2b\nFyNEqad3330XgB49egCwcmX6BI/xAdB9GV955ZVAlJVedtllAJxyyimFDTYBZZoiIgmUdabp20Ke\ney6YqXOHHYIJ8KZNm5baxn9avf766wDstttuAHTr1q1ocUr2fvghmLp65MiRADz00EMAfPDBB1nv\n44UXXgDg5ptvBmCjjTbKY4SSq2XLghmZ//SnPwFrZpjZ+O67YMbeESNGAOntn0888USuIeZEmaaI\nSAJlmWnOnTsXgP79+wPw+eefA1E22bNnz9S2r74azNL5008/AVFbmDLN8nTFFVcAcOmll9b4+kEH\nHQRAv379Uutuv/12AObMmQPAgw8GU6X7rPWxxx4rTLCStSVLlqSWjz76aABefPHFnPe7atUqAJYu\nXZrzvvJFmaaISAKqNEVEEiiry/Ply5cDUXr/8ccfA9ENoO233x6Agw8+OPUef3nujRs3DoCBAwcW\nNljJ2gYbbJBarn5TwM9NM378eCDqVvSrX/0qtc2JJ54IwOrVqwHo2LEjAM8///waxxo1ahQAf/3r\nXwHYZ599ABg7diwALVq0yOE3kdr48gOYMmVKjdv8+te/BqKbgHvuuWfqNX/eXnXVVYUKMW+UaYqI\nJFDyTPO9995LLZ988slAdDPHZx2vvPJK2nsOP/zw1PI555wDRF2PFi1aVLhgJZEvv/wSiBrzISqn\nTTfdFIhu4viriZr4DMV79NFHgajzdDxjeeONN9KO6a9E/I0KZZqFcffdd2fcplOnTkD0iGTc9OnT\n8x5ToSjTFBFJoGSZpv9kOfXUU1PrfJbgH6k65JBDAGjcOD3Mrbfeeo39+ff47ko+g/WfblJ8vlvR\njz/+mFrnM8yJEycCsOOOOybe74Ybbpi2//jVSnXHHnssAJ07d058HMmPrl27AjBhwoRat/FtzrXx\n/zflQJmmiEgCJcs0r7nmGiDKLmvyt7/9DYjukh555JFAlD3UxN+d/eSTYC54ZZrF5zuh+07pcb5N\nsXv37vXe/4ABA4CaM8xNNtkEiDrH+wEgpHQ6dOgAQPv27dPWP/nkk6nlWbNm1bmP0047Lf+B1ZMy\nTRGRBEqWac6cObPW137zm98A0KtXLwBWrFgBRO2f/lE8KU++P2W8LTMX/s73ww8/DERXETXp3bs3\nAGPGjMnLsaVub7/9NlB3mbRr167G9fE6oLb/Fd8WXU5XjMo0RUQSKFmm+dJLLwHRsG8QDbKx8847\n1/ge/8n0+9//fo3X4gOaSmm1bt0aiAZW8VMZQDTwwoUXXpj2vUmTJmn7uPzyy1PLvg/g+++/n7aN\nf88FF1yQWjd8+PDcfwHJmh9EOj5gR3V+8HA/wIq/VxG/YvS9X6rzWeoWW2yRe7B5okxTRCQBVZoi\nIglYLpdLURz5AAAIlElEQVS13bt3dzNmzMhjONnxA3sAbLzxxsCa6f3kyZMB+O1vf5vXY5vZTOdc\n/fvLVJhcytiPvN+qVavUuuoN/r6LUPXyi4+fWP1/1Hd07tu3LwAPPPBAveKrjco4uV133TW1XFc3\nwria5giqzj8Es8suu+QQ3ZpyKWNlmiIiCZR8wI76iA+6sO222wIwb968UoUjtVh//fWBaJ4XgFtv\nvRWI5pGpz4jcd955J1DzrKRSGvFMsbasMdv3lztlmiIiCVRkphm30047AWtmmr4rhJTeJZdcklr2\nj8ONHj06bRvfGd0PPF2TPfbYA4g6sEv58EM0QvrQjQ2RMk0RkQQqPtP085zfd999aev9wMV+qgQp\nD/5Ouu/gvHjxYgCeeuqpWt9z1113AdFQgZrnvPzEZw/1c5T7eek9P5DLhx9+mPV+r7vuOiD/PSRy\noUxTRCSBis80d999dwCaNm0KRI9q+bZOKW9Tp04F4K233kpbH5/iok2bNoAyzHLWrFmz1LLvP+u/\newsWLACiHi/Z+Oqrr/IQXX4p0xQRSUCVpohIAhV/ef7FF18A0WW5V1fXFSl/++23X2q5plGtpPK0\nbds28XvefPNNIL35pj7zSuWTMk0RkQQqPtP0Y2/6Ryv9YB7+E0rKk+92cu6556atr6qqAjS3jwT+\n/e9/A+mD9JSaMk0RkQQqPtNs2bIlAOussw4QDTflR4aX8uQH7qje9jx06FAAunTpUvSYRLKhTFNE\nJIGKzzSrq6QhpkTWJv7c9FeFq1atyvgezUYpIlLhGkymecQRRwBw4403ljgSycaZZ54JRG2bK1eu\nBOCmm24C4JFHHklt69utp0yZUswQJc/8oNSTJk0CYO+99651W/+o5XnnnQdoNkoRkYrVYDLNkSNH\nAtFTB3fccUcpw5EM/CAcxx9/PAA33HADEA0V579D+gC3Uvn69OkDwOrVq0scSf0o0xQRSUCVpohI\nAhU573kpaU7shk9l3PBp3nMRkSJRpSkikoAqTRGRBHJq0zSzpcAn+QunImzlnNuk1EEUi8q44VMZ\nJ5NTpSkisrbR5bmISAKqNEVEEqiz0jSzlmY2K/xabGaLYj+vU8jAzKyxmc02s8ez2PbSWGxzzOyA\nHI/9spntkGGbKjObEsb4vJltnssxS6VUZWxmZ5rZvPDrf7LYfpiZLQ3jWmBmf8jx+GPM7OAM27Qw\ns4lm9nYY53G5HLNUSljGZ4d/t7lmNtbM1s2wfWWcx865rL6Ai4Cza1hvQKNs95PgeCOAB4DHs9j2\nUuD0cHlbYClhe21sm8YJjv0ysEOGbcYDR4fL/YC78/03KPZXscoY2AF4G2gK/Bp4Hmif4T3DgOvD\n5dbAMqBVDmU8Bjg4wzZ/BUaGy5sBy5Mcoxy/iljGWwEfAE3CfY8Djsnwnoo4j+t1eW5mHcxsvpmN\nBeYB7cxsRez1gWZ2R7i8mZk9ZmYzzOx1M+uZxf63An4L3J00NufcXIJCahFmE7eY2evAZWbW3Mzu\nCeN4y8wOCo+3npk9EmYw4wgKOpNugB+r7Dng0KSxlrMCl3FX4DXn3H+ccz8BU4FDso3NObcY+Bew\nZZid3Gdm04B7wiuUa8M4ZpvZsDDGRmZ2s5m9Y2bPAK2yORSwfrjcnKCi/iXbOMtdoc9jgg/EJgQD\nA60HfJ5tbOV8HufSptkFuM451w1YVMd2o4FRLnhk6UjAF8KuZnZrLe+5HvgzwT9tImbWC/jBOffv\ncFUboKdzbgRB5jDJOdcD2Ae4xsyaAKcAy51zXQk+7XaM7e/uWlL8t4n+wIcBG5jZhknjLXOFKuM5\nQB8z29jMmgH7Ae2yDcrMOhBkMh/F4uzrnDsGOAFYEpbxLsDJZrYlcDjQnuAkOR7oFdvfSDPbv4ZD\n/R3Ywcw+Jyjv/3FhStKAFKSMnXOfEPz9PgO+ICiTrAdELefzOJeh4T50zmXzwOq+QGeLpqFoYWZN\nnXPTgenVNw7bmT5zzs0ys30TxPNnMxsCfAscFVv/iHPOj0HVD9jPzPy8sU2ALYE9gVEAzrm3zGye\nf7Nz7vhajncGcKOZDQVeBBbTgLKQUEHK2Dk318yuBZ4FVgJvkd3f7mgz2wtYBQxzzq0IjznBOfdD\nuE0/oKuZDQx/3hDoSFDGD4b/CwvN7IVYPBfUcrz9gdeBPkAnYJKZbeecW5lFrJWiUOdxS+BAgg+q\nb4BxZjbQOfdQhuOU/XmcS6X5XWx5NUEq7cXTYgN6OOd+zHK/vYBDzax/uJ8NzOxe59zgDO+7yjl3\nfYY4jaAd68P4BlaPeYWcc4sILynNbAPgsAZ2MkHhyhjn3O3A7QBmNoqg/SuTsc650zPEacBw59xz\n8Q3MLOvL/5jjgYvC7PJdM/uMoPJ8sx77KleFKuN+wPvOuWUAZjae4NzOVGmW/Xmcly5H4SfAcjPr\naGaNSG+fehY42f9QS4oc39cI59wWzrkq4Bjgn77CNLNRvv2iniYDqTu1ZubT96nAoHDdfwHbZNqR\nmbWyqJTOJ7xcaajyWcbhNpuG36uA/oQnk5mdZmYn5RDqZGC4mTUO99fZzJoSlPFRYdtmW4LsMZNP\ngb7hftoAHYCP63xHBctzGX8K7GZmTcPzpC+wIHxvRZ/H+eyneQ7BL/MKsDC2/mRg97BRfj7wxzDY\nuto0a7M9QfpcXxcDzSzozjCP4E4iwI1ASzNbAPyF4HKRMM7a2kL6EmQf7wEbA1fkEFelyGcZPx5u\n+zhwknPum3B9V+CrHGK8DXgfmGVmc4FbCK6oHiU4kecT3GB81b+hjjbNiwjaXmcDzxDcdV6eQ2yV\nIC9l7JybBjxBcC7NAX4G7gxfrujzuGIeoww/DZ52zv2+1LFI4ZjZRGCAc+7nUsci+dcQzuOKqTRF\nRMqBHqMUEUlAlaaISAKqNEVEElClKSKSgCpNEZEEVGmKiCSgSlNEJIH/D4iDu1++8jz6AAAAAElF\nTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f0240724c18>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_example_errors(images, cls_true, img_shape, cls_pred, data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualization of Layer Weights and Outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Helper-function for plotting convolutional weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def plot_conv_weights(weights, input_channel=0):\n",
    "    # Get the lowest and highest values for the weights.\n",
    "    # This is used to correct the colour intensity across\n",
    "    # the images so they can be compared with each other.\n",
    "    w_min = np.min(weights)\n",
    "    w_max = np.max(weights)\n",
    "\n",
    "    # Number of filters used in the conv. layer.\n",
    "    num_filters = weights.shape[3]\n",
    "\n",
    "    # Number of grids to plot.\n",
    "    # Rounded-up, square-root of the number of filters.\n",
    "    num_grids = math.ceil(math.sqrt(num_filters))\n",
    "    \n",
    "    # Create figure with a grid of sub-plots.\n",
    "    fig, axes = plt.subplots(num_grids, num_grids)\n",
    "\n",
    "    # Plot all the filter-weights.\n",
    "    for i, ax in enumerate(axes.flat):\n",
    "        # Only plot the valid filter-weights.\n",
    "        if i<num_filters:\n",
    "            # Get the weights for the i'th filter of the input channel.\n",
    "            # See new_conv_layer() for details on the format\n",
    "            # of this 4-dim tensor.\n",
    "            img = weights[:, :, input_channel, i]\n",
    "\n",
    "            # Plot image.\n",
    "            ax.imshow(img, vmin=w_min, vmax=w_max,\n",
    "                      interpolation='nearest', cmap='seismic')\n",
    "        \n",
    "        # Remove ticks from the plot.\n",
    "        ax.set_xticks([])\n",
    "        ax.set_yticks([])\n",
    "    \n",
    "    # Ensure the plot is shown correctly with multiple plots\n",
    "    # in a single Notebook cell.\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get Layers\n",
    "\n",
    "Keras has a simple way of listing the layers in the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_1 (InputLayer)         (None, 784)               0         \n",
      "_________________________________________________________________\n",
      "reshape_1 (Reshape)          (None, 28, 28, 1)         0         \n",
      "_________________________________________________________________\n",
      "layer_conv1 (Conv2D)         (None, 28, 28, 16)        416       \n",
      "_________________________________________________________________\n",
      "max_pooling2d_1 (MaxPooling2 (None, 14, 14, 16)        0         \n",
      "_________________________________________________________________\n",
      "layer_conv2 (Conv2D)         (None, 14, 14, 36)        14436     \n",
      "_________________________________________________________________\n",
      "max_pooling2d_2 (MaxPooling2 (None, 7, 7, 36)          0         \n",
      "_________________________________________________________________\n",
      "flatten_1 (Flatten)          (None, 1764)              0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 128)               225920    \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 10)                1290      \n",
      "=================================================================\n",
      "Total params: 242,062\n",
      "Trainable params: 242,062\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We count the indices to get the layers we want.\n",
    "\n",
    "The input-layer has index 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "layer_input = model.layers[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first convolutional layer has index 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras._impl.keras.layers.convolutional.Conv2D at 0x7f024cd56278>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "layer_conv1 = model.layers[2]\n",
    "layer_conv1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The second convolutional layer has index 4."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "layer_conv2 = model.layers[4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convolutional Weights\n",
    "\n",
    "Now that we have the layers we can easily get their weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "weights_conv1 = layer_conv1.get_weights()[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This gives us a 4-rank tensor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5, 5, 1, 16)"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weights_conv1.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the weights using the helper-function from above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUoAAADuCAYAAABf005JAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEIlJREFUeJzt3ftz1fWdx/H3N4QQQrAhhCQ0QM6CAy1SjCXL0hTxUot4\nxWGply5lhh1FurVYF2eLyrRoWdJpKcilxdlBudWFrQwgZRXFBRrEUuZgKwQE5BLCxZAcQ8g9EPLd\nv8B5vc8P6fc4Ph8/v+bzDt9z8uKbmc/3+wnCMDQAwOdLi/oHAIBUR1ECgEBRAoBAUQKAQFECgEBR\nAoBAUQKAQFECgEBRAoCQnkw474Ybwlh+vsydvZIjM4mEb+aIETqTff6YzFR1dFji2rXAN/XvLy8v\nL4wVF+tg4PgnXLrkmtl1/rzMXL1pjMxcuFBlly8nUvfa9u8fxgYP1sHKShm58g8lrpm9Tx6UmYzM\nTNdaB9vbE2EYDnCFI5DXr18YKyqSueYjR2Qm+ytfcc28cuWKXsuxTrWZfRaG8rubVFHG8vMtvnix\nzM3c/qDMrFrlm/naazpTNudbMlN6+LBvYERixcUW//OfZa4rPUNm0l7Wn5GZWfOcOTJzcXNcZqZM\nKXXNi0ps8GCL79qlg8OGychbS/X1MDMbeZ/+fyMWi7nWCo4dO+sKRiRWVGTxzZtl7gPHXU/ZhAmu\nmdv/+EeZ8ax0m2saf3oDgERRAoBAUQKAQFECgEBRAoBAUQKAQFECgJDUPsqDp/pYMPkfZS5saZWZ\n5uYs18yyr9Xr0Kuv6szDD7vmRaW5JbCK/XqP5IQdz+vFfvIT18zstjaZGd5+SGYyQ71OlLrS0q01\nM1fmshz7GnMd+yPNzGKODdEfODdXp7y6OrMVK2RMb+c3K3XsjzQzu/+552TmtfJymbnsmsYdJQBI\nFCUACBQlAAgUJQAIFCUACBQlAAgUJQAIFCUACEltOB8ypKf99KcDZS7oc1xmZs1yvLrczOr795eZ\nBsc6Ha5p0Tl92mzaNJ07cE5voq12bLQ1Mxv72Wc65Nm8XlPjmheVtKOVllUyXAfPnZORcSW+N5zb\nxo0yUnbHHb61du/25SLSUVtrJ5cvl7mZjz0mM9UbNrhmxh3f8Wkdocz89lu+l05zRwkAAkUJAAJF\nCQACRQkAAkUJAAJFCQACRQkAAkUJAAJFCQBCUk/mNDWZ7dmjc/v26aduyr7te6W+ZWbKyLH2dpm5\n5psWmZ49zQoLda6wfYDMVNXV+YbOmycjNevXy0yqX9sTGaNsYiwuczs/OSozDwwe6Zq5LfauDv3+\n9661rKjIl4tI18gx1rpBX1/b8SsZuX3/f7tmNjgex7vcq9Gx0nXXPO4oAUCgKAFAoCgBQKAoAUCg\nKAFAoCgBQKAoAUCgKAFASGrDeUuLWdyxr7Tslek6dN230XNPjx4yU+VYJ9WPgvj6DRfswF3Py9x7\n5Xoz+V3PPeea2Th3ocykrVwpM85HByKTn282a5bO9eihN5Nvm/uBb2h6joxcTfGN5F69rzXa6Bq9\nwX7oK/8hM2fOPOCcqs9NWbr0EZlZtEj3ixl3lAAgUZQAIFCUACBQlAAgUJQAIFCUACBQlAAgUJQA\nIFCUACAEYRj6w0FQZ2Znu+/H6VbFYRjqcxQiwrXtPl/wa2vG9e1OrmubVFECwJcRf3oDgEBRAoBA\nUQKAQFECgEBRAoBAUQKAQFECgEBRAoCQ1Jk5ebm5YcxzzsfHH8tI840lrpnZF0/ITG3OcJmpr6+y\n5uZEyh7vkpedHcZyc3Uw0P+EM82+hzgGDdKZlkMHZabWzBrDMHWvbU5OGBs4UAevXtUZx/U3Mzt6\nUZ+ZMzJdf7fNzA42NSVS+cmcvMzMMNanjw5evqwzBQW+oWmOe7xEQkaqOjstcf26/FCTKspYUZHF\n33xTB8eMkZGK/3KcUmZmE+bfKTPLHtolM4sWlbrmRSWWm2vxZ5/VwcxMGfn+npmumUuW6My+Ql0M\n+sioaMUGDrT42rU6WFWlM+m+X5nR86fITLxwomutYOfOlH48MNanj8Xvu08HN27UmRkzfEMdvwe2\nerWMlF686BrHn94AIFCUACBQlAAgUJQAIFCUACBQlAAgUJQAICS1j7L2Si9btn2ozE09ojeWTvjl\nbNfMxO7dMjN7lF5rXcc517zI5OaaPfqozt19t4xsOuLbR/nXv+rMx6++KjMLX3rJNS8yDQ1mW7fK\n2MMnF8rMG294tzR+qiNPTvYttXOnc2Y0Wuvr7W/r18uc3v5tNnyW/gzMzIYc/l+ZmXn+ZzJzdotv\nfzV3lAAgUJQAIFCUACBQlAAgUJQAIFCUACBQlAAgUJQAICS14Tz/yic2e8e9OrhHv1Tz3S1bXDMn\n5ug3Rb+/fLnMNLumRaf5o4+swvF25wkrVshMzou+mU88oTNHx/2rzLRn/843MCqXL5tt2iRjv9yh\nNzv/YX6La+TsV4plZvpffuRay+wpZy4aGWbmeFm+lfz4xzJTVex7g/z3HwtlZscOvU5jo2scd5QA\noFCUACBQlAAgUJQAIFCUACBQlAAgUJQAIFCUACBQlAAgJPVkjrW2msXjMlZdVyczoy7onfVmZvb+\nH2Rk/JNPykx2U5NvXkTazeykIzfhhz+UmfsP+mZWVenMkJv0kxIZvnHRSUszy9RPi2UN0//WD50j\n5zky+Y6nzszM9CEL0er42hg7tUb3Qt57/ykz1c6Zazboz2ruM7pjXn/dN487SgAQKEoAEChKABAo\nSgAQKEoAEChKABAoSgAQKEoAEJLacH6i9802sURvLN26Va81aZJvZsVD53Xonnt05p13fAMjkm5m\nuY5c9Xn9f9tJz851M6tYdUKH1utN0Wkpvpm/o63NTh4+LHNHHWs9uHatb+hT+viGzoYG31oprrbW\nbOlSnVu2QW/DL3POrHRkFj+uP9GKd9td87ijBACBogQAgaIEAIGiBACBogQAgaIEAIGiBACBogQA\ngaIEACEIQ+eRDGYWBEGdmZ3tvh+nWxWHYTgg6h/i83Btu88X/NqacX27k+vaJlWUAPBlxJ/eACBQ\nlAAgUJQAIFCUACBQlAAgUJQAIFCUACAkdRREXl5eGIvFdPCo46X6w4b5hlZXy8jBRs9a1RaGicA3\n9O8vCPLCIIjJ3ADHtuPB1077hnZ2ykiH45iHC2ZWH4Ype21zgiD8qiOXlZ0tM1eam10z+94yRmY+\n/dS1lNXUHEyk8obzvN69w1jfvjJ3vG2IzLS0+GYWFOhMTY3nmIcLFoaX5Xc3qaKMxWIWP3BAB0tK\ndMZzsI6Z6+yR4O03HAvd6psXkSCIWa9e+jyiRx7Ray2redg3NJGQkdO7d8vMZN+0yHzVzNY5cqW3\n3CIzb+3d65p5+/v6s1ywwLWUlZcHKf3US6xvX4tPnSpzEyp/JzP79/tmzpihM+Xlxx0r/bNrHn96\nA4BAUQKAQFECgEBRAoBAUQKAQFECgEBRAoBAUQKAkNSG8+Zmsw/2624tW7RIL7ZmjWvmpdVvycwD\nT+h1KipS+/+Eb9qHFr+eIXOJ5df0YkuXumb+7emnZabKsU6ba1p0soYNs9LFi2WucrLeOn/v977n\nG/reNhlZeKPe8G9mVu6bGJ3CQrO5c2Us83G91He+4xtZXn5SZvr1GyEzjY2Zrnmp3R4AkAIoSgAQ\nKEoAEChKABAoSgAQKEoAEChKABAoSgAQktpwnt3ZYGUJvZHWHMdFdP3iF66ZBfPny8y2jfqV76Xj\nu1zzonLtG9+0S+/qt2IXPPGgzOx3bCQ3MxvqyJT85jcys2DJEte8qLSfOmUnHJvJszyLeY5CMbPF\nJ/XnNGdOSr+43K3z8GFLFBfL3CTHWhOdM3/kyNw/VT+JUrrF9xlwRwkAAkUJAAJFCQACRQkAAkUJ\nAAJFCQACRQkAAkUJAAJFCQBCUk/mXGjJsef36ycOHp0cyMzo48ddM7f26CEzD916q17ok09c86LS\n0WF26pTOFTz1lMx0zXU8PWVmiRydyc88rUOrV7vmRaXdzCoduSme79G0aa6Z/1KgM1u36qdZzMz2\n7nXFItNlZs2O3BBHxltI97/4og55vpeNja553FECgEBRAoBAUQKAQFECgEBRAoBAUQKAQFECgEBR\nAoCQ1IbzopYTtjDueFn7uHE6s2aNa+a9nlBVlc50dLjmRSW7R5uVZR/SwbkLZGTqSd8L9R2nbNjI\nSY6vSKAfMIhSTnq6TenXT+YqHTu7R919t2vmM3d8KjN7937mWivVtQ4bY4cW62NMpsYc3+/x431D\nV63SmaYmnbl+3TWOO0oAEChKABAoSgAQKEoAEChKABAoSgAQKEoAEChKABAoSgAQgjAM/eEgqDOz\ns93343Sr4jAMB0T9Q3werm33+YJfWzOub3dyXdukihIAvoz40xsABIoSAASKEgAEihIABIoSAASK\nEgAEihIAhKSOgsjOzgtzc2MyV1en1xo61DczkdCZwVn6lfpViYQlmppS9syCvCAIY57goEE607On\na+bVM2dkJmPgQJmpamiwRGtryl7bIMgNzYpkrrCwl8xkZPhmDmg/JzNttbWutY6aJVJ5w3lOEIT6\nW2LW5+abZab1o49cMzsdmUxH5ryZ1Yeh/O4mVZS5uTF79ll9NsbKlXqtDRt8Mz1H6ywuWSczpT//\nuW9gRGJmpq+smT3zjM4UFLhmVk+bJjNDHn9cZko955dEqsjMtsjUjBk3yozn/ykzs387NltmKpcv\nd631jRR/6mWgma115Mbu2iUz8f79XTPrHZnhjsyDrmn86Q0AEkUJAAJFCQACRQkAAkUJAAJFCQAC\nRQkAQlL7KPOt1mbbMpmb/fp4vVhzu2vmwiXflpnTjnU6XNMiNHq02dtvy9jRIr1xuss5clRhocy8\nFntJZhIZbzknRiXNzPrI1PbteqWXX3aO3HpMRhqdS6W6PunpNrZfP5mrqMyVmQnvvOMb6vkgHL9P\nzucHuKMEAIWiBACBogQAgaIEAIGiBACBogQAgaIEAIGiBAAhqQ3nnefOWe3TT8tc/ne/65jsG535\npz/JzFDHTuFe6/TLfaMUHjpknY7N5DX/F8rMndkHXDP/Eo6VmZJx+sXlWa5pUeoyM/2Aw+HD+2Tm\nzry+rok7du6Umf2ulVJfQ/HNtvlX+rXTU6xCZtbVTHTNnH5XpSsn7dOfuRl3lAAgUZQAIFCUACBQ\nlAAgUJQAIFCUACBQlAAgUJQAIFCUACAk9WROekGB5U+f7gg6ll250jXzwG23yczYN9/UC23e7JoX\nlSvDxtg2x9MNmZ4TNBbMdc38p5wcHerZU0ayOjtd86ITmlmbI1cgE/WDbnRNnPSDH8jMXevXu9Z6\n0ZWKTs6FIzblha/roOOYh+l5R10zhz/07zKzaZPOtD1W6prHHSUACBQlAAgUJQAIFCUACBQlAAgU\nJQAIFCUACBQlAAhJbThvv3TJjv361zI3yLFW9oIFrpkZ8+bJTP3kyTKT6lui29rMjh3TuRde8GzI\n9W3mD1sGy8zsufqgh3P/49u0G5UxQ5os/sL7MnfPlpkys2KFb+bPxo+XmfQdO3yL1dX5chH5sOMm\n612lH5YYeLtea8QI30zP70rayt/KTO+mWtc87igBQKAoAUCgKAFAoCgBQKAoAUCgKAFAoCgBQKAo\nAUCgKAFACMIw9IeDoM7Mznbfj9OtisMwHBD1D/F5uLbd5wt+bc24vt3JdW2TKkoA+DLiT28AEChK\nABAoSgAQKEoAEChKABAoSgAQKEoAEChKABAoSgAQ/h8IrdSrQ5nAhAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f024ce1dd30>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_conv_weights(weights=weights_conv1, input_channel=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also get the weights for the second convolutional layer and plot them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "weights_conv2 = layer_conv2.get_weights()[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVMAAADuCAYAAACEaORrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl0VWWaLvDnkITMISQnkBCGo0YMg4oaS8ShcCjExoGi\nugQtlmV7aUuUVuRagkApTVE4NNeBppClLColFhUVwavggBYLUatAAwKKFZnqgEwhB8hIEhLY9w8M\nq5fv++6T0345vbw+vz+fk5f97Zzk5WTvb39fwPM8EBHRd9Ppf3oARET/P2AzJSJygM2UiMgBNlMi\nIgfYTImIHGAzJSJygM2UiMgBNlMiIgfYTImIHEiM5YuDKSleKD1dvpCTY9bsOtxFzc/sFDZrDqWF\nRHbkSBj19ZFAtDF+V9nZQa+gQB4/IcGuSdm3U3+hk/1/VVXGGSI7fDg+5xjs2tULFRbKF7Zvt4u6\n6O9jS0FvsySpqU7NN2zbFvE8L893kA6kpwe97OyQyAtSq+2iw4fVeF/KWWZJa6vM6urCaGqKw3uZ\nnOyF0tJEvjvRHq/xViIlxT5Oyu6vRBZuakKkpaXDzxHw+Zn1+cU8Wp+k5n4PfSrfSuzbF8bRo9Hf\ny5iaaSg9HeUjRsgXRo82a275o/L1AF7JuNOsmTtokcjmzCmJPkAHCgpCKC0tF3lmpl3Tf/oo/YWM\nDLNm/uAXRfbEE/E5x1BhIcqXLZMvXHedXXT99Wpc+eh8s6T71tVqHrjmmt2+A3QkOzuEe++V7+XU\nYuXc27z0khr71UQiMlu+PE7vZVoayq+6SuR3Be3x3nCDnhcX28fpO+5KkZV89lnU8bli/sz6/I69\n/GEPNT9xwj7OoEEyu+WW9r2X/DOfiMgBNlMiIgfYTImIHIjpmil69wbmzZP5woVmySsTjIuNgxeY\nNffdMExkL1Zvizo8FyoqjmLw4KXKK2f6VP0fNX32WXmTqc3778ispsZ/bK40nEjB+qN9RX6Jz4Xh\nVSP1a6P3D7WPU1BwdaxDc6qxEdi0SXlhaL5Zs3aifq1xdtMq+0ADB4po4/qWaMNzo7AQmDVLxNUz\n7JIZxmsbr33ILqpTbib6XXx0bOvOFPS7Wf7M+g1hW+lf1XzqiiFmzeLFMvv666jDA8BPpkRETrCZ\nEhE5wGZKROQAmykRkQNspkREDrCZEhE5ENPUqKojCXi+LEvklw+fZNb0Lzqu5oFkfdoCABQUyGko\nkab4PJ5XVNQVzz77zyLPt2fT4MIyY0rJXrvmvrIZIiu5/GSU0bmR3lKNS/YpU4DUeUSnRMr0fKex\nLAEAXHBBjANzLDkZKCpSXvjoI7Pmb54+bebKn2v/0CmVCfKxxRboz4V3iET5a/zKzX8yv7zHr3+h\n5k/kPmnWTH5ZmZo4yniMugMMOKcV5asOyRe0Z3m/MfUl/b2cPf2YWRNIT455bG34yZSIyAE2UyIi\nB9hMiYgcYDMlInKAzZSIyIGY7ubv2dOKX/1KrkT+k5/kmjXvvbdSzTdv/qlZ0/d8uaj1Ze0Ynwtd\nOjfin3puEfnx4vPMmuOz9LugPuu/IF9Z6KS6Nk7/t6WlASVydsTsx+3ja+vbAMCcOfZh7mvSvy9/\n/rPv6Jwp7HQAs1NminzL8EfMmskZu9S873B7oRttUeXdcVn+GsDJk0B9vcyT7NkE/ywnq0Qr0Wd6\nNDb6j82hzVsT0W1gN5FPmSKzNo89pvee1lZ9wXoAKFNmrUydGn18AD+ZEhE5wWZKROQAmykRkQNs\npkREDrCZEhE5wGZKRORATFOjLurbiPLnNou8coC910/3gD6pqdKzj5NyxRUiC8Rpj+7mTqnYlSGn\nQU2/w65RtgACYO9PDgDnn6/NndEXhXFtw+edEOiTLvIHHrBrrHN57jm75vUCa0+hyXaRS+npwODB\nIj6vzJ7rct6K2Wre2mof5s03tRVt4vNeHk9Mw57ghSLvXVpq1gwceIua+2xBr0+/iuMeULm5wO23\ny3wSnjJrrt2sL8CkTWVrU1Ehs5SUaKM7hZ9MiYgcYDMlInKAzZSIyAE2UyIiB9hMiYgcCHiez231\nb39xIFAFIF5LOHxbH8/z8jr6IDzHuPghnCfP0aHvw3nG1EyJiEjHP/OJiBxgMyUicoDNlIjIATZT\nIiIHYno2PyUl6KWnh2I6wBndj8X09QBQdyJNZAcPhlFTE5H7mTgW7NLFC+Xni3xvTaZdE4z9OC0t\nMovXOQYCQQ/oLfIBA+z/W63nkzdssJ9BT0zsrOatrRsi8bgLHExJ8ULpcg0C1NSYNQ1nD9LzBvs4\n2jPt+/eHUV0dh5/X5GQvpA1AO+9v1CTrW32kptrH6dxUK7JwZSUiNTUdfo4AEExK8kLJySL/8qT9\noH3//sYLfmsK1CrnWVWFSG1t1POMqZmmp4cwYkS5yP0WgVjy4MZYDgEAWF0tF24YP17uWdQRQvn5\nKFdW73joHXsxl3HjYj/OXmVtjHid46lG+qFIX3nF/gW0FodISNAW+Tila9eeal5VFYjLFJdQejrK\nRyj7/bz5plmzvlT+fAPA+vX2cS6/XGZjx8bp5zUjA+XXXSdfUBZ4afNW0X1qbi3YAwC9K1aJrGTC\nhKjjcyWUnIzyc88V+XkNfzNryj85qb+gLdrS5h25OVvJww9HHR/AP/OJiJxgMyUicoDNlIjIgZiu\nmeblAXffLfMhJfZNiP0Ref0T8L9scc1Fh5XU58KsQxu2pSFwjXa9y75onZ2doObTpjWZNcOHyzs6\nBw9GHZ4jTQC+EumAAX4LGsuvB4AHHvilWTFxop736eNzGJdycoDRo0W8aOiLZsma3+v5yJH2Yb74\nQmZx21K+a1dgzBgR31Z2k1ny5/uVu58AHn44yayZNWuYDLOyoo/PlXPOAT74QMQjZ9kly17XPyuO\nOrDYLnr3XZkd1vqRxE+mREQOsJkSETnAZkpE5ACbKRGRA2ymREQOsJkSETkQ09SojE7HMCRFeTz0\noP1w+vtr5DPgAKA9AdemuTlXZJdeGtNQ/9vOOisBTz4pp3yMGmk8mgYAFV+q8VSf6TQbm+SDw2PH\nRh2eE+edl4a335ZT1rTHItu89JL+eOKQbP3cAQALXop1aE79fX8X/Ojf5eOkTfaMNWxp6afmRxYr\nG6p/I0d51nRuis/D/A5VtWTj+YNyGpTf1EPLY4/ZjwY/9tgOJa2L/SD/XVVVwIIFIv70U/3RWAC4\n9lrjhWz9PQYAfPyxzBL0qY/fxk+mREQOsJkSETnAZkpE5ACbKRGRA2ymREQOxHSLvKohDfPXybvA\nH31k11hr1IZCds348TLTFlPuCNmBGoxKXilfmKBk39j14Hw1P/P9582aC4Py7nDa8eroA3Qgqbke\nPcJ/FfmuNfpizgBw5wx9VsaQBUVmzZdjZ+svPPaY/wAdSUoClE0TfO/mTxr4dzV/6oq77KJq5X3z\nW83doaoqYN48mT/+uF0zYYK+oMmwpTP9ikRUcmt8ZtgAwJaD3dB7jrxz/+qrds0lV+g7PZgrnQM4\nXr5FZN6l7Vvom59MiYgcYDMlInKAzZSIyAE2UyIiB9hMiYgcYDMlInIgprkNlZXAM8/IXNsXqs0O\nbX0EAI3vrjVrRj1zpciO+21P5FBLWhfsv0AujjHnL8r+6994+qwDan7VVfZ0mtWPfyLDJHsPHqfS\n0oBBg2QeDpsl5fp28thSYUw/AXDeQnsRinjIztb3bvKZGaPNAAIAbFzoM83to7ky9Jt/5VBSElBY\nKPPSUrtm0asBNffbZa32hRdi+nrXEhOBoLKe0tln+xRFInrus6FXcvLnStq+Db34yZSIyAE2UyIi\nB9hMiYgcYDMlInKAzZSIyIGA53nt/+JAoArA7o4bjq8+nufldfRBeI5x8UM4T56jQ9+H84ypmRIR\nkY5/5hMROcBmSkTkAJspEZEDbKZERA7E9Gx+585BLzU1JPKcHLvm5Ek9z821azrVym0gwocOIVJb\nqz9U7FB2dtDLzw+JPCPQYBclJ6txQ7P97U2v2S+ycHU1IseOdfg5BnNyvFBPuUXJV/+wn7M/5xzj\nBev5ZwB7jikPUwOoqtoQicdd4GBurhfq1UvkG7fY78uZZ+p5dqDGPlB9vYjCNTXxeS+zsrxQt24i\nb0nPNmuSAvpT9TUN9vdFu0996FAYtbWRDj9HwH4v0cnn8+CmTWrccq7ceun0ay0y278/jOrq6OcZ\nUzNNTQ1hyBC54sWtt9o11noPY8faNWnvLBNZyUMPRRueE/n5ISxaJM9xSKKyMEkbY0Or9f+QP+Rt\nLln5iMhKFi6MOj4XQj17onzFCpFfOVbf5wkA1qzR806li8ya8Z/eqeYLFgTiMsUl1KsXylevFnnn\nfPt//6ee0vObEuw9wPDxxyIq+cMfoo7PhVC3bihXBl15yU1mTffAITV/q9z+eW1V+u+kSe3bG8mF\nUK9eKF+1Sr6QkWEXGZ/YKlcZq/YA2LdPZmPHcg8oIqK4YTMlInKAzZSIyIGYrpmenVeNt/9VXs/E\n4MFmTWVCDzVPu6iffSBtz/G9e6MNz4kM1GNIq7Jwdc+QWfNlRL/WtNLnMhtGyD3KG159K8ro3GgJ\ndEZlsrw+qtyTOq2iQs/7Kzdf2jz3rxvVfMEC3+E5s3lrIroVy+ujS5faNUNuNu4zvPuuXaStgN7c\nHGV0brSkZ2N/ibw+2mP9G2bN9c/p11NvvNE+zj31T4psZtPB6AN0pLo+Ecs+kr9nfu/lkosvVvPu\nLyuLebe9piwcnZbYvpXp+cmUiMgBNlMiIgfYTImIHGAzJSJygM2UiMgBNlMiIgdimhq1/1g2Htk0\nSuQzs+Uje226ZxhTmn7/e/tA2nSbSZOiDc+Nkyf1Z2A/1/bTPqXoJ/pjmA8+aB/mhhtkFqfZX0iq\nrkL3/yv3gS8tvcus6fy4nMoFAG+VyMdi2yycFfvYXEpKAvLzZX7TrwrMmmNGPn/HMPtAQ+VrVevj\n86hlQwNQrjwdWV9vP0769mvGWVrPfgNA6x0yW7LEf3AOHTumP2q/ZMY2s2bSAmWKI4CnmuQ0r9OC\nynoSie1rk/xkSkTkAJspEZEDbKZERA6wmRIROcBmSkTkQEx38+vrgXXrZL5s0NVmzaiSWuPI9qED\n6ZVKmhpldG7sq8vC1DXy7uzsInsRZOuuvbJmwmlrJ2oLYMsdBjpETg4wZoyIO88zVkYGgEGDYj7M\n8uWHY65xqV8/4G9/U14YeYFZE56jLzZz74C/mDW/+901ItMWU+4I2Ql1uClDzqb5a9D+nUS18XPm\nt2qIttByrfG73QEyM4GhQ2U+akpfs6Z7dz2fm+Kz0LyyPvuhSPs+c/KTKRGRA2ymREQOsJkSETnA\nZkpE5ACbKRGRA2ymREQOxDQ1qq7Ow3vvtYh8Vd2VdtH/1veJCYRfMUtuvVXu3fLuu8nRB+hAYyPw\nxRcyH1Wh7wEP2LO8iot9DlStvJiS4j84V06c0KfHTJhglqxa01nN/X6AHnhA37f86af9BudOZaV+\nrIIx9l5b9WusVw6YNVOmyGyZslVahwgE1B/AIfNus2uMfbueGmrvGxXZKrN9jXF6IwFkpp/E1YPl\nAi2hUJpZYy0cNMtnAR5tdtgf/hBtdKfwkykRkQNspkREDrCZEhE5wGZKROQAmykRkQMBz/Pa/8WB\nQBWA3R03HF99PM/L6+iD8Bzj4odwnjxHh74P5xlTMyUiIh3/zCcicoDNlIjIATZTIiIH2EyJiBxg\nMyUiciCmhU4Cga4eUCjy4mJ7gY70xoj+wvHj9oEKCkQU3r0bkUgkEHWQ31FGRtDLzQ2JPC+zyS4K\nGMM6JhdmOK1S7nMVbm5GpKWlw88xmJnphfKUmR4H9UVpAJxaHEX9x4JmydEU+T4CwK5dGyLxmFKT\nlhb0unQJidxv66KsLD0vzGk0azZ8qS0Csweed7jD38tAIN0DskWemKh/7wHg/AHGBlVffWXWtPQd\nILK9e8M4fLjjfycBIDU16GVlhUTeK9FegOZIsv498Nl+Dtu3n1TSPfC86OcZUzM91Ujlak+lpf3N\niku2GhvRhcP2YaZPF1HJpZdGGZsbubkhTJtWLvK7hm6zi6x3p1z+O6cpyxmVfP55tOE5EcrLQ/lv\nfytfmDPHLrI2YRs3zix5uWiamo8ZE4jLfMEuXUL4l3+R78H779s1116r57PHbDFrAuf3UlKfDe2c\nygZwj0i7dtW/9wBQvvqI/sJll5k1lavk93HYsJKoo3MlKyuE0aPlGOZ2/51Zs+QM/Xvg8/8/rruu\nQUmviDY8APwzn4jICTZTIiIH2EyJiByI8ZrpSQA+N2I0paV6XuJzvUW7Bmnd5HGsqQmoqJD56qK+\nZs2nn+r55Bvt71Vg3UQlnRpldI7U1QEffCDzl1+2a4zrqdt+bl+bG12hr9w+xndw7mRmAkOHynz2\nZSvtIuNG29Sym8yS5maZXXppQpTRuZGaWoDiYvkefPaZfVk6kPufxiu32gfK136W4/coen09sG6d\nzHeV2T9/7xsr6i+asNGs2bDhQpGNHdu+z5z8ZEpE5ACbKRGRA2ymREQOsJkSETnAZkpE5ACbKRGR\nAzFNjerRIw333iunDlySZE81UOczALg9tNYsWZywRknroozOjaoq4LnnZD5Rm8n0jb179Tww4Byz\nxvujfDSu5FHtuWD3DqX2wdyBz4v8kqN2zSXnn6/mPXv6HOgln8dp4yA1FRg4UHlhedis+fKqe9V8\n9oJ/MmuGDH1LZD6PuTuVl6c/0XvPJvsxSyxYoOetxjP7AMbfL9cfeO21+ExXBIB+/YBP1snfj+Ot\n9ufBRcPlo+8AcPWDt5g1q1fI9TTSUtr3e8lPpkREDrCZEhE5wGZKROQAmykRkQNspkREDsR0N78A\nBzDVU+4SfqEtjvsNY/X2xbn2IiDnnjtUZDt2ZEYbnhMXZm5H+WXyzm2gz5tmTXGxvqiF9xdlMZE2\nYx6U2VGf2+kOff31Cdx/v3Yse9Vyb+tVav766/ZxzhoxU39BW5i6AyRt/xI9Rlwg8jsv+MysqTfe\nsld27DBrypTFpm+4IerwnMjr2op7xiiLPUd8plk8qPzsATg55ymz5N/+TWYffhhtdA5t2gTk5op4\n6e/t35lwWL9rn5xsH2ZRWZrIIke40AkRUdywmRIROcBmSkTkAJspEZEDbKZERA6wmRIRORDT1KiK\nmgIMWSn3XFm+3K7pDn2Pbq/BXjygUlnTZNiwqMNz4+yzgRUrRPzsPPv/nftC+l5HmFdqH+fRR2X2\nxBNRBudKEwC5Esdrrw22S1Y8qca3NS01SyaVPxLrwJyq7tEfy34jF1s58IJds3On8cLIkWbNhAky\n27MnyuAc2bw1Ed2Kc0ReVmZ/77Oz9XxMsX0cbS+tysoog3Mo0nMQFj0i38vsFLtm2jR9BaKHH7an\njYXDMjt+PNroTuEnUyIiB9hMiYgcYDMlInKAzZSIyAE2UyIiBwKe57X/iwOBKgC7O244vvp4npfX\n0QfhOcbFD+E8eY4OfR/OM6ZmSkREOv6ZT0TkAJspEZEDbKZERA6wmRIRORDTs/m5uUGvV6+Q/Efq\nq82awyf0B4FzW30e7E1PF1H44EFEqqsDUQf5HXXtGvQKC0Mi93sOuUcPPd+rPxoMAOjWTWYHDoRR\nXR3p8HMMdunihbQBNDaaNTsb8tX8rB52zf6jqWp+4MCGSDzuAmdmBr3c3JDIE/RdZgAAXZv1bXY2\n7tfPHwD695fZvn1hHD0ah/cyI8ML5chn8+vTlPf3Gxn7t+kvWA/tA6g4Iv+95uYwWlo6/hwBICsr\n6OXlhUTeNa3ZrGlo1fcnqa+3j9M9ISKy8OHDiNTVRT3PmJppr14hrF4tFxvIWbPMrHmxfpSa3x6x\n95tBSYmM7ror+gAdKCwMYdkyeY5z5tg11pZGv/61XXPvvTK74w553h0h1K0byp95Rr6wdatZM2rd\nQ2q+bMYWs+aRpeep+W9/G4jLFJfc3BAeURbHUP6vPm30bn1Bl87T9fMHgGXKj/+oUXF6L3NyUD55\nssjXnqv8gH3jylnGqkE+G1cNKbtPZFu2xOccASAvL4TZs+V7OfriXWbN+qoz1fzjj+3jTMpeJLKS\nmcZeZt/CP/OJiBxgMyUicoDNlIjIgZiumdbXA2vWyPxnPzts1vzxj3r+o7JJZk1Tqcx27MnwH5wj\ne/boi/2umvWJWXPf736k5osX23utDx9eJLJm+1q6U63pXXDk0hEi39tLZm1GGvdf+o3Wr4sC9rUp\n6xqzaykpQJH8NuPHP7bfl4fP0K+NapeY/+txvq1TvD6mNDWp17oXrrdLLn9nlZpPmWLXBIMyS4yp\ne3w3Xb0jGN36J5FXpv/CrDlcoed+48558E6R1dbOjzo+gJ9MiYicYDMlInKAzZSIyAE2UyIiB9hM\niYgcYDMlInIgpskNO3fW4mc/06ZVjDZrrMcwtwy63T7Qe++JqKRFPjPbEfr2asSqOfIRycru+vQn\nABhsbDc/d8JJs6bHUJlF4nOKSDx+DDnhjSKfUXqhWTM3qD9SV/FTe3/2iRNjH5tL+/cDs2bJ3NtQ\naxf9rwvU+PnEz8yS3q3ykcbOXpzmuSUlAYWFIj74gV2iTeUC/KdGvTFdTg0s+WVDtNG506ULcOON\nIl6/JvZ/at06+7Ujr68VWcldPg/z/xf8ZEpE5ACbKRGRA2ymREQOsJkSETnAZkpE5EBMd/NTU7NQ\nVCQXlp03z645cMB44StlBYo2GcqiJkuX+g/OlUBAvd3Z/d/vMUt+8dx/qnnw3b5mzf6Sm0RWstZe\ngMOllqQ0VBbKO/d3323XvFiu37Vv/cKn5pkjar54se/wnOmbvBurQsqi4juutYsq9NUxyuW6xKeN\nG6csQpysr/LuWlViAZ7PmybyFSvsmjFj9PyOO+yavsPlbJY9e3xW2XbswKEEzHwmS+SPXPtXs2b+\npiFqvqTEZ2H6D5WdI+rqoo4P4CdTIiIn2EyJiBxgMyUicoDNlIjIATZTIiIH2EyJiByIaWpU//5A\n+Sdy8Y6+xXZP3r79hJqvf8BeIOPp57Q9pewpEE41NanTY47MsveBucBYOMF3jxxtPpnPvuUuNTTo\n+zO98IJdo+39BQC/+Y1dM78sJ6ZxOXfy5KmNy75N2Wf+tM2b1fgGYz8hABg5UmY74jPLDXmda3BX\n4Ur5Qpm9L9vbb+uLDPlNjdq+fa+SHvcfnEMFBcAj02XvObNIn/4EAF8Y0/b2V9v7z02fLrNw8/Ko\n4wP4yZSIyAk2UyIiB9hMiYgcYDMlInKAzZSIyIGA53nt/+JAoArA7o4bjq8+nufldfRBeI5x8UM4\nT56jQ9+H84ypmRIRkY5/5hMROcBmSkTkAJspEZEDbKZERA7E9Gx+WlrQy84Oibypya9Gz1ta7Jrs\nbJkdPBhGTU0k4D/C7y4zM+jl5YVEnnP8oFlTnZKv5jt36usSAED37gkiq6kJo7Gx488xOTnoZWSE\nRN6rl11TWannhZ72zPYprfk91Xzz5g2ReNwFDiYmeiFl+5CmM/qZNUeP6nm3bvZxEk7KH+bw118j\ncuRIh7+XGRlBLzc3JPKqKrsmQf7oAQAalR072pw4cUhJj8Dz6jv8HAEgMTHode4cEnn/1H/YRX36\nqHHVYfszZEODzOrrw2hqiv57GVMzzc4OYdw4uRmOsW0OAOCii/Tc+uUE9PU+xo8viTI6N/LyQpg1\nS57jbXufNGveKH5IzW++udasuf12uZ/Niy/G5xwzMkK47jp5jn57ec2Zo+ezW/VzB4AjU/TvWW5u\nIC5TXELJySgfMEDk2176xKyxthqbMME+Tlb9fpGVXH991PG5kJsbwuTJ8r1csMCu0T6sAPbCIABw\n9Ki2z9l/+A/Ooc6dQyguludZXnybXbRwoRrPLzU+4UHf6+uNN9r3e8k/84mIHGAzJSJygM2UiMiB\nmK6ZpqcDgwfL3O9a26xZer7cZ73Vq4v2iCwzOT4L0eZktuC2ofIaWG2GfW3wpgy5aC0ANDTI66Jt\n0nZsEdnqlT53ABzKy9OvAWrrKLfJyNDzJSH7WvJtX6yNcWRufZ3RD5Mul9dHW31+XocP1/OsKffY\nRaGQzA7bizO7lJgI5ObKXFuwus1/GJc6G/cdsYve7y6ikoeToozOnaIi4PXXZd7a589mTeLjj6t5\nRUVvs2bRHPk9KNnYGn2A4CdTIiIn2EyJiBxgMyUicoDNlIjIATZTIiIH2EyJiByIaWrUjh3VGDFC\nzmn6+c9/atb0/WiRmk/et8msWX9grsgaWjq3Y4Tf3cmEJBzL7iHyrHn2FCCkp6tx2kplP/M22jyP\nlJRow3OisRHYpHz7777brtm+Xc8vvtiuuW1pcWwDc+zQoWY8/bR8dts7oT+zDQBZ2frni2nT5ps1\nkx9NlWFzc/QBOtC1/muMXq/sA289/wtg5tgd+gsP6lOJAABnnSUzv0U5HOv0+Qak9JGPxxtnAgCY\nOUWfArXkGW2dgVO25copYO09S34yJSJygM2UiMgBNlMiIgfYTImIHGAzJSJyIKa7+UAmgGtE+uqr\n9qolvdfdqebGGgQAgNavZBavG4cnTgDV1TIfu85e6ES7MQ8AGRn3mjUzlMU2Kg/FZdFy5CVV4578\nZSIvLxll1gwcqOfaGh9tnn/dZ3n6ODjzzGTMnn2GyIcZi5kAQG35NjW//v6+Zk3leLlAzdd/is9C\n38jL06dhWKtcA+Y2F2vv0GfeAMCPf6wtdO5zDMcSAwF0S5TtqpvPyvTnv2q88M47Zk3fJLl4S0or\nFzohIoobNlMiIgfYTImIHGAzJSJygM2UiMgBNlMiIgdimhp1zjkJWLRI7ms0ZJ1cTOK04V/q+cGD\ndk2jnGoyN7Em2vCc+Pvf9X2u/KYAjRun58+/Y+81g8FlIlqS7rMJk0PbDmVj2AI5Dcpv+tmMGXru\ntzaL3z5E8XDiBNDQIPNVGfYUMPzygBq//cEHdk2ZfC/XLo/PHlDYtQsYPVrEM3/6mVmi7Q0PABMn\n+h3oUyU75js0pzIzgcsuk7kyXapNv37GC3UJZs0hZdpY+yZG8ZMpEZETbKZERA6wmRIROcBmSkTk\nAJspEZG7aoR0AAAAYUlEQVQDAc/z2v/FgUAVgN0dNxxffTzPy+vog/Ac4+KHcJ48R4e+D+cZUzMl\nIiId/8wnInKAzZSIyAE2UyIiB9hMiYgcYDMlInKAzZSIyAE2UyIiB9hMiYgcYDMlInLg/wG5h3Aq\nt90azAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f02402de048>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_conv_weights(weights=weights_conv2, input_channel=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Helper-function for plotting the output of a convolutional layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def plot_conv_output(values):\n",
    "    # Number of filters used in the conv. layer.\n",
    "    num_filters = values.shape[3]\n",
    "\n",
    "    # Number of grids to plot.\n",
    "    # Rounded-up, square-root of the number of filters.\n",
    "    num_grids = math.ceil(math.sqrt(num_filters))\n",
    "    \n",
    "    # Create figure with a grid of sub-plots.\n",
    "    fig, axes = plt.subplots(num_grids, num_grids)\n",
    "\n",
    "    # Plot the output images of all the filters.\n",
    "    for i, ax in enumerate(axes.flat):\n",
    "        # Only plot the images for valid filters.\n",
    "        if i<num_filters:\n",
    "            # Get the output image of using the i'th filter.\n",
    "            img = values[0, :, :, i]\n",
    "\n",
    "            # Plot image.\n",
    "            ax.imshow(img, interpolation='nearest', cmap='binary')\n",
    "        \n",
    "        # Remove ticks from the plot.\n",
    "        ax.set_xticks([])\n",
    "        ax.set_yticks([])\n",
    "    \n",
    "    # Ensure the plot is shown correctly with multiple plots\n",
    "    # in a single Notebook cell.\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Input Image\n",
    "\n",
    "Helper-function for plotting a single image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def plot_image(image):\n",
    "    plt.imshow(image.reshape(img_shape),\n",
    "               interpolation='nearest',\n",
    "               cmap='binary')\n",
    "\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot an image from the test-set which will be used as an example below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADV5JREFUeJzt3X+oXPWZx/HPUzeNYKrmNtMYbextc0UJwabLEFYra1dt\nuAmB6D+SICUFaQoqrlB0xaKr+E9YbYqgVG80NC6tbTGVBAmubqhooJaMJv6Ku+uvG5twzZ0YoSkI\nadJn/5iTcqv3fGecc2bO3DzvF1xm5jznzHlyyOeemfmeO19zdwGI5wtVNwCgGoQfCIrwA0ERfiAo\nwg8ERfiBoAg/EBThB4Ii/EBQ/9DPnc2bN8+Hh4f7uUsglPHxcR0+fNg6WbdQ+M1sVNIDkk6T9Ki7\nb0itPzw8rEajUWSXABLq9XrH63b9st/MTpP0kKQVkhZLWmtmi7t9PgD9VeQ9/zJJ77j7e+5+TNKv\nJK0upy0AvVYk/OdJ+uOUxweyZX/HzNabWcPMGs1ms8DuAJSp55/2u/uYu9fdvV6r1Xq9OwAdKhL+\ng5IWTnn81WwZgBmgSPh3S7rAzL5uZl+UtEbS9nLaAtBrXQ/1uftxM7tJ0n+pNdS32d3fLK0zAD1V\naJzf3XdI2lFSLwD6iMt7gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQ\nhB8IivADQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCIvxA\nUIQfCKrQLL1mNi7pqKQTko67e72MpgD0XqHwZ/7F3Q+X8DwA+oiX/UBQRcPvkp41s5fNbH0ZDQHo\nj6Iv+y9z94Nm9hVJz5nZ/7j7C1NXyH4prJek888/v+DuAJSl0Jnf3Q9mt5OSnpK0bJp1xty97u71\nWq1WZHcAStR1+M3sDDP70sn7kpZLeqOsxgD0VpGX/fMlPWVmJ5/nl+7+TCldAei5rsPv7u9J+maJ\nvQDoI4b6gKAIPxAU4QeCIvxAUIQfCIrwA0GV8Vd9ITz55JO5tU2bNiW3Pffcc5P1008/PVm/7rrr\nkvVzzjkntzYyMpLcFnFx5geCIvxAUIQfCIrwA0ERfiAowg8ERfiBoBjn79Ctt96aWxsfH+/pvh9+\n+OFk/cwzz8ytLV68uOx2ZoyFCxfm1m677bbktvX6qf8t9Jz5gaAIPxAU4QeCIvxAUIQfCIrwA0ER\nfiAoxvk79Oijj+bWXn311eS27cba9+3bl6zv2bMnWX/++edzay+99FJy23ZTqH3wwQfJehGzZs1K\n1ufNm5esT0xMJOupf3vqGgCJcX4ApzDCDwRF+IGgCD8QFOEHgiL8QFCEHwiq7Ti/mW2WtErSpLsv\nyZYNSfq1pGFJ45KudfePe9dm9a688squap0YHR0ttP3HH+cf+nbXCLQbz969e3dXPXVi9uzZyfqF\nF16YrF900UXJ+pEjR3JrixYtSm4bQSdn/p9L+vT/ztsl7XT3CyTtzB4DmEHaht/dX5D06V+hqyVt\nye5vkXR1yX0B6LFu3/PPd/eT11Z+KGl+Sf0A6JPCH/i5u0vyvLqZrTezhpk1ms1m0d0BKEm34T9k\nZgskKbudzFvR3cfcve7u9Vqt1uXuAJSt2/Bvl7Quu79O0rZy2gHQL23Db2ZPSPq9pAvN7ICZXS9p\ng6Tvmtnbkq7KHgOYQdqO87v72pxSscFtlGbu3Lm5tSuuuKLQcxe9hqGIrVu3Juup6xsk6eKLL86t\nrVmzpqueTiVc4QcERfiBoAg/EBThB4Ii/EBQhB8Iiq/uRmUmJ3MvDJUk3XDDDcl668ryfHfddVdu\nbWhoKLltBJz5gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiAoxvlRmYceeihZb3cdwNlnn52st/vq7+g4\n8wNBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUIzzo6d27dqVW9uwodh0D9u2peeKWbJkSaHnP9Vx5geC\nIvxAUIQfCIrwA0ERfiAowg8ERfiBoNqO85vZZkmrJE26+5Js2d2SfiCpma12h7vv6FWTmLl27Mj/\nb3Hs2LHktldddVWyfskll3TVE1o6OfP/XNLoNMt/6u5Lsx+CD8wwbcPv7i9IOtKHXgD0UZH3/DeZ\n2WtmttnM5pbWEYC+6Db8P5O0SNJSSROSfpK3opmtN7OGmTWazWbeagD6rKvwu/shdz/h7n+VtEnS\nssS6Y+5ed/d6rVbrtk8AJesq/Ga2YMrDayS9UU47APqlk6G+JyR9R9I8Mzsg6d8lfcfMlkpySeOS\nftjDHgH0QNvwu/vaaRY/1oNeMAN98sknyfozzzyTW5s9e3Zy23vuuSdZnzVrVrKONK7wA4Ii/EBQ\nhB8IivADQRF+ICjCDwTFV3ejkPvuuy9Z37NnT25txYoVyW0vvfTSrnpCZzjzA0ERfiAowg8ERfiB\noAg/EBThB4Ii/EBQjPMj6emnn07W77333mT9rLPOyq3deeedXfWEcnDmB4Ii/EBQhB8IivADQRF+\nICjCDwRF+IGgGOcP7qOPPkrWb7755mT9+PHjyfrKlStza0yxXS3O/EBQhB8IivADQRF+ICjCDwRF\n+IGgCD8QVNtxfjNbKOlxSfMluaQxd3/AzIYk/VrSsKRxSde6+8e9axXdOHHiRLI+OjqarL///vvJ\n+sjISLLe7u/9UZ1OzvzHJf3I3RdL+idJN5rZYkm3S9rp7hdI2pk9BjBDtA2/u0+4+yvZ/aOS3pJ0\nnqTVkrZkq22RdHWvmgRQvs/1nt/MhiV9S9IfJM1394ms9KFabwsAzBAdh9/M5kjaKukWd//T1Jq7\nu1qfB0y33Xoza5hZo9lsFmoWQHk6Cr+ZzVIr+L9w999miw+Z2YKsvkDS5HTbuvuYu9fdvV6r1cro\nGUAJ2obfzEzSY5LecveNU0rbJa3L7q+TtK389gD0Sid/0vttSd+T9LqZ7c2W3SFpg6TfmNn1kvZL\nurY3LaKId999N1lvNBqFnn/jxo3J+qJFiwo9P3qnbfjdfZckyylfWW47APqFK/yAoAg/EBThB4Ii\n/EBQhB8IivADQfHV3aeA/fv359aWL19e6Lnvv//+ZH3VqlWFnh/V4cwPBEX4gaAIPxAU4QeCIvxA\nUIQfCIrwA0Exzn8KeOSRR3JrqWsAOnH55Zcn663vesFMxJkfCIrwA0ERfiAowg8ERfiBoAg/EBTh\nB4JinH8GePHFF5P1Bx98sE+d4FTCmR8IivADQRF+ICjCDwRF+IGgCD8QFOEHgmo7zm9mCyU9Lmm+\nJJc05u4PmNndkn4gqZmteoe77+hVo5Ht2rUrWT969GjXzz0yMpKsz5kzp+vnxmDr5CKf45J+5O6v\nmNmXJL1sZs9ltZ+6e3pWBwADqW343X1C0kR2/6iZvSXpvF43BqC3Ptd7fjMblvQtSX/IFt1kZq+Z\n2WYzm5uzzXoza5hZo9lsTrcKgAp0HH4zmyNpq6Rb3P1Pkn4maZGkpWq9MvjJdNu5+5i71929XqvV\nSmgZQBk6Cr+ZzVIr+L9w999KkrsfcvcT7v5XSZskLetdmwDK1jb81vp61sckveXuG6csXzBltWsk\nvVF+ewB6pZNP+78t6XuSXjezvdmyOyStNbOlag3/jUv6YU86RCFLly5N1nfu3JmsDw0NldkOBkgn\nn/bvkjTdl7Mzpg/MYFzhBwRF+IGgCD8QFOEHgiL8QFCEHwjK3L1vO6vX695oNPq2PyCaer2uRqPR\n0bzpnPmBoAg/EBThB4Ii/EBQhB8IivADQRF+IKi+jvObWVPS/imL5kk63LcGPp9B7W1Q+5LorVtl\n9vY1d+/o+/L6Gv7P7Nys4e71yhpIGNTeBrUvid66VVVvvOwHgiL8QFBVh3+s4v2nDGpvg9qXRG/d\nqqS3St/zA6hO1Wd+ABWpJPxmNmpm/2tm75jZ7VX0kMfMxs3sdTPba2aV/v1xNg3apJm9MWXZkJk9\nZ2ZvZ7fTTpNWUW93m9nB7NjtNbOVFfW20Mx+Z2b7zOxNM/vXbHmlxy7RVyXHre8v+83sNEn/J+m7\nkg5I2i1prbvv62sjOcxsXFLd3SsfEzazf5b0Z0mPu/uSbNl/SDri7huyX5xz3f3fBqS3uyX9ueqZ\nm7MJZRZMnVla0tWSvq8Kj12ir2tVwXGr4sy/TNI77v6eux+T9CtJqyvoY+C5+wuSjnxq8WpJW7L7\nW9T6z9N3Ob0NBHefcPdXsvtHJZ2cWbrSY5foqxJVhP88SX+c8viABmvKb5f0rJm9bGbrq25mGvOz\nadMl6UNJ86tsZhptZ27up0/NLD0wx66bGa/Lxgd+n3WZu/+jpBWSbsxe3g4kb71nG6Thmo5mbu6X\naWaW/psqj123M16XrYrwH5S0cMrjr2bLBoK7H8xuJyU9pcGbffjQyUlSs9vJivv5m0GauXm6maU1\nAMdukGa8riL8uyVdYGZfN7MvSlojaXsFfXyGmZ2RfRAjMztD0nIN3uzD2yWty+6vk7Stwl7+zqDM\n3Jw3s7QqPnYDN+O1u/f9R9JKtT7xf1fSj6voIaevb0h6Nft5s+reJD2h1svAv6j12cj1kr4saaek\ntyX9t6ShAertPyW9Luk1tYK2oKLeLlPrJf1rkvZmPyurPnaJvio5blzhBwTFB35AUIQfCIrwA0ER\nfiAowg8ERfiBoAg/EBThB4L6f6yMEem39pFEAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f0239829c18>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "image1 = data.test.images[0]\n",
    "plot_image(image1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Output of Convolutional Layer\n",
    "\n",
    "There are different ways of getting the output of a layer in a Keras model. This method uses a so-called K-function which turns a part of the Keras model into a function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from tensorflow.python.keras import backend as K"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "output_conv1 = K.function(inputs=[layer_input.input],\n",
    "                          outputs=[layer_conv1.output])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then call this function with the input image. Note that the image is wrapped in two lists because the function expects an array of that dimensionality. Likewise, the function returns an array with one more dimensionality than we want so we just take the first element."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 28, 28, 16)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "layer_output1 = output_conv1([[image1]])[0]\n",
    "layer_output1.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then plot the output of all 16 channels of the convolutional layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUoAAADuCAYAAABf005JAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnVlwlOeVsB9tCCEhgdSsQkiIHbMYDDYGxzjBW+zYzlqu\nLE4lF7lwypWZxFWxU0mlcjFVk0riXNqp1GQm23jijO0kticJCTHxgsEYsFnMIkBISEICNWhfW1L/\nF99/ztcSrf66pd7UOs9Ng9RqvXr77fOe/WQFg0EMwzCM8clO9QIMwzDSHROUhmEYHpigNAzD8MAE\npWEYhgcmKA3DMDwwQWkYhuGBCUrDMAwPTFAahmF4YILSMAzDg9xYnuzz+YJVVVUJWkpiqaurw+/3\nZ6V6HeNhe5s4pvLeAhw5csQfDAbnpXod4+Hz+YKVlZWpXsaEqK+vj+rsxiQoq6qqOHz48MRXlUK2\nbt2a6iVExPY2cUzlvQXIysqqT/UaIlFZWcn+/ftTvYwJsXPnzqieZ6a3YRiGByYoDcMwPDBBaRiG\n4YEJSsMwDA9MUBqGYXhggtIwDMODmNKDItHW1gZATU0NfX19AMyYMQNw0jMWL14cr1817RgcHASg\np6eHmTNnAtDc3AxAd3e3Pq+oqAgAn88HQGFhITk5Oclc6pSjs7MTgI6ODvLz8wGYN89JWczKStvU\n0CmDnNdYGRkZIRAIAJAOUxjiJiiPHz8OwGuvvUZPTw+APn7qU59ixYoVAJSXlwOuEO3q6mJgYED/\nPR6hQlf+PV2oqakBnP1samoC4MUXXwSgoKCAlStXArBkyRIASkpKABgeHmbWrFmAK2zDkZeXB0Bp\naSnZ2Y6RIUJj2bJlFBYWxvXvSSeuXbsGQE5ODsXFxYAJyHQgOztbz+BkENkyWWFrprdhGIYHcdMo\nh4eHAUdzaW9vB2DLli2Ao5XMnj0bgLNnz+rzwNGSRJMMfQ3RYkZGRkYvODeX6urqeC17SiB70NHR\nwalTpwA4cOAAAKtXr1YNW9wfstdXrlzRPRUTPRAIqMYkpk1BQQEA69evR0r9xFWSmxu3I5KWLFiw\nAID29vYJm4lG+iKup6GhoUm9jmmUhmEYHsRNXVi4cCEAO3bsYNWqVQBs3rz5hueFK57v6OgY9f/u\n7m71ZYoGJY/ih5tOLF26FIBFixapz+WTn/wkALfccosGcU6ePAm4GmJBQYFq5hLgaWtrU43zzJkz\ngOtwLyoq4sKFCwDMmTMHIC5+onTm0qVLADQ2NvLGG28A6B6UlJQwd+5cAG666SYg/Jk2xkcsmpqa\nGvWvy55fu3ZNrcMNGzYAqOyI1++OVyAoboJSTLaFCxdSWloa089K8CHc/9etWwegGzrdAjngCi1w\ni/hFeK5du1b35M477wRck7q2tlYjuGKCNDU1qWAV8132u6Ojg6NHjwKwa9euxP1BacTFixcBpwPS\nvn37APfCGR4eVtNcPnCyjzk5OXohiWskLy9P997v9wPoRfXtb3+bj3/84wn/e9KNI0eOAHDixAkO\nHjwIoI+XL1/WAJpc5LK/hYWF6vaRyzrUfL777rsB+OIXvwg4LpTe3l7Ade/l5eWpAidnfqKY6W0Y\nhuFB3IM5sWqT0WKOdudGlfQdMVXk/3Dj3os2FMqyZctoaGgAnEAQoClEv/jFLzTos2PHjjivPj2p\nqKgAoKGhgUcffRRw3TszZszQPMvW1lbA1R5ra2tVS+nv7wegr69PzUsJaIqWLylc0w3Zo/z8fDZt\n2gSgedYVFRW6d3Lurl+/Djh9IsXNJM/PyclRy1LcR/K+1NTUcO7cOcBxowBs375dLarJYhqlYRiG\nB3HTKEUrCQaDcUnYlVSXl19+GYCbb74ZcIIX05Xc3Fz1eYVqkmMRP5kkkofS3t7O+fPnAbcg4N13\n3wXg0KFDPPjgg3Fdc7ojhRBz5sxRn9gdd9wBQEtLi2o8EoAQTaa+vl7PuexzXV2dvu4777wDwBNP\nPDHq94QSCATCvkeZhGjnRUVFmhIoWnZHR4dqkrJ34odsbm7W8yk++Ly8PL7whS8ATipbKP/85z9V\nk5T3bGRkZNK+ScE0SsMwDA/iplFGW1Ms0cNIWufevXv5+c9/DrjRw4ceemiSK8wMotFAwiWJy019\n9uxZLduT13rhhRcA51Z/6qmn4rXUKYH4vsOlnS1YsEB9kpIeJP/fvXs3LS0tgBuVPX/+vH5t+/bt\nANx+++3j/u5M1ybB1Qbnz58f9vsiD0QLlFjHyMiIFqKIFSUR7HB0dnaqj1KszxUrVugejy1ciZWk\nl11EEpBS0/zSSy9x7NgxAB555BFg/I02biTcHsuFc+XKFb3UxOSWD/fjjz9+Q6rWdEdcHOFMOAkE\nhaYCyYdbAhehqV0iDOSiklzh6YycVUm1CkVShyIFcsXF8Ze//EXdIpI6N2/evLjlUZrpbRiG4UFa\nFfKePn0agGPHjmlqyw9/+MNULiljEEd3MBjU21sSrMU8sr2ODTHnJIXozJkzupeSHhSKPM80yeiI\nZH1KipsUSFy/fl1T5kSbz8/PN43SMAwjWaSFRinajpTU9fb28thjjwGR02AMb0SLEQ1n7ty5WkIm\npV7f+MY3UrO4KY74Gi9fvqyPklgeLo0t0+vm402k/ZJ4xqFDhwAnhiEapXS+ysrKSr9a74nS2dmp\nJqD80aWlpTz55JOpXFZGMDQ0pDmTklvZ2dnJH//4R8DNff3BD36QkvVNZbq6urSZy5UrVwBYvnw5\nu3fvDvv83t5eC5TFwHhCUoJl9fX1gJs5sHr1arZt2zbqa/HsjG7qmmEYhgcp0yjFEX7kyBH+7//+\nD3DTVMTsNibH+fPn1SyUutnXXntN24g9/fTTKVvbVEXy/Px+v2rpwj333BM2zQUmn8c3XRBX23iB\nHMmVFLeRpA2uWbOGsrKyxK0rYa9sGIaRIaRMo5RhZC+//LLeEpJ5/53vfCdVy8oIJHWipaVFgzkf\nfPABAK+//rr2+PzmN7+ZmgVOQcTfJTXfw8PD+u/77rsPCJ80LX5M809GRjTISP1mr169qulA0jVo\n69atgFP7La+RCO3dNErDMAwPkq5Riq9MItzvvPOO1iH/+7//e7KXk5FI6sT169e1A8vevXsBp/uN\nJZbHjvgjRVtpaGiI2D070uhl40ai6RWxZ88e/va3vwFOhgE4vklwMmUS6QdOmqAUYSj1xRLAGRkZ\n4ctf/jIAt912W7KWk5HU1tYCbvu0wsJCnn/+ecCtenrmmWdsn2NABKS4MyQftaurSxtlhCKBMjEh\npR7cCI8IyEjTPn/1q18B8Kc//UlT2mT0jOROJhozvQ3DMDxImkYpN7KYgKL9lJWVWXL5JJEKERmU\nJVpPW1ubJpyL0/vrX/96ClY4dZG9lEDBiRMnACeAM7ajVW1trSZKJzJVJZOI1GpOZISkAvX19bFx\n40YATS6XIFmi069MozQMw/AgKRplT08Pb7/9NuCkp4A7EvS//uu/krGEjGVkZOQGTVIGXP3ud7/T\nhPNvfetbqVngFKa3t1fTe65evQq4vnbxlYUig6+M6PAaPS19KGUM7ebNm3XonQzGi2eZYiSSIiib\nm5s1iCPqtAyW/8pXvpKMJWQsfr9fhaFEWuUyqqur0yam99xzT2oWOIXp7+8f1UIN3Cir5UVOHq+G\nN5IrKS0Xly9fzubNmwHi1rk8Wsz0NgzD8CApGqXf71dNUtRoy5mMD52dnZrCIlPuZK87Ozv54he/\nCISfCWNEZnh4mFOnTgE3tlQzEsv777/Pb37zG8DtJLR48eJR016TiWmUhmEYHiRFoywrK+Puu+8G\n4KMf/SjgDLIyJk9paakm3UrATKbQPfLII9YhaBIEg0H1o8mc+Q8//BBw/Gbz5s0DXN/wTTfdFLc5\n0tMFCZa99957APziF78AnLMslU/S4zM3N1d9kqZRGoZhpBlJ0yhlvrHMOzbiQ2lpKaWlpYAbRZRk\n50hjPg1vGhsbtbPV/v37AbeO/t133+XRRx8FXI3HtMnYqK+v19rt3//+94BbkFJSUqKWkWQYVFRU\npGwWelIEZXFxccRB8EZ8EFNFTEJLYZkceXl5Oltazu/OnTsBZ163uI9MQE6Mnp4eNaFlxpDUcPf1\n9eleS+7k3LlztULKTG/DMIw0IysWyZyVldUK1CduOQmlMhgMzkv1IsbD9jZxTPG9BdvfRBLV3sYk\nKA3DMKYjZnobhmF4YILSMAzDAxOUhmEYHpigNAzD8MAEpWEYhgcmKA3DMDwwQWkYhuGBCUrDMAwP\nYqr19vl8QanFnGrU1dXh9/uzUr2O8bC9TRxTeW8Bjhw54k/nypypvL/Rnt2YBGVVVRWHDx+e+KpS\niIxrTVdsbxPHVN5bgKysrLQuD5zK+xvt2TXT2zAMwwMTlIZhGB6YoDQMw/DABKVhGIYHJigNwzA8\nMEFpGIbhQdxm5gQCAQB6e3ttVkuckXGoHR0dLFmy5Ibv9/f33/A8AJ/PR26u8xbbXJfwyN5du3aN\ngYEBADo7OwFobW3VgW2yfwsWLACcoW6zZs0C0D02xmd4eJj6eifLqbW1FYCsrCxWr14NuPubk5OT\nmgV6ELd32O/3A3D16lU2bdoUr5c1cGdJZ2dnhxWUcgDlsrpy5QrgfPjz8/MBWLp06bivL5PtZs+e\nrV87f/68vsaqVasAZ7hTpnH16lUAjh8/TmNjIwD79u0D4MSJE7p/lZWVgDvALXT6ZaRplyJES0pK\ndP9EKJSXl1NeXg7AjBkz4vdHpSH9/f0cO3YMcOfPL126lLq6OsAZ1gYwf/58wNlTEZq9vb3jvq5M\naMjJyblhuF48MdPbMAzDg7hplM3NzYCj/Zw+fRqAI0eOAI4pKLOm165dO+px+fLlejMb4RGNReYc\nj2V4eBhATcFFixbp19va2gBHMwRndLCYmIODg6NeJ1SjPHPmDOBoV/feey8A99133+T/mDRD3ETr\n1q3Tv39oaAiAkZER1ayPHz8OwBtvvAHAwMAAfX19AKp1BoNB1XDkUcarVlZW6vu3Zs0aADZv3qzv\nmZj0mUphYaGeI9H4urq61BqS/ZWzXFVVpXsj+9vX16fWjzzKWW5vb9dz//DDD496TjwwjdIwDMOD\nuGmUPT09ADQ1NfH6668Dru/M7/ertJfvyS3R3d2tzl3ROgcGBtSPI5qQ3N5f/epX9WZat24d4Ph3\nxMeRiYynSQqyDxMlnA/o6NGjALzwwgvq38xkjbKkpITq6moAdu3aBcATTzyhz5NAmVhJDQ0Ner5F\ncxkeHmZkZARwz634iwsKCtSHLD9XXl5+g1afyRQWFgJwxx13AI5/ePHixQCcPXsWgOvXrwNQXV2t\nn2mfzwc4QaCCggLA1Url5w4dOsTzzz8PuP7O3bt3x23tcROUGzduBJzDIk5VMWEGBwd1A2pqagBX\nUDY1NelBk0MzY8YMFbwXL14EXAd4aWmpOny7u7sBx8EuBznUfMwULl++DKCHKt6IiROKuFKam5sT\n4hyfasi5uuuuuyb8Gm+99RYABw8eBGDJkiUZH8QJh3zOGxsb1a300Y9+FHDPOrgCUi4fn8+nWQoi\nD8T03rNnj8qWP//5zwCsXLlSlS8R0hPFTG/DMAwP4qZRigkTeuOKqZGXl6faX21tLeBqlP39/RQX\nFwNu/lowGNRcwDfffBOAFStWAI5WJTeMOMzDaUSZhGgglZWV3HLLLUn5nRcuXAAcs3779u1J+Z2p\nQM6cnMFEIvsopmFxcXHGn12xKoeGhlQGiJV44cIF3QuRFZKu5ff7NbAjWmNOTo5ajpJqJcG2ixcv\napBMfuff/vY3lUciPyaKaZSGYRgeJLSkIDQ8Lz4F8WVGYnBwUJNT169fD7g+hvb2dvVjfP7zn4/r\netOVnTt3Am7ADEb7eSQYI1p9RUWFPk98OuIL6u/vVwd4pMIASfH6yle+osnWmchkfVfjIQHKUP+u\nfB5EGyooKEjY708XJOF+ZGREtXYJmvl8PtUkxUoUX3B+fr76yUV2ZGdna4K+PG/v3r2A4+OU15WA\nT2dnZ9xiFqZRGoZheJCyIlW5QbKzs0f9G5z0CUnDkFpQKZHMy8vT5FzxeWR6HXO4ZGSJllZXV+ut\n3N7ePuo5gUBA/Tvit3nvvfc0whiO//zP/wTQUsnPfe5z+r5kIhOtLQ4Gg5pMLtTW1qpvXfZbNKq5\nc+eqRSDv3cyZMzN6b0MJje5LJFoewzE0NDTKNwmjS0XFVymZINXV1ervlfSrtWvXaobMZEmZoAw9\nIPJv+aCfPHlSN1ZMGPleMBjU/KhMd4RHi5h0Y9N48vLy1ByX4IyXYPjtb38LwKOPPgo49bhjBYLB\nqD2RnMnr16/r5S0uptD6eFEI5P2KVCM+3cnNzY3YbETqxeVCHx4e5tKlS4Drglq/fn3czu70uM4M\nwzAmQVr1h5LuLefOndOAhFTkSCrQzTffnJFJ5YlGNHNxZYTjZz/7mboxRKPM5IqneCEmYm5urnZa\nCte/QIIMQrq2FJsKiGtO5EJxcbG6PT7+8Y8DbqeneGAapWEYhgdpoVGK1ijJ6CUlJZo2ITXi4ry9\n7bbbUrDCqYvUGsseR+pS8/zzz/Ov//qvQHiNyBjN2EbJubm545aZdnZ2agJ1Jvb1TBZ79uwB3D2X\nOEVjY6P2JLj99tuB+HYPSrmgDAQCNDU1Aa4Js2HDBv2+fNClLdvIyMi0iRROlu7ubm36u3z58nGf\n92//9m+Ak7smlQxSOWWEZ2BgQKOrUk0Sqfv80NBQxudMJpquri4++OADwDWrJeDT3t6uZzeeJrdg\nEscwDMODlGmU4oS9du2aaj2S3jI4OKjapZiMK1euBDBtMgokDeX06dMajIlUXfPSSy8B8NRTT1nw\nxgM5ty0tLZoWFNoIObS/Abgt7LKyskxLnyR/+MMfNPVH9lJGSVRVVY2yROONSR3DMAwPUqZRijP2\n1KlTWsUg6RI9PT3q//nIRz4CuDe54Y10VOno6ODuu+8e93lPPfUU4Grr9913nyWXeyD+yKamJj2T\nEqQpKipSTVI0y3BVJUZsSCrQ4OCg7q9UOYkcuf/++xM6/dU0SsMwDA+SrlHK7SsaY01NjUZkT5w4\nATi3sNSByvdsdrI30lFI9vEzn/lM2OdJGlZDQwMAX/7ylwEnLcs0yvDIuZUsjO7ubk0gv+mmmwDH\nTzYW8ambfzJ2xNKUsSSFhYWq0Us3c+lBKV3GEkXSpI8EGGQkhLRR8/l8Wh8rh6m1tVXns9gB80YO\nlBweMaXHQ2aLbNu2DXDb8JuQvBE5t9LaTwRmWVmZtg1btmzZuD9vJvfEeeWVVwA377SpqUnrueV9\nkfk7ia7WM9PbMAzDg6RplOJ8lUCDaEHFxcXaQk20x09/+tNhzRgjPNLgVG7bBx54YNznPvvss3ob\nf/rTnwbcGmTTKG9ERkWIm0K6WFVUVITV3MfO9Z6Ow8PiwbFjx7TySdKwuru79d9Szz3ZCaTRYhql\nYRiGB0nRKIeGhrQzkIwYkFGUly9f1t594qiVDiyGN319fdprMtIcY/EDP/PMM/z0pz8F3JI7S+IP\nz+DgoAZvRNsWDbOvr0/3VPbv8uXLeq4tcX9y/PKXv1R/sFifc+bM0QFtUq6YLB9wUgRlT08PZ86c\nAW40ZUpLS9U8sYYXsdPS0qIZAZECX8899xzgRLylMYYJyMh0d3ery0hqtqV6bMmSJbS1tQFuVHbG\njBlx66g9XTl58iTgXDoSnJTGOA899BD33nsvkJh67kjYJ8UwDMODpGiUra2tah7KDS03x8aNG7Uj\niKjRw8PDasIYkQkEAlHdrr/+9a8BePDBB1WjtOBNZHJycjTQJWczNHggKW4yGbCoqMjS2SaJdBKb\nOXPmqFQscHKpU6Wxm0ZpGIbhQVI0yv7+fm2wKf4cCfO/+uqrWo8sCdCmTUZPeXl5VFqMdFb57Gc/\nG7F5r+FSUlISsX547Fz0zs5OqyCbJFLlFJrQLylZAwMDKfOrm0ZpGIbhQVKuP5/Pp23aJWoovp6S\nkhIef/xxZzF2G8dMdnZ2VPv22GOPAU65opXVJQbRgIyJI37zlpYWzYaR8sS1a9feMKAtWSRFMi1c\nuFDNaylel0Nls1kmx7lz57TS5uabbx73eVLJIE1mDW/6+vqi+mDK/lu61eSRYM6WLVu0vn7Xrl0A\n3HrrrWZ6G4ZhpCtZsTTEzcrKagXqE7echFIZDAbnpXoR42F7mzim+N6C7W8iiWpvYxKUhmEY0xEz\nvQ3DMDwwQWkYhuGBCUrDMAwPTFAahmF4YILSMAzDAxOUhmEYHpigNAzD8CCmEkafzxecqkO/6urq\n8Pv9aduA0fY2cUzlvQU4cuSIP50Tzqfy/kZ7dmMSlFVVVbz77rsTX1UKSfcxE1VVVbz33nupXsaE\nkPZ46UpVVRWHDx9O9TImTFZWVlpXvUwHuWCmt2EYhgcmKA3DMDwwQWkYhuGBCUrDMAwPTFAahmF4\nYILSMAzDAxOUhmEYHsRtZk5XVxcA9fX1+P1+AHp6egAYGhrSAUEyilYeS0pKdNyqDRcLjwxcipWR\nkRGdMWINmmNnYGCA69evA+57IGOXi4qKbEZOFAwODgKOfBB5MDw8DIweByyDxEKRMysziVJJ3CRT\nfb2TE/vCCy/wwQcfAHD27FkAmpub+chHPgK4Q8Vkfm9fXx+FhYWAu4HhyMvLA2DBggVIFYAI23nz\n5lFWVgaE3/DpSugH2YRt7OTn55OTkwO4l35fXx/g7Kdc/tFw7do1VSBEMfD5fBQVFcVzyWlHR0cH\n4MiAxsZGAK5cuaLfl3Mp01nnz58POHsjn+VIZ1fO5qxZs1S2yHsWSiTZEg12JRqGYXgQN41Sbtfi\n4mLVFj/88EPAKXF6//33AfcGeOONNwAIBAL09/cD6GMk1q9fz6OPPgrALbfcAji30EQ1JiMy0928\nnDNnDoCaiKJRguNSgtEuo/FG13Z1dennobu7G4Dq6modMZypmqVYIYODg9TV1QHw5ptvAnDixAn9\nvrg0Fi1aBEB5eblqmTKHPhgM3qAtyj7Pnj2bxYsXA7Bq1SoAli9frq87Wab3p8AwDCMK4qZRVldX\nA/D000/r17773e8Cjv9SnLp79+4FoLOzE3Ac5i0tLYB7c3R3d6vfsrW1FYAjR44Azg0iQ9LXrVsH\nODeN/OxkfRHpyNWrVwF46623VDOX4fBz5sxhyZIlAFRUVACwYcMGACorK3VfTOOOnZ6eHvWTyaP4\nFyP9DHCD/3L27NmcP38egIaGBsCxpiorK4HM1SjF51hUVKTnVDS+kydPqpZ58eJFAC5dugTA4cOH\n9bMsWuPQ0JC+DyI/xAqdOXOmauf33XcfAPfee6/KiFj8yeFIaJhZzBZ5hNg7zYgz+Nlnn9WvSeBI\nnLfyBmQqEhQ7dOgQ586dA9wPW1lZmQrNAwcOAPAf//EfALS1tekhE/Owr6+PgoICAEpLSwHYunUr\nADt37mRgYABwgmbgujemE/LhvXz5sprSItB6e3sBxxQfG9zKzc3VD+7YD2Zvby9tbW2A+96tXLky\n4y8wMZVnz56te1JeXg64Ai0cTU1NNDc3A67yEwgEdP/lzF+7dk0fZe/l0uns7FRXyWQFpZnehmEY\nHqR94qI40Z944gkA/vjHP6oTfao2C40VuSE3bdqkmo1o6aWlpWqGiPbd3t4OOLespGKISTg4OKia\n5MqVKwF3H8+dO6cpV+vXr0/o35TOiLsiNH9PTEPZ62AwqNqgmIPBYFBdUGN5/fXX1e0kwZwdO3Zk\nrMk9WcrLy1XzjBbRLsXFUVhYqC68yWIapWEYhgdpq1GKn0gCPeKUbWtr08DQihUrUrK2ZCPO78WL\nF6tfTHw/RUVFqkFKkEu0TdGMQsnJydEUDOHYsWOAkxQsv8sraJHJLFy4UB/H+iElabytrU19YuJD\n8/l8GlAbS319vXZZF61z8eLF0z79Kp5I0Ymcf7E844G9S4ZhGB7ETaP0SssZmygaCAQAtzQxlHfe\neYc9e/YA8OSTTwJw9OhR/bmNGzdOer1TCUmaHS95VlIwJLInex0IBPRnxJ8W+j6I//IPf/gD4PjY\nRHM3HMZGpSUJWh6jpampSX3BMqdlzZo1Ga+5xyoXIhEMBrlw4QLgao9z587V3yPaubxn2dnZcSu7\nTZrpPXbD5I8K/bqYjidPntRggqQAiWANBAJ87nOfu+H106FwPlXIXo51XIea3qEfeNlzEZASqNi4\ncaMGc4z4IEO3Dh48qBeapMUsWrQo4wWlF9HkPUvDHXHHgZu+JgHMvLw8VQpEVoTmXU4WM70NwzA8\nSItgjqjH0tIqPz9ftUZJLJXb5M4774z4GoY3b731FuDW3Irm+dBDD4U1721vJ85zzz0HOGlF27dv\nB2DLli2AqxUZ4ZFzJ+lUwWBQA22REshDuw7FK6HfNErDMAwP0kKjlERRqevevHmzfk/K96T8a8eO\nHUle3dRm7I3a0NCggTHx64qGI6lBxuTZt28f4HTIASfgtnv3bsANQFij6shI+pUUSyxcuPCG1LZw\niI8ynqT8neru7tY8Pgnm3HPPPZw6dQpwKhoAHn744XFfIxMbYcSDcGbHlStXtHJB8lClbV04zOye\nGL/5zW8At0rqkUce0aYNsVacTEcCgYA2g5ELXSrKIv1MKPHMUTXT2zAMw4OUaZSiBdbW1rJ//37A\nNQHb29t1nITUwobrOmSaZHgiObAPHjyoJp/saaw5gcb4vPzyy4DbFnDp0qUAfOxjH9PUq0zvGDQZ\nxIJpa2vTPF8JzgwNDenZHZszGQwGVfNMhEvDNErDMAwPkq5Ryo0hPSX379+vfhzpffjOO+9ol5YH\nH3ww2UvMSGT0xtGjR7VCJDRoNhbzTcbO9evX+e///u9RX5N0tjVr1mgQxxgfSSC/fPmyygU5r9nZ\n2fo1qcwRAoFAQuvmTaM0DMPwIOkapUSmJPL6/vvv8y//8i+A25mluLhYfZPh+iKabzI84Xxfklrx\n0ksvAU7KhURfxX8WimmSE+fZZ5/VbI01a9YAcMcddwCOBmS+yfGRz7QUnfT29mpSuWiPw8PD40a+\nR0ZGwvbwB0ojAAANuElEQVSNiBdJE5QiIGtqagB0pMG9996rDWRlSl1jYyNf+tKXkrW0KU+kD+BP\nfvITwG0q8uCDD7Jr166krGu6IJVOr7zyirb4uuuuuwC3OXLoOBTDRQIwkictAZxgMKjNb6R5dzik\nlVoszTUmgpnehmEYHiRNo5Tqm0OHDgGu0/bxxx/n9OnTgGsmRgoymNkdPfv27dPKJjFZdu/ePe4N\nbWZ3bEhXm5/+9Kf6f9EkJYgjARwzu8Mjw78kuVxSgSoqKiJqkoJolIk0u8E0SsMwDE+SolH29fWp\nb1IeH3nkEQBeeOEFDdyIZvmd73znhtcwbSc845UpgqNRytgB0XRuvfXWpK0t03nmmWcAt8y2oqKC\n+++/H0BrkqPRiqYrQ0NDGsAVjVL6c+bk5Kj/Mlzaj8Q8klUvn5Tf0tbWpm3SZIqgCL49e/awePFi\nAG1DFY7p3Jg3VqTSacaMGRp9jdTwwi6hifHiiy8C6AynDRs26Pm2aidv+vv7tYWaCEhpPu33+zXA\nI63VCgoKbhCeyZo5ZKa3YRiGB0nRKIeHh1VrlJkskgpUWFiooX0xD8f+rBEZCSq8/fbbgFPPDY6m\nI6M0LGcyvvzkJz/h0qVLADp5cdu2bdoZKNHpKpnA4OCgBmGWLFkCuMGcjo4OPZ8yJyc7O5tly5YB\nJH0eummUhmEYHiRFo6yqqtJ6TUFu4QULFmjqitXCxk5XVxevvvoq4PrMDhw4AMDWrVsjtsw3Jk4g\nEFDf2YYNG/TREsujp6SkZFxfbllZmaYUhk4XTdUwNtMoDcMwPEh5h/No27sb4WlubtbendIpvqWl\nBYDly5eHrZU3Jk5jYyPgRGw3bdoEoCMeqqqqEp74nElEGiWbn5+vcY1QJME82aRcUNrBmhyDg4Pq\nsrj77rsBN2D20EMPmekdZyQVaPXq1Rog27p1KxB5MqAxeQKBQMoqnMz0NgzD8CArlhSRrKysVqA+\ncctJKJXBYDBts4BtbxPHFN9bsP1NJFHtbUyC0jAMYzpiprdhGIYHJigNwzA8MEFpGIbhgQlKwzAM\nD0xQGoZheGCC0jAMwwMTlIZhGB7EVMLo8/mCY7sATRXq6urw+/1pO+HJ9jZxTOW9BThy5Ig/nRPO\nfT5fUDq7TzXq6+ujOrsxCcqqqioOHz488VWlEKnHTVdsbxPHVN5bgKysrLSueqmsrOTdd99N9TIm\nxG233RbV88z0NgzD8MAEpWEYhgcmKA3DMDwwQWkYhuGBCUrDMAwPTFAahmF4ELdREH19fQA0NDTQ\n1tY26mvd3d06H0PGFsj0tbKyMgoKCpzF5KZ8MoUxzQgEAoBzRmVWtI0niR8jIyOAM0FxorPOUzUn\nJ5S4SSa/3w/A0aNHOXPmDABnz54F4MSJExQWFgLoiE+fzwc4g8xnzZoFoAIz7EL/vxCdN2+ejroV\noTt//nwdRJSqcZbG1EQGsV25ckVn4MyfPz+VS8oo5NKZzKybeChQkxW2ZnobhmF4EDeNsrS0FIDi\n4uJRmh440+kuXLgAQG1tLQDZ2Y6MnjFjBj09PfpvCH/7DA8PA45GetdddwFw8803A9DV1aVa6YIF\nC+L1J6UNMob20qVLOulP9sCYHFeuXAHgH//4h1o9X/va1wCzTuJBqqYmxhvTKA3DMDyIezCnuLiY\n4uJiAFauXAnArFmz2L59OwDt7e0AdHR0AM6NI8EfXVRurvoUrl+/Drj+zvz8fGpqagAYGBjQn1m9\nenW8/pS0o6urC4AlS5boPvz85z8H4MCBA6qJL1myBHD3ffv27SxatAgw7Wg8GhoaAMe3Lj7yLVu2\nALBjx46UrStTaGpqAqC3t1d9wFPxLMZNUIqwCwaDbNu2DUAFZmFhIZ2dnQBqZp8+fRpwundcvnwZ\ncCKP4JjlIkhFAPf39wPuAPrQ1woGgxOOqE0F7rjjDgBOnjype/XXv/4VcIIRYj7Kfsul8corr6gr\nQvaqqKiI6upqwHV/iNti586dKmRDkcilPD+TkL996dKl+kGeqp1w0hERlP/4xz/0UhI3XWdnJzNn\nzgRg8+bNAKxYsQKAiooKPc/yHqWSzDv5hmEYcSZuGqWk54TTSMDJlwxl/fr1EV/v2rVrgHMTATQ2\nNgJOCpFoSeJ8X7ZsGSUlJRNc+dShs7NTc/3ELKypqWHt2rX6b3ADZo2Njfp8yRecPXs2y5YtA+D1\n118HXI21urr6hvfv/PnzqgHIYyYhroldu3axfPlyAMrLy1O5pIxCgjnDw8NqDYk7bf/+/fq5FnNc\nztjs2bPV5SSf7ZycHA1mivb/iU98AoDHHnvsht/d0tKilqYEmCeKaZSGYRgexE2jFO0uXogGKhqO\n3C4rVqxQLWBslU+mIqlRK1asUM1dEnmbmpo02V/8t+fOnQOcYJfc4sFgEHD8PZcuXQLge9/7HgAP\nPPAAALfffrv+TikaOH36dEYHNVatWgU4WqRoK+EQ/7lo6F786U9/AmDDhg0A6heebsjnd/bs2Xzs\nYx8DXOvmM5/5DMePHwdci1G0za6uLg38SlHAvHnzNOAm8kGef+HCBbUI5Ky3traqRmsapWEYRoJJ\ny+Lq9vZ2XnvtNQD27NkDuCVIV69eVQ1ItIFMR/ws4UrrQv1pki4lyegzZ87UbARJISorK9MUo3Ap\nVRJBF42yoqJCfzYTkairPI5HNFkV586d03O7b98+AH784x9PcoVTG9HAJZo9locffhhwLUaxigYH\nB/Uz39zcDDjFJtGkGB09ehRwyqrXrVs32T8BSDNBKar2W2+9xYsvvgi4gYl77rkHcDZc6sWN0cjh\nCT1E4qYQOjo6Iuac9vb2jnqcNWvWtAiUeRGpD4FcRvv37+d///d/AdfkzOT83nggF9CcOXPGfY6Y\n1LFW+ZSVlcXNLWemt2EYhgdppVFKUvqxY8c0QVpSX6Tb0KpVqzKynjtZ9Pb2jqshnjt3jj//+c8A\n3HLLLYCThpGJiebxpK6uDoC//OUvaiY+99xzKVzR9ETcRa2trYATSItX60b7BBiGYXiQFhqllCue\nOnUKcPpXilNX0iskdeWmm25KwQozh7E+S3DLQ9va2jQ4JIm/1sR2fMTqke5Op06d4s477wRg06ZN\nKVtXpiCls9FohXV1darNSwBJSiDjQcoFZX9/v5oub7/9NuDkRskHVqJc4tC1LujxRy6lmpoajRJW\nVVUBU7OBQTIYGhrS2uX3338fcHKJn3766VQuK2MIBoMxZVt0dXVpfqZcUpECcLFiprdhGIYHKVfP\nWlpa1HT58MMPAafOW0xu0SwzvfomUcgtG8mEPn/+POCYLBI0S4eOLelMS0sL7733HoA2pX7ggQc0\n+GhMjmgtR6mY6ujo0LMbqcJqophGaRiG4UHKNEqpQT558iT79+8HXI1y7ty5mlQuqUBelRNGeCJp\nkuL8Fo1o5cqVeisb4ZG6+hMnTqhGKXv2/e9/P2XryhSkr0G0GqXUiPf19WlHskSMnzCN0jAMw4Ok\na5RyY0hK0BtvvKGapGiNW7Zs0Xb8Xn0rjfG5evVq2PpwScyVqK3c3mVlZeabHAfpSCMZGn//+9+1\n/+eTTz6ZqmVlHNFmWUhps5SP+ny+qDs7TYSkCUrJiZKDdvjwYcD5sEomveRIbt++nd27dydraRmH\nHJ5wQvL48eNqaouglPZXZnaPj7T6evPNNwHHZSTnVfoQGIlH5Ie4QMZW8CUKM70NwzA8SJpGKXXc\n0jT21VdfBeDQoUPaYeXWW28FnCqcTJkHnAoiNSnt7u7m4sWLgDttUJ4fzwTdTEHOrZjZhw4dAhw3\n0Y9+9KOUrSvTiDZ4IyNipLuVBH0TaXaDaZSGYRieJEWjHBwc1DC+pFScPHkScAIOt912G+AGbmy4\n08SIZlxBcXGxapKyz+F8mYYTeBSf2N69ewH3/P7gBz+wOvhJEgwGY9rDjo4OLbeV8kYZlJdokiIo\n+/r61NyTzs8y17u8vFzVZ9k0iTAa0SHmYTRNAFatWqXZBZKjah/48HR1dWnQUaaByiUUbuqfERsT\n6dsgskGmGySrF4GZ3oZhGB4kRaMcGRmhvr4ecMP6GzduBJw2atIZSG4Jq8KJnt7eXs1JFXOkqakJ\ncDsugTuT5OLFi9pqLRE1sZlES0uLzmw6cOAAAI8//ngql5RRhAZsJX0wUpPoo0ePqqkdaXREIjCN\n0jAMw4OkaJS5ubnaV1ImKMpQoU9+8pMaxLGRA7FTW1urDY9ffvllwK1/Xbp0Kdu2bQPc1J8FCxYk\nzQE+1eno6NB0NkGGhhmTp6enR4seZJ9lgmJ1dbWeU7EwCwoKkq5JCiaZDMMwPEiKRtnb26v9JD/1\nqU8BsHjxYsBSUyZLRUWFRr1FM6+oqADg97//vWYX7Ny5E3CS+q1reXT09/erJi4pVffff38ql5RR\nXLp0if/5n/8B3ER+ydyYP3++aptSnuj3+3XWd7JJiqDMy8vT6g+pJ7a64vhQUlLCmjVrADh48CAA\nv/vd7/R7X/rSlwC0dt4CZdGzcOFCvvCFLwBu9ZI15o0f4cY9SPrV1atXNQApLrmKioqUnV8zvQ3D\nMDzIiiW5OysrqxWoT9xyEkplMBhM23kStreJY4rvLdj+JpKo9jYmQWkYhjEdMdPbMAzDAxOUhmEY\nHpigNAzD8MAEpWEYhgcmKA3DMDwwQWkYhuGBCUrDMAwPTFAahmF4YILSMAzDg/8HJG4XH75mrHYA\nAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f02397fd2b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_conv_output(values=layer_output1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_conv_output(values=layer_output2)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}