tutorials_previous/3_tensorflow_simple_neural_network.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6TuWv0Y0sY8n"
   },
   "source": [
    "# Getting Started in TensorFlow\n",
    "## A look at a very simple neural network in TensorFlow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "u9J5e2mQsYsQ"
   },
   "source": [
    "This is an introduction to working with TensorFlow. It works through an example of a very simple neural network, walking through the steps of setting up the input, adding operators, setting up gradient descent, and running the computation graph. \n",
    "\n",
    "This tutorial presumes some familiarity with the TensorFlow computational model, which is introduced in the [Hello, TensorFlow](../notebooks/1_hello_tensorflow.ipynb) notebook, also available in this bundle."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Dr2Sv0vD8rT-"
   },
   "source": [
    "## A simple neural network\n",
    "\n",
    "Let's start with code. We're going to construct a very simple neural network computing a linear regression between two variables, y and x. The function it tries to compute is the best $w_1$ and $w_2$ it can find for the function $y = w_2 x + w_1$ for the data. The data we're going to give it is toy data, linear perturbed with random noise.\n",
    "\n",
    "This is what the network looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "cellView": "both",
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "output_extras": [
      {
       "item_id": 1
      }
     ]
    },
    "colab_type": "code",
    "collapsed": false,
    "executionInfo": {
     "elapsed": 681,
     "status": "ok",
     "timestamp": 1474671827305,
     "user": {
      "displayName": "",
      "photoUrl": "",
      "userId": ""
     },
     "user_tz": 420
    },
    "id": "q09my4JYtKXw",
    "outputId": "4938066b-231d-4078-e2dd-fd223eca7c9f"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAJYAAABkCAYAAABkW8nwAAAO90lEQVR4Xu2dT5Dc1J3Hv+YQT8VJ\nZUhVdprLWs4FTSrGGv4ql9CuHBCH4GaTFCLZwnIcjOAy8l6Q/1SlU4XHcg6xJgtY2OOik2KxSGoT\nGWrXzYFC2T2MDAtWitRavmQ0e9k2SYGowom4hNRPtqA9TE+rW3/cPfPepcfup6f3fu/Tv9/T+/PV\npo8//vhjsMQsULAFNjGwCrYoKy6xAAOLgVCKBRhYpZiVFcrAYgyUYgEGVilmZYUysBgDpViAgVWK\nWVmhDCzGQCkWGEuwrly5gtf++zW887/vYOn/lnD5T5cT40x9ZQrb/nEbxDtFiHeI2LJlSylGY4X2\nt8BYgUVAvfzqy3i5/TI+vPLhmq37wpYv4AHpATxw3wMMsP4cFJ5jbMAiqA4eOYg/Lv8xMcL26e34\n+vTXk8+vbv1q8n/03TsX38EfLv4h+aRE380dmmNwFY7O2gWOBVgE1Y/2/yjxUls+vwXaY1oS7tZK\n3v94MJ8zceUvV0Dea+H4AoOrQrhGHqxuT0Xjp0P7D2HqH6Yymejyu5dx5PiRZBxGnmt+bj7TdSxT\nfgv0ASuAzglwmyE8pfbZu3VaEDkDdT+AweevzGolvPjvL+LMb84knmr+yHxmqNKyCK7ZQ7OJ5yIo\n+3m6clqx8UrNB1bso2W64FQN9cnijdcdAvNAQWGRPBcLicX3Ua8S84FVcj3PnjuLhRcWkgH63OG5\nXHc7+NTBZEBP47NvffNbucpiF/e3QCaw2g0NfNvES5c+wtQ9u2G0LCj8BLAiFEaeBU0zYJ9fxkfY\njKl7FZgtCzIHIA7QUmXov/g9LmMztt6rwLBMyFROj3TkZ0fgveXh4X96GN//zvf7t2aNHGlI7VlW\n0pYmRC+AKUwAsQu5thOuvIjQEjGBGJ7CQYptdOw6etc6VzXXzcUZwJrGseWt2P28DV2I4OgyDgQK\nFgMTYtQ1xqq10eDuR6j8Fi1NxGTkwpAfRos7h05bQscQIFgibEeHMBHCVhs4EBtY8lQQd6ulvbN7\n8e6f302mC7Z/bXsuo9NkKk1X9PZ+IUyeR0sN4GscYl8DPzOP5VuPYynQwMU+dL4O3wzRbpQQ93O1\nbvQuzgRWS0p/tQA6Nuqcilq7A5u3Px28T7qw7BB1VUHqhEKTB2+pCAIVHZVD3dPgujpE6peOBzes\nQRS5nr/+b//g24nF7JN27qkCGq/J++RknHXm5JlVeiKGr/MQPQMdV0ZkCRBbNUwEMYzQhRyZEHgH\nOv29ynPM6HXtja1Rf7B4AZ7RgZv+SuMAOj+NtrYEX3avfyqMfDi2DdcLEAQBvPOX8MGtR3Ex0MEF\nJiRxP373wWZsvaeBhixDVRrg1/jxlwEWPV3ap+xVrR57Cjgpht2xEDV4mLIFvqkiaoUwwzp4U4Hv\n9/awN7YrR+vuGcAS4ZsdtKV0VNEFVqMLrIkWJGEPPP4hKA0RgiCAc1XsdJQErGQ2Ig7hOQ5sx4Hz\n0u+wvHX2akjtMWCpNhQCiCicq+AcCx1Fh9B2IegcNN6B4Teg1z0EeknzKqPFRe7a9AeLm4ajXvzU\noJEDqUahMESrKxSqbQHbDBGLoXUNlBiuUsNOT8fFQEVsNdHmdOjStTgSGOCnLTQuBDBosLxKqnTw\nntw/glPnoHMS4E6iFVjgbBGcwUGMPAjtawP73GZf/wVkAutYtAvPezYUPoKjipBdGZ5vQOgavGte\nHbfsiXD09TZUIUbg6JD3vITlrU/iYthErPOYaQk44ZhocDF8U0HDqsEOHfQaC7/2X68lyzJVTjd0\nWiJu2XMem++7+tAxSd52+hguTe3GYtjq6V3XPyqDtbA/WLyAtqRg0rHhLceo3avCsk0kjqd7uoEL\n0FJkaC/9Hh/gS9ixS0dTCaDKHVidNhoTNN2gQP/FedAmly/t2IWm2YK2xswqDbj3antzz5oToD/9\n15/i5smbcdo8vfaDQGiC37YfEyeW4KtcMu2g1HbCrp9Dx5Fw3ZCw04ZSb0Jse6CsLH1qgZFfK0zn\nn+hpznzKHGpJRzus4YJ/AX/78G94ofUC7r777pwMxAhdE6pyAK8u78CJJZ+BtcKiIw8Wea0DTx34\nZCH5oHYwM1y0TjhnziXbaWgB+4cP/RCPPfYYtm/fjpMnT+Kmm24aDrDYhdpoQdAbaMtNSB4Da6Uh\nRx4sqnB3SCTPNbtvtu9iMoU/Wg5Kt9p0h8DTp09j3759ePrpp/H4448PB1fylOtC5jTUGVifseFY\ngJXClXou+jcN6Gk2nj7JG1Gi7TG0Hkiz7OlGP/ru6OGjq46rnnjiCSwuLibe66677hocMAZWT5uN\nDVgpXGfbZ5OtybQNZq1EE6G0NXmXtGvNwbrv+4n3uu222wYPjwys9QFW2goKjbQ4Tdth6CAFeSpK\n5J3oQMUwhynS8PjMM89AVdVs3ouBtb7Aytbrw+WiMZfnednCIwOLgTUIZml43LFjB5577rnhnx4H\nuek6yztWY6yqbb+wsJBMTwwUHquu5Ijej4GVoWMoPJ4/fz7xXkM9PWa4x3rLwsDK2KMXLlxIvBeF\nR5qe2LRpU8YrN2Y2BtaA/U7hkaYnnn322exPjwPeYz1kZ2AN2YtpeCTvdeeddw5Zyvq9jIGVo28p\nPJL3ok2NLDxeb0gGVg6w0kvT8HjixIlkHJY1lauaE8GRangwsvD/noKqt+kzsLJSkCEfzdi/8cYb\nifdaKzxWoppDmxJ5FT54NH06YZShAQVmYWAVaEwqKg2PMzMzyfTEyqfHqlRzAoOH6OqwJnXoNQeB\nSWcjq0sMrJJsferUqSQsdofHylRzYg8aLyG0QtiTOvhGhFZglyKD0Mt8DKySwEqLpfD45ptvYn5+\nHr/+z19/sukwj2pOP72vyJXBy4BNME340Pg6AiNAu8IDkQysksGi4t9++2189wffxee++DkIO4Tc\nqjlrSw504Eg81FobYetq+KOwKDgagjVOnRdtBgZW0RZdpbw0BL73/nv4yZM/6bv7tVeVxkk1h4FV\nAVgbUTWHgVUBWGUcvCVV6EP/cuiztQ9NCNsMiIshrPSIeaK3oUNIlXQqaDMDqwIjlyEV0Fv6MoQl\nbENT/FTIhWSXOF2AF5jocei8cCswsAo36WcLLEPchO7yyr+9smrt6TQ3geQmcgcd2CQbIHoIDKGy\nuSwG1joEi06oU+jj3RAWR2HQgFiiTuxqJmRgVQBWGaGQDo78/OjPe9T+qpfSeBeeqIM3JPip4k8F\n7aVbMLAqMHSlg/dr7YkcCZxWg1Jz0G5UL7/EwKoArBuhmoNEbupBvPrRDhxf8qFVLFrCwKoArFQi\n4P3o/VwTpCmgdBi3r2oOIrQbNdwfGljytZ46r2U1n4FVlmW7yn3rrbfwvX/+XrKkMyPM5FLNIS2K\nbCrSNI8loKX48G6AxhIDq2SwaIcDgWWaJn71H78qRDWnlxbF1aaQxJILj6TRjRhm0L4hYrwMrJLA\nos1+BBXtyaLty5SKVs1Zverx1RB4dhIPPe/CVioeXF2rFAOrYLDIOxFQd9xxRwLVytSt90XfFaGa\nU3ATCimOgVWIGa8WkoY9AorA6pUIrqJVcwpsRiFFMbAKMONqYS9LsWWo5mS5bxV5GFg5rExhj8ZP\ndHBitbCXo+ixv5SBNWQXpmGPvNXtt98+ZCnr9zIG1oB9O2zYG/A2Y5+dgZWxC1nYy2goNt2Q3VA0\njqIDESzsZbcZ81hr2CoNe/T56KOPZrcqy8m2zazGAAt7+X8ZzGOtsCELe/mhohLGEqwyVFpY2CsG\nqLSUsQKrDJUWFvaKBWrswCpDpYWFvXKgKiYUxh5U/huwhd8idBqYRARX4bHTldd8Le8gTSpapYWW\nX0is47qnveTdi02I6aFOejlAbSdcOT2fF8NTOEixDTqnV6Uk0CC2GpW8hYTCyFXA72yj8XoAAzoE\n+nsxgNnrZc8DtL7bU9HJlDwqLY9855FkbY8ktS3LWlGLECbPo6UG8DUOsa+Bn5nH8q3HsRRo4GIS\nL6vDN0O0e70SdoB2rfeshYBF71Juyzzu90TcF59FIC8WJvSVvgiT9nnPH5nP/K7CtOPonYWzh2aT\nF2Fu+usmvPjLF3us7cXwdR6iZ6DjyogsAWKrhokghhG6kCMTAu9Ap7+r1l0cQwoLAote4+ugwT+I\nsxO78XrQKkTkqzsEkqeily8Nk0il5cfHfowv3/xlLBxf6Pk2sNhTwEkx7I6FqMHDlC3wTRVRK4QZ\n1sGbCnxfrfxgwjBtvtHXFAZW7OsQZo7hEm7Fkxf8nm+mH6TBlau0RG00OBWcY6Gj6BDaLgSdDn46\nMPwG9Hr15/MGsdco5S0GrDiAIU7D5M/AgIo9gY6Lng4+5wi3jIOea59wieCQzgEnAe4kWoEFzhbB\nGRzEyIPQDmBWpaoxSpQMUZdCwCLh1OlmDWcCBzJsSNzDiIyL8LR8Ur1lHE2nPeZzh+d6mooENW7Z\ncx6b7zuHTlvCJB1Nnz6GS1O7sUhKxDl/LEP00Vhekh8sUjThNUyYAdxr59dCSwSvAWbg5Xq7exkq\nLfRO6TMnz/TurNAEv20/Jk4swaf2xC6U2k7Y9XPoOBIm6crYh6UoaLodABOoSU3YlpLbQ48lQT0q\nnR+sEq1RBlj0dGmfsnPVOtB51IMmfEdGLQ7RkkSYkps8VbJ01QIjDdaNCIVZwOi4DnxOgsRRXIzh\nazwakY3gmphsljLWe56RBqv6wfvg3R0HFqS6CcHxC5kQHrwGo3nFSIN1Q1RaBuinyDchSyYmDRct\nhWPLPF22G2mwuo+k55kgHUylJRtZoa1A0kI0bAdGPRnSszQuYFE90yUdepoznzKHWtLRDmsglZY8\ncHZTE7UVCGqEpmtDScZZLK20wEh7LKpst9YBKQUf1A5mhovWCefMuU9eM9JbWnEQMAIY/DQOXLr+\nmqmHXkfIdj18YpSRByuFa6+2F1f+cgXkuWb3zfZdN6Twt/DCQuKpsgmVDQIXy9vPAmMB1krPRf9e\nryot/TpsXL4fG7BSuNa7Ssu4gNOvnmMFVtqY9azS0q/DxuX7sQRrXIy7kevJwNrIvV9i2xlYJRp3\nIxfNwNrIvV9i2xlYJRp3IxfNwNrIvV9i2xlYJRp3IxfNwNrIvV9i2xlYJRp3Ixf9d0NIelzdt4X5\nAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from __future__ import print_function\n",
    "\n",
    "from IPython.display import Image\n",
    "import base64\n",
    "Image(data=base64.decodestring(\"iVBORw0KGgoAAAANSUhEUgAAAJYAAABkCAYAAABkW8nwAAAO90lEQVR4Xu2dT5Dc1J3Hv+YQT8VJZUhVdprLWs4FTSrGGv4ql9CuHBCH4GaTFCLZwnIcjOAy8l6Q/1SlU4XHcg6xJgtY2OOik2KxSGoTGWrXzYFC2T2MDAtWitRavmQ0e9k2SYGowom4hNRPtqA9TE+rW3/cPfPepcfup6f3fu/Tv9/T+/PVpo8//vhjsMQsULAFNjGwCrYoKy6xAAOLgVCKBRhYpZiVFcrAYgyUYgEGVilmZYUysBgDpViAgVWKWVmhDCzGQCkWGEuwrly5gtf++zW887/vYOn/lnD5T5cT40x9ZQrb/nEbxDtFiHeI2LJlSylGY4X2t8BYgUVAvfzqy3i5/TI+vPLhmq37wpYv4AHpATxw3wMMsP4cFJ5jbMAiqA4eOYg/Lv8xMcL26e34+vTXk8+vbv1q8n/03TsX38EfLv4h+aRE380dmmNwFY7O2gWOBVgE1Y/2/yjxUls+vwXaY1oS7tZK3v94MJ8zceUvV0Dea+H4AoOrQrhGHqxuT0Xjp0P7D2HqH6Yymejyu5dx5PiRZBxGnmt+bj7TdSxTfgv0ASuAzglwmyE8pfbZu3VaEDkDdT+AweevzGolvPjvL+LMb84knmr+yHxmqNKyCK7ZQ7OJ5yIo+3m6clqx8UrNB1bso2W64FQN9cnijdcdAvNAQWGRPBcLicX3Ua8S84FVcj3PnjuLhRcWkgH63OG5XHc7+NTBZEBP47NvffNbucpiF/e3QCaw2g0NfNvES5c+wtQ9u2G0LCj8BLAiFEaeBU0zYJ9fxkfYjKl7FZgtCzIHIA7QUmXov/g9LmMztt6rwLBMyFROj3TkZ0fgveXh4X96GN//zvf7t2aNHGlI7VlW0pYmRC+AKUwAsQu5thOuvIjQEjGBGJ7CQYptdOw6etc6VzXXzcUZwJrGseWt2P28DV2I4OgyDgQKFgMTYtQ1xqq10eDuR6j8Fi1NxGTkwpAfRos7h05bQscQIFgibEeHMBHCVhs4EBtY8lQQd6ulvbN78e6f302mC7Z/bXsuo9NkKk1X9PZ+IUyeR0sN4GscYl8DPzOP5VuPYynQwMU+dL4O3wzRbpQQ93O1bvQuzgRWS0p/tQA6Nuqcilq7A5u3Px28T7qw7BB1VUHqhEKTB2+pCAIVHZVD3dPgujpE6peOBzesQRS5nr/+b//g24nF7JN27qkCGq/J++RknHXm5JlVeiKGr/MQPQMdV0ZkCRBbNUwEMYzQhRyZEHgHOv29ynPM6HXtja1Rf7B4AZ7RgZv+SuMAOj+NtrYEX3avfyqMfDi2DdcLEAQBvPOX8MGtR3Ex0MEFJiRxP373wWZsvaeBhixDVRrg1/jxlwEWPV3ap+xVrR57Cjgpht2xEDV4mLIFvqkiaoUwwzp4U4Hv9/awN7YrR+vuGcAS4ZsdtKV0VNEFVqMLrIkWJGEPPP4hKA0RgiCAc1XsdJQErGQ2Ig7hOQ5sx4Hz0u+wvHX2akjtMWCpNhQCiCicq+AcCx1Fh9B2IegcNN6B4Teg1z0EeknzKqPFRe7a9AeLm4ajXvzUoJEDqUahMESrKxSqbQHbDBGLoXUNlBiuUsNOT8fFQEVsNdHmdOjStTgSGOCnLTQuBDBosLxKqnTwntw/glPnoHMS4E6iFVjgbBGcwUGMPAjtawP73GZf/wVkAutYtAvPezYUPoKjipBdGZ5vQOgavGteHbfsiXD09TZUIUbg6JD3vITlrU/iYthErPOYaQk44ZhocDF8U0HDqsEOHfQaC7/2X68lyzJVTjd0WiJu2XMem++7+tAxSd52+hguTe3GYtjq6V3XPyqDtbA/WLyAtqRg0rHhLceo3avCsk0kjqd7uoEL0FJkaC/9Hh/gS9ixS0dTCaDKHVidNhoTNN2gQP/FedAmly/t2IWm2YK2xswqDbj3antzz5oToD/915/i5smbcdo8vfaDQGiC37YfEyeW4KtcMu2g1HbCrp9Dx5Fw3ZCw04ZSb0Jse6CsLH1qgZFfK0znn+hpznzKHGpJRzus4YJ/AX/78G94ofUC7r777pwMxAhdE6pyAK8u78CJJZ+BtcKiIw8Wea0DTx34ZCH5oHYwM1y0TjhnziXbaWgB+4cP/RCPPfYYtm/fjpMnT+Kmm24aDrDYhdpoQdAbaMtNSB4Da6UhRx4sqnB3SCTPNbtvtu9iMoU/Wg5Kt9p0h8DTp09j3759ePrpp/H4448PB1fylOtC5jTUGVifseFYgJXClXou+jcN6Gk2nj7JG1Gi7TG0Hkiz7OlGP/ru6OGjq46rnnjiCSwuLibe66677hocMAZWT5uNDVgpXGfbZ5OtybQNZq1EE6G0NXmXtGvNwbrv+4n3uu222wYPjwys9QFW2goKjbQ4Tdth6CAFeSpK5J3oQMUwhynS8PjMM89AVdVs3ouBtb7Aytbrw+WiMZfnednCIwOLgTUIZml43LFjB5577rnhnx4Huek6yztWY6yqbb+wsJBMTwwUHquu5Ijej4GVoWMoPJ4/fz7xXkM9PWa4x3rLwsDK2KMXLlxIvBeFR5qe2LRpU8YrN2Y2BtaA/U7hkaYnnn322exPjwPeYz1kZ2AN2YtpeCTvdeeddw5Zyvq9jIGVo28pPJL3ok2NLDxeb0gGVg6w0kvT8HjixIlkHJY1lauaE8GRangwsvD/noKqt+kzsLJSkCEfzdi/8cYbifdaKzxWoppDmxJ5FT54NH06YZShAQVmYWAVaEwqKg2PMzMzyfTEyqfHqlRzAoOH6OqwJnXoNQeBSWcjq0sMrJJsferUqSQsdofHylRzYg8aLyG0QtiTOvhGhFZglyKD0Mt8DKySwEqLpfD45ptvYn5+Hr/+z19/sukwj2pOP72vyJXBy4BNME340Pg6AiNAu8IDkQysksGi4t9++2189wffxee++DkIO4TcqjlrSw504Eg81FobYetq+KOwKDgagjVOnRdtBgZW0RZdpbw0BL73/nv4yZM/6bv7tVeVxkk1h4FVAVgbUTWHgVUBWGUcvCVV6EP/cuiztQ9NCNsMiIshrPSIeaK3oUNIlXQqaDMDqwIjlyEV0Fv6MoQlbENT/FTIhWSXOF2AF5jocei8cCswsAo36WcLLEPchO7yyr+9smrt6TQ3geQmcgcd2CQbIHoIDKGyuSwG1joEi06oU+jj3RAWR2HQgFiiTuxqJmRgVQBWGaGQDo78/OjPe9T+qpfSeBeeqIM3JPip4k8F7aVbMLAqMHSlg/dr7YkcCZxWg1Jz0G5UL7/EwKoArBuhmoNEbupBvPrRDhxf8qFVLFrCwKoArFQi4P3o/VwTpCmgdBi3r2oOIrQbNdwfGljytZ46r2U1n4FVlmW7yn3rrbfwvX/+XrKkMyPM5FLNIS2KbCrSNI8loKX48G6AxhIDq2SwaIcDgWWaJn71H78qRDWnlxbF1aaQxJILj6TRjRhm0L4hYrwMrJLAos1+BBXtyaLty5SKVs1Zverx1RB4dhIPPe/CVioeXF2rFAOrYLDIOxFQd9xxRwLVytSt90XfFaGaU3ATCimOgVWIGa8WkoY9AorA6pUIrqJVcwpsRiFFMbAKMONqYS9LsWWo5mS5bxV5GFg5rExhj8ZPdHBitbCXo+ixv5SBNWQXpmGPvNXtt98+ZCnr9zIG1oB9O2zYG/A2Y5+dgZWxC1nYy2goNt2Q3VA0jqIDESzsZbcZ81hr2CoNe/T56KOPZrcqy8m2zazGAAt7+X8ZzGOtsCELe/mhohLGEqwyVFpY2CsGqLSUsQKrDJUWFvaKBWrswCpDpYWFvXKgKiYUxh5U/huwhd8idBqYRARX4bHTldd8Le8gTSpapYWWX0is47qnveTdi02I6aFOejlAbSdcOT2fF8NTOEixDTqnV6Uk0CC2GpW8hYTCyFXA72yj8XoAAzoE+nsxgNnrZc8DtL7bU9HJlDwqLY9855FkbY8ktS3LWlGLECbPo6UG8DUOsa+Bn5nH8q3HsRRo4GISL6vDN0O0e70SdoB2rfeshYBF71Juyzzu90TcF59FIC8WJvSVvgiT9nnPH5nP/K7CtOPonYWzh2aTF2Fu+usmvPjLF3us7cXwdR6iZ6DjyogsAWKrhokghhG6kCMTAu9Ap7+r1l0cQwoLAote4+ugwT+IsxO78XrQKkTkqzsEkqeily8Nk0il5cfHfowv3/xlLBxf6Pk2sNhTwEkx7I6FqMHDlC3wTRVRK4QZ1sGbCnxfrfxgwjBtvtHXFAZW7OsQZo7hEm7Fkxf8nm+mH6TBlau0RG00OBWcY6Gj6BDaLgSdDn46MPwG9Hr15/MGsdco5S0GrDiAIU7D5M/AgIo9gY6Lng4+5wi3jIOea59wieCQzgEnAe4kWoEFzhbBGRzEyIPQDmBWpaoxSpQMUZdCwCLh1OlmDWcCBzJsSNzDiIyL8LR8Ur1lHE2nPeZzh+d6mooENW7Zcx6b7zuHTlvCJB1Nnz6GS1O7sUhKxDl/LEP00Vhekh8sUjThNUyYAdxr59dCSwSvAWbg5Xq7exkqLfRO6TMnz/TurNAEv20/Jk4swaf2xC6U2k7Y9XPoOBIm6crYh6UoaLodABOoSU3YlpLbQ48lQT0qnR+sEq1RBlj0dGmfsnPVOtB51IMmfEdGLQ7RkkSYkps8VbJ01QIjDdaNCIVZwOi4DnxOgsRRXIzhazwakY3gmphsljLWe56RBqv6wfvg3R0HFqS6CcHxC5kQHrwGo3nFSIN1Q1RaBuinyDchSyYmDRcthWPLPF22G2mwuo+k55kgHUylJRtZoa1A0kI0bAdGPRnSszQuYFE90yUdepoznzKHWtLRDmsglZY8cHZTE7UVCGqEpmtDScZZLK20wEh7LKpst9YBKQUf1A5mhovWCefMuU9eM9JbWnEQMAIY/DQOXLr+mqmHXkfIdj18YpSRByuFa6+2F1f+cgXkuWb3zfZdN6Twt/DCQuKpsgmVDQIXy9vPAmMB1krPRf9eryot/TpsXL4fG7BSuNa7Ssu4gNOvnmMFVtqY9azS0q/DxuX7sQRrXIy7kevJwNrIvV9i2xlYJRp3IxfNwNrIvV9i2xlYJRp3IxfNwNrIvV9i2xlYJRp3IxfNwNrIvV9i2xlYJRp3Ixf9d0NIelzdt4X5AAAAAElFTkSuQmCC\".encode('utf-8')), embed=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fBQq_R8B8rRf"
   },
   "source": [
    "Here is the TensorFlow code for this simple neural network and the results of running this code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "cellView": "both",
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "output_extras": [
      {
       "item_id": 1
      }
     ]
    },
    "colab_type": "code",
    "collapsed": false,
    "executionInfo": {
     "elapsed": 7741,
     "status": "ok",
     "timestamp": 1474671834967,
     "user": {
      "displayName": "",
      "photoUrl": "",
      "userId": ""
     },
     "user_tz": 420
    },
    "id": "Dy8pFefa_Ho_",
    "outputId": "318456b0-f9de-4717-d9c7-956b5d390d05"
   },
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "'module' object has no attribute 'subtract'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-12-313910ecc1ea>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     30\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     31\u001b[0m     \u001b[0myhat\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmatmul\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mweights\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 32\u001b[0;31m     \u001b[0myerror\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msubtract\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0myhat\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtarget\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     33\u001b[0m     \u001b[0mloss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0ml2_loss\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0myerror\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     34\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mAttributeError\u001b[0m: 'module' object has no attribute 'subtract'"
     ]
    }
   ],
   "source": [
    "#@test {\"output\": \"ignore\"}\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from pkg_resources import parse_version\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "# Set up the data with a noisy linear relationship between X and Y.\n",
    "num_examples = 50\n",
    "X = np.array([np.linspace(-2, 4, num_examples), np.linspace(-6, 6, num_examples)])\n",
    "X += np.random.randn(2, num_examples)\n",
    "x, y = X\n",
    "x_with_bias = np.array([(1., a) for a in x]).astype(np.float32)\n",
    "\n",
    "losses = []\n",
    "training_steps = 50\n",
    "learning_rate = 0.002\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    # Set up all the tensors, variables, and operations.\n",
    "    input = tf.constant(x_with_bias)\n",
    "    target = tf.constant(np.transpose([y]).astype(np.float32))\n",
    "    weights = tf.Variable(tf.random_normal([2, 1], 0, 0.1))\n",
    "\n",
    "    if parse_version(tf.__version__) >= parse_version('0.12'):\n",
    "        tf.global_variables_initializer().run()\n",
    "    else:\n",
    "        tf.initialize_all_variables().run()\n",
    "\n",
    "    yhat = tf.matmul(input, weights)\n",
    "    yerror = tf.subtract(yhat, target)\n",
    "    loss = tf.nn.l2_loss(yerror)\n",
    "  \n",
    "    update_weights = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)\n",
    "  \n",
    "    for _ in range(training_steps):\n",
    "        # Repeatedly run the operations, updating the TensorFlow variable.\n",
    "        update_weights.run()\n",
    "        losses.append(loss.eval())\n",
    "\n",
    "    # Training is done, get the final values for the graphs\n",
    "    betas = weights.eval()\n",
    "    yhat = yhat.eval()\n",
    "\n",
    "# Show the fit and the loss over time.\n",
    "fig, (ax1, ax2) = plt.subplots(1, 2)\n",
    "plt.subplots_adjust(wspace=.3)\n",
    "fig.set_size_inches(10, 4)\n",
    "ax1.scatter(x, y, alpha=.7)\n",
    "ax1.scatter(x, np.transpose(yhat)[0], c=\"g\", alpha=.6)\n",
    "line_x_range = (-4, 6)\n",
    "ax1.plot(line_x_range, [betas[0] + a * betas[1] for a in line_x_range], \"g\", alpha=0.6)\n",
    "ax2.plot(range(0, training_steps), losses)\n",
    "ax2.set_ylabel(\"Loss\")\n",
    "ax2.set_xlabel(\"Training steps\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vNtkU8h18rOv"
   },
   "source": [
    "In the remainder of this notebook, we'll go through this example in more detail."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "r6rsv-q5gnn-"
   },
   "source": [
    "## From the beginning\n",
    "\n",
    "Let's walk through exactly what this is doing from the beginning. We'll start with what the data looks like, then we'll look at this neural network, what is executed when, what gradient descent is doing, and how it all works together."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "UgtkJKqAjuDj"
   },
   "source": [
    "## The data\n",
    "\n",
    "This is a toy data set here. We have 50 (x,y) data points. At first, the data is perfectly linear."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "cellView": "form",
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "output_extras": [
      {
       "item_id": 1
      }
     ]
    },
    "colab_type": "code",
    "collapsed": false,
    "executionInfo": {
     "elapsed": 271,
     "status": "ok",
     "timestamp": 1474671835304,
     "user": {
      "displayName": "",
      "photoUrl": "",
      "userId": ""
     },
     "user_tz": 420
    },
    "id": "-uoBWol3klhA",
    "outputId": "cc31ce5d-9b65-4ef6-8475-643be268569a"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQYAAAEDCAYAAADX+KqPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFTlJREFUeJzt3XtQVVXfB/Dv4aaOVIJcjxneIrFQZzSbUUd9EHUMDRTQ\nd3LSkVJ7HWREnVMes2w0SHzK6TKDt9SBmpyilIhynJCLqXjNSy8xjg7pKAIhkogXhLPfP0gCtsDB\nvfbZZ+/z/fwlPbB/y8t8n7X2Wue3TJIkSSAiasVN6wEQkfNhMBCRDIOBiGQYDEQkw2AgIhkGAxHJ\neIh4yO7du5GVlQWTyYTQ0FCkpqbCy8tLxKOJSAOKZwyVlZXIzMzE999/j5ycHDQ1NeGnn34SMTYi\n0oiQGYPNZsPdu3fh5uaGe/fuISAgQMRjiUgjioMhMDAQCxcuxKRJk9CrVy+MGzcOY8eOFTE2ItKI\n4qXErVu3kJeXh/z8fBw6dAh37txBTk6OiLERkUYUB8ORI0fQv39/9OnTB+7u7pgyZQp+++23Tn+G\nH88gcm6KlxJmsxlnz57F/fv34eXlheLiYoSHh3f6MyaTCX/9Vae0tN38/Z9gPZ3WM/LvTat69lAc\nDMOHD8e0adMQExMDDw8PDBs2DHPmzFH6WCLSkJBdicTERCQmJop4FBE5AZ58JCIZBgMRyTAYiEiG\nwUBEMgwGIpJhMBCRDIOBiGQYDEQkw2AgIhkGAxHJMBiISIbBQEQyDAYikmEwEJEMg4GIZBgMRCTD\nYCAiGQYDEckwGIhIRkgw1NXVISkpCdOnT0dUVBTOnj0r4rFEpBEhzWA/+OADTJw4EZ9++ikaGxtx\n7949EY8lIo0onjHcvn0bJ0+eRGxsLADAw8MD3t7eigdGRNpRPGO4evUqfHx8sHr1apSWluKFF17A\nmjVr0LNnTxHjIzKkmppavPVWPsrLfWA21yAtLQI+Pn20HlYLxTOGxsZGlJSU4NVXX8XevXvRs2dP\nbNu2TcTYiAzrrbfykZ39Gk6cmIns7PmwWPK1HlIbimcMQUFBCAoKarmWbtq0adixY0eXP2fvVVmi\nsJ5+6xnx91Ze7gPA9M9XJpSX+zj899kZxcHg5+eH4OBglJWVYeDAgSguLsbgwYO7/Dmj3w/Ievqr\npXa9h8uHy5efRFXV/wGYgeZwkGA233TI79Nhd1cCwDvvvINVq1ahsbER/fv3R2pqqojHEhnKw+VD\ncxhMhNmcin79wmE230Ra2n+0Hl4bQoJh6NCh+O6770Q8isiwLl9+Ev8uH3wQEDAMx4/PdOiMyF5C\ngoGI5FovHUJC/kZwcAPOnJHwcPkQEnJL6yF2iMFApJLWS4czZyRMn74F0dGZ/wTFLadbPrTGYCBS\nSdulgwnXrwfjwIHJWg7JbvwQFZFKQkL+BiD985VzLx3a44yBSKDW7xWCg+sxffoXuH7dz+mXDu0x\nGIgEav9eITo6UzfLh9a4lCASqP17heav9YczBiKF5Ccao6GHLcnOMBiIFHrUicaAgGG6e6/QGoOB\nSKFHnWjU43uF1viOgUghPW9LdoQzBqJuan/U2WodBUAfJxrtxWAg6qb2W5JAJrZvn6X1sITiUoKo\nm4yyJdkZzhiI7GDELcnOMBiI7GDELcnOMBiI7GDELcnOMBiIHkHPTVZEYDAQPYKem6yIICwYbDYb\nYmNjERgYiC1btoh6LJEm9NxkRQRh25UZGRl2tY0n0gMjnmbsDiHBUFFRgcLCQsTHx4t4HJEmampq\nMXfu15g6NQ8NDc1NVkaO3Ifo6EzDLx3aE7KUSElJgcViQV2d87XBJrKXUZqsiKA4GAoKCuDn54ew\nsDAcO3bM7p8z4rVjrKfvWlpdG+dMV9M9pDgYTp8+jYMHD6KwsBD3799HfX09LBYL0tLSOv05o1w7\nxnr6rqX1tXFa/N3ZQ3EwrFixAitWrAAAHD9+HDt37uwyFIicRfsTjU8/vRF+fkNdYkuyMzzHQC6t\n/YnG4ODnkZs7ScMROQehwTBmzBiMGTNG5COJVBUS8nebE40DB97WekhOgTMGcildNVlJT38FTU1a\nj1J7DAZyKV01WfH1dezLQGfFRi3kUlyhyYoInDGQ4blakxURGAxkeK7WZEUEBgMZnqs1WRGBwUCG\n4+pNVkRgMJDhuHqTFREYDGQ4rt5kRQRuV5LhuHqTFRE4YyBDaP1eITi4ucnK9et+XDo8JgYDGQKb\nrIjFpQQZAk80isUZA+kStyTVxWAgXeKWpLoYDKRL3JJUF98xkC5xS1JdnDGQbnBL0nEYDKQb3JJ0\nHMXBUFFRAYvFgurqari7uyM+Ph7z588XMTaiNrgl6TiKg8Hd3R2rV69GWFgY6uvrMXv2bIwbN473\nWJIQN27UYtGiH9hkxcEUB4O/vz/8/f0BAL1798bgwYNRVVXFYCAhli79mU1WNCD0HcPVq1dRWlqK\n4cOHi3wsubCyMm+wyYrjCQuG+vp6JCUlwWq1onfv3l1+v5HvWmQ9cQYOrMOJE/+eaAwNvat6baP+\nWXaHkGBobGxEUlISoqOjERkZadfPGPWuRdZTpv1R548+moT79/890bh+/X9U/b0a6c+yo3r2EBIM\nVqsVQ4YMwYIFC0Q8jlxY+y3JHj32tLn3gRxD8cnHU6dOIScnB8XFxYiJicGsWbNQVFQkYmzkgtpv\nSTa/YyBHUzxjGDVqFP744w8RYyEX1dm9D7xLUhs8+Uia6+zeB94lqQ0GA2mus3sfeJekNhgM5HBs\nsuL8GAzkcGyy4vwYDORwbLLi/NiohRyOTVacH2cM5BBssqIvDAZyCDZZ0RcuJcgh2GRFXzhjINV0\ndqKR7xWcG4OBVNPZiUa+V3BuDAZSTWcnGsm58R0DqYbbkvrFGQMJ0/6os9U6CgBPNOoRg4GEab8l\nCWSyyYpOcSlBwnBL0jg4YyBFuCVpTAwGUoRbksbEYCBFuCVpTEKCoaioCCkpKZAkCbGxsVi8eLGI\nx5ITqqmpRWLij7hwoRebrBiY4mCw2WxYv349du/ejYCAAMTFxWHy5Mm8os6g2GTFNSgOhnPnziEk\nJAT9+vUDAERFRSEvL4/BYFBssuIaFG9XVlZWIjg4uOXrwMBAVFVVKX0sOSmeZnQNimcMkiR1/U2P\nYPT7AY1U78aNWixd+jPKyrxhNjcgJiYD1675YuDA20hPfwW+vrxLUk/17KE4GIKCglBeXt7ydWVl\nJQICArr8OaPfD2ikeosW/dBqS1LCnDl7kJs7CQDQ1KTu36XR/iydoZ49FC8lwsPDceXKFVy7dg0N\nDQ3Izc3F5MlccxoJr41zPYpnDO7u7li7di0SEhIgSRLi4uL44lHnurr3gdfGGZ+QcwwTJkzAhAkT\nRDyKnEBXW5K8Ns74ePKRZLrakuS1ccbHT1eSDLckiTMGAsB7H6gtBgMB4L0P1BaXEgSATVaoLc4Y\nXBibrFBHGAwujE1WqCMMBhfGJivUEb5jcGHclqSOcMbgQnjvA9mLweBCeO8D2YtLCRfCLUmyF2cM\nBsctSXocDAaD45YkPQ4Gg8FxS5IeB4PBYLpqssLlA9mDwWAwvPeBRGAwGAzvfSARuF1pMDzNSCIo\nmjGkpaUhPz8fXl5eeOaZZ5Camgpvb3YQdrQbN2qxaNEPbLJCwiiaMYwfPx65ubnIzs5GSEgItm7d\nKmpc1A1Ll/6M7OzXcOZMDH7++X/h5eWFAwcmY/v2WfDx6aP18EiHFAXD2LFj4ebW/IiRI0eioqJC\nyKCoe5rveeCJRhJH2MvHrKwsREVFiXocdaH1tmR19R8AZoBbkiSKSeri8smFCxeiurpa9t+Tk5MR\nEREBAEhPT0dJSQk+++wzdUZJMnPnfo1vvvkfNIfBTTz99FYEBz//z32S0+HryyUEPb4uZwy7du3q\n9H/fu3cvCgsLkZGR0a3CRr8fUO16Fy70QusTjX5+Qw15n6QR/+60rmcPRe8YioqKsGPHDqSnp8PL\ny0vJo6ibuC1JalL0jmHDhg148OABEhISAAAjRozAunXrRIyL2umsyUpo6F2sX89tSRJHUTAcOHBA\n1DioC501WXH0dJSMjycfdYJNVsiR+FkJJ8YmK6QVBoMTY5MV0gqDwYmxyQpphcHgRNhkhZwFg8GJ\nsMkKOQsGgxNhkxVyFtyudCI8zUjOgjMGjbV+r8AmK+QsGAwaa/9eITo6k8sH0hyXEhrjiUZyRpwx\nOBi3JEkPGAwOxi1J0gMGg4NxS5L0gO8YHIxbkqQHnDE4ALckSW8YDA7ALUnSGy4lHIBbkqQ3nDGo\nhE1WSM+EBMMXX3yBTZs2obi4GH368D4DgE1WSN8UB0NFRQWOHDkCs9ksYjyGwSYrpGeK3zGkpKTA\nYrGIGIuhcFuS9EzRjOHgwYMIDg7Gc889J2o8utX6nUJo6J029z5w+UB602UwdHR35fLly7F161bs\n3Lmz5b91cQ1mG/ZelSWK2vUSE39sd+/DHuzbN1/Vmq0Z7c9Tq1quUM8ej3135YULF3Dt2jVER0dD\nkiRUVlYiNjYW3377Lfr27dtlYaPdD9j2LkkTLlzoxfsddVbLVerZ47GXEqGhoTh8+HDL1xEREdi7\ndy+eeuqpx32k7nBLkoxK2DkGk8nUraWEEXS0Jcm7JEnvhAVDXl6eqEfpRkdbkrxLkvSOJx+7gU1W\nyFUwGLqBTVbIVTAYuoFNVshV8NOV3cDTjOQqOGPoApuskCtiMHSBTVbIFXEp0QU2WSFXxBnDI/BE\nI7k6BsMjsMkKuToGwyOwyQq5Or5jeARuS5Kr44wB8qPObLJCro7BAPmWJJCJ7dtnaT0sIs1wKQFu\nSRK157IzBm5JEnXMZYOBW5JEHXPZYOCWJFHHXCYY2GSFyH4uEwxsskJkP8XBkJmZia+++gqenp6Y\nOHEiVq1aJWJcwrHJCpH9FAXDsWPHkJ+fjx9//BEeHh6oqakRNS7hQkL+5tKByE6KguHrr7/GokWL\n4OHR/BhfX18hgxLl4XuF8nIf+PmxyQqRvRQFw59//omTJ09i8+bN6NGjBywWC8LDw0WNTbG2W5Js\nskJkL0V3VzY1NeHWrVv45ptvcO7cOSxfvtzu+yUccV9febkPWr9XKC/3cdg9gUa//5B3V+q3nj0e\n++5KANizZw+mTp0KABg+fDjc3Nxw8+ZN+Pj4dFnYEReymM01aP6UZPOMwWy+6ZC6rnD/Ie+u1G89\neyhaSkRGRuLo0aN48cUXUVZWhsbGRrtCwVHS0iIAZKK83Adm802+VyCyk6JgmD17NqxWK2bOnAlP\nT09s3LhR1LiE8PHpg+3bZ/HKOKJuUhQMnp6e2LRpk6ixEJGT4MeuiUiGwUBEMgwGIpJhMBCRDIOB\niGQYDEQkw2AgIhkGAxHJMBiISIbBQEQyDAYikmEwEJEMg4GIZBgMRCTDYCAiGQYDEckwGIhIhsFA\nRDKKgqG0tBRz585FTEwM4uLicP78eVHjIiINKQqGTZs2YdmyZdi3bx+WLVuGtLQ0UeMiIg0pCgaT\nyYS6uubuy3V1dQgMDBQyKCLSlqIu0atXr8Ybb7yBjRs3QpIk7NmzR9S4iEhDj31FXXJyMo4cOYI1\na9YgMjIS+/fvh9Vq7fTmKiLSB5MkSdLj/vDo0aNx8uTJlq9HjRqFU6dOCRkYEWlH0TuGwMBAHD9+\nHABw9OhRDBgwQMSYiEhjimYMp0+fxoYNG2Cz2dCjRw+89957GDZsmMjxEZEGFAUDERkTTz4SkQyD\ngYhkGAxEJKPogJMSn3zyCfLy8uDm5oa+ffviww8/hL+/v2r10tLSkJ+fDy8vLzzzzDNITU2Ft7e3\nKrX279+Pzz//HJcuXUJWVhaef/55VeoUFRUhJSUFkiQhNjYWixcvVqUOAFitVhQUFKBv377IyclR\nrc5DFRUVsFgsqK6uhru7O+Lj4zF//nzV6jU0NGDevHl48OABmpqaMG3aNCQmJqpW7yGbzYbY2FgE\nBgZiy5YtqtaKiIiAt7c33Nzc4OHhgaysrI6/WdLI7du3W36dkZEhvfvuu6rWO3z4sNTU1CRJkiRt\n2rRJ+u9//6tarUuXLkllZWXSa6+9Jv3++++q1GhqapIiIyOlq1evSg0NDdIrr7wiXbx4UZVakiRJ\nJ06ckEpKSqQZM2aoVqO1qqoqqaSkRJKk5n8rU6dOVfX3J0mSdOfOHUmSJKmxsVGKj4+Xzp49q2o9\nSZKkXbt2SStXrpSWLFmieq2IiAiptrbWru/VbCnRu3fvll/fvXsXbm7qDmXs2LEtNUaOHImKigrV\nag0aNAgDBgyApOKGz7lz5xASEoJ+/frB09MTUVFRyMvLU63e6NGj8eSTT6r2/Pb8/f0RFhYGoPnf\nyuDBg1FVVaVqzV69egFonj00NjaqWgtonhUVFhYiPj5e9VoAIEkSbDabXd+r2VICADZv3ozs7Gw8\n8cQTyMjIcFjdrKwsREVFOayeGiorKxEcHNzydWBgoGE/9n716lWUlpZi+PDhqtax2WyYPXs2rly5\ngnnz5qleLyUlBRaLpeWDiGozmUx4/fXXYTKZMHfuXMyZM6fD71U1GDr7nEVERASSk5ORnJyMbdu2\n4csvv8SyZctUrQcA6enp8PT0xMyZM1WvpSY1ZyPOpL6+HklJSbBarW1mmWpwc3PDvn37cPv2bSxd\nuhQXL17EkCFDVKlVUFAAPz8/hIWF4dixY6rUaG/Pnj3w9/dHTU0NFi5ciEGDBmH06NGP/F5Vg8He\nD1TNmDEDS5YsURwMXdXbu3cvCgsLhcxOtP6wWFBQEMrLy1u+rqysREBAgIYjEq+xsRFJSUmIjo5G\nZGSkw+p6e3tjzJgxOHTokGrBcPr0aRw8eBCFhYW4f/8+6uvrYbFYVO1p8vDlvq+vL6ZMmYLz5893\nGAyavWO4fPlyy6/z8vIwaNAgVesVFRVhx44dSE9Ph5eXl6q1WlPr/9nDw8Nx5coVXLt2DQ0NDcjN\nzcXkyZNVqfWQo2cpVqsVQ4YMwYIFC1SvVVNT0zKlv3fvHo4eParqv8kVK1agoKAAeXl5+Pjjj/HS\nSy+pGgp3795FfX09AODOnTv49ddf8eyzz3b4/Zq9Y/joo49QVlYGNzc3mM1mvP/++6rW27BhAx48\neICEhAQAwIgRI7Bu3TpVav3yyy9Yv349bt68iTfffBNDhw7Fjh07hNZwd3fH2rVrkZCQAEmSEBcX\nh8GDBwut0drKlStx7Ngx1NbWYtKkSVi2bBliY2NVq3fq1Cnk5OQgNDQUMTExMJlMSE5OxoQJE1Sp\n99dff+Htt9+GzWaDzWbDyy+/jIkTJ6pSSwvV1dVITEyEyWRCU1MTZs6cifHjx3f4/fysBBHJ8OQj\nEckwGIhIhsFARDIMBiKSYTAQkQyDgYhkGAxEJMNgICKZ/wfuExw3vDkyWAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xa5aa810>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#@test {\"output\": \"ignore\"}\n",
    "num_examples = 50\n",
    "X = np.array([np.linspace(-2, 4, num_examples), np.linspace(-6, 6, num_examples)])\n",
    "plt.figure(figsize=(4,4))\n",
    "plt.scatter(X[0], X[1])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AId3xHBNlcnk"
   },
   "source": [
    "Then we perturb it with noise:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "cellView": "form",
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "output_extras": [
      {
       "item_id": 1
      }
     ]
    },
    "colab_type": "code",
    "collapsed": false,
    "executionInfo": {
     "elapsed": 375,
     "status": "ok",
     "timestamp": 1474671835705,
     "user": {
      "displayName": "",
      "photoUrl": "",
      "userId": ""
     },
     "user_tz": 420
    },
    "id": "fXcGNNtjlX63",
    "outputId": "455c3e70-a724-4e0a-d08e-9bf6bd1aa7e9"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQYAAAEDCAYAAADX+KqPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFp1JREFUeJzt3X9QVOX+B/D3CvjjijcWRX6Ud+PSUDpBztjtD+trSQih\nIqBmNU02cqPsjm46FQ6U1YwKtTb9UXfGNENHm0nTREKrYQYJmyzL394xYyqqiwReW0ggQBee7x8I\n4R7ZPXjOnuec3ffrP+rsOZ+wffs8z3l+2IQQAkREg4yQXQARmQ+DgYgUGAxEpMBgICIFBgMRKTAY\niEghXI+bbN26Fbt374bNZkNycjJKS0sxcuRIPW5NRBJobjE0Nzdj+/bt2LNnDyorK9HT04OPP/5Y\nj9qISBJdWgy9vb3o7OzEiBEj0NXVhYkTJ+pxWyKSRHMwxMbGYsmSJbjvvvswZswY3H333Zg+fboe\ntRGRJJq7EhcvXkR1dTVqamrw+eef448//kBlZaUetRGRJJqD4dChQ5g0aRKioqIQFhaGWbNm4fjx\n4z4/w+UZROamuSuRkJCAkydPoru7GyNHjsRXX32FlJQUn5+x2Wz43//atD5aVzEx41iTSmasizWp\nExMzTtV1moMhNTUVmZmZyM3NRXh4OKZMmYJFixZpvS0RSaTLW4lly5Zh2bJletyKiEyAMx+JSIHB\nQEQKDAYiUmAwEJECg4GIFBgMRKTAYCAiBQYDESkwGIhIgcFARAoMBiJSYDAQkQKDgYgUGAxEpMBg\nICIFBgMRKTAYiEhBlx2ciPTkdrdi1aoa/PzzX+Fw/A6XKw12e5TPaxsb7UhIcPu8ltRjMJDprFpV\ng4qKxwDYcOKEALAd77yT5/dawPe1pJ4uXYm2tjY4nU5kZWVhzpw5OHnypB63pRD1889/Rd8XHQBs\nV37Wfq3e3O5WFBSUIyOjGgUFe9DS0mrYswNNlxbDunXrcO+99+LNN9+Ex+NBV1eXHrelEOVw/H6l\npdDXCnA4Lupyrd6G07KxGs3B0N7ejiNHjuDVV1/tu2F4OCIjIzUXRqHL5UoDsP3KGMNFuFwz/V7b\nN8bQ4vNavclsrQSa5mBoaGiA3W5HUVERzp49i9tvvx0vvPACRo8erUd9ZAHDGSxUw26PUvU37+Dn\nJie3Yc2amYYOPMpsrQSa5mDweDw4c+YMXnrpJaSkpGDdunXYtGkTnE6nHvWRBchqUns/t7vb2Kb8\ncFo2VqM5GOLi4hAXFzdwLF1mZiY2b97s93Nqj8oyEmtSb3BdjY12DG5SNzbaDalb1nP7xcSMw969\ni/1eY0Wag2HChAmIj49HfX09EhMT8dVXXyEpKcnv58x4ph9rUse7roQEN/peFfY1qRMSWgypW9Zz\n1TLjn59hZ1cCwIsvvojnnnsOHo8HkyZNQmlpqR63JYuQ1aQe/Nzk5E6sWRM8TXnZbELSmfRmTFLW\npI4Z62JN6qhtMXCtBBEpMBiISIHBQEQKDAYKGT/88DOmTn0LDkc5pk59E/X1P8suybS4upJCxoIF\nH6GxsQiADZ2dAnl5pThxYrnsskyJLQYKGS0tN2HwhKi+n+laGAwUMuz2/6JvQhQACNjtDTLLMTV2\nJShklJfnIC+vFC0tN8Fub0B5+TzZJZkWg4FCRmKig2MKKrErQUQKDAYiUmAwEJECg4GIFBgMRKTA\nYCAiBQYDESkwGIhIgROcyC/v7eHLynIAhKm+3ujzJGU/PxgwGMgv723an356B/7977mqrzf6hCbZ\nzw8GunUlent7kZeXh6VLl+p1SzIJ7xOX6ut9nzQm+4Qm2c8PBroFw7Zt21RtG0/W43D8jsGrEhMT\n24d1vdEnNMl+fjDQpSvR1NSE2tpaLF26FFu2bNHjlmQi3tvDb9gwDz096q83+oQm2c8PBroEQ0lJ\nCQoLC9HWZq6tskkf3mdJRkf73hZd7dmTgSL7+cFAczB89tlnmDBhAiZPnozDhw+r/pwZj+5iTeqZ\nsS7WpB/NB8688cYb+OijjxAWFobu7m50dHRg1qxZcLlcPj9nxoM4WJM6ZqyLNamjNqh0PYnq66+/\nRllZGd5++22/15rxF8aa1DFjXaxJHZ5ERUTXTdcJTnfddRfuuusuPW9JRBJw5iNJxynM5sNgIOk4\nhdl8OMZA0nEKs/kwGEg6TmE2H3YlyK/hLrseLk5hNh8GA/k13GXXQxlqkJFTmM2HwUB+DXfZ9VA4\nyGgdHGMgv4a77HooHGS0DrYYyK/hLrseisPx+5WWgg0cZDQ3BgP5Ndxl10PhIKN1MBjIMBxktA6O\nMRCRAoOBiBQYDESkwDGGEMUVjeQLgyFEcbIR+cKuRIjiZCPyhcEQoriikXxhVyJEcbIR+aI5GJqa\nmlBYWIgLFy4gLCwMDz74IBYvXqxHbRRAnGxEvmgOhrCwMBQVFWHy5Mno6OjA/Pnzcffdd/McyxDF\ntx3BQXMwxMTEICYmBgAwduxYJCUl4fz58wyGIOV2t2LZsn2oqxtzzS8+33YEB13HGBoaGnD27Fmk\npqbqeVsyEX9ffL7tCA66BUNHRwecTieKi4sxduxYv9eb8Uw/1uRfY6Mdg7/4jY32q2pMTv7jqqXV\nycmdhv03mO13BZizJjV0CQaPxwOn04mcnBykp6er+owZj+5iTf4lJLjR95qz74ufkNCC777778C4\nQnx8B7Ky3savv8bD4biINWtmGvLfYMbflVlrUkOXYCguLsYtt9yCxx9/XI/bkYm5XGkYNWrHlTGG\nvtechYVXdy9ycrajqup+2aWSBpqD4ejRo6isrERycjJyc3Nhs9mwcuVKzJgxQ4/6yGTs9ijs3PnI\nVX8Tclwh+GgOhmnTpuHbb7/VoxayKG7ZFnw485E04yzK4MNgIM04izL4MBhIgbMXicFACpy9SAyG\nEDZUy8DfW4bffmtFQcFHAW9R9Nf3ww9/gdv9HaKjb0ZSkoctGAMwGELYUC0Df28Z/vWvTwxpUQyu\nDxBobNyB//xnccCeR39iMISwoVoG/t4y9J1dGfh5C971AZEBfR79icEQwoZqGfh7y5CY2IZvvgn8\nvAXv+oD2gD6P/sRgCGHXO/9gw4bZ6O4O/LyF/vp+/PEv+O23OkRHO5CUtJ3zJAxgE0II/5fpz4yL\nSwJd03BfA16rJjO8SjTr4iDW5J+hi6hIHT1eA/JVIhmBu0QbSI/FRlywREZgMBhIjy3bue07GYFd\nCQPpsdiIC5bICBx8vMKsA0VmqwkwZ12sSR21g4/sShCRAoOBiBQYDESkwMFHCzPDZCcKTroEw8GD\nB1FSUgIhBBYsWIAnn3xSj9uSH5zsRIGiORh6e3uxZs0abN26FRMnTsTChQtx//3384g6A8ic7OTd\nWikqmobS0mOK1gtbNdakORhOnToFh8OBG2+8EQAwZ84cVFdXMxgMIHN3Zu/WyjfflKKxsQjerRe2\naqxJczA0NzcjPj5+4OfY2FicPn1a621JBZmTnbxbKy0tN+FarRdO4bYmzcFwvfOjzHimn9VqiokZ\nh717FxtYzZ+8z6gcP/4cGhqUZ1YaeZal1f78zExzMMTFxaGxsXHg5+bmZkycONHv58w4I4w1qRMT\nMw5r1vzfVXsyFBdno6Tkz5/7z6z0vi5QZ1ma8Xdl1prU0BwMKSkp+OWXX3Du3DnExMRg//79eOON\nN7TelkzuWrs8vfOOQ9V1ZH6agyEsLAyrV69Gfn4+hBBYuHAhBx6JLE6XeQwzZszgIbYmwdeDpAfO\nfAwyfD1IeuBaiSAT6NeDbncrHnrofWRkVKOgYA9aWlp1vT+ZA1sMQSbQk57YIgkNDIYgE+hJT5yw\nFBoYDEEm0K8HZU7DJuMwGGhYXK40jBq1A3V1Y7jnZBBjMNCw2O1R2LnzEdPN6CN98a0EESmwxRBA\nnGxEVsVgCCC+2iOrYlcigPhqj6yKwRBAVjpOzu1uRUFBOWc0EgB2JQLKSsfJsdtDgzEYAshKexGw\n20ODsStBAKzV7aHAY4uBAFir20OBx2AgANbq9lDgsStBRAoMBiJS0NSVcLlcqKmpwciRI/G3v/0N\npaWliIyM1Ks2IpJEU4vhnnvuwf79+1FRUQGHw4GNGzfqVVfAeU/ocbs5oYeon6ZgmD59OkaM6LvF\n1KlT0dTUpEtRRuif0HPiRC4qKhbj6ac/kV0SkWnoNsawe/duS20h7z2hp76eXSCifn7HGJYsWYIL\nFy4o/vnKlSuRlpYGANiwYQMiIiKQnZ2t+sGyz/TzPlMxMbFdek3XYsaaAHPWxZr0YxPXeyrtFeXl\n5di5cye2bduGkSNHqv6c7B2AWlpaUVhYMzChp6xsHnp6wqTW5M2MZx8C5qyLNaljyNmVBw8exObN\nm/Hee+8NKxTMwHtCT3S0+f4QiWTRFAxr167F5cuXkZ+fDwC444478Morr+hRFxFJpCkYqqqq9KqD\niEyEMx+JSIHBQEQKXF1pEFk7RnOnaroeDAaDyNo6jVu20fVgV8IgsrZO835uVRW42Sv5xWDQSO3u\nyrK2TvN+bmdnBCoqFqOwsMaQ55M1sSuhkdqmuqyt0/qfW1UFdHZGAMgCN3slfxgMKvgawFPbRZC1\ndVr/cwsK9qCi4mHw+HpSg8Gggq9WgcPx+1WLscz6heNmrzQcDAYVfLUKrPKF42avNBwMBhV8tQr4\nhaNgxGDwoX9s4YcfwpGQUIrx45Px97//YdpWAZFeGAw+DB5bAAT+8Q/9JgdxRiKZGYPBh0BOSuKM\nRDIzTnDyIZCTkniILJkZWww+BPKNg1Vec1JoYjD4EMg3Dv5Cx+1uxbJl+1BXN4ZjEGS4oA0Gsw/u\n+QsdjkGQTEEbDFb/YnEMgmTSZfDx3XffxW233YbWVvMs5e37Iv0O4H0A+1Bb22SppcayVmMSATq0\nGJqamnDo0CEkJCToUY9u+gb3PgbwCAAbWlvnorDQOq0GlysNo0btuDLGYN6p1hScNLcYSkpKUFhY\nqEctunK50hAV1QWrNsft9ijs3PkIqqruxzvv5JlqfISCn6ZgOHDgAOLj43HrrbfqVY9u7PYo3Htv\nGNgcJxo+v0fUDXV25YoVK7Bx40aUlZUhMjISaWlp+PDDD2G32wNW7HC53a14+ulPUF8ficTEdmzY\nkIXoaP7NS+TPdZ9dWVdXhyVLlmD06NEQQqC5uRmxsbHYtWsXxo8f7/fz/o6DM/p1o1nPGTRbTYA5\n62JN6gT87Mrk5GR88cUXAz+npaWhvLwcN9xww/Xe8ipWf91IZGW6rZWw2WzQeHD2Vfgen0ge3YKh\nuroaUVH6NfX5Hp9IHtPOfLTKlmlEwci0wcAt04jk4X4MRKTAYCAiBQYDESkwGIhIwbSDj4OZfdMV\nomBjiWAI9VmQDEYymiWCIdRnQYZ6MJLxLDHGEOqzIEM9GMl4lmgxhPosSO+t5uPiLqCgoJxdCwoY\nSwSDllmQwdA/9w7GS5cuo6Lin2DXggLFEsGgRTD0z72DMSOjGuxaUCBZYoxBC+/+eVUVUFCwx1I7\nRnsL9TEXCrygbzF49887OyNQUfEwrNhy6BfqYy4UeEEfDP1foqoqoLMzAkAWrN785spTCrSg70r0\nf4kyMgSAhwFEgc1vIt+CvsXQj81vIvVCJhjY/CZST3NXYvv27XjggQeQnZ2N119/XY+aiEgyTS2G\nw4cPo6amBvv27UN4eDjcbrdedRGRRJpaDO+//z4KCgoQHt6XL9HR0boURURyaQqGn376CUeOHMGi\nRYvw2GOP4fTp03rVRUQS+e1K+Dq7sqenBxcvXsQHH3yAU6dOYcWKFaiurlb1YLVHZRmJNalnxrpY\nk378BsOWLVuG/Hc7duxARkYGACA1NRUjRoxAS0uLqoNtzXimH2tSx4x1sSZ11AaVpq5Eeno6vvzy\nSwBAfX09PB6PqU67JqLro+mtxPz581FcXIzs7GxERETgtdde06suIpJIUzBERERg/fr1etVCRCYR\n9GsliGj4GAxEpMBgICIFBgMRKTAYiEghZJZdB4Ng2PGarIHBYCHBsOM1WQO7EhbCE6nIKAwGC+G2\n8WQUdiUshPtWklEYDBbCfSvJKOxKEJECg4GIFBgMRKTAYCAiBQYDESkwGIhIgcFARAqaguHs2bN4\n6KGHkJubi4ULF/JcCaIgoSkY1q9fj+XLl2Pv3r1Yvnw5XC6XXnURkUSagsFms6GtrW/f/La2NsTG\nxupSFBHJpWlKdFFREZ544gm89tprEEJgx44detVFRBJd9xF1K1euxKFDh/DCCy8gPT0dn376KYqL\ni32eXEVE1mATQgj/l13bnXfeiSNHjgz8PG3aNBw9elSXwohIHk1jDLGxsfj6668BAF9++SVuvvlm\nPWoiIsk0tRiOHTuGtWvXore3F6NGjcLLL7+MKVOm6FkfEUmgKRiIKDhx5iMRKTAYiEiBwUBECtKD\n4d1338Vtt92G1tZW2aXA5XIhKysLOTk5WL58Odrb26XVcvDgQTzwwAPIzMzEpk2bpNXRr6mpCYsX\nL8bs2bORnZ2Nbdu2yS5pQG9vL/Ly8rB06VLZpQxoa2uD0+lEVlYW5syZg5MnT8ouCVu3bsXcuXOR\nnZ2NZ599FpcuXRr6YiHRr7/+KvLz88XMmTNFS0uLzFKEEEJ88cUXoqenRwghxPr168Xrr78upY6e\nnh6Rnp4uGhoaxKVLl8S8efPE999/L6WWfufPnxdnzpwRQgjR3t4uMjIypNfUb8uWLeLZZ58VTz31\nlOxSBqxatUrs3r1bCCHE5cuXRVtbm9R6mpqaRFpamuju7hZCCPHMM8+I8vLyIa+X2mIoKSlBYWGh\nzBKuMn36dIwY0fcrmTp1KpqamqTUcerUKTgcDtx4442IiIjAnDlzUF1dLaWWfjExMZg8eTIAYOzY\nsUhKSsL58+el1gT0tWRqa2vx4IMPyi5lQHt7O44cOYIFCxYAAMLDwxEZGSm5qr6WVWdnJzweD7q6\nujBx4sQhr5UWDAcOHEB8fDxuvfVWWSX4tHv3bsyYMUPKs5ubmxEfHz/wc2xsrCm+hP0aGhpw9uxZ\npKamyi5l4C8Xm83m/2KDNDQ0wG63o6ioCHl5eVi9ejW6urqk1hQbG4slS5bgvvvuw4wZMzBu3DhM\nnz59yOsDeq7EUOssVqxYgY0bN6KsrGzgnwmDplP4WvuRlpYGANiwYQMiIiKQnZ1tSE3ejPpdXI+O\njg44nU4UFxdj7NixUmv57LPPMGHCBEyePBmHDx+WWstgHo8HZ86cwUsvvYSUlBSsW7cOmzZtgtPp\nlFbTxYsXUV1djZqaGowbNw5OpxOVlZVD/j8e0GAYakFVXV0dzp07h5ycHAgh0NzcjAULFmDXrl0Y\nP358IEvyu8irvLwctbW1UgfX4uLi0NjYOPBzc3Ozz2afUTweD5xOJ3JycpCeni67HBw7dgwHDhxA\nbW0turu70dHRgcLCQun7gsTFxSEuLg4pKSkAgMzMTGzevFlqTYcOHcKkSZMQFdV3OvqsWbNw/Pjx\nof/yM2Tkw4+ZM2eK1tZW2WWI2tpaMXv2bOF2u6XW4fF4BgYfu7u7TTH4KIQQzz//vCgpKZFdxjUd\nPnzYVIOPjz76qPjxxx+FEEK89dZbwuVySa3n5MmTYu7cuaKrq0v09vaKVatWiffee2/I601xRJ3N\nZjNF83nt2rW4fPky8vPzAQB33HEHXnnlFcPrCAsLw+rVq5Gfnw8hBBYuXIikpCTD6xjs6NGjqKys\nRHJyMnJzc2Gz2bBy5Upp4zBm9+KLL+K5556Dx+PBpEmTUFpaKrWe1NRUZGZmIjc3F+Hh4ZgyZQoW\nLVo05PVcK0FECtInOBGR+TAYiEiBwUBECgwGIlJgMBCRAoOBiBQYDESkwGAgIoX/B0/dgNBk20eF\nAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xbdce750>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#@test {\"output\": \"ignore\"}\n",
    "X += np.random.randn(2, num_examples)\n",
    "plt.figure(figsize=(4,4))\n",
    "plt.scatter(X[0], X[1])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3dc1cl5imNLM"
   },
   "source": [
    "## What we want to do\n",
    "\n",
    "What we're trying to do is calculate the green line below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "cellView": "form",
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "output_extras": [
      {
       "item_id": 1
      }
     ]
    },
    "colab_type": "code",
    "collapsed": false,
    "executionInfo": {
     "elapsed": 1150,
     "status": "ok",
     "timestamp": 1474671836784,
     "user": {
      "displayName": "",
      "photoUrl": "",
      "userId": ""
     },
     "user_tz": 420
    },
    "id": "P0m-3Mf8sQaA",
    "outputId": "32a8a45d-ba64-4286-acf7-0883d9184693"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQYAAAEDCAYAAADX+KqPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XlYE+faP/BvCJsFlIDI4oKIoqAgde1BBUXUuqDi2uXo\nqVZa7VEEhSAo7oKAWrtcx2Nr1Z/2bfXUI1W7vfRFDG7gjrYuVEUUkEUhsi+B+f1BoUqEBDOZmZD7\nc139IzqZuYXy5Zknz9yPiGEYBoQQ8hwDvgsghAgPBQMhRAkFAyFECQUDIUQJBQMhRAkFAyFEiSEb\nJ9m/fz+OHDkCkUgEFxcXxMTEwNjYmI1TE0J4oPGIIT8/HwcPHsTRo0dx4sQJ1NXV4aeffmKjNkII\nT1gZMdTX16OyshIGBgaoqqpCly5d2DgtIYQnGgeDra0tFixYgNGjR6NDhw4YMWIEvLy82KiNEMIT\njW8lSkpKkJSUhOTkZJw+fRoVFRU4ceIEG7URQniicTCcO3cO3bt3h6WlJcRiMcaNG4erV6+2+h56\nPIMQYdP4VsLBwQHp6emorq6GsbExUlNT4e7u3up7RCIRCgtLNb00q2xsLKgmNQmxLqpJPTY2Fmod\np3EweHh4YMKECZg+fToMDQ3h5uaGOXPmaHpaQgiPWPlUYunSpVi6dCkbpyKECACtfCSEKKFgIIQo\noWAghCihYCCEKKFgIIQooWAghCihYCCEKKFgIIQooWAghCihYCCEKKFgIIQooWAghCihYCCEKKFg\nIIQooWAghCihYCCEKKFgIIQoYaWDEyFsKiqSIzw8GVlZHeHo+Axxcb6QSCxbPTY3VwIHh6JWjyXq\no2AgghMenoxjx+YBEOHaNQbAQXz5ZYDKY4HWjyXqY+VWorS0FEFBQZg4cSImT56M9PR0Nk5L9FRW\nVkc0/KADgOjP15ofy7aiIjkCAxMwfnwSAgOPorhYztm1tY2VEcOWLVvg4+ODTz/9FAqFAlVVVWyc\nlugpR8dnf44UGkYBjo4lrBzLtraMbHSNxsFQVlaGS5cuYevWrQ0nNDSEubm5xoUR/RUX5wvg4J9z\nDCWIixuj8tiGOYbiVo9lG5+jFW3TOBiys7MhkUgQERGB27dvY8CAAVi9ejVMTU3ZqI/ogLZMFqpD\nIrFU6zfv89d1cSnFpk1jOJ145HO0om0aB4NCocDNmzexdu1auLu7Y8uWLfjiiy8QFBTERn1EB/A1\npG5+3epqbofybRnZ6BqNg8HOzg52dnZN29JNmDABe/bsUfk+dbfK4hLVpL7n68rNleD5IXVuroST\nuvm6biMbGwt8//18lcfoIo2DoXPnzrC3t0dmZiacnJyQmpoKZ2dnle8T4p5+VJN6mtfl4FCEho8K\nG4bUDg7FnNTN13XVJcTvH2d7VwLAmjVrEBoaCoVCge7duyMmJoaN0xIdwdeQ+vnrurhUYtOm9jOU\n55uI4WlPeiEmKdWkHiHWRTWpR90RAz0rQQhRQsFACFFCwUAIUULBQPTGvXtZ8PT8DI6OCfD0/BSZ\nmVkv/P2Je8ew+Nf3UVL9jKcKhYOeriR6Y+bM48jNjQAgQmUlg4CAGFy7tgwAcPxuAj67uhNWplYQ\nG9CPBY0YiN4oLu6G5xdENbwGfrh3HJ9d3QlLE0vE++yEmZEZbzUKBQUD0RsSySM0LIgCAAYSSTZ+\nyfwJn1zZjk5/hkKPjo58ligYNGYieiMhYRoCAmJQXNwNEkk2Vu7uhB2XYtHRuCPifHagZycnvksU\nDAoGojecnByb5hSSshIRe2ELzI0tEOezA706qV7Gr0/oVoLoneSHSYi7GA0zI3PEeu+As2UfvksS\nHAoGoldSsk9h64VNMDXsgK3e29FH4sJ3SYJEwUD0xpmcFESnboCJ2BQxo7ahr1U/vksSLAoGohfO\n557FltT1MBIbI3pUPNys+/NdkqBRMJB2L+1xKjaeXwuxyBBbRsZiQGd3vksSPAoG0q5dzEvDhnNr\nYCAywJaRsfCw8eS7JJ1AwUDarSv5l7Du7GoAwKYRMRjY5XWeK9IdtI6BtEvpBVcRdTYCDBhsHBGN\nQbZD+C5Jp1AwEJWat4ffu3caALHax3O9n+SZe2ew6L/LUVUjwqAns9Hbty9n124vKBiISs3btC9Z\ncgiffz5F7eO53KHptyc3sOi/wSiSWwG/xuFM1ihIy9vPDlFcYW2Oob6+HgEBAVi8eDFbpyQC0XzH\npczM1nca42uHpptPf0fk6TBU1TDA/8UAWd6cXr89YS0YDhw4oFbbeKJ7HB2f4fmnEp2cytp0PBc7\nNN0svImI06GorquC59MA4MFoTq/f3rByK5GXlweZTIbFixdj3759bJySCEjz9vC7dk1FXZ36x2u7\nnfwfxRlYfT4UVYpKrBoWBc+xgyEta587RHGFlWCIjo6GVCpFaamwWmUTdjTfS9LKqvW26OruPcmG\ne/I/EJ6yAlVMBaRDIzGmx1gAoDkFDWkcDKdOnULnzp3h6uqKtLQ0td8nxK27qCb1CaGuu0V3sfp8\nGKqYCqzzWYcpLi1PiPJFCF+nV6HxhjM7duzA8ePHIRaLUV1djfLycowbNw5xcXGtvk+IG3FQTeoR\nQl0PnmUiVBaMZ9VyrBgixfxhb/FeU3NC+Do1p25QsboT1YULF7B37178+9//VnmsEL9gVJN6+K7r\nYUkWVp4KgrxajuWDVmKK81Tea3oZodakDloSTXRKdukjhMmCIa+WY9nrwZjiPJXvktolVhc4DRs2\nDMOGDWPzlIQ0yS3LQZgsGEVVRfjIcxmm9qYJRm2hlY+Ed+osoc4rf4zQU8vxpPIJPhz4EQL6zOKp\nWv1AwUB4p2oJdX55HkJPLUdhZSEWuX+IWS5zeatVX9AcA+Fda0uoCyoKECpbjvyKfCwYsAhz+73D\nS436hoKB8K6lJdSFFYUIkwUjrzwP89zewzuu83irUd/QrQRRqa2PXbfVy5ZQP618ijBZMHLLcvCO\n6zzMc3uPtesR1SgYiEptfey6JS1NMjZfQl1cVYSwU8HIKcvGW/3ewXv934dIJGrlzIRtFAxEpbY+\ndt0Sdfo0yKuKESYLwaPSh5jlMgcLB3xAocADmmMgKrX1seuWqOrT8KxaDmlKCLJKHiCgzyx84PER\nhQJPaMRAVGrrY9ctcXR89udIQYTmfRJKa0oQnrICmc8yMbV3AJYMXEqhwCMKBqJSWx+7bklLfRpK\na0ogla3APfk9TOk1FUs9l1Mo8IyCgXDmZX0aymrLsColFHflf2Ci02QsGxRCoSAANMdAeFNeW47I\n02HIKL6D8T3fRPDgUBiI6H9JIaDvAuFFRW0FVp+W4tbTmxjrOA4rh4RTKAgIfScI5yoVlYg6uwq/\nP/0No7v7Qjo0kkJBYGiOQU/xtSlMlaIKUWcicL0wHd7dRmPVsDUUCgJEwaCn+NgUprquGmvPRiC9\n8CpGdB2FiOFREBuwt7SasIeiWk9xvSlMTV0NNpxbg6sFV/CGgxdWD18HQwP6vSRUFAx6istNYWrq\narDxfBQu5l3AMLvhiHpjA4zERlq7HtEcRbae4mpTmNq6WmxOXY+0x6kYYjcU67w2w1hsrJVrEfZo\nHAx5eXmQSqV48uQJxGIxZs+ejfnz57NRG9EiLjaFUdQrEJ22Eedzz+L1LoOw3msLhYKO0DgYxGIx\nIiIi4OrqivLycsyYMQMjRoygfSz1VOOnHQ+yzFD+t70wcX2EwXZDsXFEDEzEJnyXR9Sk8RyDjY0N\nXF1dAQBmZmZwdnZGQUGBxoURYSoqkmPu3G8xfnwSAgOPorhY/sLfh4cn49jxd5FudQV36/JRlmGF\nTSNjYGpoylPF5FWwOseQnZ2N27dvw8PDg83TEgFR9THngyxzYPQGoPf/AnkD0SlnPDoYduCtXvJq\nWAuG8vJyBAUFITIyEmZmZiqPF+KeflSTarm5Ejz/MWdurqSpxnqmHgqv/wEUWUC+O/DzJ+gXcIKz\nf4PQvlaAMGtSByvBoFAoEBQUhGnTpsHPz0+t9whx6y6qSTUHhyI0fMzZ0FPBwaEYd+48gjQ8CRdM\nfkVFj8twqLSFdfZ49Jp0FJs2jeHk3yDEr5VQa1IHK8EQGRmJ3r174x//+AcbpyMCFhfnCxOTQ8jI\n6ND0MWeY9CSOP80F3O4A90diJDMK+38W3s7TRH0aB8Ply5dx4sQJuLi4YPr06RCJRAgJCYG3tzcb\n9RGBkUgscfjw202/CRmGwUWjk4Db78BTF+DHfyHX9STPVRJNaRwMgwcPxq1bt9iohegYhmGwK/1z\nVPRKAe4PB378F1BtodVVlIQbtPKRvBKGYfDl9V1I+OMI/ta3P0R3vfG43ymtrqIk3KFgIG3GMAz2\n/vYFvss4jO4WPbB99CeQ+FvxXRZhEQUDUaKqV8Puy7tx6PY36GreDfE+OyExpVBobygYiJLWFjEd\nvLkf39z5f7A3d0C8z05Yd7DmtVaiHRQMeqylkUFLvRq+uXUQB37fBzszWzDHJ+Ddj69rtftTY333\n7r2GoqI7sLLqCWdnBWfdpvQZBYMea2lk8LKNYQ7f/gb7ftsD29dsYZzojxP/XaL0Pm3WBzDIzT2E\n336br7Xrkb9QMOixlkYGzXs1jPxIjj039sOmgw3ifXZi0b8yXvo+bdcHmGv1euQv1MFJj7XUxamx\nV0Ni4lhMktbh67v7Yd3BGvE+O2Fv7gAnp9KXvk/b9QFlWr0e+QuNGPSYqi5Ox+8m4F/XPoPEVIJt\nPp+gq0U3AMCuXZNQXa397k+N9d2//xqePs2AlZUjnJ0P0joJDogYhmFUH8Y+IT5cou2a2tqy/WU1\ncdX2/Yd7x/HJle2wNLHE9tGfokdHx1br4hvVpB5OH6Ii6mGjZTsXbd9/yfwJn1zZjk4mloj32flC\nKBD9QHMMHGKjZbu2274nPvgZOy7FoqNxR8T57EDPTk6snp/oBgoGDrHRsl2bbd+TshKx7eJWmBtb\nIM5nB3p1or6d+opuJTjERst2bbV9T36YhLiL0TAzMkes9w44W/Zh5bxEN1EwcIiNlu3aaPuekn0K\nWy9sgqlhB2z13o4+EhdWz090D91K6LmzOacRnboBJmJTxIzahr5W/fguiQgABYMeO597FptT18FI\nbIzoUfFws+7Pd0lEICgY9FTa41RsPL8WYpEhtoyMxYDO7nyXRASE5hh02KsudrqYl4YN59bAQGSA\nzSO3wsPGk4NqiS5hJRhSUlIQHR0NhmEwc+ZMfPDBB2yclqjwKoudruRfwrqzqwEAm0bEwLPLIO0X\nSnSOxsFQX1+PTZs2Yf/+/ejSpQtmzZqFsWPH0t6VHGjrYqf0gquIOhsBBgw2jojGINshr3zt5qOV\niIjBiIm5ojR64WoJN2GXxsFw/fp1ODo6omvXrgCAyZMnIykpiYKBAy/rm9CSG4XpWH0mHPVMHdZ7\nRWOo3XCNrt18tHLxYgxycyPQfPTCxRJuwj6NgyE/Px/29vZNr21tbXHjxg1NT0vUoO5ip9+e3EDk\nGSkU9bVY57UZw+3f0PjazUcrxcXd8LLRi7aXcBPt0DgYXvXhTCHu6adrNdnYWOD77+e3+v4b+Tew\nLnUV6kUKbHszHmOc2Fkp6eJS8cJoxdo6B9nZf712camEjY2F0nGNf64Nuvb9EzKNg8HOzg65ublN\nr/Pz89GlSxeV7xPi46jtraY7RbchTQlBlaISkcPXYYD5EFb+jTY2Fti0adQLPRkiI/0RHf3X68Y9\nK5sfp629LNvj908bOHvs2t3dHQ8fPkROTg5sbGzw448/YseOHZqelmjoj+IMrEpZiSpFJVYNi4JP\nd3abm7xsafaXXyo/nq2NJdxE+zQOBrFYjKioKCxcuBAMw2DWrFk08cize/I/EJ6yAuW1ZZAOi8SY\nHmP5LonoGFbWMXh7e9MmtgJxNesa3j20GOWKGngUTcFgP80+fSD6iZZEtyMPnmXi798uwZPSDqj8\nZRvSDu6AVJrMd1lEB1EwtBMPS7IQJgtGWV0VcDoSuDMN2vh4sKhIjrlzv8X48UkIDDyK4mI5q+cn\nwkDPSrQD2aWPECYLhrxajv7yCbh0q3Gyj/1W67RgST9QMOi43LIchMmCUVRVhCWeSzHGbxykcu21\ndqcFS/qBgkGH5ZU/Ruip5XhS+QQfDvwIM/rMBgCt/gZvyzJsorsoGHRUfnkeQk8tR2FlId53/wCz\nXOZyct24OF+YmBxCRkYHrW42Q/hFwaCDCioKECpbjvyKfCwYsAhv9XuXs2tLJJY4fPhtwa3oI+yi\nTyV0TGFFIcJkwcgrz8M8t/fwjus8vksi7RCNGLSI7V4ETyufIkwWjNyyHLzt+nfMc3uPvWIJeQ4F\ngxax+dFecVURwmTByCnLxty+b2NB/0UQiUSq30jIK6BbCS1i66M9eVUxwmQheFT6ELNc5uB99w8p\nFIhWUTBoERvbyT2rlkOaEoKskgcI6DMLH3h8pJVQKCqSIzAwgVY0EgB0K6FVmm4nV1JdgvCUFch8\nlompvQOwZOBSrY0UaEUjeR4FgxZp0ougrKYUK34Kxz35PUzu5Y9/egZp9faBVjSS59GthACV1ZZh\n1elQ3Cq8hTedJiFo0AoYiLT7rdLmLtpE99CIQWDKa8sReToMd4puY7rbVCxxC9F6KADa20Wb6CYK\nBgGpqK3A6tNS3Hp6E2N7+GGtz1o8fVLOybWpBRt5Ht1KCESlohJRZ1fh96e/YXR3X4QNjeRkpEDI\ny9D/eQJQpahC1JkIXC9Mh3e30Vg1bA3EBmK+yyJ6TKNbibi4OCQnJ8PY2Bg9evRATEwMzM3N2apN\nL1TXVWPt2QikF17FiK6jEDE8ikKB8E6jEcPIkSPx448/4tixY3B0dMTu3bvZqkvrmi/oKSrifkFP\nTV0NNpxbg6sFV/CGgxdWD18HQwOa9iH80ygYvLy8YGDQcApPT0/k5eWxUhQXGhf0XLs2HceOzceS\nJT9zev2auhpsPB+Fi3kXMMxuOKLe2AAjsRGnNRDSEtbmGI4cOaJTLeSbL+jJzOTuFqi2rhabU9cj\n7XEqhtgNxTqvzTAWG3N2fUJUUTluXbBgAZ48eaL05yEhIfD19QUA7Nq1C0ZGRvD391f7wnzv6dd8\nT0UnpzJOalLUKxCZtAkXC87Dy/ENfDzhY5gYmrR4PN9fp5YIsS6qiT0i5lV3pf1TQkICDh8+jAMH\nDsDYWP3fenx3ACoulkMqTW5a0LN371TU1Wl30q+uvg7RaRuRkn0KHjae2DIyFqaGpi0eL8S9DwFh\n1kU1qYeTvStTUlKwZ88efP31120KBSFovqDHykq738R6ph6xF7YgJfsU3Dt7YPPIra2GAiF80igY\nNm/ejNraWixcuBAAMHDgQKxfv56NutqVeqYecRejkfwoCW7W/bF5ZCw6GHbguyxCWqRRMCQmJrJV\nR7tVz9Rj+6VYJGX9in5WrogeFY/XjF7juyxCWkUrH7WonqnHzsvbkPjgF7hI+iLGexvMjMz4LosQ\nlSgYtIRhGHx25WP8nPkjelv2wVbvbTA3olWhRDfQMjstYBgGn1/7BD/cPw5nS2fE+exAbVk9AsMT\nWOsYrS62O1UT/UDBwDKGYbAr/XMcv5sAp05OiPXeAQvjjggMT+CldRq1bCOvgm4lWMQwDL68vgsJ\nfxyBY8eeiPXegU4mDb+d+Wqd1vy6iYmgZq9EJRoxaKhxqP4gywL1g4+ifmA6nCx7Id7nY0hMrZqO\n42sz2ObXraw0wrFjb4FGDqQ1FAwaahqqD9kNiG/B/r4xjoTsfCEUAP5apzVeNzERqKw0AjAR1OyV\nqELBoIbWJvCysjoCg/Y0/FfSDVbXpsM60lrpHHy1Tmu8bmDg0T9HCrR9PVGNgkENrU7gvX4CMEoH\nSh2AE/9Gbz9hLvqiZq+kLSgY1NDSxOHh29+g9vWLsM00QOfr09HbL1GwP3DU7JW0BQWDGl42cXgk\n4zD23NgNO3M7fLPsM9hHOPBdJiGsoWBoRePcwr17hnBwiIG1tQt69aqAzz9LsTv9K1h3sMY2n09g\nb06hQNoXCoZWPD+3ADAYOvQg/MOBz65+BYmpBNt8PkFXi26vdG5akUiEjIKhFc3nFtJrfseDq7/C\n0sQS23w+QTeL7q98blqRSISMVj624oX9HPseQ7H7f9DRpBPifXaiR0dHjc5Nm8gSIaMRQysaP+K7\nXnUHxR7fop+TA+K9d6BnJyeNz83XSkhC1EHB0AqJxBJvrTFD1oWf0dW4O+J9PkYvy96snFvVuoKi\nIjmWLv0BGRkdaA6CcK7dBgMbk3vJD5MQdzEarxmZYeuo7XC27MNafarWFdAcBOFTuw0GTX+wUrJP\nYeuFTTA17ICt3tvhYtVXa7W+DM1BED6xMvn41VdfoV+/fpDLhfMob8MP0jMA3wL4ATJZntqPGp/N\nOY3o1A0wEZsiZtQ29LNy1WapL/XCxCfNQRCOaTxiyMvLw7lz5+DgIKxFPg2Tez8BeBuACHL5FEil\nqkcN53PPYnPqOhiJjRE9Kh5u1v05qbe5uDhfmJgc+nOOgZ5tINzSeMQQHR0NqVTKRi2siovzhaVl\nFdoyHE97nIqN59dCLDLElpGxGNDZXet1tkQiscThw28jMXEsvvwygCYeCac0CoaTJ0/C3t4effty\ne/+tDonEEj4+Yqg7HE/NTsWGc2tgIDLA5pFb4WHjyUmdhAjRK+9dGRwcjN27d2Pv3r1Nf9aW3e64\n2NNv795pWLLkEDIzzeHkVIZdu6bCykr5uhdyLmDF/66AkZEYO9/ciWFdh2m9NnUJde9DIdZFNbHn\nlfeuzMjIwIIFC2BqagqGYZCfnw9bW1t89913sLZWblTSnKrt4Lh6liC94Coiz0ghNhQhatgmDLUb\nzvo1XpUQ9z4EhFkX1aQere9d6eLigrNnzza99vX1RUJCAjp16vSqp3wBF5/j3yhMx+oz4ahn6rBj\n3Mfo22Egq+cnRFex9qyESCRq062EKtr+HP/3J78h8owUivparP3bJozsMZLV8xOiy1gLhqSkJFha\nsjfU1+bn+Lee3kTE6VDU1FVj9Rvr8TeHEaydm5D2QLArH7XVo/BO0W2sOr0S1XVViBy+DqO6+bBy\nXkLaE8EGgzZ6FP5RnIFVKStRpahE+LA18OlOi4YIeRm96cdwT/4HwlNWoLy2DGFDI+Dbw4/vkggR\nLL0IhvvP7kEqW4GymlKsHLoKfo4T+C6JEEFr98Hw4FkmpLIVKKkpQfDgMEzoOZHvkggRvHYdDA9L\nshAmC8azajmWD1qJSb2m8F0SITpBsJOPz3uVVZDZpY8QJguGvFqOpa8vxxTnqRxVS4ju04lgaOsq\nyNyyHITJglFUVYQlnksxrfcMzmrVBmo1T7imE8HQllWQeeWPEXpqOZ5UPsEHHkswo89sTmrUJmrz\nRrimE3MM6q6CzC/PQ+ip5SisLMT77h9gdt+3OKtRm6jNG+GaTowY1FkFWVBRgDBZMPIr8vFe//fx\nVr93uS9US5q3mreze4LAwAS6tSBaoxPBoGoVZGFFIcJkwXhc/hjz3N7Du27zm/6uPdyfNw/Gmppa\nHDv2PujWgmiLTgRDa55WPkWYLBi5ZTl42/XvmOf23gt/3x7uz5sH4/jxSaBbC6JNOjHH0JLiqiKE\nyYKRU5aNuX3fxoL+iyASiV44pvn9eWIiEBh4VO2O0UJEHaSJtunsiKGitgJhshA8Kn2ImS6z8b77\nh0qhACjfn1dWGuHYsbegiyOHRtp68pSQRjobDNmlj5BV8gAz+szGhx7/fGkoAH/9ECUmApWVRgAm\nQteH39p48pSQ5+nsrYSLVV8kTPsBSzyXthgKwF8/ROPHMwDeAmAJGn4T0jqdHTEAgLmx+h14afhN\niPp0OhjagobfhKhP41uJgwcP4s0334S/vz+2bdvGRk2EEJ5pNGJIS0tDcnIyfvjhBxgaGqKoqIit\nugghPNJoxPDtt98iMDAQhoYN+WJlZcVKUYQQfmkUDA8ePMClS5cwZ84czJs3Dzdu3GCrLkIIjzTa\nu7Kurg4lJSX4z3/+g+vXryM4OBhJSUlqXViIe/pRTeoTYl1UE3tUBsO+ffta/LtDhw5h/PjxAAAP\nDw8YGBiguLgYEolE5YWFuKcf1aQeIdZFNalH3aDS6FbCz88P58+fBwBkZmZCoVCoFQqEEGHT6FOJ\nGTNmIDIyEv7+/jAyMkJsbCxbdRFCeKRRMBgZGSE+Pp6tWgghAqGzz0oQQrSHgoEQooSCgRCihIKB\nEKKEgoEQokRvHrtuD9pDx2uiGygYdEh76HhNdAPdSugQ2pGKcIWCQYdQ23jCFbqV0CHUt5JwhYJB\nh1DfSsIVupUghCihYCCEKKFgIIQooWAghCihYCCEKKFgIIQooWAghCjRKBhu376NuXPnYvr06Zg1\naxbtK0FIO6FRMMTHx2PZsmX4/vvvsWzZMsTFxbFVFyGERxoFg0gkQmlpQ9/80tJS2NraslIUIYRf\nGi2JjoiIwKJFixAbGwuGYXDo0CG26iKE8OiVt6gLCQnBuXPnsHr1avj5+eGXX35BZGRkqztXEUJ0\ng4hhGEb1YS83ZMgQXLp0qen14MGDcfnyZVYKI4TwR6M5BltbW1y4cAEAcP78efTs2ZONmgghPNNo\nxHDlyhVs3rwZ9fX1MDExwbp16+Dm5sZmfYQQHmgUDISQ9olWPhJClFAwEEKUUDAQQpTwHgxfffUV\n+vXrB7lczncpiIuLw8SJEzFt2jQsW7YMZWVlvNWSkpKCN998ExMmTMAXX3zBWx2N8vLyMH/+fEya\nNAn+/v44cOAA3yU1qa+vR0BAABYvXsx3KU1KS0sRFBSEiRMnYvLkyUhPT+e7JOzfvx9TpkyBv78/\nVq5ciZqampYPZnj0+PFjZuHChcyYMWOY4uJiPkthGIZhzp49y9TV1TEMwzDx8fHMtm3beKmjrq6O\n8fPzY7Kzs5mamhpm6tSpzN27d3mppVFBQQFz8+ZNhmEYpqysjBk/fjzvNTXat28fs3LlSubDDz/k\nu5Qm4eHhzJEjRxiGYZja2lqmtLSU13ry8vIYX19fprq6mmEYhlm+fDmTkJDQ4vG8jhiio6MhlUr5\nLOEFXl7UuJp4AAADF0lEQVReMDBo+JJ4enoiLy+PlzquX78OR0dHdO3aFUZGRpg8eTKSkpJ4qaWR\njY0NXF1dAQBmZmZwdnZGQUEBrzUBDSMZmUyG2bNn811Kk7KyMly6dAkzZ84EABgaGsLc3JznqhpG\nVpWVlVAoFKiqqkKXLl1aPJa3YDh58iTs7e3Rt29fvkpo1ZEjR+Dt7c3LtfPz82Fvb9/02tbWVhA/\nhI2ys7Nx+/ZteHh48F1K0y8XkUik+mCOZGdnQyKRICIiAgEBAYiKikJVVRWvNdna2mLBggUYPXo0\nvL29YWFhAS8vrxaP1+q+Ei09ZxEcHIzdu3dj7969TX/GcLScorVnP3x9fQEAu3btgpGREfz9/Tmp\nqTmuvhavory8HEFBQYiMjISZmRmvtZw6dQqdO3eGq6sr0tLSeK3leQqFAjdv3sTatWvh7u6OLVu2\n4IsvvkBQUBBvNZWUlCApKQnJycmwsLBAUFAQTpw40eL/41oNhpYeqMrIyEBOTg6mTZsGhmGQn5+P\nmTNn4rvvvoO1tbU2S1L5kFdCQgJkMhmvk2t2dnbIzc1tep2fn9/qsI8rCoUCQUFBmDZtGvz8/Pgu\nB1euXMHJkychk8lQXV2N8vJySKVS3vuC2NnZwc7ODu7u7gCACRMmYM+ePbzWdO7cOXTv3h2Wlg27\no48bNw5Xr15t+ZcfJzMfKowZM4aRy+V8l8HIZDJm0qRJTFFREa91KBSKpsnH6upqQUw+MgzDhIWF\nMdHR0XyX8VJpaWmCmnx89913mfv37zMMwzCfffYZExcXx2s96enpzJQpU5iqqiqmvr6eCQ8PZ77+\n+usWjxfEFnUikUgQw+fNmzejtrYWCxcuBAAMHDgQ69ev57wOsViMqKgoLFy4EAzDYNasWXB2dua8\njuddvnwZJ06cgIuLC6ZPnw6RSISQkBDe5mGEbs2aNQgNDYVCoUD37t0RExPDaz0eHh6YMGECpk+f\nDkNDQ7i5uWHOnDktHk/PShBClPC+wIkQIjwUDIQQJRQMhBAlFAyEECUUDIQQJRQMhBAlFAyEECUU\nDIQQJf8fkI+nnJy5GlIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xa5aa650>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#@test {\"output\": \"ignore\"}\n",
    "weights = np.polyfit(X[0], X[1], 1)\n",
    "plt.figure(figsize=(4,4))\n",
    "plt.scatter(X[0], X[1])\n",
    "line_x_range = (-3, 5)\n",
    "plt.plot(line_x_range, [weights[1] + a * weights[0] for a in line_x_range], \"g\", alpha=0.8)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VYUr2uPA9ah8"
   },
   "source": [
    "Remember that our simple network looks like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "cellView": "form",
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "output_extras": [
      {
       "item_id": 1
      }
     ]
    },
    "colab_type": "code",
    "collapsed": false,
    "executionInfo": {
     "elapsed": 898,
     "status": "ok",
     "timestamp": 1474671837740,
     "user": {
      "displayName": "",
      "photoUrl": "",
      "userId": ""
     },
     "user_tz": 420
    },
    "id": "gt8UuSQA9frA",
    "outputId": "6eb7616b-25a9-4845-aeab-7472201c60f6"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAJYAAABkCAYAAABkW8nwAAAO90lEQVR4Xu2dT5Dc1J3Hv+YQT8VJ\nZUhVdprLWs4FTSrGGv4ql9CuHBCH4GaTFCLZwnIcjOAy8l6Q/1SlU4XHcg6xJgtY2OOik2KxSGoT\nGWrXzYFC2T2MDAtWitRavmQ0e9k2SYGowom4hNRPtqA9TE+rW3/cPfPepcfup6f3fu/Tv9/T+/PV\npo8//vhjsMQsULAFNjGwCrYoKy6xAAOLgVCKBRhYpZiVFcrAYgyUYgEGVilmZYUysBgDpViAgVWK\nWVmhDCzGQCkWGEuwrly5gtf++zW887/vYOn/lnD5T5cT40x9ZQrb/nEbxDtFiHeI2LJlSylGY4X2\nt8BYgUVAvfzqy3i5/TI+vPLhmq37wpYv4AHpATxw3wMMsP4cFJ5jbMAiqA4eOYg/Lv8xMcL26e34\n+vTXk8+vbv1q8n/03TsX38EfLv4h+aRE380dmmNwFY7O2gWOBVgE1Y/2/yjxUls+vwXaY1oS7tZK\n3v94MJ8zceUvV0Dea+H4AoOrQrhGHqxuT0Xjp0P7D2HqH6Yymejyu5dx5PiRZBxGnmt+bj7TdSxT\nfgv0ASuAzglwmyE8pfbZu3VaEDkDdT+AweevzGolvPjvL+LMb84knmr+yHxmqNKyCK7ZQ7OJ5yIo\n+3m6clqx8UrNB1bso2W64FQN9cnijdcdAvNAQWGRPBcLicX3Ua8S84FVcj3PnjuLhRcWkgH63OG5\nXHc7+NTBZEBP47NvffNbucpiF/e3QCaw2g0NfNvES5c+wtQ9u2G0LCj8BLAiFEaeBU0zYJ9fxkfY\njKl7FZgtCzIHIA7QUmXov/g9LmMztt6rwLBMyFROj3TkZ0fgveXh4X96GN//zvf7t2aNHGlI7VlW\n0pYmRC+AKUwAsQu5thOuvIjQEjGBGJ7CQYptdOw6etc6VzXXzcUZwJrGseWt2P28DV2I4OgyDgQK\nFgMTYtQ1xqq10eDuR6j8Fi1NxGTkwpAfRos7h05bQscQIFgibEeHMBHCVhs4EBtY8lQQd6ulvbN7\n8e6f302mC7Z/bXsuo9NkKk1X9PZ+IUyeR0sN4GscYl8DPzOP5VuPYynQwMU+dL4O3wzRbpQQ93O1\nbvQuzgRWS0p/tQA6Nuqcilq7A5u3Px28T7qw7BB1VUHqhEKTB2+pCAIVHZVD3dPgujpE6peOBzes\nQRS5nr/+b//g24nF7JN27qkCGq/J++RknHXm5JlVeiKGr/MQPQMdV0ZkCRBbNUwEMYzQhRyZEHgH\nOv29ynPM6HXtja1Rf7B4AZ7RgZv+SuMAOj+NtrYEX3avfyqMfDi2DdcLEAQBvPOX8MGtR3Ex0MEF\nJiRxP373wWZsvaeBhixDVRrg1/jxlwEWPV3ap+xVrR57Cjgpht2xEDV4mLIFvqkiaoUwwzp4U4Hv\n9/awN7YrR+vuGcAS4ZsdtKV0VNEFVqMLrIkWJGEPPP4hKA0RgiCAc1XsdJQErGQ2Ig7hOQ5sx4Hz\n0u+wvHX2akjtMWCpNhQCiCicq+AcCx1Fh9B2IegcNN6B4Teg1z0EeknzKqPFRe7a9AeLm4ajXvzU\noJEDqUahMESrKxSqbQHbDBGLoXUNlBiuUsNOT8fFQEVsNdHmdOjStTgSGOCnLTQuBDBosLxKqnTw\nntw/glPnoHMS4E6iFVjgbBGcwUGMPAjtawP73GZf/wVkAutYtAvPezYUPoKjipBdGZ5vQOgavGte\nHbfsiXD09TZUIUbg6JD3vITlrU/iYthErPOYaQk44ZhocDF8U0HDqsEOHfQaC7/2X68lyzJVTjd0\nWiJu2XMem++7+tAxSd52+hguTe3GYtjq6V3XPyqDtbA/WLyAtqRg0rHhLceo3avCsk0kjqd7uoEL\n0FJkaC/9Hh/gS9ixS0dTCaDKHVidNhoTNN2gQP/FedAmly/t2IWm2YK2xswqDbj3antzz5oToD/9\n15/i5smbcdo8vfaDQGiC37YfEyeW4KtcMu2g1HbCrp9Dx5Fw3ZCw04ZSb0Jse6CsLH1qgZFfK0zn\nn+hpznzKHGpJRzus4YJ/AX/78G94ofUC7r777pwMxAhdE6pyAK8u78CJJZ+BtcKiIw8Wea0DTx34\nZCH5oHYwM1y0TjhnziXbaWgB+4cP/RCPPfYYtm/fjpMnT+Kmm24aDrDYhdpoQdAbaMtNSB4Da6Uh\nRx4sqnB3SCTPNbtvtu9iMoU/Wg5Kt9p0h8DTp09j3759ePrpp/H4448PB1fylOtC5jTUGVifseFY\ngJXClXou+jcN6Gk2nj7JG1Gi7TG0Hkiz7OlGP/ru6OGjq46rnnjiCSwuLibe66677hocMAZWT5uN\nDVgpXGfbZ5OtybQNZq1EE6G0NXmXtGvNwbrv+4n3uu222wYPjwys9QFW2goKjbQ4Tdth6CAFeSpK\n5J3oQMUwhynS8PjMM89AVdVs3ouBtb7Aytbrw+WiMZfnednCIwOLgTUIZml43LFjB5577rnhnx4H\nuek6yztWY6yqbb+wsJBMTwwUHquu5Ijej4GVoWMoPJ4/fz7xXkM9PWa4x3rLwsDK2KMXLlxIvBeF\nR5qe2LRpU8YrN2Y2BtaA/U7hkaYnnn322exPjwPeYz1kZ2AN2YtpeCTvdeeddw5Zyvq9jIGVo28p\nPJL3ok2NLDxeb0gGVg6w0kvT8HjixIlkHJY1lauaE8GRangwsvD/noKqt+kzsLJSkCEfzdi/8cYb\nifdaKzxWoppDmxJ5FT54NH06YZShAQVmYWAVaEwqKg2PMzMzyfTEyqfHqlRzAoOH6OqwJnXoNQeB\nSWcjq0sMrJJsferUqSQsdofHylRzYg8aLyG0QtiTOvhGhFZglyKD0Mt8DKySwEqLpfD45ptvYn5+\nHr/+z19/sukwj2pOP72vyJXBy4BNME340Pg6AiNAu8IDkQysksGi4t9++2189wffxee++DkIO4Tc\nqjlrSw504Eg81FobYetq+KOwKDgagjVOnRdtBgZW0RZdpbw0BL73/nv4yZM/6bv7tVeVxkk1h4FV\nAVgbUTWHgVUBWGUcvCVV6EP/cuiztQ9NCNsMiIshrPSIeaK3oUNIlXQqaDMDqwIjlyEV0Fv6MoQl\nbENT/FTIhWSXOF2AF5jocei8cCswsAo36WcLLEPchO7yyr+9smrt6TQ3geQmcgcd2CQbIHoIDKGy\nuSwG1joEi06oU+jj3RAWR2HQgFiiTuxqJmRgVQBWGaGQDo78/OjPe9T+qpfSeBeeqIM3JPip4k8F\n7aVbMLAqMHSlg/dr7YkcCZxWg1Jz0G5UL7/EwKoArBuhmoNEbupBvPrRDhxf8qFVLFrCwKoArFQi\n4P3o/VwTpCmgdBi3r2oOIrQbNdwfGljytZ46r2U1n4FVlmW7yn3rrbfwvX/+XrKkMyPM5FLNIS2K\nbCrSNI8loKX48G6AxhIDq2SwaIcDgWWaJn71H78qRDWnlxbF1aaQxJILj6TRjRhm0L4hYrwMrJLA\nos1+BBXtyaLty5SKVs1Zverx1RB4dhIPPe/CVioeXF2rFAOrYLDIOxFQd9xxRwLVytSt90XfFaGa\nU3ATCimOgVWIGa8WkoY9AorA6pUIrqJVcwpsRiFFMbAKMONqYS9LsWWo5mS5bxV5GFg5rExhj8ZP\ndHBitbCXo+ixv5SBNWQXpmGPvNXtt98+ZCnr9zIG1oB9O2zYG/A2Y5+dgZWxC1nYy2goNt2Q3VA0\njqIDESzsZbcZ81hr2CoNe/T56KOPZrcqy8m2zazGAAt7+X8ZzGOtsCELe/mhohLGEqwyVFpY2CsG\nqLSUsQKrDJUWFvaKBWrswCpDpYWFvXKgKiYUxh5U/huwhd8idBqYRARX4bHTldd8Le8gTSpapYWW\nX0is47qnveTdi02I6aFOejlAbSdcOT2fF8NTOEixDTqnV6Uk0CC2GpW8hYTCyFXA72yj8XoAAzoE\n+nsxgNnrZc8DtL7bU9HJlDwqLY9855FkbY8ktS3LWlGLECbPo6UG8DUOsa+Bn5nH8q3HsRRo4GIS\nL6vDN0O0e70SdoB2rfeshYBF71Juyzzu90TcF59FIC8WJvSVvgiT9nnPH5nP/K7CtOPonYWzh2aT\nF2Fu+usmvPjLF3us7cXwdR6iZ6DjyogsAWKrhokghhG6kCMTAu9Ap7+r1l0cQwoLAote4+ugwT+I\nsxO78XrQKkTkqzsEkqeily8Nk0il5cfHfowv3/xlLBxf6Pk2sNhTwEkx7I6FqMHDlC3wTRVRK4QZ\n1sGbCnxfrfxgwjBtvtHXFAZW7OsQZo7hEm7Fkxf8nm+mH6TBlau0RG00OBWcY6Gj6BDaLgSdDn46\nMPwG9Hr15/MGsdco5S0GrDiAIU7D5M/AgIo9gY6Lng4+5wi3jIOea59wieCQzgEnAe4kWoEFzhbB\nGRzEyIPQDmBWpaoxSpQMUZdCwCLh1OlmDWcCBzJsSNzDiIyL8LR8Ur1lHE2nPeZzh+d6mooENW7Z\ncx6b7zuHTlvCJB1Nnz6GS1O7sUhKxDl/LEP00Vhekh8sUjThNUyYAdxr59dCSwSvAWbg5Xq7exkq\nLfRO6TMnz/TurNAEv20/Jk4swaf2xC6U2k7Y9XPoOBIm6crYh6UoaLodABOoSU3YlpLbQ48lQT0q\nnR+sEq1RBlj0dGmfsnPVOtB51IMmfEdGLQ7RkkSYkps8VbJ01QIjDdaNCIVZwOi4DnxOgsRRXIzh\nazwakY3gmphsljLWe56RBqv6wfvg3R0HFqS6CcHxC5kQHrwGo3nFSIN1Q1RaBuinyDchSyYmDRct\nhWPLPF22G2mwuo+k55kgHUylJRtZoa1A0kI0bAdGPRnSszQuYFE90yUdepoznzKHWtLRDmsglZY8\ncHZTE7UVCGqEpmtDScZZLK20wEh7LKpst9YBKQUf1A5mhovWCefMuU9eM9JbWnEQMAIY/DQOXLr+\nmqmHXkfIdj18YpSRByuFa6+2F1f+cgXkuWb3zfZdN6Twt/DCQuKpsgmVDQIXy9vPAmMB1krPRf9e\nryot/TpsXL4fG7BSuNa7Ssu4gNOvnmMFVtqY9azS0q/DxuX7sQRrXIy7kevJwNrIvV9i2xlYJRp3\nIxfNwNrIvV9i2xlYJRp3IxfNwNrIvV9i2xlYJRp3IxfNwNrIvV9i2xlYJRp3Ixf9d0NIelzdt4X5\nAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<IPython.core.display.Image at 0xbde1fd0>"
      ]
     },
     "execution_count": 6,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import Image\n",
    "import base64\n",
    "Image(data=base64.decodestring(\"iVBORw0KGgoAAAANSUhEUgAAAJYAAABkCAYAAABkW8nwAAAO90lEQVR4Xu2dT5Dc1J3Hv+YQT8VJZUhVdprLWs4FTSrGGv4ql9CuHBCH4GaTFCLZwnIcjOAy8l6Q/1SlU4XHcg6xJgtY2OOik2KxSGoTGWrXzYFC2T2MDAtWitRavmQ0e9k2SYGowom4hNRPtqA9TE+rW3/cPfPepcfup6f3fu/Tv9/T+/PVpo8//vhjsMQsULAFNjGwCrYoKy6xAAOLgVCKBRhYpZiVFcrAYgyUYgEGVilmZYUysBgDpViAgVWKWVmhDCzGQCkWGEuwrly5gtf++zW887/vYOn/lnD5T5cT40x9ZQrb/nEbxDtFiHeI2LJlSylGY4X2t8BYgUVAvfzqy3i5/TI+vPLhmq37wpYv4AHpATxw3wMMsP4cFJ5jbMAiqA4eOYg/Lv8xMcL26e34+vTXk8+vbv1q8n/03TsX38EfLv4h+aRE380dmmNwFY7O2gWOBVgE1Y/2/yjxUls+vwXaY1oS7tZK3v94MJ8zceUvV0Dea+H4AoOrQrhGHqxuT0Xjp0P7D2HqH6Yymejyu5dx5PiRZBxGnmt+bj7TdSxTfgv0ASuAzglwmyE8pfbZu3VaEDkDdT+AweevzGolvPjvL+LMb84knmr+yHxmqNKyCK7ZQ7OJ5yIo+3m6clqx8UrNB1bso2W64FQN9cnijdcdAvNAQWGRPBcLicX3Ua8S84FVcj3PnjuLhRcWkgH63OG5XHc7+NTBZEBP47NvffNbucpiF/e3QCaw2g0NfNvES5c+wtQ9u2G0LCj8BLAiFEaeBU0zYJ9fxkfYjKl7FZgtCzIHIA7QUmXov/g9LmMztt6rwLBMyFROj3TkZ0fgveXh4X96GN//zvf7t2aNHGlI7VlW0pYmRC+AKUwAsQu5thOuvIjQEjGBGJ7CQYptdOw6etc6VzXXzcUZwJrGseWt2P28DV2I4OgyDgQKFgMTYtQ1xqq10eDuR6j8Fi1NxGTkwpAfRos7h05bQscQIFgibEeHMBHCVhs4EBtY8lQQd6ulvbN78e6f302mC7Z/bXsuo9NkKk1X9PZ+IUyeR0sN4GscYl8DPzOP5VuPYynQwMU+dL4O3wzRbpQQ93O1bvQuzgRWS0p/tQA6Nuqcilq7A5u3Px28T7qw7BB1VUHqhEKTB2+pCAIVHZVD3dPgujpE6peOBzesQRS5nr/+b//g24nF7JN27qkCGq/J++RknHXm5JlVeiKGr/MQPQMdV0ZkCRBbNUwEMYzQhRyZEHgHOv29ynPM6HXtja1Rf7B4AZ7RgZv+SuMAOj+NtrYEX3avfyqMfDi2DdcLEAQBvPOX8MGtR3Ex0MEFJiRxP373wWZsvaeBhixDVRrg1/jxlwEWPV3ap+xVrR57Cjgpht2xEDV4mLIFvqkiaoUwwzp4U4Hv9/awN7YrR+vuGcAS4ZsdtKV0VNEFVqMLrIkWJGEPPP4hKA0RgiCAc1XsdJQErGQ2Ig7hOQ5sx4Hz0u+wvHX2akjtMWCpNhQCiCicq+AcCx1Fh9B2IegcNN6B4Teg1z0EeknzKqPFRe7a9AeLm4ajXvzUoJEDqUahMESrKxSqbQHbDBGLoXUNlBiuUsNOT8fFQEVsNdHmdOjStTgSGOCnLTQuBDBosLxKqnTwntw/glPnoHMS4E6iFVjgbBGcwUGMPAjtawP73GZf/wVkAutYtAvPezYUPoKjipBdGZ5vQOgavGteHbfsiXD09TZUIUbg6JD3vITlrU/iYthErPOYaQk44ZhocDF8U0HDqsEOHfQaC7/2X68lyzJVTjd0WiJu2XMem++7+tAxSd52+hguTe3GYtjq6V3XPyqDtbA/WLyAtqRg0rHhLceo3avCsk0kjqd7uoEL0FJkaC/9Hh/gS9ixS0dTCaDKHVidNhoTNN2gQP/FedAmly/t2IWm2YK2xswqDbj3antzz5oToD/915/i5smbcdo8vfaDQGiC37YfEyeW4KtcMu2g1HbCrp9Dx5Fw3ZCw04ZSb0Jse6CsLH1qgZFfK0znn+hpznzKHGpJRzus4YJ/AX/78G94ofUC7r777pwMxAhdE6pyAK8u78CJJZ+BtcKiIw8Wea0DTx34ZCH5oHYwM1y0TjhnziXbaWgB+4cP/RCPPfYYtm/fjpMnT+Kmm24aDrDYhdpoQdAbaMtNSB4Da6UhRx4sqnB3SCTPNbtvtu9iMoU/Wg5Kt9p0h8DTp09j3759ePrpp/H4448PB1fylOtC5jTUGVifseFYgJXClXou+jcN6Gk2nj7JG1Gi7TG0Hkiz7OlGP/ru6OGjq46rnnjiCSwuLibe66677hocMAZWT5uNDVgpXGfbZ5OtybQNZq1EE6G0NXmXtGvNwbrv+4n3uu222wYPjwys9QFW2goKjbQ4Tdth6CAFeSpK5J3oQMUwhynS8PjMM89AVdVs3ouBtb7Aytbrw+WiMZfnednCIwOLgTUIZml43LFjB5577rnhnx4Huek6yztWY6yqbb+wsJBMTwwUHquu5Ijej4GVoWMoPJ4/fz7xXkM9PWa4x3rLwsDK2KMXLlxIvBeFR5qe2LRpU8YrN2Y2BtaA/U7hkaYnnn322exPjwPeYz1kZ2AN2YtpeCTvdeeddw5Zyvq9jIGVo28pPJL3ok2NLDxeb0gGVg6w0kvT8HjixIlkHJY1lauaE8GRangwsvD/noKqt+kzsLJSkCEfzdi/8cYbifdaKzxWoppDmxJ5FT54NH06YZShAQVmYWAVaEwqKg2PMzMzyfTEyqfHqlRzAoOH6OqwJnXoNQeBSWcjq0sMrJJsferUqSQsdofHylRzYg8aLyG0QtiTOvhGhFZglyKD0Mt8DKySwEqLpfD45ptvYn5+Hr/+z19/sukwj2pOP72vyJXBy4BNME340Pg6AiNAu8IDkQysksGi4t9++2189wffxee++DkIO4TcqjlrSw504Eg81FobYetq+KOwKDgagjVOnRdtBgZW0RZdpbw0BL73/nv4yZM/6bv7tVeVxkk1h4FVAVgbUTWHgVUBWGUcvCVV6EP/cuiztQ9NCNsMiIshrPSIeaK3oUNIlXQqaDMDqwIjlyEV0Fv6MoQlbENT/FTIhWSXOF2AF5jocei8cCswsAo36WcLLEPchO7yyr+9smrt6TQ3geQmcgcd2CQbIHoIDKGyuSwG1joEi06oU+jj3RAWR2HQgFiiTuxqJmRgVQBWGaGQDo78/OjPe9T+qpfSeBeeqIM3JPip4k8F7aVbMLAqMHSlg/dr7YkcCZxWg1Jz0G5UL7/EwKoArBuhmoNEbupBvPrRDhxf8qFVLFrCwKoArFQi4P3o/VwTpCmgdBi3r2oOIrQbNdwfGljytZ46r2U1n4FVlmW7yn3rrbfwvX/+XrKkMyPM5FLNIS2KbCrSNI8loKX48G6AxhIDq2SwaIcDgWWaJn71H78qRDWnlxbF1aaQxJILj6TRjRhm0L4hYrwMrJLAos1+BBXtyaLty5SKVs1Zverx1RB4dhIPPe/CVioeXF2rFAOrYLDIOxFQd9xxRwLVytSt90XfFaGaU3ATCimOgVWIGa8WkoY9AorA6pUIrqJVcwpsRiFFMbAKMONqYS9LsWWo5mS5bxV5GFg5rExhj8ZPdHBitbCXo+ixv5SBNWQXpmGPvNXtt98+ZCnr9zIG1oB9O2zYG/A2Y5+dgZWxC1nYy2goNt2Q3VA0jqIDESzsZbcZ81hr2CoNe/T56KOPZrcqy8m2zazGAAt7+X8ZzGOtsCELe/mhohLGEqwyVFpY2CsGqLSUsQKrDJUWFvaKBWrswCpDpYWFvXKgKiYUxh5U/huwhd8idBqYRARX4bHTldd8Le8gTSpapYWWX0is47qnveTdi02I6aFOejlAbSdcOT2fF8NTOEixDTqnV6Uk0CC2GpW8hYTCyFXA72yj8XoAAzoE+nsxgNnrZc8DtL7bU9HJlDwqLY9855FkbY8ktS3LWlGLECbPo6UG8DUOsa+Bn5nH8q3HsRRo4GISL6vDN0O0e70SdoB2rfeshYBF71Juyzzu90TcF59FIC8WJvSVvgiT9nnPH5nP/K7CtOPonYWzh2aTF2Fu+usmvPjLF3us7cXwdR6iZ6DjyogsAWKrhokghhG6kCMTAu9Ap7+r1l0cQwoLAote4+ugwT+IsxO78XrQKkTkqzsEkqeily8Nk0il5cfHfowv3/xlLBxf6Pk2sNhTwEkx7I6FqMHDlC3wTRVRK4QZ1sGbCnxfrfxgwjBtvtHXFAZW7OsQZo7hEm7Fkxf8nm+mH6TBlau0RG00OBWcY6Gj6BDaLgSdDn46MPwG9Hr15/MGsdco5S0GrDiAIU7D5M/AgIo9gY6Lng4+5wi3jIOea59wieCQzgEnAe4kWoEFzhbBGRzEyIPQDmBWpaoxSpQMUZdCwCLh1OlmDWcCBzJsSNzDiIyL8LR8Ur1lHE2nPeZzh+d6mooENW7Zcx6b7zuHTlvCJB1Nnz6GS1O7sUhKxDl/LEP00Vhekh8sUjThNUyYAdxr59dCSwSvAWbg5Xq7exkqLfRO6TMnz/TurNAEv20/Jk4swaf2xC6U2k7Y9XPoOBIm6crYh6UoaLodABOoSU3YlpLbQ48lQT0qnR+sEq1RBlj0dGmfsnPVOtB51IMmfEdGLQ7RkkSYkps8VbJ01QIjDdaNCIVZwOi4DnxOgsRRXIzhazwakY3gmphsljLWe56RBqv6wfvg3R0HFqS6CcHxC5kQHrwGo3nFSIN1Q1RaBuinyDchSyYmDRcthWPLPF22G2mwuo+k55kgHUylJRtZoa1A0kI0bAdGPRnSszQuYFE90yUdepoznzKHWtLRDmsglZY8cHZTE7UVCGqEpmtDScZZLK20wEh7LKpst9YBKQUf1A5mhovWCefMuU9eM9JbWnEQMAIY/DQOXLr+mqmHXkfIdj18YpSRByuFa6+2F1f+cgXkuWb3zfZdN6Twt/DCQuKpsgmVDQIXy9vPAmMB1krPRf9eryot/TpsXL4fG7BSuNa7Ssu4gNOvnmMFVtqY9azS0q/DxuX7sQRrXIy7kevJwNrIvV9i2xlYJRp3IxfNwNrIvV9i2xlYJRp3IxfNwNrIvV9i2xlYJRp3IxfNwNrIvV9i2xlYJRp3Ixf9d0NIelzdt4X5AAAAAElFTkSuQmCC\".encode('utf-8')), embed=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ft95NDUZy4Rr"
   },
   "source": [
    "That's equivalent to the function $\\hat{y} = w_2 x + w_1$. What we're trying to do is find the \"best\" weights $w_1$ and $w_2$. That will give us that green regression line above.\n",
    "\n",
    "What are the best weights? They're the weights that minimize the difference between our estimate $\\hat{y}$ and the actual y. Specifically, we want to minimize the sum of the squared errors, so minimize $\\sum{(\\hat{y} - y)^2}$, which is known as the *L2 loss*. So, the best weights are the weights that minimize the L2 loss."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RHDGz_14vGNg"
   },
   "source": [
    "## Gradient descent\n",
    "\n",
    "What gradient descent does is start with random weights for $\\hat{y} = w_2 x + w_1$ and gradually moves those weights toward better values.\n",
    "\n",
    "It does that by following the downward slope of the error curves. Imagine that the possible errors we could get with different weights as a landscape. From whatever weights we have, moving in some directions will increase the error, like going uphill, and some directions will decrease the error, like going downhill. We want to roll downhill, always moving the weights toward lower error.\n",
    "\n",
    "How does gradient descent know which way is downhill? It follows the partial derivatives of the L2 loss. The partial derivative is like a velocity, saying which way the error will change if we change the weight. We want to move in the direction of lower error. The partial derivative points the way.\n",
    "\n",
    "So, what gradient descent does is start with random weights and gradually walk those weights toward lower error, using the partial derivatives to know which direction to go."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "W7SgnPAWBX2M"
   },
   "source": [
    "## The code again\n",
    "\n",
    "Let's go back to the code now, walking through it with many more comments in the code this time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "cellView": "both",
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "output_extras": [
      {
       "item_id": 1
      }
     ]
    },
    "colab_type": "code",
    "collapsed": false,
    "executionInfo": {
     "elapsed": 548,
     "status": "ok",
     "timestamp": 1474671838303,
     "user": {
      "displayName": "",
      "photoUrl": "",
      "userId": ""
     },
     "user_tz": 420
    },
    "id": "4qtXAPGmBWUW",
    "outputId": "841884bc-9a23-4627-cdec-8e7d4261a3c0"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAl4AAAESCAYAAAAsU9sMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xl8E3X+P/DX5GzaND3TlNJSzlJA2ir1QuUUELRyiOiu\ni4ooKIoX6v7EXXCV1V2Q/equawFFEGR1FWVVqIJWW+QoiAot900pkPRuk6ZHjvn9URoLFHqmk+P1\nfMijZJKZz3ukHd585jPvtyCKoggiIiIicjuZ1AEQERER+QsmXkRERESdhIkXERERUSdh4kVERETU\nSZh4EREREXUSJl5EREREnaRDEq+5c+di8ODBSEtLc22rqKjAQw89hDFjxmD69Okwm80dMRQRkVsZ\njUbcf//9GDduHNLS0rBq1SoAV76mLViwAKNHj8b48eNx4MABqUInIi/QIYnXpEmTsHz58gu2LVu2\nDDfeeCM2btyI66+/HkuXLu2IoYiI3Eoul+PFF19ERkYGPv74Y6xZswbHjh277DUtOzsb+fn52LRp\nE1555RXMnz9f4jMgIk/WIYlXamoqdDrdBdsyMzMxceJEAMDEiRPx3XffdcRQRERupdfr0a9fPwBA\nUFAQevXqBZPJdMk1LTMzE0D9tW7ChAkAgOTkZJjNZhQXF0sTPBF5PLet8SotLUVkZCSA+gtZWVmZ\nu4YiInKLgoICHDx4EMnJySgpKbngmlZaWgoAKCwsRHR0tGsfg8EAk8kkSbxE5Pm4uJ6IqAlVVVV4\n8sknMXfuXAQFBUEQhCY/11TXtct9lojIbYlXRESEa7q9qKgI4eHhze7DtpFE5AnsdjuefPJJjB8/\nHrfeeiuAy1/TDAYDjEaja1+j0YioqKhmx+D1jsg/KTrqQBdfREaMGIHPP/8cM2bMwLp16zBy5Mhm\njyEIAoqKPOPpR70+mLF4aByA58TiKXEAnheLN5s7dy569+6NBx54wLXtcte0kSNHYs2aNRg3bhx2\n794NnU7nuiV5JZ50vWsLT/p+aytvPwdvjx/w/nNoy7WuQxKvOXPmYMeOHSgvL8ewYcMwe/ZszJgx\nA0899RQ+++wzxMTE4K233uqIoYiI3Ornn3/GV199hYSEBEyYMAGCIOCZZ57BI488gqeffvqSa9rQ\noUORnZ2NUaNGQaPR4PXXX2/RONYamztPg4g8lCB62Hy3p2S+npSFe0osnhIH4DmxeEocgOfFQlf2\nxeZjGNyv+VuSnsqTvt/aytvPwdvjB7z/HNpyrePieiIiCZRV1kgdAhFJgIkXEZEEqmrsUodARBJg\n4kVEJIGqaq7xIvJHTLyIiCTAxIvIPzHxIiKSABMvIv/ExIuISAIWJl5EfomJFxGRBKpYx4vILzHx\nIiKSAG81EvknJl5ERBKw2Z2w2R1Sh0FEnYyJF5EP+MW0C18d+krqMKiVrKzlReR3mHgReblfTT9j\nzYHVyCnIuaRZPXk2ay0TLyJ/w8SLyIsdLj2Ejw5+iAC5Go9d+xgEQZA6JGoFzngR+R8mXkReKr/y\nFFbuew+CIGD6wBmI1cVKHRK1Eme8iPwPEy8iL1RoLcR7eUtR57DhD/0fRM/Q3lKHRG3AGS8i/8PE\ni8jLVNSW493cdFTZqjA5YQoGRiZJHRK1EWe8iPwPEy8iL2K1WfFu7hKU1pRibI9xuCFmsNQhUTtY\nWUSVyO8w8SLyEjaHDSv2vodzVedwc9dbMLLbaKlDonbijBeR/2HiReQFnKITqw+sxPGKY0iJuhoT\net/FJxh9QDXXeBH5HSZeRB5OFEWsPfxf7Cveiz5hffC7xD8w6fIRnPEi8j9MvIg83DcnM7DjXA7i\nguPw4ICHoZAppA6JOgifaiTyP0y8iDzYljOb8d2pTdBrIvHwwJkIUARIHRJ1EIVcxhkvIj/ExIvI\nQ/1q+hn/O/I5glXBeCRpFrSqYKlDog6k1Sg540Xkh5h4EXmghlZAarkKM5IeQ4QmQuqQqIMFaRSc\n8SLyQ0y8iDzMaXO+qxXQQwMfQYy2q9QhkRsEccaLyC8x8SLyIIXWQrybu6S+FVC/B9ArtI/UIZGb\nBAUoYXc4YbM7pA6FiDoREy8iD3FJKyB9stQhkRsFaZQA+GQjkb9h4kXkAaw2K97LW4rSmlLc1n0s\nWwH5AVfixXVeRH6FiReRxBpaAZ21nMVNXW/GrfFjpA6JOoGWM15EfomJF5GEnKITHx74gK2APMzc\nuXMxePBgpKWlubYdPHgQ99xzDyZMmIDJkycjNzfX9d6CBQswevRojB8/HgcOHGjRGJzxIvJPTLyI\nJCKKIj47/An2Fue5WgHJBP5IeoJJkyZh+fLlF2xbtGgRZs+ejf/973+YPXs2Fi1aBADIzs5Gfn4+\nNm3ahFdeeQXz589v0Rhc40Xkn3iVJ5LINyczkHNuO2K1sWwF5GFSU1Oh0+ku2CYIAsxmMwDAbDbD\nYDAAADIzMzFhwgQAQHJyMsxmM4qLi5sdIyiAM15E/ohXeiIJNLQCitRE4OEktgLyBi+++CIefvhh\n/P3vf4coivj4448BAIWFhYiOjnZ9zmAwwGQyITIy8orH+23Gy+a+oInI43DGi6iT7S78xdUKaEbS\n4whW6ZrfiST30Ucf4aWXXkJWVhZefPFFzJ07F0D9LeOLtWSdnpZrvIj8Eme8iDrR4dJD+M+B1VDL\nVXhk4KNsBeRF/ve//+FPf/oTAOC2225z/d5gMMBoNLo+ZzQaERUV1ezxGma8REEGvd47+3B6a9yN\nefs5eHv8gG+cQ2sw8SLqJBe3AuoaHCt1SHQFF89kGQwG7Ny5E9dddx22b9+O+Ph4AMDIkSOxZs0a\njBs3Drt374ZOp2v2NiPwW+JVWm5FUZG540/AzfT6YK+MuzFvPwdvjx/w/nNoS9LIxIuoExRZi/Be\n7lLUOWx4YMA0tgLycHPmzMGOHTtQXl6OYcOGYfbs2Xj11VexYMECOJ1OqNVqvPrqqwCAoUOHIjs7\nG6NGjYJGo8Hrr7/eojH4VCORf2LiReRGlioL3v9hGb4pzQAAPHfT/2MrIC+wePHiJrd//vnnTW6f\nN29eq8dQKWRQyAWu8SLyM1xcT+RGK394D19av0CFohza0GAcPXRY6pDIQwiCgEC1gjNeRH6GiReR\nm9gcNnxf/h2q7VUwBHVBbHAcCusKpQ6LPIgmQMkZLyI/w1uN5PHM5iosWZIHo1EDg8GKWbOSPP4p\nGKfoxJoDq1DjrEFYYAS663oAAKJUzT/tRv4jUK1ASUWN1GEQUSdi4kUeb8mSPOzePQyCIMBoFJGe\nnoWFC6Ob31EiDa2A8opzcVviOCiLlCipKEGUKgr3DX1A6vDIgwQGKGB3OGGzO6BUyKUOh4g6ARMv\n8nhGo8ZVkLI++dJIHNGVbTz5NXLObUdXbVfMSJnFqvR0WYHq+kuwtcaOEC0TLyJ/wDVe5PEMBqur\nppIoioiOrpY4osvbcmYzvj21EZGaCDyS9CiTLrqiwIDziRfXeRH5Dc54kcebNSsJ6elZMBo1iI6u\nxmOPDZQ6pCaxFRC1VuMZLyLyD0y8yONptUF4/vkbpA7jio6UHWYrIGo1zngR+R8mXkTtYKmy4O9f\n/hXflW2EWhaA10YuYisgajHOeBH5H67xImqHN9e/iQzrelhV1YiI0CNn91apQyIvouGMF5HfcfuM\n14gRI6DVaiGTyaBQKLB27Vp3D0nUpKbqgWm1QW06ltFUiMcX/xU/B60HdDZcF38NIgMjUVjOAqnU\ncoHqhn6NNokjIaLO4vbESxAErF69GiEhIe4eiuiKmqoH1ta1Y08s/ht2R+yB3SEAzkAcPnwOsdd3\nY4FUahWu8SLyP25PvERRhNPpdPcw5MdaOpPVUfXAbA4bDmt+gVNuRZA9EY6TtahzWpCYlMgCqdQq\nDWu8qrnGi8hvdMqM1/Tp0yEIAu655x5MmTLF3UOSn3nrrZ1Yvz4YtbVyqFQK1NXtxJ/+NPySzxkM\nVhiNIgRBaHM9sIZWQEJANRS2CGjtfeEIdaJvjRIzxs3qiNMhP8IZLyL/4/bE6+OPP4Zer0dpaSmm\nTZuGnj17IjU19bKf96QefIzlUp4SB/BbLD/+aIHZnAZBEFBXJ+LHHz9qMs7582/CG29sx9mzAYiJ\nqcFzzw1u1Rovs8WMWf99Ansr9yIpvhtse7qhpK4O0ZpgfPD3P3vE/xtPiIFajk81Evkftydeer0e\nABAeHo5Ro0YhLy/violXUZHZ3SG1iF4fzFg8NA7gwljsdjVEUXRVt3c4Ai4b5+OPX+36fXW1E9XV\nLT+fZ9c+gZy6nQhSByE8XI8BQ3vhpQf+6BpL6v83nvbnQ81TKmRQyAXOeBH5EbeWk6iurkZVVRUA\nwGq1YsuWLejTp487hyQ/NHx4AHS6YqjV5dDpijF8uLrDx9h65kfstvyKAEUAEsP7QyFXorCOTzBS\n+wiCgEC1gjNeRH7ErTNexcXFeOKJJyAIAhwOB9LS0nDzzTe7c0jyQ08/PQhq9Z5GLYUGdejx9xT+\ninVHPkOIIhRdwrpCJVdBFEU+wUgdQhOg5IwXkR9xa+IVFxeHL774wp1DELm1pdCRssP4z8H6VkAL\nR/8fftj1HQqrChGliuITjNQhAtUKlFTUSB0GEXUStgwin9WegqmWKgveyfwnvi37xtUKqE9UAvqM\nS3Bz1ORvAgMUsDucsNkdUCrkUodDRG7GlkHksxoKpppMN2DPnuFIT89r8b7Lvn8HG6xftbsVkNlc\nhUWLcjBnzh4sXLgdFktVm45DvotPNhL5FyZe5LPaWjDVXFeJ78o2we60oYeuZ30roDYupG9P8kf+\nQaNmLS8if8LEi3yWwWB1lZhoacHUans1luWmAwC6auNgCIpu10L6jqqWT77LVUSVM15EfoGJF/ms\nWbOSkJKSBYMhBykpWXjssYFX/LzNYcOKve/hrOUs7up3D/Bjb/z6mQ0FGwIwIXVym2JoS/JH/iWQ\nM15EfoWL68lnteZpx4ZWQMfKjyJJnwxTRiKUZ2cj5nx7oQ8+yMKjjwZdslg/IECGRYtyLruAf9as\nJKSnZzUqdXHl5I/8D2e8iPwLEy/ye6Io4vMjnyKvOBe9Qnvj94lT8f/e33/JLcKG9Vr1r0Wkp2ch\nMFB9ybbGyZ47S12Qb+CMF5F/4a1G8nubTn2D7We3oau2K6Zd9TCUcmWTtwibWq919mwA13BRu/w2\n42WTOBIi6gxMvMjnXamkw7YzW7Dp5DeICIjAI0mPQqOoT5yaWh/WVDIWE1PNNVzULoFqJQDOeBH5\nC95qJI/WniKoDZq6Rfj88zcgt2g3Pj+yFlqlFjOSZyFYpXPt09QtwqbWa+n1wXjlFa7horbTnJ/x\nquYaLyK/wMSLPNrlkqbWaOoW4ZGyw1hzYBXUchVmJD2GSE1ks8dpKhnjGi7fNHfuXGRlZSEiIgJf\nffWVa/vq1auxZs0aKJVKDB06FM899xwAYOnSpfjss88gl8vx0ksvtaonLdd4EfkXJl7k0TqiDlZY\nWCm2bv0R1mon6qI+R9fYE5izvgK9YxLw8DWPomtwbEeHTV5u0qRJmDp1Kl544QXXth07duCHH37A\n+vXroVAoUFpaCgA4duwYvv76a2RkZMBoNGLatGnYtGmT6/u2OXyqkci/cI0XebSOqIMlCHIAA1Ad\nvgO1AwtxNuYASuTFCK7Uok8Yey/SpVJTU6HT6S7Y9tFHH+GRRx6BQlGfKIWHhwMAMjMzMW7cOCgU\nCsTGxiI+Ph65ubktHkulkEEuEzjjReQnmHiRR2ttEdSmmAqVUPT8EM6+n8MevQ02hwPdQ3pCKajc\nEDH5qpMnT2LXrl2YMmUKpk6dir179wIATCYTunTp4vqcwWCAyWRq8XEFQUBggIIzXkR+grcayaN1\nxBqqM7L/4lzPPFiVh+FU26Cpi4AhMBpR1W1rA0T+yeFwoLKyEp988glyc3Px1FNPITMz0zUj21hL\nbzPq9cEAgOBAFay1dtdrb+Ft8TbF28/B2+MHfOMcWoOJF/k8e8xB2DWHITicUFQKCLSKSKxOxH1D\nH5A6NPIi0dHRGD16NAAgKSkJcrkcZWVliI6Oxrlz51yfMxqNiIpqWVJfVGQGAKiVMhSW2VyvvYFe\nH+xV8TbF28/B2+MHvP8c2pI08lYj+SxLlQVLNryNA7UH4KytRWigBt16dEWv0ATMGDcLQUGtK0tB\n/uXimaxbb70V27dvBwCcOHECNpsNYWFhGDFiBDIyMlBXV4fTp08jPz8fSUlJrRorUK2A3eGEze7o\nsPiJyDNxxos8VuMaXqGhFRAEB8rKwmEwWDF//k3N7r8m6wNstH0NuV4OuU0G+XEFQruEYVif4Z0Q\nPXmzOXPmYMeOHSgvL8ewYcMwe/Zs3HXXXXjxxReRlpYGpVKJv//97wCA3r17Y+zYsbj99tuhUCgw\nf/78Ft9qbKAJOF9EtcaOEK28w8+HiDwHEy/yWI1reG3dWgpgH3r3vgFGo4g33tiOxx+/+or77zTv\nQKHThJjQGChrVBDCZZjYexJvMVKzFi9e3OT2RYsWNbl95syZmDlzZpvHa1zLK0SrbvNxiMjzMfEi\nj9W4hldtrQJAAOx2O06dMuP06QpUVW2/bCX77We34lzdWajVAegfORBKmRKJ+kTMGDerk8+CqHms\n5UXkP7jGizxW4xpeKpUdanUNTp0yo7w8HA5HCPbsGY709LxL9sst2o3PDn+KlG7X4I7AO2EwG7iY\nnjwaq9cT+Q/OeJHHatwbsVevYvz8sxEHDxZCJqtGXNy4JivZHy07gjUHVkElV2JWymzEBsdJFD1R\ny+mC6mvKlZtrJY6EiNyNiRd5rMY1vBYtykFIyHRERpajoiIcp09vRffuN7oq2VuqLHgn85/4tmwj\n1DI1Xhu5kEkXeQ19aP0/IArLW9+ZgYi8C281kldoWO/VvXswQkNLIZdXuCrZm4pMGPPKMLx98C0c\nNR5GkEqLnN3bpA6ZqMUMYecTrzImXkS+jjNe5BUMBiuMRhEKhQI9e4Zi8OAQ11ONM5c8iPz4U3DK\nRAgCcHT3YQy8vnV1lIikFBqshkIu44wXkR9g4kVeofF6r+joajz33GBUVztRY6/BPsVeCIIAhUwO\npUwJq7waUSq2AyLvIRME6EMDUFhWDVEUW10HjIi8BxMv8goX92zUaoNgrirDir3vQS7IEazWwWlz\nwmG3I6pWzycYyesYwgJxrsSKqho7tBql1OEQkZsw8aLLalw5vqXV4juDpcqCD7Pfw1cFGTA7KnHf\n1fdjz85fUSgWwSBEYdELb7IdEHmdhgX2pjIrtJoQiaMhIndh4kWX1bhyfEurxXekixO/WbOSYCoq\nxO2v3AlzbBGgcqJPXG9ozBp88NxHnRYXkTtEnV9gX1RWjV4xTLyIfBUTL7qsxpXjBUHA2bMBnTr+\nxYlfenoW1h17GeUDiyGq7IBTQP5OE0pGl3RqXETuEMUnG4n8AstJ0GU1rhwviiJiYmo6dfzGiZ/D\nWYUfjq5HvnYvnEItIAIyMQA2pY0L6cknuBIvPtlI5NOYeNFlzZqVhJSULBgMOUhJycJzzw3q1PEb\nJ375zndwsueXsIdbAZkTqFQANhm0laFcSE8+IUIXAJkgcMaLyMfxViNdVlNPElZXmztt/IYSEqcL\nRGy3v4VaZTmUMjlsZwXIyoHuQj/8Z/4SLqQnn6CQyxARokZhmVXqUIjIjTjjRR5Lqw3C/Q/2wC7N\nE7D0KoJNboNMK0eYNgTDEgdj25Jv0L17d6nDJOowUWGBqLTaUM1m2UQ+i4kXeSxTkQnDX74RR0OP\nQlSIgADYy2xQ1CowasAoqcMj6nBR50tKFHGdF5HPYuJFHuvZD55AWXwZIAcgBwS7ALlcgUGhqXh8\nwuNSh0fU4fhkI5HvY+JFHqlxKyBBJUB2Tg4UCehR3AP/fuxdrusin8QnG4l8HxMv8jh2px0r99W3\nAgrRhUJTHAClXIHIsgisfeFLJl3ksxpuNXLGi8h38alG8ihO0Yn/HFiNI2VH8OD107Fry04U6s63\nAnriTej1rNlFvkvvSrz4ZCORr2LiRR7BVGTCCx88jYOyg7DLbbhn0O/x6LVPQHkDmwWT/1Ap5QgL\nVnNxPZEPY+JFnaKpvota7W+3DP/4wTP4NfJXWO1VkAty5OXsgfJmJl3kf6JCNTh8uhw2uwNKhVzq\ncIiogzHxok7RVN/FRx8d6ErGclS5sIdYIJcpEBYQhmKxvv9icwkbka/Rh2lw6HQ5isprEBPJ73Ui\nX8PF9dQpLm64bTRqXMnYEYsaVXKgymKHrTIQZSW1iHCGA/gtYTOZbsCePcORnp4n5WkQuZ2BTzYS\n+TQmXtShzOYqLFqUgzlz9mDhwu2wWKoAXNpwOzq6GsdPVmGXeix+jrkD9lonkN0HsgNdodg2CL1q\npgFoOmEjaqmMjAxYLBYAwFtvvYXp06dj7969Ekd1ZVFhgQD4ZCORr2LiRR3qcjNUFzfcfuyxgdgl\nLEBl0s8QQ+vgDJNB7dDiBusWXK1YC0tlDICmEzailkpPT4dWq0Vubi62bNmCCRMmYMGCBVKHdUWu\n6vVMvIh8Etd4UYe63AzVxQ23S6pLUN2rAAqFDQp7MBwKG+SGOqDmwgSroVG20ahBdHQ1HntsYOef\nFHkthaL+Erd161bcfffdSEtLw/vvvy9xVFfWUFLCVM6SEkS+iIkXdSiDwQqjUYQgCJedobLUmfFu\n7jtQy9SICFYgSBkEh8MB2XElDIacCxKsixM2otYQBAEZGRnIyMjAO++8AwCw2WwSR3VlgQEKaDVK\nzngR+Si332rcvHkzbrvtNowZMwbLli1z93AksaZuKTZWY6/Bu7lLUFRdjBeGvoiUoquhOxOKRFM/\nfPHqKixenIznn7+BTy5Sh/jTn/6E9evXY/LkyYiLi8PJkydx/fXXN7vf3LlzMXjwYKSlpV3y3vLl\ny5GYmIjy8nLXtgULFmD06NEYP348Dhw40O64DWEaFFfUwOF0tvtYRORZ3Drj5XQ68eqrr2LlypWI\niorC5MmTMXLkSPTq1cudw5KELjdDZamyYHXWCnxXtgnVzmr8LuUPmDLw97gn5T4JoiR/cc0117hm\nugCge/fu+POf/9zsfpMmTcLUqVPxwgsvXLDdaDRi27ZtiImJcW3Lzs5Gfn4+Nm3ahD179mD+/Pn4\n5JNP2hW3PkyDY2crUVpZ67r1SES+wa0zXrm5uYiPj0fXrl2hVCpx++23IzMz051Dkof6MGslMmo3\nwCg7B5lWhqrTFtdaMCJ3+dvf/gaz2Qy73Y7f//73SElJwRdffNHsfqmpqdDpdJdsf+211y5JxjIz\nMzFhwgQAQHJyMsxmM4qLi9sVN3s2EvkutyZeJpMJXbp0cb02GAwoLCx055DkgURRxNaKLSitKUaw\nSoc+YX1RbGvfX0xELbFt2zYEBwdjy5YtMBgM2LhxY5sX13///ffo0qUL+vbte8H2wsJCREdHu14b\nDAaYTKZ2xW1wlZTgAnsiX+PWW40NZQBaQ68PdkMkbcNYLtWWODKOZKBSKINOHYzk6CTIBTl6yru1\n+5y8+f+Ju3hSLJ7kp59+wqhRo2AwGNo001pTU4MlS5Y0mbQ1dZ1r6RiX+/NK6FH/AIC51uHRf6ae\nHFtLefs5eHv8gG+cQ2u4NfGKjo7G2bNnXa9NJhOioqKuuE9RkdmdIbWYXh/MWFoZR1PtffZW7sGn\nh9fi6thUhJWHw1xoRpQqCuOH3tOuc/KW/yedydNi8QQRERGYP38+fvzxR8yYMQN2ux0Oh6PVx8nP\nz8eZM2cwfvx4iKIIk8mESZMm4dNPP4XBYIDRaHR91mg0Nnuda3C5Py+lUJ/MnTpb4TF/phfzpO+3\ntvL2c/D2+AHvP4e2XOvcmngNHDjQdcHS6/XYsGED/vGPf7hzSOpEFydaNpsN+/ePdvVjfHnpe8Cg\nXQhSBmHW1U8hKrBlfxkRdZTFixfjyy+/xMSJExESEoKCggJMmzatRfs2nslKSEjA1q1bXa9HjBiB\ndevWISQkBCNHjsSaNWswbtw47N69GzqdDpGRke2KO1ijRIBKzrZBRD7IrYmXXC7Hn//8Zzz00EMQ\nRRGTJ0/mE40+5OLG1ybTV4jUV6FAeB9VoQewu2oHxjmvw8MDZzLpIkmEh4fjD3/4A06cOIGjR4+i\ne/fumDRpUrP7zZkzBzt27EB5eTmGDRuG2bNn46677nK931CnDgCGDh2K7OxsjBo1ChqNBq+//nq7\n4xYEAVFhGhhL6js38EEUIt/h9gKqQ4YMwZAhQ9w9DEng4ir1omhBAd5HZZ+fYNbmQR1gRkhlCLrp\n4iWOlPxVXl4ennzySahUKoiiCLvdjn/9618YMGDAFfdbvHjxFd+/+OnsefPmtTvWi0WFBSLfZEG5\npQ5hweoOPz4RSYOV66nNLq5Sf9PNanxVuRJlIachlwFXx13b4mM1tT6MRVSpvf7617/itddew403\n3ggAyMnJwauvvoqPP/5Y4sia1zUyCLsAnDxXibBgvdThEFEHYZNsarOLq9TrknPh6FkMtc6J0DAt\nyivKEaVq2S3GyzXXJmqP6upqV9IFADfccAOqq71j3VTfuFAAwKHT5c18koi8CRMvarOGKvV/eaUX\nAvvtwIf5q2BT2hBRpUd4RRgCjUG4b+gDLTrW5ZprE7WHRqNBTk6O6/XOnTuh0XjH91bPGB0UcgEH\n88ukDoWIOhBvNVK7rc5agfU1X6JOXgu5Wo5wTTj6dRuAxOpEBAVd/nZh49uLR48eQkDAVVAqtZdt\nrk3UWnPnzsVTTz0FlUoFoL5B9j//+U+Jo2oZlVKOnl10OFJQAWuNDYEBSqlDIqIOwMSL2sxUZMIL\nHzyNLTVb4NTY0UvfG8pyFVArIFGf2OxsV+OnIgMCBqG29mPExvZBdHT1Jc21idoiKSkJmzZtwokT\nJyCKInr06IHRo0cjKytL6tBaJKFbGA4XVOBwQQVSerevRAUReQYmXtRmL3zwNH6O+Bm2qlqIChGl\nJ0oxbMhIJFYnYsa4Wc3u3/j2olKpRGxsHyxenOzusMnPKJVKJCQkuF63paOGVBK7hWL9NuBQfhkT\nLyIfwTV510I9AAAgAElEQVRe1CaWKgt2Wnai3FoGQZQhqE4LW40didXNz3Q1MBisrr8EeXuROos3\n1cTq1TUEcpmAQ/lcYE/kKzjjRW3y2sa/oCbYCshFKFRyCDUCBkUMatFMV4NZs5KQnp4Fo1HD24vU\noY4ePXrZ9+x2eydG0j5qpRw9uuhw7GwFqmvt0Kh5ySbydvwpplbLK85FTuV2xER1RdWJKtQJddBV\n6rDohTdbdZyGpyKJOtqMGTMu+55a7V3FSPt2C8XRMxU4UlCBpF4RUodDRO3ExMvPtLdQ6fHyo/hw\n/0po5UHoHdUHwXE6iKKIxOpE6PVsC0Se4fvvv5c6hA7Tt1soNmw/hUP5ZUy8iHwA13j5mfYUKi2o\nLMD7e9+FKIp4ZcTrSHVci7DysFat6yKi1undNQQyQWAhVSIfwRkvP9PaQqWWKgve37gM3x3dhHOB\nZxChjcS84QvQS9cX3+6rQ5lRA5XBCpHtOIncIkClQPcuwTh5zoyaOjsCVLxsE3kz/gT7mYv7Kzb3\nJOF/sldhfcGXOBVxEiKcUDkDsHffHmzdJ3fV4DIaRaSnZ3Xaei2zuQr//vevOH5cxr6O5Bf6xoXi\n+NlKHD1Tgat68HYjkTfjrUY/c3F/xSs9SWipsuDrvA04bD2I6gorNIIGapkKhXWFkrb4WbIkD7t2\n3cK+juQ3+nYLAwCWlSDyAZzx8mJtWSjfmicJV2etwKmIk3DUOGGHEyWHqyGG2BASGgJVK2fOOhL7\nOpK/6RMbAkFg4kXkCzjj5cXas1C+OaIoIrs8C2qNGupaHWRFEZCXhkC7+35U7k+5YOasX79NqK2t\nxpw5e7Bw4XZYLFUdFkdTWHiV/I1GrUC8IRgnzlWi1uaQOhwiagfOeHkxd8z8WKosWJP1AbZW/ogj\nxYcQ3iUCQl0iqp0GhFQPRKzycZSW5Fwwc7ZoUU6nrveaNSsJH3zwI44fl7kKr7a3TAaRp+vbLRQn\njWYcO1OB/t3DpQ6HiNqIiZcXa+1C+Zb4T/YqfO/MRIGYD50+BNH50SirDEL5yf7opnoYDvul43T2\nrT+tNggvvzwERUVm17bOTv6IOlvfbmHYuPM0DuaXM/Ei8mJMvLxYR7fcMRWZ8M7Of6E0ogRKqDCi\n562ID+yOpx99HunpeSgv34fQ0IpLxnFHAthaXPdFvi4hNgQCgMP5ZVKHQkTtwMTLi3V0y53HVk1H\nSXAJnEonFHInfj7yE65NuM41jl4ffMEsUwNP6LnoCckfkTsFBigRZ9Di+LlK9m0k8mL8ySVYqiz4\n53f/wM/2XYBcRKApEDKlDCgTcN8jzVek94Sei56Q/BG52zV99Mg3WfDL4SLcNLCL1OEQURsw8SKk\nZ/4L31g3QBYog6AGlDVKhMSHoq+qL4KCvGOBuickf0TudsMAA/635QS27zMy8SLyUiwn4edKa0qQ\nWfYtHKIDqV2ug8IUjKqzTmCzHi9Pfk3q8IiokaiwQPTqqsOBk2UoM9dKHQ4RtQETLz9mqTNjWW46\nZIIM3XTdUXYmANq6OxBXPhc9LVn49FOj1CES0UUGD4iGCGDHfpPUoRBRGzDx8lM19hq8l7cURdYi\nPJw6E8NlI+A8Gg7tsf6IFafzyUAiD3VtPwPkMgE5+/gPIyJvxDVefsZSZcHqrBX4ruxbVDutuDf5\nPkzsNxlCfwHle7djT+lwCDI+GUjkqbQaJQb2jMDuo8UoKLIgVq+VOiQiagUmXh7ObK7Cv//9K44f\nl3VIRfY1WR8go3YDSmXFCNWEwVpQBUtPK5YsyUNBgRIWy2rEx8chLg58MpDIQ914VTR2Hy1Gzj4T\nJg9j4kXkTXir0cMtWZKHXbtu6ZB+jGaLGWuOrMbh4oOwWqvRQ9cTxbZiV8/H0tJboNVORVwc8Pzz\nN7DlDpGHSu4VAY1ajpz9RjjP9y0lIu/AxMvDdWRF9r988ycUy4sgyAG1Ro1jZ48iShXFqu9EF5k7\ndy4GDx6MtLQ017aFCxdi7NixGD9+PGbPng2LxeJ6b+nSpRg9ejTGjh2LLVu2uD0+lVKOQX2jUFpZ\niyOny90+HhF1HCZeHs5gsEI8/y/a9qy72nEuB79YdkEfEYVuVd0RWKJBoDEI9w19oMPGIPIVkyZN\nwvLlyy/YdvPNN2PDhg344osvEB8fj6VLlwIAjh49iq+//hoZGRl499138Ze//MX18+RONw6IBgBs\n5yJ7Iq/CxMvDzZqVhNTUH2Ew5CAlJatN6672Fufh00MfQ6fQob9+AAYmJSNl4CCMGXgbgoKCMGtW\nElJSsto1BpEvSU1NhU6nu2Db4MGDIZPVXzJTUlJgNNYnPN9//z3GjRsHhUKB2NhYxMfHIzc31+0x\n9u0WirBgNX46WASb3eH28YioY3BxvYfTaoPw8stDmuyR2BLHy4/iw/0roZQr8Ldb/4HNv/yAwqpC\nRKmicN/QB1xjsOo7UcutXbsWd9xxBwDAZDIhJSXF9Z7BYIDJ5P4aWzJBwA39Dfh6Rz72HC1BamKU\n28ckovZj4uVDzOYqLFmSB6NRA4PBikkP6rHy8LtwiE5Mv2oGEsP7IXFcP6nDJPJq6enpUCqVrsSr\nqduKDWsmm6PXB7crlnG39MLXO/Lx85FijL2lV7uO1Rbtjd8TePs5eHv8gG+cQ2sw8fIhDU8nCoKA\n/OJiZP/nSSSmaPH7fn9AYjgTLqL2WrduHbKzs7Fq1SrXtujoaJw7d8712mg0IiqqZbNPbZ3JbhCk\nENDNoMVP+03Yf6QQ+tDOezBGrw9ud/xS8/Zz8Pb4Ae8/h7YkjVzj5UOMRg0cziocFxZid9wknDDv\nx+iut2GQ4VqpQyPyOhfPZG3evBnvvfce0tPToVKpXNtHjBiBjIwM1NXV4fTp08jPz0dSUlKnxTnm\num5wiiI2/XS608YkorbjjJcPCQsvxtdnp6IyYRdEmRMGTTROHTkB9JE6MiLvMmfOHOzYsQPl5eUY\nNmwYZs+ejaVLl8Jms+Ghhx4CACQnJ+Pll19G7969MXbsWNx+++1QKBSYP39+i281doRrE6PwefYx\n/LjnLO68qTuCA1XN70REkmHi5UOC+/+Kas12iMpaKKFEpEOBwrrCFu9/8Rqx9lbJJ/JWixcvvmTb\nXXfdddnPz5w5EzNnznRnSJelkMsw+rpu+Oi7I/j+lzMYf3MPSeIgopbhrUYfIYoitlu3Q6OVIUQb\niJjwSNgEG6JULX/SqWGNWEdUySeizjMkKQZBAQpk/lyA2jqWliDyZEy8fIAoivjy2DpUOirQNaQr\nusm6QV2nRrwt3lUyoiVYwZ7IO6lVcoy4JhaWahu25J1rfgcikgxvNXoxS5UF/8lehZ2VO3C27gyu\n73YDIiyRKFeX19fpmvIAgoJafqvQYLDCaBQhCEKLKtjz1iSR5xg5KBbf7MzHxp35GHZ1DOQy/rua\nyBMx8fJi/8lehc1iNk44j0GlViHUHIon0p5u8/FmzUrCm29+jaysCgBBqK1Vw2Kpumwy1bh8hdEo\nIj09i4VYiSSiC1Lh5qQu+OGXM/jpYCFu6B8tdUhE1AQmXl5sX9U+nLAfh0KmRL+IATA3atrbFlpt\nENRqDQyGsRAEAQcOiHjzzU1Qq1UwGjXo2dOBBx7o60rEeGuSyLOMuTYOWb+ewTc5+bi+n6FTn64k\nopbhXLSXOl5xDMdrjkAGAX3D+yFArmnVQvrLuTiZysqqdS2437VryAUL7tlcm8izRIUFIrVvFPIL\nLdh3slTqcIioCUy8vNA5y1msyHsXvbskYLRmLLpZuyGxOrFVC+kv5+JkCqi67KwWm2sTeZ6xN3QD\nAHy19WST7YyISFq81ehlSmtK8G7eEljt1fjDwAc6vCr9rFlJSE/PgtGoQXR0NWprQ3DgQNML7tlc\nm8jzdI/WIaV3JHYfLcYvh4sxqK9e6pCIqBEmXl7EYrNgWW46KmorcGevCW5pBXRxMmWxVLkSsZ49\nnXjgAc5qEXm6u4f3Qt7xEnyadRTJvSOgkPPmBpGnYOLlJWodtVieuxRF1iIMjxuJoXHDO2XcxomY\ntzczJfIXXSKCMCylKzJ/KcD3v5zB6GvjpA6JiM5z2z+D3n77bQwZMgQTJ07ExIkTsXnzZncN5dNM\nRSaMnzceN/5zEFZtWYG+QYm4vWea1GERkYe78+bu0KgV+GrrCViqbVKHQ0TnuXX+edq0aVi3bh3W\nrVuHIUOGuHMon2SpsuDuhXfiO+E7FCsLYQ2yYmv2Zj4iTkTNCg5UIW1wd1TV2LF+20mpwyGi89ya\nePGJmvZZvnEZjgWfRA3qYIMDYrWIQrFI6rCIyEuMHBSLyJAAZP5cAFOZVepwiAhuTrzWrFmD8ePH\n46WXXoLZzLVBDczmKixalIM5c/Zg4cLtsFiqmvzcp3k/wCGXQxTlgCMQVdW1MAhXrtXV0mMTke9T\nKmSYPKwXHE4Ra384JnU4RARAENsxLTVt2jQUFxdfsv2ZZ55BSkoKwsLCIAgC/u///g9FRUV47bXX\n2hWsr3j55c3YtesWV4mG1NQf8fLLF96K3XZ6Gya+MRNVCifs5yrhVNZBa9Li4KrtiIq6fPLVkmMT\nkWfojIdVRFHEax/+jGNnKvH/7rsGCXGhHXJcX3jYxtvPwdvjB7z/HPT64Fbv066nGlesWNGiz02Z\nMgWPPvpoiz7rKX8A7vxmOH5cBpvNccHrxmPtK96LlfveR0xgT1TuvhoydR1klkiMHdgbgqC5YlzN\nHbs9POkHxFNi8ZQ4AM+LhTyDIAi4d0Qf/HX1z1i18RDmP3gtlAqWlyCSitt++oqKfluL9O233yIh\nIcFdQ3mdK7XaOVFxHKv3r4BCJsebv5uLuwbehOFREzD+moF46snr2nVsIvJPvbqGYPjVXXG2uApf\nbTshdThEfs1tdbwWLVqEAwcOQCaToWvXrnjllVfcNZTXubg6/GOPDYSlyoIlmW9jU9k3UMqUeHX4\n39C/S3/0f751MxlNHZuIaPKwXsg9VoKM7fm4JkGP7tE6qUMi8ktuS7wWLlzorkN7vaZa7bz51SKs\nr/4SNmUdeob2xq7cHbgmdlCHHJuISKNW4MFxiVj88W68v+EA5j14LSvaE0mAP3UewGKz4LuyTbA5\n6tBN1x1RgQYU1hV2+Dh84pHIvw3oHo4hyTEoKKpibS8iiTDxklhDKyCbw47aoiDk5wVg924jQhDS\n4WMtWZKH3buHwWS6AXv2DEd6el6Hj0FEnm3K8N4IC1Zjw/ZTyDd5xsMYRP6EiZeEHE4HVu17H/nm\nfESUDIY65z7YjiaidsetqNyf0uHjGY0aV9V7QRBgNGo6fAwi8myBAQo8ODYRDqeI9zccgN3hlDok\nIr/CJtkSsFRZsCbrA2RV/IAKeznu6D8eznM3QSa/CagFIAdKS3I6fFyDwQqjUXTV+OITj0T+aWDP\nCNw0MBpb84z4YssJ3DW0l9QhEfkNznhJYE3WB9hk34h8nIRD4wBMIroYat1eBmLWrCSkpGTBYMhB\nSkoWn3gk8mO/G9kHUaEabNh+CruPXFoIm4jcgzNeEthl/glG51loFIHoG9EPpZWleLoTykDwiUci\nahAYoMSsiVfhr6t/xrvr92P+tGsRFcrlB0TuxsSrk+08twNn6gqgUquQGNEfCkGBKFUUkyIi6nTd\nDMGYOrov3s84gHc+z8PcqYOgUsqlDovIp/FWYyfaV7wXnx7+GMndUnCH5k5Em6ORWJ2I+4Y+IHVo\nROSnbk7qgiHJMcgvtODDbw9LHQ6Rz+OMVydpaAUkF2SYmfI4uof0kDokIiIAwH2j+uCU0YwtuefQ\nu2sIhiTHSB0Skc/ijFcnMFadw/t5y+AQnXhgwHQmXUQebu7cuRg8eDDS0tJc2yoqKvDQQw9hzJgx\nmD59Oszm32pgLViwAKNHj8b48eNx4MABKUJuF6VCjlkTr0JQgAIfbjqMY2cqpA6JyGcx8XKzsppS\nLMtNh9VejXv6/g79IvpLHRIRNWPSpElYvnz5BduWLVuGG2+8ERs3bsT111+PpUuXAgCys7ORn5+P\nTZs24ZVXXsH8+fOlCLnd9KEazLhzABxOJ95am4tzJexsQeQOTLzcyGKzYFluOipqK3BHzzuRGn2d\n1CERUQukpqZCp7uwiXRmZiYmTpwIAJg4cSIyMzNd2ydMmAAASE5OhtlsRnGxd5ZnGNgzAg/clghL\ntQ3/+O8elJlrpQ6JyOcw8XKTWkct3s9bhkJrIYbFjcDwbiOlDomI2qG0tBSRkZEAAL1ej9LSUgBA\nYWEhoqOjXZ8zGAwwmUySxNgRhiTHYOItPVBSWYP/+2QPrDV2qUMi8ilMvNygoRXQqcpTSDVcizt6\n3il1SETkJg2FjxtraM3lre4Y3B3Dr+mKgiIL/vVZLmx2h9QhEfkMPtXYwURRxH8P/QcHSw+iX3g/\nTOn7u3ZdhM3mKvz737/i+HEZDAYrZs1KglYb1IERE1FLREREoLi4GJGRkSgqKkJ4eDiA+hkuo9Ho\n+pzRaERUVFSLjqnXB7sl1o7w1O8GodbuxLbcc1i16Qie/8MgyOUX/lvdk+NvKW8/B2+PH/CNc2gN\nJl4dSBRFfHXsf/jZtAvxuu6YOmAa5LL2FSNcsiQP+/ffCpvNAaNRRHp6FgutEnWCi2eyRowYgc8/\n/xwzZszAunXrMHJk/fKBkSNHYs2aNRg3bhx2794NnU7nuiXZnKIic/MfktADoxNQUlaNrblnUbvc\nhhl3DoDifPKl1wd7fPzN8fZz8Pb4Ae8/h7YkjbzV2IGyTn+P7IIsGAINmD5wBtRydbuPaTRqXDNm\ngiDAaGRLDyJ3mzNnDu69916cOHECw4YNw2effYYZM2Zg27ZtGDNmDLZv344ZM2YAAIYOHYrY2FiM\nGjUK8+bN89qnGpuiVMjx5OQkJMSFYtehIrz9eR7qbLztSNQenPHqID8Zd2D98S8Rqg7FjKTHEKTs\nmNuBBoMVpaXubZ5NRBdavHhxk9tXrlzZ5PZ58+a5MRppadQKPDMlGf/+PA+5x0rw5qd78OTkJKnD\nIvJanPHqAPuK9+KTQx8jUKHBjKTHEBoQ1mHHnjUrCampP8JgyEFKSpZbmmcTEV2JWinH7LuScE2C\nHgfzy7H4v7thqbZJHRaRV+KMVzs1bgU0feBMGIKim9+pFbTaILz88hCvvgdORN5PqZDhsQkDsHzD\nAeTsM2HuO1vw+ISrEK4LkDo0Iq/CGa92aNwK6P4BD7EVEBH5NLlMhodv749hV3fFibOVeOWDXTha\nwPZCRK3BxKuNGrcCmtL3XvSPGCB1SEREbieTCZg6OgEzJgyExWrDwo9+wY+5Z6UOi8hrMPFqgypb\n1QWtgK6Nvl7qkIiIOo0gCEi7pSeeuScZaqUcKzIO4uPMI3A4nVKHRuTxmHi1Uq2jFsvzlp5vBTSc\nrYCIyG8N6B6OP92fii4Rgdj002n2dyRqASZereBwOrB63wqcqjyFQYZU3NFzvNQhERFJyhAeiJem\npiKldyQOnCrDvOU7sOtgodRhEXksJl4tJIoiPjn0EQ6UHkC/8H64p+/vvb4fGxFRRwgMUGD2XQMx\ndXQCbHYn3vnfXizfsB/VtWywTXQxlpNoofXHv8Au00+I18V3SCsgIiJfIggChl8Ti8T4MCz7cj+2\n5hlx+HQ5pt/eHwlxoVKHR+QxOOPVAj/kZyLr9A+ICozC9IEzO6QVEBGRL+oSEYSX7h+EcTfEo7i8\nBn9b8wuWb9iPSmud1KEReQTOeDVjl3En1h//EiHqkA5tBURE5KsUchkmD+uFlD6RWL3xELbmGbH7\nSDEmDe2FockxkMm4TIP8F2e8rmB/yT7899BHrlZAYQHhUodEROQ1encNwbwHU/G7W/vA4RSxeuMh\nLFi1C0cKyqUOjUgynPG6jONlx7Fq3/uQCzI8NHAGooO6SB0SEZHXkctkGJUah2sTo/DJ90eRs9+E\n1z/8BUm9IjBpSE90MwRLHSJRp2Li1QRTlRHLD74Dh+jEgwOmo0dIT6lDIiLyaqFaNWbcOQDDr+mK\nz7KPI/dYCXKPleC6flGYcEtPRIcHSh0iUadg4nURURSxYt97sDqtuDvhXgyIvErqkIiIfEaf2FD8\n8fdXY9/JUnyWfRw7DxRi18EipCbqMea6bujRRSd1iERuxcTrIoIgYGBkMhJiuqNPwECpwyEi8jmC\nIOCqHhEY0D0cvxwuxpdbT2DngULsPFCIhNgQjLmuG5L7RELGWonkg5h4NeH2nmnQ64NRVGSWOhQi\nIp8lCAIG9dXjmoT6qvff7MzH3uOlOFyQh6gwDYYmx2DwVdEI0bKED/kOJl5ERCQpQRDQv3s4+ncP\nx5kiCzb9dBrb95nwadYxfJZ9HMm9I3BLUgwG9gqHXMaH8cm7MfEiIiKP0VWvxbRx/TBlRG/k7DPh\nx9yz+PVIMX49UgxdkAqD+upxXWIU+sSGsh4YeSUmXkRE5HGCApQYOSgWIwfF4pTRjB9zz2LngUL8\n8MsZ/PDLGVcSlpqgR5+4UCjknAkj78DEi4iIPFp8dDDio/vi3pF9cCi/HD8dLMQvh4tcSZhaJUf/\n+DAM7BWBpJ4RCNcFSB0y0WUx8SIiIq+gkMswoEc4BvQIx9QxCTiYX449R4qRd7zEdTsSALpEBKJv\nXCgSuoWib1wYwoK5OJ88BxMvIiLyOnKZDAO6h2NA9/pWbqYyK/KOlSDveCkOF5Qja/dZZO0+CwCI\nCtOgT9cQ9IjRoUcXHeKitLw1SZJh4kVERF7PEBYIQ2ogbk2Ng8PpxCmjBYdOl+FQfjmOFJRj614j\ntu41AqifOYs3aJHQPRyRwWrERWkRqw9CgIp/JZL78buMiIh8ilwmQ88YHXrG6DD2+ng4nSLOlVpx\n4mwlTpyrxPFzlThpNOPY2coL9osK1aBLRCC6RAS5vkZHBCIoQAGBxVypgzDxIiIinyaTCegaGYSu\nkUG4OakLAMBmd6DaAeQeKsTpQgtOF5pRUFSFPcdKsOdYyQX7a9QKRIVqoA8NgD5MA32IBuG6AITr\n1IjQBUCj5l+l1HL8biEiIr+jVMgR0yUYOrX8gu1max2MpVacK7HCWGKFsdSKovJqnCupwilT091M\nNGo5woMDEKJVIVSr/u1rkAq6QBWCg1TQBSoRpFGyDRIx8SIiImoQHKhCcKAKfWJDL9guiiIqqupQ\nWFaN4opqlFbWorSyBqXmWpRU1KDMXIszxVVXPLZMEBCkUUCrqU/CtAFKBGkUCApQIlCtQGDA+V9q\nJTRqOTRqBQLUCmhUcgSoFFDIBd7y9AHtSry++eYbvP322zh27BjWrl2LAQMGuN5bunQpPvvsM8jl\ncrz00ku4+eab2x0sEZHUVq5cibVr10IQBCQkJOD1119HYWEhnn32WVRUVGDAgAFYuHAhFAr+u9aX\nCIKAUK0aoVo1EuJCm/xMnc2B8qo6VFhqUW6pQ2VV/S+ztQ6VVhsqrXWwWG2wVNtgKq2GUxRbFYNc\nJkCtlEOtkiNAJXfNoKmUMqgVcqiUcqiUMigVMqgUcqgUMiiV9b9XKmRQyAUoFfL6r3IZFA2/FAIU\nchnkMsG1TS4TIJcL9V9lMsjlAmfrOki7rgwJCQl4++23MW/evAu2Hzt2DF9//TUyMjJgNBoxbdo0\nbNq0iZk6EXk1k8mE1atX4+uvv4ZKpcLTTz+NDRs2IDs7G9OmTcPYsWMxf/58rF27Fvfee6/U4VIn\nUynliArVICpU0+xnnaKImlo7LNU2WGvtsNac/3X+99W1dlTX2VFT6zj/1Y4amwO1dQ7U1Dlgqbah\npLIWdTZHJ5xZPUGof3ChPhkTIGv0VSZcuE0QBMhk9bN8De/LhPr1dsL53wsyAQFqJWx1dtd2QYDr\nfVmj1wLqv0JA/b7nt6HhM2i8DRDQsG/978//d8F7Ded0/hNonKK4juXaB9AFqXBLcky7E9B2JV49\ne/YEUD8F21hmZibGjRsHhUKB2NhYxMfHIzc3F8nJye0ZjohIck6nE9XV1ZDJZKipqUFUVBR27NiB\nf/zjHwCAiRMn4l//+hcTL7oimSAgMECJwABlm4+h1wfDVFgJm92JOpsDtTYH6mxO2Oz1v+rsDtTZ\nnbCff21z/Pae3dHwS4TdUf+ewyHC4XDC4RRd7znFRtucIhwOEU6nCIfz/HvO+vdFUUSd3QmnCDgc\n9V+dogjx/OecTrR6hs8TXdUjAhEh7euM4Ja5cJPJhJSUFNdrg8EAk8nkjqGIiDqNwWDAtGnTMGzY\nMGg0Gtx0003o378/dDodZLL6gpzR0dEoLCyUOFLyFzLh/O1HpRzBUgfTAqJ4YSIWEaFFUZH5/Pbz\nyZpY/7mGr06nCBGo/yWKQOPP1R8UovhbYteQ3zV+LaL+w/Wfr3/d8LnGk0eNj4mGMc+/F6xRtjvp\nAlqQeE2bNg3FxcWXbH/mmWcwYsSIJve5eAYMAG8zEpHXq6ysRGZmJn744QcEBwfjqaeewubNmy/5\nHK93RE0TBAFyQUBD4wCNWuF35TiaPdsVK1a0+qDR0dE4d+6c67XRaERUVFSL9tXrPSdnZyyX8pQ4\nAM+JxVPiADwrFl+0bds2xMXFITS0fnH1rbfeil9//RWVlZVwOp2QyWRee71rC2+PH/D+c/D2+AHf\nOIfW6LBmVY1nuUaMGIGMjAzU1dXh9OnTyM/PR1JSUkcNRUQkiZiYGOzZswe1tbUQRRE5OTno06cP\nrr/+enzzzTcAgHXr1mHkyJESR0pEnkoQm7ov2ELfffcdXn31VZSVlUGn0yExMRHvvfcegPpyEmvX\nroVCoWA5CSLyGW+//TY2bNgAhUKB/v37Y8GCBTAajXj22WdRWVmJfv36YdGiRVAq275omoh8V7sS\nLyIiIiJquQ671UhEREREV8bEi4iIiKiTMPEiIiIi6iQemXitXr0at912G9LS0vDGG29IHQ6WL1+O\nxMRElJeXSzL+woULMXbsWIwfPx6zZ8+GxWLp9Bg2b96M2267DWPGjMGyZcs6fXygvizJ/fffj3Hj\nxshCFA0AAAwvSURBVCEtLQ2rVq2SJI7GnE4nJk6ciEcffVTSOMxmM5588kmMHTsWt99+O/bs2SNJ\nHCtXrsQdd9yBtLQ0zJkzB3V1dZLE4ck84WeptebOnYvBgwcjLS3Nta2iogIPPfQQxowZg+nTp8Ns\nNksY4ZVd7trhTedQV1eHu+++GxMmTEBaWhrefvttAEBBQQGmTJmCMWPG4Nlnn4Xdbpc40iu7+Jrp\nbfGPGDECd955JyZMmIDJkycDaMP3kehhcnJyxGnTpok2m00URVEsKSmRNJ5z586JDz30kDh8+HCx\nrKxMkhi2bt0qOhwOURRFcdGiReIbb7zRqeM7HA7x1ltvFQsKCsS6ujrxzjvvFI8ePdqpMYiiKBYW\nFor79+8XRVEULRaLOHr0aEniaGzFihXinDlzxJkzZ0oaxx//+Edx7dq1oiiKos1mE81mc6fHYDQa\nxREjRoi1tbWiKIriU089Ja5bt67T4/BknvKz1Fo//fSTuH//fvGOO+5wbVu4cKG4bNkyURRFcenS\npeKiRYukCq9Zl7t2eNM5iKIoWq1WURRF0W63i3fffbe4e/du8amnnhIzMjJEURTFefPmiR999JGU\nITbr4mumt8U/YsQIsby8/IJtrf0+8rgZr48++giPPPIIFIr62q7h4eGSxvPaa6/hhRdekDSGwYMH\nu9qRpKSkwGg0dur4ubm5iI+PR9euXaFUKnH77bcjMzOzU2MAAL1ej379+gEAgoKC0KtXL0lbsxiN\nRmRnZ+Puu++WLAYAsFgs2LVrF+666y4AgEKhgFarlSSWhj6Gdrvd1ceQfuMpP0utlZqaCp1Od8G2\nzMxMTJw4EUB9f8rvvvtOitBapKlrh8lk8qpzAACNpr75dl1dHex2OwRBwI4dOzBmzBgA9efw7bff\nShniFTV1zczJyfGa+IGGFkbOC7a19vvI4xKvkydPYteuXZgyZQqmTp2KvLw8yWL5/vvv0aVLF/Tt\n21eyGC62du1aDBkypFPHNJlM6NKli+u1wWCQvBddQUEBDh48KGlh3oakXOr2MAUFBQgLC8OLL76I\niRMn4s9//jNqamo6PY7GfQyHDBmC4OBgDB48uNPj8GSe+LPUVqWlpYiMjARQn9iUlZVJHFHLNFw7\nkpOTUVJS4lXn4HQ6MWHCBNx000246aabEBcX51V9Qi++ZpaVlSEkJMRr4gfqWx5Nnz4dd911Fz79\n9FMAaPX3kSQNki7X//Hpp5+Gw+FAZWUlPvnkE+Tm5uLpp592678IrxTL0qVL8f7777u2iW4sedaS\nnpjp6elQKpUXrLPoDO4877aoqqrCk08+iblz5yLo/7d3/yF11X8cx5/33tawKVFemS0lLBFq80fG\n3CJxYXcbc2sm7AdIEJQIWdnskt3dWIuZCVZC/bFkWFsbDiHUjSgINreJ605pWRLixtzGVatrQw3d\nzLze2x/DS87rmn3nuffu+3r8dX98zufzPnjOx/c953Pve8mSkMRw8uRJrFYrjz76KO3t7SGJYZrX\n66W7u5t33nmH1NRUKisr2bdvH6WlpYbGcWMdw9LSUr766ivDj9dwFm7n0v+bG+eOUH9omi+z2cyR\nI0cYGxvjlVdeobe3d1abcN2nYHOm3++fdU6Ea/zTGhoaiIuLY2hoiBdffJGkpKR5xxySxOtm9R8b\nGhpYt24dAGlpaZjNZoaHh7nvvvsMjeX8+fMMDAyQn5+P3+/H4/EEMtzY2FjD4pjW3NzMqVOnQrKg\nPD4+nl9++SXw3OPxhOwWktfrpbS0lPz8fGw2W0hiAPjhhx9oaWnh1KlTTExMcPXqVcrLy6murjY8\nlvj4eOLj40lNTQVg/fr1gQoSRrqxjuHatWvp7OxU4vUP4XQu/a9iY2O5cuUKVquV33//PeTLQv5N\nsLkj0vZhWnR0NCtXruSnn376z3VCjRZsznz//fcZHR2NiPinxcXFAdeXQdlsNrq6uuZ9HIXdrUab\nzYbL5QLg0qVLeL3eBUu6biYlJYXTp09z/PhxWlpaWLp0Kc3NzQuSdP2b1tZW6urq+PTTT7n77rsN\nHz81NRW3283AwAB//fUXX3/9dchq0TmdTpKTk3nhhRdCMv60N954g5MnT3L8+HFqampYtWpVSJIu\nAKvVygMPPMClS5eA62smHnnkEcPjCFbHMBRxhLNwOpfm68YrE7m5uTQ1NQGRUZ8y2NwRSfswNDQU\n+Lbcn3/+icvlIjk5OWLqhAabMz/88MOIiR9gfHycq1evAnDt2jXa2tpISUmZ93EUdiWDJicncTqd\n9PT0sGjRIhwOB1lZWaEOi2eeeYbGxsbAp3kjrVu3jsnJycDY6enpvPvuu4bG0NraSmVlJX6/ny1b\ntlBcXGzo+ABnz57l+eefJyUlBZPJhMlkoqyszPA1bzfq6Ojg888/p7a2NmQx9PT08Pbbb+P1eklM\nTKSqqoqYmBjD4whWx1A1C2cKh3Npvux2O+3t7YyMjGC1Wnnttdew2Wy8/vrr/PrrryxbtoyPP/54\n1gL8cDHX3JGWlsaOHTsiYh/OnTuHw+HA5/Ph8/nIy8vj5Zdfpq+vL+LqhP5zzoyk+Pv6+nj11Vcx\nmUxMTU3x7LPPUlxczMjIyLyOo7BLvERERETuVGF3q1FERETkTqXES0RERMQgSrxEREREDKLES0RE\nRMQgSrxEREREDKLES0RERMQgSrxERCRibdu2jYKCAjZu3Mjy5cspKCigoKAAp9M5776KiopmVBaY\ny86dO/nxxx//S7jz0t3dzbfffrvg44ix9DteIiIS8QYGBtiyZUug8kkw06VpIsWXX36Jy+WipqYm\n1KHIbRSSWo0iIiILzeVyUV1dTUpKCufOncNutzM0NER9fT1erxdgRnWUNWvWcODAAZKSkigsLOTx\nxx+ns7OTwcFBNm3axI4dOwAoLCykpKSE7Oxs3nzzTaKjo+nt7cXj8ZCZmUlVVRUAv/32G+Xl5QwP\nD5OYmMjU1BS5ubls3759RpxXrlzBbrczPDwMQHZ2NkVFRezdu5dr165RUFDAqlWrcDgcdHZ2UlNT\nw/j4OAClpaXk5OTgdrspLCwkLy8Pl8uF2Wxm9+7dZGZmBu2/vLx84f8AEpQSLxERuWOdP3+eiooK\nVqxYAcAff/xBfn4+AL29vRQVFXHixImg23o8Hg4fPszo6Cg2m42tW7fy4IMPzmp34cIF9u/fj8/n\nIz8/n46ODrKystizZw85OTkUFRXR39/P5s2byc3NnbX90aNHSU5OZteuXQCMjo4SExNDSUkJZ86c\n4aOPPgrEvmfPHj777DPuv/9+PB4P27Zt45tvvgGuJ3AZGRk4nU5cLhd2u51jx44F7V9CR4mXiIjc\nsR5++OFA0gVw+fJlPvnkEwYHB7FYLAwODjIyMhK0Du+GDRsAiImJISkpCbfbHTTxWrt2LXfddf3f\n6WOPPYbb7SYrK4v29nbee+89ABISEuasO5yRkUF9fT1RUVGsXLmS7OzsoO3Onj1Lf38/L730UqBo\nucVioa+vj3vuuYeoqCjy8vIAePLJJ7FYLFy+fPmW+xdjKPESEZE71pIlS2Y8LysrY/fu3axZswaf\nz0daWhoTExNBt128eHHgsdlsZmpqal7tTCbTLcX4xBNP0NTUxHfffUdjYyN1dXUcOnRoVju/38/y\n5cs5cODArPfcbves13w+HyaT6Zb7F2NEzipDERGRm7iV74qNjY2RkJAAQENDw5zJ1O2QlZVFU1MT\ncH3xf0dHR9B2/f39REdHk5eXh8Ph4OeffwYgOjp6xm3BzMxMLly4wPfffx94raurK/B4fHw8cNvx\nzJkzADz00ENz9i+hoSteIiJyR7iVK0xOp5Pi4mLuvfdenn76aWJiYoJuf2Nfc713s3a7du3irbfe\n4ujRoyQkJJCenj5jvGkul4uDBw9isVjw+/1UVFQA8NRTT/HFF1/w3HPPsXr1ahwOB3v37uWDDz5g\ndHSUyclJEhMTqa2tBcBqtdLV1UVtbS0mk4mamhosFsuc/Uto6OckREREFsDExASLFi3CbDbj8XjY\nunUr9fX1JCYm3vaxpr/V2NbWdtv7lttLV7xEREQWwMWLF9m5cyd+vx+fz0dZWdmCJF0SWXTFS0RE\nRMQgWlwvIiIiYhAlXiIiIiIGUeIlIiIiYhAlXiIiIiIGUeIlIiIiYhAlXiIiIiIG+RunOLafdYsI\nRAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xc447410>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#@test {\"output\": \"ignore\"}\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Set up the data with a noisy linear relationship between X and Y.\n",
    "num_examples = 50\n",
    "X = np.array([np.linspace(-2, 4, num_examples), np.linspace(-6, 6, num_examples)])\n",
    "# Add random noise (gaussian, mean 0, stdev 1)\n",
    "X += np.random.randn(2, num_examples)\n",
    "# Split into x and y\n",
    "x, y = X\n",
    "# Add the bias node which always has a value of 1\n",
    "x_with_bias = np.array([(1., a) for a in x]).astype(np.float32)\n",
    "\n",
    "# Keep track of the loss at each iteration so we can chart it later\n",
    "losses = []\n",
    "# How many iterations to run our training\n",
    "training_steps = 50\n",
    "# The learning rate. Also known has the step size. This changes how far\n",
    "# we move down the gradient toward lower error at each step. Too large\n",
    "# jumps risk inaccuracy, too small slow the learning.\n",
    "learning_rate = 0.002\n",
    "\n",
    "# In TensorFlow, we need to run everything in the context of a session.\n",
    "with tf.Session() as sess:\n",
    "    # Set up all the tensors.\n",
    "    # Our input layer is the x value and the bias node.\n",
    "    input = tf.constant(x_with_bias)\n",
    "    # Our target is the y values. They need to be massaged to the right shape.\n",
    "    target = tf.constant(np.transpose([y]).astype(np.float32))\n",
    "    # Weights are a variable. They change every time through the loop.\n",
    "    # Weights are initialized to random values (gaussian, mean 0, stdev 0.1)\n",
    "    weights = tf.Variable(tf.random_normal([2, 1], 0, 0.1))\n",
    "\n",
    "    # Initialize all the variables defined above.\n",
    "    tf.global_variables_initializer().run()\n",
    "\n",
    "    # Set up all operations that will run in the loop.\n",
    "    # For all x values, generate our estimate on all y given our current\n",
    "    # weights. So, this is computing y = w2 * x + w1 * bias\n",
    "    yhat = tf.matmul(input, weights)\n",
    "    # Compute the error, which is just the difference between our \n",
    "    # estimate of y and what y actually is.\n",
    "    yerror = tf.subtract(yhat, target)\n",
    "    # We are going to minimize the L2 loss. The L2 loss is the sum of the\n",
    "    # squared error for all our estimates of y. This penalizes large errors\n",
    "    # a lot, but small errors only a little.\n",
    "    loss = tf.nn.l2_loss(yerror)\n",
    "\n",
    "    # Perform gradient descent. \n",
    "    # This essentially just updates weights, like weights -= grads * learning_rate\n",
    "    # using the partial derivative of the loss with respect to the\n",
    "    # weights. It's the direction we want to go to move toward lower error.\n",
    "    update_weights = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)\n",
    "\n",
    "    # At this point, we've defined all our tensors and run our initialization\n",
    "    # operations. We've also set up the operations that will repeatedly be run\n",
    "    # inside the training loop. All the training loop is going to do is \n",
    "    # repeatedly call run, inducing the gradient descent operation, which has the effect of\n",
    "    # repeatedly changing weights by a small amount in the direction (the\n",
    "    # partial derivative or gradient) that will reduce the error (the L2 loss).\n",
    "    for _ in range(training_steps):\n",
    "        # Repeatedly run the operations, updating the TensorFlow variable.\n",
    "        sess.run(update_weights)\n",
    "\n",
    "        # Here, we're keeping a history of the losses to plot later\n",
    "        # so we can see the change in loss as training progresses.\n",
    "        losses.append(loss.eval())\n",
    "\n",
    "    # Training is done, get the final values for the charts\n",
    "    betas = weights.eval()\n",
    "    yhat = yhat.eval()\n",
    "\n",
    "# Show the results.\n",
    "fig, (ax1, ax2) = plt.subplots(1, 2)\n",
    "plt.subplots_adjust(wspace=.3)\n",
    "fig.set_size_inches(10, 4)\n",
    "ax1.scatter(x, y, alpha=.7)\n",
    "ax1.scatter(x, np.transpose(yhat)[0], c=\"g\", alpha=.6)\n",
    "line_x_range = (-4, 6)\n",
    "ax1.plot(line_x_range, [betas[0] + a * betas[1] for a in line_x_range], \"g\", alpha=0.6)\n",
    "ax2.plot(range(0, training_steps), losses)\n",
    "ax2.set_ylabel(\"Loss\")\n",
    "ax2.set_xlabel(\"Training steps\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lSWT9YsLP1de"
   },
   "source": [
    "This version of the code has a lot more comments at each step. Read through the code and the comments.\n",
    "\n",
    "The core piece is the loop, which contains a single `run` call. `run` executes the operations necessary for the `GradientDescentOptimizer` operation. That includes several other operations, all of which are also executed each time through the loop. The `GradientDescentOptimizer` execution has a side effect of assigning to weights, so the variable weights changes each time in the loop.\n",
    "\n",
    "The result is that, in each iteration of the loop, the code processes the entire input data set, generates all the estimates $\\hat{y}$ for each $x$ given the current weights $w_i$, finds all the errors and L2 losses $(\\hat{y} - y)^2$, and then changes the weights $w_i$ by a small amount in the direction of that will reduce the L2 loss.\n",
    "\n",
    "After many iterations of the loop, the amount we are changing the weights gets smaller and smaller, and the loss gets smaller and smaller, as we narrow in on near optimal values for the weights. By the end of the loop, we should be near the lowest possible values for the L2 loss, and near the best possible weights we could have."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dFOk7ERATLk2"
   },
   "source": [
    "## The details\n",
    "\n",
    "This code works, but there are still a few black boxes that are worth diving into here. `l2_loss`? `GradientDescentOptimizer`? What exactly are those doing?\n",
    "\n",
    "One way to understand exactly what those are doing is to do the same thing without using those functions. Here is equivalent code that calculates the gradients (derivatives), L2 loss (sum squared error), and `GradientDescentOptimizer` from scratch without using those functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "cellView": "both",
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "output_extras": [
      {
       "item_id": 1
      }
     ]
    },
    "colab_type": "code",
    "collapsed": false,
    "executionInfo": {
     "elapsed": 4219,
     "status": "ok",
     "timestamp": 1474671842604,
     "user": {
      "displayName": "",
      "photoUrl": "",
      "userId": ""
     },
     "user_tz": 420
    },
    "id": "_geHN4sPTeRk",
    "outputId": "3ee8e5e5-0db0-4e6b-ef7e-f9e530fa7bee"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAl4AAAESCAYAAAAsU9sMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xl4VOX1wPHvnTWTfZ8AgSCbQSCgUBFEWUVBkcWli3UB\nFBXFpVh/FVuhatWCtrW1ZVHccKuiqAgIGiSAsgiVhFV2wjaThGwzk22W+/sjMAYIEpKZzEzmfJ7H\nh+TOnfueC0k8ed/3nqOoqqoihBBCCCH8ThPoAIQQQgghwoUkXkIIIYQQzUQSLyGEEEKIZiKJlxBC\nCCFEM5HESwghhBCimUjiJYQQQgjRTHySeE2bNo3+/fszatQo77GysjImTJjAtddey8SJE7HZbL4Y\nSggh/MpisXDHHXcwcuRIRo0axYIFCwCYOXMmI0aMYPTo0UyZMgW73e59z9y5cxk+fDgjRoxg7dq1\ngQpdCBECfJJ4jRs3jvnz5592bN68efTr14/ly5fTt29f5s6d64uhhBDCr7RaLU888QRLly7lgw8+\n4J133mHfvn0MGDCAJUuW8Nlnn5GRkeH9mbZ3716WLVvG0qVLefXVV/nzn/+MlEcUQpyLTxKvPn36\nEBsbe9qx7Oxsxo4dC8DYsWP5+uuvfTGUEEL4VUpKCl27dgUgKiqKjh07UlBQQP/+/dFoan9k9urV\nC4vFAsDKlSsZOXIkOp2O9PR0MjIyyMvLC1j8Qojg5rc9XsXFxSQnJwO1P8hKSkr8NZQQQvjFkSNH\n2LVrF1lZWacdX7hwIQMHDgTAarXSqlUr72tmsxmr1dqscQohQodsrhdCiHo4HA4eeughpk2bRlRU\nlPf47Nmz0ev13HDDDQD1LisqitJscQohQovfEq+kpCSKiooAKCwsJDEx8bzvkX0RQohg4HK5eOih\nhxg9ejTDhg3zHl+0aBE5OTm89NJL3mNpaWkcP37c+7nFYiE1NfW8Y8jPOyHCk85XFzrzh8iQIUP4\n5JNPmDRpEosWLWLo0KHnvYaiKBQWBsfTjykpMRJLkMYBwRNLsMQBwRdLKJs2bRqdOnXizjvv9B5b\nvXo1r732Gu+88w4Gg8F7fMiQITz22GPcddddWK1W8vPzz1qarE8w/bxrjGD6emusUL+HUI8fQv8e\nGvOzzieJ19SpU9mwYQOlpaUMGjSIKVOmMGnSJB5++GE+/vhjWrduzcsvv+yLoYQQwq82b97M4sWL\n6dKlC2PGjEFRFB555BH+8pe/4HQ6mTBhAgA9e/ZkxowZdOrUiREjRnD99dej0+mYPn26LDUKIc5J\nUYNsvjtYMt9gysKDJZZgiQOCJ5ZgiQOCLxZxfsHy79UYwfT11lihfg+hHj+E/j005medbK4XQggh\nhGgmkngJIYQQQjQTSbyEEEIIIZqJJF5CCCGEEM1EEi8hhBBCiGYiiZcQQgghRDORxEsIIYQQoplI\n4iWEEEII0Uwk8RJCCCGEaCY+69UohAiczdbvqSm20y9xcKBDEUII8TNkxkuIEPeDdTPv7XyHDUc2\nnNWsXgghRHCRxEuIEPZj8S7e3/UOJl0E9//ifmnOLIQQQU4SLyFCVH75Id7aPh9FUZjQ/R7SY9MD\nHZIQQojzkMRLiBBUUFHAa1vnUuN28ttL7qJDfKdAhySEEKIBJPESIsSUVZfyat5sHE4HN1/8S3ok\nZwU6JCGEEA0kiZcQIaTCWcG8vNkUVxUz4qKRXNGqX6BDEkIIcQEk8RIiRDjdTl7f9ioWh4UBba5i\naLvhgQ5JCCHEBZLES4gQ4FE9LNj5JgfK9tMr9VLGdLpJnmAUQogQJImXEEFOVVU++vEDthdto0tC\nF36d+VtJulqAnP8dCXQIQogAkMRLiCC37MASNlo20DamLXd2m4hOIw0nWoJ8qy3QIQghAkASLyGC\n2JojOWTnf0WKKZm7e9xLhC4i0CEJH7FX1AQ6BCFEAEjiJUSQ+sG6mc/2LiLGEMM9WZOJNsQEOiTh\nQ45KV6BDEEIEgCReQgSh3cU/8v6udzBqDUzKup8kU1KgQxI+5qhyBjoEIUQASOIlRJDJLz/Em9tf\nQ1EUJvaYROvoNoEOKaxYLBbuuOMORo4cyahRo3j77bcBKCsrY8KECVx77bVMnDgRm+2nPVrPPvss\nw4cPZ/To0ezcubNB48hSoxDhSRIvIYKItAIKPK1WyxNPPMHSpUv54IMPePfdd9m3bx/z5s2jX79+\nLF++nL59+zJ37lwAcnJyyM/PZ8WKFTz99NNMnz69QePIjJcQ4UkSLyGCxGmtgLrcKq2AAiQlJYWu\nXbsCEBUVRceOHbFarWRnZzN27FgAxo4dS3Z2NgDZ2dmMGTMGgJ49e2Kz2SgqKjrvOI5KSbyECEeS\neAkRBCqcFbyaN+enVkCt+wc6JAEcOXKEXbt20bNnT06cOEFycjJQm5wVFxcDUFBQQFpamvc9ZrMZ\nq9V63mvbKyTxEiIcSUEgIQLM6XbyxrbXOO44zpVtBkgroCDhcDh46KGHmDZtGlFRUecsWquq6lnH\nGlLgtsblIS4+EoNe2+RYAyUlJfSftA31ewj1+KFl3MOFkMRLiADyqB7e2fkW+8v2SSugIOJyuXjo\noYcYPXo0w4YNAyApKYmioiKSk5MpLCwkMTERqJ3hslgs3vdaLBZSU1MbNE7+kRLioo2+v4FmkJIS\nQ2FhaBeBDfV7CPX4IfTvoTFJoyw1ChEgqqqycPd/2Va0lc4Jnfl15m/RKPItGQymTZtGp06duPPO\nO73HhgwZwieffALAokWLGDp0KABDhw7l008/BWDLli3ExsZ6lyTPp6JaankJEW5kxkuIAPny4FI2\nHF9PenQ6d3W7W1oBBYnNmzezePFiunTpwpgxY1AUhUcffZR77rmHRx55hI8//pjWrVvz8ssvAzBw\n4EBycnK45pprMJlMPP/88w0eq6JKEi8hwo38pBciANYeXc3Xh1aQYkrmnqz7pBVQEOndu/c5a3G9\n+eab9R5/6qmnGjWWQxIvIcKOrGsI0cy2FPyPT/d8Iq2ABBXV8mSjEOFGEi8hmtHu4h95b+cCaQUk\nAFlqFCIcSeIlRDM5bMv3tgKa0OMeaQUkJPESIgxJ4iVEMyisKOS1vJOtgLreScf4zoEOSQQBeapR\niPAjm+uF8CO7w87r38zjy+KlAPx+wBP0SOkZ4KhEsKiQfo1ChB2Z8RLCj9785jU+r/iMMl0p0fEx\n7Nn1Y6BDEkFElhqFCD+SeAnhJ063k5WlX1PpcmCOakV6TFsKagoCHZYIIrLUKET4kaVGEfRsNgdz\n5mzFYjFhNlcweXJW0Pf28qge3t35NlWeKhIjk2gfexEAqYaGtZIRLZ/RoJU6XkKEIUm8RNCbM2cr\nW7YMQlEULBaV2bNXMXNmWqDDOidVVfl494dsLcrjusyR6Av1nCg7QaohldsG3nn+C4iwEBWhp1IS\nLyHCjiReIuhZLCZv4+ja5MsU4Ih+3vKDy1h/fB3p0elM6jVZqtKLekWZ9BSXVQY6DCFEM5M9XiLo\nmc0VqKoK1M4mpaUF7/+s1h5dzVeHlpNsSuLurHsl6RLnFG3SU1Ht8n5tCyHCg8x4iaA3eXIWs2ev\nwmIxkZZWyf339wh0SPWq2wpoUtYDxBhiAx2SCGJRJj2qClU1bkxG+VEsRLiQ73YR9KKjo/j9768I\ndBg/q24roHt63CetgMR5RUfqgdqSEpJ4CRE+5LtdiCawO+z89fO/8HXJcoyaCJ4f+iJtYtIDHZYI\nAdERJxOvaheSpgsRPmSPlxBN8I8v/sHSii+oMFSSlJTCui1rAx2SCBFRplMzXlK9Xohw4vcZryFD\nhhAdHY1Go0Gn07Fw4UJ/DylEveqrBxYdHdWoa1msBTzw0l/YHPUFxDq5POMykiOTKSiVAqmiYeou\nNQohwoffEy9FUViwYAFxcXH+HkqIn1VfPbDG7h178KUX2JKUi8utgCeS3buPk963nRRIFQ0WVWep\nUQgRPvyeeKmqisfj8fcwIow1dCbLV/XAnG4nu03/w6OtIMrVFffBKmo8djKzMqVAqmiwUzNeUr1e\niPDSLDNeEydORFEUfvnLX3Lrrbf6e0gRZl5+eSNffBFDdbUWg0FHTc1G/vjHwWedZzZXYLGoKIrS\n6Hpgp1oBKRGV6JxJRLu64I73cHGVnkkjJ/vidkSYkD1eQoQnvydeH3zwASkpKRQXFzN+/Hg6dOhA\nnz59znl+MPXgk1jOFixxwE+xrFljx2YbhaIo1NSorFnzfr1xTp9+JS++uI5jxyJo3bqKxx7rf0F7\nvGx2G5P/+yDbyreRldEOZ247TtTUkGaK4a2//iko/m6CIQbRMLLUKER48nvilZKSAkBiYiLXXHMN\nW7du/dnEq7DQ5u+QGiQlJUZiCdI44PRYXC4jqqp6K4C73RHnjPOBBy71flxZ6aGysuH387uFD7K+\nZiNRxigSE1PoNrAjT975f96xAv13E2z/PuLnRUcaANlcL0S48Ws5icrKShwOBwAVFRWsXbuWzp07\n+3NIEYYGD44gNrYIo7GU2NgiBg82+nyMb4+uYYv9ByJ0EWQmXoJOq6egRp5gFI3301KjJF5ChBO/\nzngVFRXx4IMPoigKbrebUaNGMWDAAH8OKcLQI4/0xmjMrdNSqLdPr59b8AOL9nxMnC6eVgltMGgN\nqKoqTzCKJok06lCQPV5ChBu/Jl5t27bls88+8+cQQvi1pdCekt28t6u2FdDM4X/nm01fU+AoINWQ\nKk8wtmDTpk1j1apVJCUlsXjxYgB27drF9OnTqa6uRqfT8dRTT5GVlQXAs88+y+rVqzGZTLzwwgt0\n7dr1vGNoNAomo072eAkRZqRlkGixmlIw1e6w85/sf/JVyZcYNRE8N3QWnVO70HlkFz9HLYLBuHHj\nuP3223n88ce9x2bNmsWUKVMYMGAAOTk5zJo1iwULFpCTk0N+fj4rVqwgNzeX6dOn8+GHHzZonMgI\nSbyECDfSMki0WKcKplqtV5CbO5jZs7c2+L3zVv6HJRWLva2A1m/5tlEx2GwOZs1az9SpucycuQ67\n3dGo64jm1adPH2JjY087pigKNlvtwws2mw2z2QxAdnY2Y8aMAaBnz57YbDaKiooaNE5khE7qeAkR\nZmTGS7RYjS2Yaqsp5+uSFbh0Ti6K69ikVkC+rJYvAuuJJ57g7rvv5q9//SuqqvLBBx8AUFBQQFpa\nmvc8s9mM1WolOTn5vNeMNOqornHj9njQauT3YCHCgSReosVqTMHUSlcl8/JmA9Amui3mqLQmbaT3\nVbV8EXjvv/8+Tz75JMOGDePLL79k2rRpvPHGG94yJnWd+jc/n4Q4E1CKKSqCuGjfP43rby2hbEio\n30Ooxw8t4x4uhCReosWaPDmL2bNX1XnascfPnu90O3lj22scsx/jpq6/ZPmHu/mhykGyMZo/PH5z\no2LwRbV8ERw+/fRT/vjHPwJw3XXXeT82m81YLBbveRaLhdTUhiXq2pP52eFjpdQkRPo2YD8Lprpx\njRXq9xDq8UPo30NjkkZJvESLdSFPO55qBbSvdC9ZKT2xLs1Ef2wKrU8mTG+9tYr77os6a7N+RISG\nWbPWn3MD/4UmfyJ4nDmTZTab2bhxI5dffjnr1q0jIyMDgKFDh/Luu+8ycuRItmzZQmxsbIOWGaF2\nqRGklpcQ4UQSLxH2VFXlkz0fsbUoj47xnfhN5u384fUdZy0R1rdfKzLS+LN7uPxZ6kL4z9SpU9mw\nYQOlpaUMGjSIKVOm8Mwzz/Dss8/i8XgwGo0888wzAAwcOJCcnByuueYaTCYTzz//fIPHiYqQxEuI\ncCOJlwh7Kw59ybpj39Emug3ju9+NXquvd4mwvv1aEREG2cPVAr300kv1Hv/kk0/qPf7UU081apxI\n6dcoRNiRx2hEi/dzJR2+O7qWFQe/JCkiiXuy7sOkq02cJk/OolevVZjN6+nVaxX3398Ds7nCu/x0\nKhlr3bryrGNCNFTkyRkvh1SvFyJsyIyXCGpNKYJ6yrlKOuQVbuGTPQuJ1kdzT9b9xBh+qttU3xJh\nffu1UlJiePpp2cMlGufUHq9KWWoUImxI4iWCmi/qYNW3RLinZDfv7nwbo9bAPVn3kRKZct7r1JeM\nyR4u0RSnZrxkqVGI8CGJlwhqvqiDlZBQzLffrqGi0kNN6ie0ST/A1C/K6NS6C3dfdh/pMW19HbYQ\nDXJqj5dUrxcifMgeLxHU6ttXdaEURQt0ozJxA9U9CjjeeicntEXElEfTOUF6L4rA+amchOzxEiJc\nyIyXCGq+qINlLdCj6/AOnthPcKUdp8atpX1cB/ROgx8iFqLhomSpUYiwI4mXCGq+2EN1VPNfjnfY\nSoV+Nx6jE1NNEubINFIrG9cGSAhf0es06LSK1PESIoxI4iVaPFfrXbhMe1DcHnTlCpEVKpmVmdw2\n8M5AhybCnKIoRBp1kngJEUYk8RItlt1h551Vb7KzeiceTRXxsdHEJcXT+mg6k0ZODnR4QgC1G+xl\nj5cQ4UMSLxG06tbwio8vQ1HclJQkYjZXMH36led9/7ur3mK5cxnaFC1apwbtfh3xrRIY1HlwM0Qv\nRMNERugoKqstxHvqCV4hRMsliZcIWnVreH37bTGwnU6drsBiUXnxxXU88MClP/v+jbYNFHistI5v\njb7KgJKoYWyncbLEKIJKZIQOl1vF6fJg0GsDHY4Qws8k8RJBq24Nr+pqHRCBy+Xi0CEbhw+X4XCs\nO2cl+3XHvuV4zTGMxgguSe6BXqMnMyVTlhhF0DlVUsJR5ZLES4gwIHW8RNCqW8PLYHBhNFZx6JCN\n0tJE3O44cnMHM3v21rPel1e4hY93f0SvdpdxQ+SNmG1m2UwvgpY0yhYivMiMlwhadWt4dexYxObN\nFnbtKkCjqaRt25H1VrLfW7KHd3e+jUGr5/5eD9I2pl2AoheiYby1vGSDvRBhQRIvEbTq1vCaNWs9\ncXETSU4upawskcOHv6V9+37eSvZ2h53/ZP+Tr0qWY9QYeG7oLEm6REj4qXq9zHgJEQ5kqVGEhFP7\nvdq3jyE+vhittoxevVZx//09sBZaufbpQbyy62X2WnYTZYhh/ZbvAh2yEA0ijbKFCC8y4yVCgtlc\ngcWiotPp6NAhnv7947xPNd475y7yMw7h0agoCuzdspsefbMCHLEQDePd4yUzXkKEBUm8REg4s2fj\nY4/1p7LSQ5Wriu26bSiKgk6jRa/RU6GtJNUg7YBEaJBG2UKEF0m8REg4s2djdHQUNkcJb25/Da2i\nJcYYi8fpwe1ykVqdIk8wipAhS41ChBdJvMQ51a0c39Bq8c3B7rDzTs5rLD6yFJu7nNsuvYPcjT9Q\noBZiVlKZ9fg/iIo6u7aXEMHoVOLlkKVGIcKCJF7inOpWjm9otXhfOjPxmzw5C2thAdc/fSO29EIw\neOjcthMmm4m3Hnu/2eISwpeiTu7xqpTES4iwIImXOKe6leMVReHYsYhmHf/MxG/27FUs2jeD0h5F\nqAYXeBTyN1o5MfxEs8YlhC+ZjLXV6h2yx0uIsCDlJMQ51a0cr6oqrVtXNev4dRM/t8fBN3u/ID96\nGx6lGlTQqBE49U7ZSC9CmlajwWjQyh4vIcKEJF7inCZPzqJXr1WYzevp1WsVjz3Wu1nHr5v45Xv+\nw8EOn+NKrACNB8p14NQQXR4vG+mFz02bNo3+/fszatSo044vWLCA6667jlGjRvHiiy96j8+dO5fh\nw4czYsQI1q5de8HjRUXopJyEEGFClhrFOdX3JGFlpa3Zxj9VQuLwEZV1rpep1pei12hxHlPQlEJ7\npSvvTZ8jG+mFz40bN47bb7+dxx9/3Htsw4YNfPPNN3zxxRfodDqKi4sB2LdvH8uWLWPp0qVYLBbG\njx/PihUrvLO1DRFp1FFcXu3z+xBCBB+Z8RJBKzo6ijvuuohNpgexdyzEqXWiidaSEB3HoMz+fDfn\nS9q3bx/oMEUL1KdPH2JjY0879v7773PPPfeg09X+vpqYmAhAdnY2I0eORKfTkZ6eTkZGBnl5eRc0\nXmSEnspqF56TM7xCiJZLEi8RtKyFVgbP6Mfe+L2oOhUUcJU40VXruKbbNYEOT4SZgwcPsmnTJm69\n9VZuv/12tm3bBoDVaqVVq1be88xmM1ar9YKuHWnUoQJVss9LiBZPlhpF0PrdWw9SklECCqAFpVpB\nq9XRO74PD4x5gIoKT6BDFGHE7XZTXl7Ohx9+SF5eHg8//DDZ2dnefYh1NXSZMSUlBoDEeBMAEVER\npCRG+i5oPzsVfygL9XsI9fihZdzDhZDESwSl2lZA21GcCopBQTmuATdc5LqIf894laioKCoqmm+/\nmRBpaWkMHz4cgKysLLRaLSUlJaSlpXH8+HHveRaLhdTUhj1pW1hY+zWsoTZ5O3KsFI3b7ePI/SMl\nJcYbf6gK9XsI9fgh9O+hMUmjLDWKoOPyuE62AtIQFxuPqSgCvVZHckkSCx//XDbTi2Zx5kzWsGHD\nWLduHQAHDhzA6XSSkJDAkCFDWLp0KTU1NRw+fJj8/Hyysi6sSfupfo1SvV6Ilk9mvERQ8age3tu5\ngD0le7ir70Q2rd1IQezJVkAP/oOUFKnZJfxv6tSpbNiwgdLSUgYNGsSUKVO46aabeOKJJxg1ahR6\nvZ6//vWvAHTq1IkRI0Zw/fXXo9PpmD59+gU90QgQbaqtXm+vlCKqQrR0kniJoGAttPL4W4+wS7ML\nl9bJL3v/hvt+8SD6K/SBDk2EoZdeeqne47Nmzar3+L333su9997b6PFSTu7xKiipaPQ1hBChQRIv\n0Szq67sYHf3TkuH/vfUoPyT/QIXLgVbRsnV9LvoBknSJ8GA+uaHeUiyJlxAtnSReolnU13fxvvt6\neJOxDYY8nHF2tBodCcYEitTa/ovnS9iEaAmS4yLQKArWkspAhyKE8DPZXC+axZkNty0WkzcZ22M3\nYteCw+7CWR5JSXE1SZ7a4pSnzrFaryA3dzCzZ28N5G0I4Rc6rYaU+AisMuMlRIsniZfwKZvNwaxZ\n65k6NZeZM9dhtzuAsxtup6VVsv+gg03GEWxufQOuag/kdEazsw2673rTsWo8UH/CJkRDLV26FLvd\nDsDLL7/MxIkTvYVPg405MRJbhRNHlWywF6Ilk8RL+NS5ZqjObLh9//092KQ8S3nWZtT4GjwJGozu\naK6oWMuluoXYy1sD9SdsQjTU7NmziY6OJi8vj7Vr1zJmzBieffbZQIdVr7ST+7ysxfI1LkRLJnu8\nhE+da4bqzIbbJypPUNnxCDqdE50rBrfOidZcA1WnJ1inGmVbLCbS0iq5//4ezX9TImSd6qv47bff\ncssttzBq1Chef/31AEdVP7M38aqgQ+vY85wthAhVkngJnzKbK7BYVBRFOecMlb3Gxqt5/8GoMZIU\noyNKH4Xb7UazX4/ZvP60BOvMhE2IC6EoCkuXLmXp0qX85z//AcDpDM6lvLSE2l9S5MlGIVo2vyde\nq1ev5rnnnkNVVW666SYmTZrk7yFFAJ1vhqrKVcVrW+dSWFnE7wc+wZdfL8GqFtQWSH1GCqQK3/rj\nH//Ia6+9xs0330zbtm05ePAgffv2DXRY9fLOeEktLyFaNL8mXh6Ph2eeeYY333yT1NRUbr75ZoYO\nHUrHjh39OawIoHPNUNkddhaseoOvS1ZQ6ankV71u45c9fsOvet0WgChFuLjsssu8M10A7du3509/\n+lMAIzq3+BgjBp1GZryEaOH8urk+Ly+PjIwM2rRpg16v5/rrryc7O9ufQ4og9e6qt1havQSL5jhK\nlIaKw44LbqsixIV64YUXsNlsuFwufvOb39CrVy8+++yzQIdVL42ikJoQibW48qw+kUKIlsOviZfV\naqVVq1bez81mMwUFBf4cUgQhVVX5tnwNxVVFxBhi6ZJ4MUXOokCHJcLAd999R0xMDGvXrsVsNrN8\n+fKg3VwPkJZootrppsxRE+hQhBB+4telxsb81paSEuOHSBpHYjlbY+JYumcpZZQQGxFDz7QstIqW\nDtp2Tb6nUP478ZdgiiWYfP/991xzzTWYzeagnmmt+2RjfLQxwNEIIfzBr4lXWloax44d835utVpJ\nTf35zdOFhTZ/htRgKSkxEssFxlFfe59t5bl8tHshl6b3IaE0EVuBjVRDKqMH/rJJ9xQqfyfNKdhi\nCQZJSUlMnz6dNWvWMGnSJFwuF263O9BhnVNanZ6NF7dLCHA0Qgh/8Gvi1aNHD/Lz8zl69CgpKSks\nWbKEv/3tb/4cUjSjMxMtp9PJjh3Dvf0YZ8ydD72/J0ofxeRLHyY1Up5YFM3rpZde4vPPP2fs2LHE\nxcVx5MgRxo8fH+iwzsksRVSFaPH8mnhptVr+9Kc/MWHCBFRV5eabb5YnGluQMxtfW62LSU5xcER5\nHUf8TrY4NjDSczl397hXki4REImJifz2t7/lwIED7N27l/bt2zNu3LhAh3VOdWe8hBAtk9/reF19\n9dVcffXV/h5GBMCZVepV1c4RXqe88/fYordijLARVx5Lu9iMAEcqwtXWrVt56KGHMBgMqKqKy+Xi\nX//6F926dQt0aPWKNumJitBJLS8hWjCpXC8a7cwq9VcOMLK4/E1K4g6j1cClbX8BNGwjc337w6Kj\no/x7A6LF+8tf/sJzzz1Hv379AFi/fj3PPPMMH3zwQYAjOzdzYiSHLDbcHg9ajbTTFaKlke9q0Whn\nNr6O7ZmHu0MRxlgP8QnRlJaVkmpo2BLjuZprC9EUlZWV3qQL4IorrqCyMrj3T5kTInF7VE6UVQU6\nFCGEH0jiJRrtVJX6Pz/dkciuG3gn/22ceidJjhQSyxKItERx28A7G3StczXXFqIpTCYT69ev936+\nceNGTKbg/tpKSzzVszG4E0QhROPIUqNosgWr3uCL6s+p0VajNWpJNCXStV03MisziYo693Jh3eXF\nvXt/JCKiO3p99DmbawtxoaZNm8bDDz+MwWAAahtk//Of/wxwVD/v9J6NSYENRgjhc5J4iUazFlp5\n/K1HWFu1Fo/JRceUTuhLDVCtkJmSed7ZrrpPRUZE9Ka6+gPS0zvX21xbiMbIyspixYoVHDhwAFVV\nueiiixipvGqIAAAgAElEQVQ+fDirVq0KdGjnlFaniKoQouWRxEs02uNvPcLmpM04HdWoOpXiA8UM\nunoomZWZTBo5+bzvr7u8qNfrSU/vzEsv9fR32CLM6PV6unTp4v082PsgpibULjVK4iVEyyR7vESj\n2B12Nto3UlpRgqJqiKqJxlnlIrPy/DNdp5jNFd7/CcryomguwdwyCCDCoCM+2iB7vIRooWTGSzTK\nc8ufpiqmArQqOoMWpUqhd1LvBs10nTJ5chazZ6/CYjHJ8qLwqb17957zNZfLdd73T5s2jVWrVpGU\nlMTixYtPe23+/PnMmjWL9evXEx8fD8Czzz7L6tWrMZlMvPDCC3Tt2rVJ8aclRvJjfik1TjcGvbZJ\n1xJCBBdJvMQF21a0lfXl39E6tQ2OAw5qlBpiy2OZ9fg/Lug6p56KFMLXJk2adM7XjMbzN58eN24c\nt99+O48//vhpxy0WC9999x2tW7f2HsvJySE/P58VK1aQm5vL9OnT+fDDDxsfPLUb7Hfll1JQWkl6\nSnSTriWECC6SeIWZphYq3V+6l3d2vEm0NopOqZ2JaRuLqqpkVmaSkiJtgURwWLlyZZPe36dPH44e\nPXrW8eeee47HH3+c+++/33ssOzubMWPGANCzZ09sNhtFRUUkJyc3enxzwk8b7CXxEqJlkT1eYaYp\nhUqPlh/l9W2v4lY9/Hnw8/Rx/4KE0oQL2tclRKhauXIlrVq14uKLLz7teEFBAWlpad7PzWYzVqu1\nSWNJz0YhWi6Z8QozF1qo1O6w8/ryeXy95yuORx0hKTqZPw56mk5xF/P19hpKLCYM5gpUaccpWrCq\nqirmzJnD66+/ftZr9T0l2dAN/CkpMfUe73rykmUVrnOeEwyCObaGCvV7CPX4oWXcw4WQxCvMnNlf\n8XxPEr6X8zZfHPmcQ8kHUfFg8ESwY8dW1m3Xe2twWSwqs2evarb9Wjabg3//+wf279dIX0fRLPLz\n8zl69CijR49GVVWsVivjxo3jo48+wmw2Y7FYvOdaLBZSUxu27F5YaKv3uNbjQaMoHDpeds5zAi0l\nJSZoY2uoUL+HUI8fQv8eGpM0ylJjmDmzv+LPPUlod9hZtnUpuyt2UVlWgUkxYdQYKKgpCGiLnzlz\ntrJp01XS11H4Vd2ZrC5duvDtt9+SnZ3NypUrMZvNLFq0iKSkJIYOHcqnn34KwJYtW4iNjW3S/i4A\nnVZDcnyE1PISogWSGa8Q1piN8hfyJOGCVW9wKOkA7ioPLjyc2F2JGuckLj4OwwXOnPmS9HUU/jZ1\n6lQ2bNhAaWkpgwYNYsqUKdx0003e10993QMMHDiQnJwcrrnmGkwmE88//7xPYjAnRLJ1/wkqqpxE\nRuh9ck0hROBJ4hXC6rbc8fVyn6qq5JSuwmgyYiiLxVOuRVtsIPrAHZSr3Xj4oZ9qcCUklFFd7Wbq\n1NxmWfozmysoLpbCq8J/XnrppZ99PTs7+7TPn3rqKZ/H0CY5iq37T3DQYuOS9ok+v74QIjAk8Qph\n/pj5sTvsvLvqLb4tX8ueoh9JTEtEqcmk0mMmrrIH6foHKD6x/rSZs1mz1jfrfq/Jk7N466017N+v\n8RZebWqZDCGCTdf2CXy5MZ/tB4sl8RKiBZHEK4Rd6Eb5hngv521WerI5ouYTmxJH2uFWlJRHUXrw\nEtoZ7sbtOnuc5l76i46OYsaMq0/bkNncyZ8Q/talbTw6rcL2/cXcMijQ0QghfEUSrxDm65Y71kIr\n/9n4L4qTTqDHwJAOw8iIbM8j9/2e2bO3Ulq6nfj4srPG8UcCeKFk35doaYx6LZ3T49l5qIQyRw1x\nUYZAhySE8AFJvEKYr1vu3P/2RE7EnMCj96DTeti853t+0eVy7zjneuw3GHouBkPyJ4Svde+QyM5D\nJew4WEy/bmnnf4MQIuhJ4iWwO+z88+u/sdm1CbQqkdZINHoNlCjcds/5K9IHQ8/FYEj+hPC1bu0T\n+Yh9bNsviZcQLYUkXoI52f/iy8qlaCI1KEbQV+mJy4jnYsPFREWFxgb1YEj+hPC1tqnRxEYZ2H6w\nGFVVG1wRXwgRvKSAapgrrjpBdulXuD0uerf6BTprDI5jHlidwoybnwt0eEKENUVR6NY+kXJHDYcL\n7IEORwjhA5J4hTF7jY15ebNR0NAutj2lR01E19xA29JpdLCv4qOPLOe/iBDCr7p3qC0lsf1gcYAj\nEUL4giReYaraXc38rfMorChkYp9JDNYMwbM3keh9l5CuTpQnA4UIEqdqeG3bL4mXEC2B7PEKM3aH\nnQWr3iC75CsqPBX8qudtjOt6C8olCqXb1pFbPBhFI08GChEs4qIMtEuNZs+RUqqdbox6baBDEkI0\ngSReQc5mc/Dvf//A/v0an1Rkf3fVWyyrWcIJTRHxpgQqjjiwd6hgzpytHDmix25fQEZGW9q2RZ4M\nFCJIdOuQSH6BnR/zS8nqmBTocIQQTSBLjUFuzpytbNp0FVbrFeTmDmb27K2NvpbNbuO9PQv4sXAX\nFRWVXBTbgSJnkbfnY3HxVURH307btvD7318hLXeECBLdTy43bj8gy41ChDpJvIKcLyuy//nLP1Ko\nLQQtGE1G9h3bS6ohVaq+CxHkOqXHY9BrZIO9EC2AJF5BzmyuQFVVgCbtu9p4fAP/s28iJTGVDEd7\nIk+YiLREcdvAO302hhDCP/Q6DZntEjhW5KC4vCrQ4QghmkD2eAW5yZOzeOutNezfr2l0RfbtRdv4\naPcHxOpiSUtpQ1SbKFRVJbMyk6ioKKn6LkQI6HZRInn7TrD9QDFX9Wwd6HCEEI0kiVeQi46OYsaM\nq+vtkdgQB8r2s2DHG+g0Wl4Y9jdW/+8bCioKSDWkctvAO71jSNV3IYJb94tOlpWQxEuIkCaJVwti\nszmYM2crFosJs7mCm+5K5c3dr+JWPUzodg+ZSV3JHNk10GEKIRohLTGSpFgjOw4W43J70Gllp4gQ\noUi+c1uQU08nWq1X8P32LB5+73kqXJX8KvM3dE26JNDhCSGaQFEULu2SgqPKxZY9RYEORwjRSJJ4\ntSAWiwm3x8F+ZrGl7Tj227YzvM119Db/ItChCSF8YPClbQBY+b8jAY5ECNFYkni1IAmJReRxO/kX\n/5MK0z4iorQc2nMg0GEJIXykVVIUXTMS2JVfyrEiR6DDEUI0guzxakFiLvmBStM6VH01evQku3UU\n1BQ0+P1n7hFrapV8IYTvDb60DTsPlfDND0e57ZougQ5HCHGBZMarhVBVlfWV6zFFa4iLNtE6MRmn\n4iTVkNrga9TdI9bUKvlCCP/o1TmZ+GgD3207TlWNK9DhCCEukCReLYCqqize9yllrlLaxLWhnSYD\nY42RDGeGt2REQ0gFeyGCn06rYWCvNlRWu1m/wxrocIQQF0iWGkOY3WHnvZy3+b58I0drjnB5u74k\n21MoNZbW1um69U6iohq+VGg2V2CxqCiK0qAK9rI0KVqqadOmsWrVKpKSkli8eDEAM2fO5JtvvsFg\nMNCuXTuef/55oqOjAZg7dy4ff/wxWq2WJ598kgEDBvg1vqt7tmbxtwdZufkoA3u29v7CJIQIfjLj\nFcLey3mbNepqdni2UWmsIMGWwIM3PMIfx8xg0sjJF5R0QW2V/K5dl2GxvI/Vupjq6hrs9nNv4JWl\nSdFSjRs3jvnz5592bMCAASxZsoTPPvuMjIwM5s6dC8DevXtZtmwZS5cu5dVXX+XPf/6ztwWXvyTE\nGLmsSzJHCu3sO1ru17GEEL4lM14hbIdjOwdc+9Bp9GQmdsPmsDfpetHRURiNJszmESiKws6dKv/4\nxwqMRgMWi4kOHdzceefF3lktWZoULVWfPn04evToacf69+/v/bhXr14sX74cgJUrVzJy5Eh0Oh3p\n6elkZGSQl5dHz549/Rrj4MvS2fRjISt/OEKn9Di/jiWE8B2Z8QpRB8r2s69qD6BwcWJXTDrTBW2k\nP5czk6lVq6q9s1qbNl192qyWNNcW4WrhwoUMHDgQAKvVSqtWrbyvmc1mrFb/773KbBdPq6RINu0q\noNxR4/fxhBC+ITNeIcjiOM7rW+fRqVUXEu2JKBWa03ovNsWZ+7zAcc5ZLWmuLcLR7Nmz0ev13HDD\nDQD1Lis2dM9VSkpMk2IZdVVH5n26lf/tO8EtQ5u/tERT4w8GoX4PoR4/tIx7uBCSeIWYkqpi5uXN\npsJVyW3d76BP2uU+vf6ZyVR1dRw7d9a/4V6aa4tws2jRInJycnj77be9x9LS0jh+/Lj3c4vFQmpq\nw2afCwttTYonq30CRr2WxWv2c+Ulqeh12iZd70KkpMQ0Of5AC/V7CPX4IfTvoTFJoyReIcTutDM3\n7z+UVZdxQ4cbfZ50wdnJlN3u8CZiHTp4uPNOmdUS4eHMmazVq1fz2muv8c4772AwGLzHhwwZwmOP\nPcZdd92F1WolPz+frKysZokxMkLH4Mva8OWGfLI3H+W6vu2aZVwhRONJ4hUiqt3VvL51HoUVhQxq\nO4TB7YY2y7h1E7FQ/81EiIaaOnUqGzZsoLS0lEGDBjFlyhTmzp2L0+lkwoQJAPTs2ZMZM2bQqVMn\nRowYwfXXX49Op2P69OnNWt7h+n4ZrMk9xpJ1B7mqZyuiIvTNNrYQ4sIpqp+ee37llVf48MMPSUpK\nAuDRRx/l6quvPu/7guV/7MGSZNgddj7d9AGfHv6cCk8Ft2b9mjt7TghI3Z5g+TuB4IklWOKA4ItF\nnJ+v/r2+3JDPh9/s5bq+7bh1cCefXPN8gunrrbFC/R5CPX4I/XsIuqXG8ePHM378eH8O0aLZHXbu\nfWU830dtoFqtIdGYSNmBUpReUixRCPGTob3b8PXmw3y96QhDL0snKS4i0CEJIc7Br+Uk/F1EsKWb\nv3weq6u+pRw7NYqTyupK1uzPCXRYQoggo9dpGXtVB1xuD5+u3R/ocIQQP8OvM17vvvsun332Gd27\nd+cPf/gDMTGy/AANb7WzMO8bnAYPqqpF8UTgqK7mfLmstPERIjz165bG8o35fLfVwrW/aEd6anSg\nQxJC1KNJe7zGjx9PUVHRWccfffRRevXqRUJCAoqi8Pe//53CwkKee+65JgXbUsyYsZpNm67ylmjo\n02cNM2acvv9t3eF1jHlxEqXaIjylLtB6MJ4w8NRNj/L4bY836dpCiODg670teftO8I+PcsnqmMQj\nt/i3cn6o782B0L+HUI8fQv8emn2P1xtvvNGg82699Vbuu+++Bp0bLP8A/vxi2L9fg9PpPu3zumPt\nOLGdN7bNp3XkRUT+cAM2wyY8NR66J7fn1ivv+Nm4znftpgimb5BgiSVY4oDgi0U0vx4dEslsF0/e\nvhPsOlRCZkZCoEMSQpzBb3u8CgsLvR9/9dVXdOnS/FWVg9XPtdo5ULaft7e/jlbR8I9fP8ktWUMY\nl/YX7s76A28985fzNr6WNj5ChC9FUbjl5FON7329B5fbE+CIhBBn8tser1mzZrFz5040Gg1t2rTh\n6aef9tdQIae+Vjt2h5052a+wouRL9Bo9Tw9+nktaXcIlv7+wmQxp4yNEeLuoVSxX92zF6tzjLFl3\niNEDLgp0SEKIOvyWeM2cOdNflw559bXaeXnxiyyp/JwafQ0d4jqxOW8jvdP7+OTaQojwcuvgzmzd\nX8wX3x3k0s7JtDPL0q8QwcKv5SREwzicDr4qWUGNu4Z2se1JjTJTUFPg83FsNgezZq1n6tRcZs5c\nh93u8PkYQojAi4zQcdeITNweldeX7pQlRyGCiCReAVbtrmb+1rk43U6qCyPJ3xrBli0W4ojz+Vhz\n5mxly5ZBWK1XkJs7mNmzt/p8DCFEcOjRIYkre6SRb7WzbEN+oMMRQpwkiVcAuT1uFmx/g0Plh0g6\n0R/j+t/i3JtJ9YZhlO/o5fPxLBaTt9WQoihYLCafjyGECB6/GtqZuGgDn689wJFCe6DDEUIgTbID\nwu6w8+6qt8gp+4ZSVynXX3IjnuMD0GivhGpAC8Un1vt8XLO5AotF9db4kicehWjZoiL03HltJv/8\nOI/Xl+zkyTt6o9XI79tCBJJ8BwbAezlv85VrOYc4iMvkQrFCK3O138tATJ6cRa9eqzCb19Or1yp5\n4lGIMNCrczL9upk5aLGxZN2hQIcjRNiTGa8A+L58I8c9xzDpIslMuoTi8mIeaYYyEPLEoxDh6dfD\nurArv5TP1hygQ+tYul+UFOiQhAhbkng1s02WjRytOYLeaCAz6RJ0io5UQ6okRUIIv4k26XlgbA9e\neHcz8z7fwVN39SE5TvZ4ChEIstTYjHac2M5/f3yfrLY9ucF0I2m2NDIrM7lt4J2BDk0I0cJ1aB3L\nb67pgr3Syb8XbcPpcp//TUIIn5MZr2ZysOyAtxXQvZc9wEVxHQIdkhAizAzs2Zr9x8pZm3ecd1bs\nZvzIroEOSYiwIzNezcDqsDB/61zcqofbLxkvSZcQIiAUReH24V3ISIthTd5xcrYcDXRIQoQdSbz8\nrLSqhHl5s6lwVXJLl1/RLbl7oEMSQoQxvU7LA2O6ExWh492vdrP3aFmgQxIirEji5UcOp4N5ebMp\nrS7l+g6juLxV30CHJIQQJMebuG90dzweePmjXI5KcVUhmo0kXn5S2wpoHtYKKwPTBzG47dBAhySE\nEF7dLkpk/MhMHFUu/vZhLkVlUlBZiOYgiZcf/NQK6CC9zX0Y1XGMt1WPEEIEiyt7tOLWwZ0osVXz\n0n9zKa+oCXRIQrR48lSjj6mqyoc/vs/O4p1kJmbyy4t/06Sky2Zz8O9//8D+/RrM5gomT84iOjrK\nhxELIcLZdX3bYausYdn6fP7+YS6P//pSTEb5X4MQ/iIzXj72xf7P2WT9nnYx7bij2wS0Gm2Trjdn\nzlY2bboKq/UKcnMHM3v2Vh9FKoQ4l2nTptG/f39GjRrlPVZWVsaECRO49tprmThxIjabzfvas88+\ny/Dhwxk9ejQ7d+4MRMhNcvPAjlyV1YpDFhv/+jiPGqfU+BLCXyTx8qFVh1ey6vBKUiJTmJh1L0at\nscnXtFhM3hkzRVGwWKTatBD+Nm7cOObPn3/asXnz5tGvXz+WL19O3759mTt3LgA5OTnk5+ezYsUK\nnn76aaZPnx6IkJtEURTuuO5iLu2czK78Uv7xUS6V1a5AhyVEiySJl49ssmxk8b7PiDPGMSnrfqL1\n0T65rtlc4ffm2UKI0/Xp04fY2NjTjmVnZzN27FgAxo4dS3Z2tvf4mDFjAOjZsyc2m42ioqLmDdgH\ntBoN943uTu8uKezKL2XW+z9gkz1fQvicJF4+sPPEDv774/tE6kxMyrqfxAjfNaCdPDmLPn3WYDav\np1evVX5pni2EOL/i4mKSk5MBSElJobi4GICCggLS0tK855nNZqxWa0BibCq9TsN9Y7pxZY80Dlps\n/PW9HyixVQc6LCFaFNlB2UQHyw7w9o7aVkATekwiLaqVT68fHR3FjBlXU1hoO//JQohmd2pGuq6G\nPlCTkhLj63B84vE7Lmf+4m18vno/f33/B565tx+tk8+exQ/W+C9EqN9DqMcPLeMeLoQkXk1gdViY\nv20eLo+bu7pNlFZAQrRgSUlJFBUVkZycTGFhIYmJiUDtDJfFYvGeZ7FYSE1NbdA1g/kXqtH9MtCo\nKp+uOcDv/7mGB8f1oFObOO/rKSkxQR1/Q4T6PYR6/BD699CYpFGWGhvJ2wrIWSGtgIRogc6cyRoy\nZAiffPIJAIsWLWLo0NqiyEOHDuXTTz8FYMuWLcTGxnqXJEOZoijceOVF3HZNF2wVNfz13f+xOvdY\noMMSIuTJjFcjSCsgIVq2qVOnsmHDBkpLSxk0aBBTpkxh0qRJPPzww3z88ce0bt2al19+GYCBAweS\nk5PDNddcg8lk4vnnnw9w9L41tHc6aUmRzPl0G28u28VBi43fDOsc6LCECFmKWt8GhQAKlinHc01/\n1rhrmJP7bw6VH+Tq9IHc2HGs36vSB8tUbLDEAcETS7DEAcEXizi/YPn3aoiC0kpe+TiPI4UOOqXH\n8dTEK3BVOwMdVpME0/dMY4R6/BD69yBLjX7m9rhZsKO2FdBlqb2bJekSQohgkBpv4snb+/CLzFT2\nHinjkb+vYuv+E4EOS4iQI4lXA6mqyke7P2DHiR21rYAym9YKSAghQo3RoOW+0d24ZXBHyh01/P3D\nXBYs/5HqGql0L0RDSeLVQEv2L+Z7y0ZvKyCdRrbHCSHCj6IojOibwUsPD6RNchTf/HCUGW9sZN/R\nskCHJkRIkMSrAXIOf8M3h7N92gpICCFCWYc2cTx1Vx+uu7wdBSWVPPfOZhau2iezX0KchyRe57HZ\n+j2f7/vU562AhBAi1Ol1Wm4d0onHf3MpSbERLF1/iGmvruf7XQX1FpYVQkji9bN2ntjBB7ve80sr\nICGEaCkubpfAMxP7ckP/DGwVNcz+dBsvfrCFo0WOQIcmRNCRxOscDpT81ApofI97fN4KSAghWhKj\nQcu4qzvyzN19yeqYxM5DJcx4fSPvrthNqV36PQpxiuwQr4fVYeH1H2fj8ri5s9sEOsR1DHRIQggR\nEswJkTxyS0+27C3ig6/3kP2/I6zOO8bgS9sw8ooMYqMMgQ5RiICSxOsMqqryxvbXcHgc3Nzll3RP\n7hHokIQQIuT06pRM94sS+XbrcRZ/d5AV3x9m1ZajDO2dzrW/aCcJmAhbknidQVEUuidl0bl1Bheb\negY6HCGECFk6rYaBvdrQv3srVuce44t1B1m2Pp+vvj/CFd3MDO/TlvRUeWBJhBdJvOpxQ8cbQ76N\ngRBCBAu9TsPQ3ulcldWKNXnH+WrTYdbmHWdt3nG6ZiQw/Bdt6dEhCY1GilKLlk8SLyGEEM3CoNcy\ntHc6gy9rQ97eE6z4Pp+dh0rYeaiEhBgjV/ZI48oerTAnRAY6VCH8RhIvIYQQzUqjKPTqnEyvzsnk\nW22s/N9RNu608sV3h/jiu0N0To/jyh6tuKxLCtEmfaDDFcKnJPESQggRMO3MMdw1IpNfD+vM/34s\nZO3W4+w8VMKeI2UsWP4jmRkJ9L44hcs6p8iGfNEiSOIlhBAi4Ix6Lf26p9GvexpFpZVs2Gll04+F\nbD9QzPYDxSxY/iNd0uPp3iGRHh2SaJsajaLInjAReiTxEkIIEVSS401c36891/drT2FpJZt/LGTz\n7gJ2Hy7lx8OlfJyzn7goA90uSqRb+0S6tI0nKS4i0GEL0SCSeAkhhAhaKfEmruvbjuv6tqO8ooYd\nB4rZur+Y7QdO8N02C99tswCQFBtBl7bxdGkbR6c2cbRKipKnJEVQksRLCCFESIiNNHBFtzSu6JaG\nR1U5bLWzK7+E3YdL2X24lHXbLazbXpuIGQ1aMswxXNQqhotaxZJhjiEl3iTJmAg4SbyEEEKEHI2i\nkJEWQ0ZaDNde3g6PqnK8yMHuw6XsP1bOAYuNPScTslMMOg1tUqJIT4kmPSWazI7JRGoVEmKNaGS/\nmGgmkngJIYQIeRpFoU1KNG1Sohl8We2xymoX+VYbB47bOFxg50ihnXyrnQPHTxbHzt4D1CZk5sRI\n0hIjSU0wkRJvIjkugpR4E4mxRrQaTYDuSrREkngJIYRokUxGHRe3S+DidgneYy63B2txBYcL7diq\n3Ow7XILlRAWWkgoOF9jPuoZGUYiPMZAYE0FCjJHEWCMJMRHERxuIjzYSF20gPsqI0aBtzlsTIUwS\nLyGEEGFDp9V4Z8bqtobzqCqltmoKSyspLK2iqKyy9uOyKkrKq9h/rByPqp7zuka9lphIPTGRBmJP\n/hkdqSfapCcqQke0qfbjyAg9JqOWSKOeCKNWljjDUJMSry+//JJXXnmFffv2sXDhQrp16+Z9be7c\nuXz88cdotVqefPJJBgwY0ORghRAi0N58800WLlyIoih06dKF559/noKCAn73u99RVlZGt27dmDlz\nJjqd/F4bSjSKQmJsBImxEVzc7uzXPR6VMkcNxeVVlNiqKbVXU+aoqf3TXkN5RQ22CieHC2y43OdO\n0OpSqJ2VMxm1RBh0RBi1mAw6IgxajAYtEXodxpMfG/VaDHoNRl3tnwa9ltTyaioc1ei1GvR6DQad\nFr1Og16rQadTZIk0SDXpJ0OXLl145ZVXeOqpp047vm/fPpYtW8bSpUuxWCyMHz+eFStWSLE7IURI\ns1qtLFiwgGXLlmEwGHjkkUdYsmQJOTk5jB8/nhEjRjB9+nQWLlzIr371q0CHK3xIo1FIiDGSEGP8\n2fNUVaWy2o2togZ7pRN7pRNHlRN7pQt7pZPKahcVVa6TfzqpqHZRWe2m1F5NVbEbt6dhSVuDYlYU\ndDqlNhHTatBpFbTa2sRMq61NzHRaBa1GQafVoNHUfqzVatBplJ8+P/mxpu7HysmPFQVFo6BR8B73\n/qlw8jUFReHksZMfaxQURSHuuA1beVXtuYoCCt6PFeocAzh5TU4er32p9nUFTv558vip1zh1fp33\neQ+eet+psahz/unnAURF6H3SwqpJiVeHDh2A2i+0urKzsxk5ciQ6nY709HQyMjLIy8ujZ8+eTRlO\nCCECzuPxUFlZiUajoaqqitTUVDZs2MDf/vY3AMaOHcu//vUvSbzClKIoREboiIzQYb7A96qqisvt\nobLaTZXTTU1N7Z/VNW6qalzUuDzUON3UOD3UuNzo9DpKyytxujzUuDw4T/7ncv/0sdPlweXx4HKr\nuFweqp1uHJVO3B4Vl1vF7fbgu1SvZdNqFGbe3/+8yff5+GUu3Gq10qtXL+/nZrMZq9Xqj6GEEKLZ\nmM1mxo8fz6BBgzCZTFx55ZVccsklxMbGojm5rJOWlkZBQUGAIxWhSFEU9Dotep2W2AacX3ePWlN4\nPLUJn9uj/vTfyc89qorn5LEz/1RVtc45tfvk1FOvnbyuR609z+OpTSxVTp6n1r4eHW2k3FaFqta+\n7lFVUH86R+Xk5M7JYwC1k4InX1dBpfb1k4dR67x28szTXj917Kfz8Z6sej88+XmdrDQm0kBsVDPM\neAlO6GAAAA3tSURBVI0fP56ioqKzjj/66KMMGTKk3vecOQMGyDKjECLklZeXk52dzTfffENMTAwP\nP/wwq1evPus8+XknQolGo2DQBOapTF8lj6HkvInXG2+8ccEXTUtL4/jx497PLRYLqampDXpvSkrM\nBY/nLxLL2YIlDgieWIIlDgiuWFqi7777jrZt2xIfHw/AsGHD+OGHHygvL8fj8aDRaEL2511jhHr8\nEPr3EOrxQ8u4hwvhs0ce6s5yDRkyhKVLl1JTU8Phw4fJz88nKyvLV0MJIURAtG7dmtzcXKqrq1FV\nlfXr19O5c2f69u3Ll19+CcCiRYsYOnRogCMVQgQrRa1vXbCBvv76a5555hlKSkqIjY0lMzOT1157\nDagtJ7Fw4UJ0Op2UkxBCtBivvPIKS5YsQafTcckll/Dss89isVj43e9+R3l5OV27dmXWrFno9U3f\nCyKEaHmalHgJIYQQQoiGk+pqQgghhBDNRBIvIYQQQohmIomXEEIIIUQzCcrEa8GCBVx33XWMGjWK\nF198MdDhMH/+fDIzMyktLQ3I+DNnzmTEiBGMHj2aKVOmYLfbmz2G1atXc91113Httdcyb968Zh8f\nasuS3HHHHYwcOZJRo0bx9ttvBySOujweD2PHjuW+++4LaBw2m42HHnqIESNGcP3115ObmxuQON58\n801uuOEGRo0axdSpU6mpqQlIHMEsGL6XLtS0adPo378/o0aN8h4rKytjwoQJXHvttUycOBGbLXhr\nMZ3rZ0co3UNNTQ233HILY8aMYdSoUbzyyisAHDlyhFtvvZVrr72W3/3ud7hcrgBH+vPO/JkZavEP\nGTKEG2+8kTFjxnDzzTcDjfg6UoPM+vXr1fHjx6tOp1NVVVU9ceJEQOM5fvy4OmHCBHXw4MFqSUlJ\nQGL49ttvVbfbraqqqs6aNUt98cUXm3V8t9utDhs2TD1y5IhaU1Oj3njjjerevXubNQZVVdWCggJ1\nx44dqqqqqt1uV4cPHx6QOOp644031KlTp6r33ntvQOP4v//7P3XhwoWqqqqq0+lUbTZbs8dgsVjU\nIUOGqNXV1aqqqurDDz+sLlq0qNnjCGbB8r10ob7//nt1x44d6v+3d/8xVdV/HMef9yI14pJTLxOS\nO0cymhk/osQSho1QF1p4p9DGam1JbFkhxKLLbWjT1A2Lrf5QdGZWI1km5lpuLSFx1gWGYdQcOknG\nj+KScSl+SVzu+f5B3K/gRcGvnHOv3/fjL+7h3HNeZ5zz4X3POfe8165d655WXFys7N+/X1EURdm3\nb5+ye/dureLd1GRjhy9tg6IoysDAgKIoiuJ0OpX09HTl3LlzyubNm5UTJ04oiqIoW7ZsUQ4fPqxl\nxJuaOGb6Wv7k5GSlp6dn3LTp7kded8br8OHDvPTSS8yaNfps17lz52qaZ+fOnRQUFGiaYfny5e52\nJLGxsXR2dqq6/sbGRhYuXMiCBQvw9/dnzZo1VFZWqpoBIDg4mMWLFwMQGBjIokWLNG3N0tnZSXV1\nNenp6ZplAOjr66O+vp7169cDMGvWLAwGgyZZxvoYOp1Odx9D8V/ecixN16OPPsq9945vYlNZWYnZ\nbAZG+1OePHlSi2hT4mnssNvtPrUNAAEBAcDo2S+n04lOp6O2tpbVq1cDo9vw7bffahnxhjyNmTU1\nNT6TH/5ta+RyjZs23f3I6wqvlpYW6uvrycjI4Pnnn+fnn3/WLEtVVRWhoaE88MADmmWY6IsvviAp\nKUnVddrtdkJDQ92v58+fr3kvuvb2dpqamjR9MO9YUa51e5j29nbmzJlDYWEhZrOZoqIirl69qnqO\na/sYJiUlERQUxPLly1XP4c288Vi6Vd3d3RiNRmC0sHE4HBonmpqxsSMmJoY///zTp7bB5XKxbt06\nEhISSEhIwGQy+VSf0IljpsPhYPbs2T6TH0bbgW3cuJH169dz5MgRgGnvRzPSJPtmJuv/mJuby8jI\nCH///Teff/45jY2N5Obmzugnwhtl2bdvHwcPHnRPU2bwkWdT6Ym5d+9e/P39x91noYaZ3O5b0d/f\nT05ODlarlcDAQE0ynDp1CqPRyOLFi6mtrdUkwxin08n58+fZsmULUVFR7Nixg/3795OTk6Nqjol9\nDHNycvjqq69U31+9mbcdS/9vJo4dWn9omi69Xs+XX35JX18fr7zyCs3NzdfN463b5GnMVP5toH0t\nb80/pry8nODgYLq7u3nxxRcJDw+fdmZNCq8b9X8sLy9n1apVAERHR6PX63E4HMyZM0fVLBcvXqSj\no4O0tDQURcFut7sr3Hnz5qmWY8yxY8eorq7W5IbykJAQfvvtN/dru92u2SUkp9NJTk4OaWlppKSk\naJIB4Mcff6Sqqorq6mqGhobo7++noKCA4uJi1bOEhIQQEhJCVFQUAKtXr3Z3kFDTxD6GK1eupKGh\nQQqva3jTsfS/mjdvHleuXMFoNPLHH39oflvIzXgaO3xtG8YYDAaWLl3KTz/9dMt9QtXmaczcuXMn\nvb29PpF/THBwMDB6G1RKSgqNjY3T3o+87lJjSkoKNpsNgMuXL+N0Omes6LqRyMhIvv/+eyorK6mq\nqmL+/PkcO3ZsRoqumzl9+jQHDhxg79693HXXXaqvPyoqitbWVjo6Ovjnn3/4+uuvNetFZ7VaiYiI\n4IUXXtBk/WNef/11Tp06RWVlJSUlJSxbtkyTogvAaDQSGhrK5cuXgdF7JhYtWqR6Dk99DLXI4c28\n6ViarolnJpKTk6moqAB8oz+lp7HDl7ahu7vb/W25q1evYrPZiIiI8Jk+oZ7GzHfffddn8gMMDg7S\n398PwMDAAGfOnCEyMnLa+5HXtQwaHh7GarXS1NSEv78/FouF+Ph4rWPx5JNPcvToUfeneTWtWrWK\n4eFh97pjYmJ4++23Vc1w+vRpduzYgaIobNiwgezsbFXXD3D27Fmee+45IiMj0el06HQ68vLyVL/n\nbaK6ujoOHjxIaWmpZhmampp46623cDqdmEwmdu3aRVBQkOo5PPUxlJ6F43nDsTRd+fn51NbW0tPT\ng9Fo5LXXXiMlJYXNmzfz+++/c9999/H+++9fdwO+t5hs7IiOjiY3N9cntuHChQtYLBZcLhcul4vU\n1FRefvll2trafK5P6LVjpi/lb2tr49VXX0Wn0zEyMsLTTz9NdnY2PT0909qPvK7wEkIIIYS4U3nd\npUYhhBBCiDuVFF5CCCGEECqRwksIIYQQQiVSeAkhhBBCqEQKLyGEEEIIlUjhJYQQQgihEim8hBBC\n+KyMjAzMZjNr1qxhyZIlmM1mzGYzVqt12svKysoa11lgMoWFhZw7d+5W4k7L+fPn+eabb2Z8PUJd\n8hwvIYQQPq+jo4MNGza4O594MtaaxlccOXIEm81GSUmJ1lHEbaRJr0YhhBBiptlsNoqLi4mMjOTC\nhQvk5+fT3d1NWVkZTqcTYFx3lBUrVnDo0CHCw8PJzMzk4YcfpqGhga6uLtauXUtubi4AmZmZbNq0\nicTERN544w0MBgPNzc3Y7Xbi4uLYtWsXAJ2dnRQUFOBwODCZTIyMjJCcnMyzzz47LueVK1fIz8/H\n4XAAkJiYSFZWFnv27GFgYACz2cyyZcuwWCw0NDRQUlLC4OAgADk5OSQlJdHa2kpmZiapqanYbDb0\nej1bt24lLi7O4/ILCgpm/g8gPJLCSwghxB3r4sWLbN++nYceegiAv/76i7S0NACam5vJysriu+++\n8/heu93OZ599Rm9vLykpKaSnp7NgwYLr5rt06RIfffQRLpeLtLQ06urqiI+PZ9u2bSQlJZGVlUV7\nezvPPPMMycnJ173/+PHjREREUFRUBEBvby9BQUFs2rSJmpoa3nvvPXf2bdu28eGHHzJ37lzsdjsZ\nGRmcOHECGC3gYmNjsVqt2Gw28vPzOXnypMflC+1I4SWEEOKOdf/997uLLoCWlhY++OADurq68PPz\no6uri56eHo99eJ966ikAgoKCCA8Pp7W11WPhtXLlSmbNGv13+uCDD9La2kp8fDy1tbW88847AISF\nhU3adzg2NpaysjICAgJYunQpiYmJHuc7e/Ys7e3tbNy40d203M/Pj7a2Nu655x4CAgJITU0F4PHH\nH8fPz4+WlpYpL1+oQwovIYQQd6zAwMBxr/Py8ti6dSsrVqzA5XIRHR3N0NCQx/fefffd7p/1ej0j\nIyPTmk+n000p4yOPPEJFRQU//PADR48e5cCBA3z66afXzacoCkuWLOHQoUPX/a61tfW6aS6XC51O\nN+XlC3X4zl2GQgghxA1M5btifX19hIWFAVBeXj5pMXU7xMfHU1FRAYze/F9XV+dxvvb2dgwGA6mp\nqVgsFn755RcADAbDuMuCcXFxXLp0ifr6eve0xsZG98+Dg4Puy441NTUALFy4cNLlC23IGS8hhBB3\nhKmcYbJarWRnZzN79myeeOIJgoKCPL5/4rIm+92N5isqKuLNN9/k+PHjhIWFERMTM259Y2w2G598\n8gl+fn4oisL27dsBSEhI4OOPP2bdunU89thjWCwW9uzZw+7du+nt7WV4eBiTyURpaSkARqORxsZG\nSktL0el0lJSU4OfnN+nyhTbkcRJCCCHEDBgaGsLf3x+9Xo/dbic9PZ2ysjJMJtNtX9fYtxrPnDlz\n25ctbi854yWEEELMgF9//ZXCwkIURcHlcpGXlzcjRZfwLXLGSwghhBBCJXJzvRBCCCGESqTwEkII\nIYRQiRReQgghhBAqkcJLCCGEEEIlUngJIYQQQqhECi8hhBBCCJX8B3vfi4PTacKnAAAAAElFTkSu\nQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xc443950>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#@test {\"output\": \"ignore\"}\n",
    "\n",
    "# Use the same input data and parameters as the examples above.\n",
    "# We're going to build up a list of the errors over time as we train to display later.\n",
    "losses = []\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    # Set up all the tensors.\n",
    "    # The input is the x values with the bias appended on to each x.\n",
    "    input = tf.constant(x_with_bias)\n",
    "    # We're trying to find the best fit for the target y values.\n",
    "    target = tf.constant(np.transpose([y]).astype(np.float32))\n",
    "    # Let's set up the weights randomly\n",
    "    weights = tf.Variable(tf.random_normal([2, 1], 0, 0.1))\n",
    "\n",
    "    tf.global_variables_initializer().run()\n",
    "\n",
    "    # learning_rate is the step size, so how much we jump from the current spot\n",
    "    learning_rate = 0.002\n",
    "\n",
    "    # The operations in the operation graph.\n",
    "    # Compute the predicted y values given our current weights\n",
    "    yhat = tf.matmul(input, weights)\n",
    "    # How much does this differ from the actual y?\n",
    "    yerror = tf.subtract(yhat, target)\n",
    "    # Change the weights by subtracting derivative with respect to that weight\n",
    "    loss = 0.5 * tf.reduce_sum(tf.multiply(yerror, yerror))\n",
    "    gradient = tf.reduce_sum(tf.transpose(tf.multiply(input, yerror)), 1, keep_dims=True)\n",
    "    update_weights = tf.assign_sub(weights, learning_rate * gradient)\n",
    "    \n",
    "    # Repeatedly run the operation graph over the training data and weights.\n",
    "    for _ in range(training_steps):\n",
    "        sess.run(update_weights)\n",
    "    \n",
    "        # Here, we're keeping a history of the losses to plot later\n",
    "        # so we can see the change in loss as training progresses.\n",
    "        losses.append(loss.eval())\n",
    "\n",
    "    # Training is done, compute final values for the graph.\n",
    "    betas = weights.eval()\n",
    "    yhat = yhat.eval()\n",
    "\n",
    "# Show the results.\n",
    "fig, (ax1, ax2) = plt.subplots(1, 2)\n",
    "plt.subplots_adjust(wspace=.3)\n",
    "fig.set_size_inches(10, 4)\n",
    "ax1.scatter(x, y, alpha=.7)\n",
    "ax1.scatter(x, np.transpose(yhat)[0], c=\"g\", alpha=.6)\n",
    "line_x_range = (-4, 6)\n",
    "ax1.plot(line_x_range, [betas[0] + a * betas[1] for a in line_x_range], \"g\", alpha=0.6)\n",
    "ax2.plot(range(0, training_steps), losses)\n",
    "ax2.set_ylabel(\"Loss\")\n",
    "ax2.set_xlabel(\"Training steps\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "TzIETgHwTexL"
   },
   "source": [
    "This code looks very similar to the code above, but without using `l2_loss` or `GradientDescentOptimizer`. Let's look at exactly what it is doing instead.\n",
    "\n",
    "This code is the key difference:\n",
    "\n",
    ">`loss = 0.5 * tf.reduce_sum(tf.multiply(yerror, yerror))`\n",
    "\n",
    ">`gradient = tf.reduce_sum(tf.transpose(tf.multiply(input, yerror)), 1, keep_dims=True)`\n",
    "\n",
    ">`update_weights = tf.assign_sub(weights, learning_rate * gradient)`\n",
    "\n",
    "The first line calculates the L2 loss manually. It's the same as `l2_loss(yerror)`, which is half of the sum of the squared error, so $\\frac{1}{2} \\sum (\\hat{y} - y)^2$. With this code, you can see exactly what the `l2_loss` operation does. It's the total of all the squared differences between the target and our estimates. And minimizing the L2 loss will minimize how much our estimates of $y$ differ from the true values of $y$.\n",
    "\n",
    "The second line calculates $\\begin{bmatrix}\\sum{(\\hat{y} - y)*1} \\\\ \\sum{(\\hat{y} - y)*x_i}\\end{bmatrix}$. What is that? It's the partial derivatives of the L2 loss with respect to $w_1$ and $w_2$, the same thing as what `gradients(loss, weights)` does in the earlier code. Not sure about that? Let's look at it in more detail. The gradient calculation is going to get the partial derivatives of loss with respect to each of the weights so we can change those weights in the direction that will reduce the loss. L2 loss is $\\frac{1}{2} \\sum (\\hat{y} - y)^2$, where $\\hat{y} = w_2 x + w_1$. So, using the chain rule and substituting in for $\\hat{y}$ in the derivative, $\\frac{\\partial}{\\partial w_2} = \\sum{(\\hat{y} - y)\\, *x_i}$ and $\\frac{\\partial}{\\partial w_1} = \\sum{(\\hat{y} - y)\\, *1}$. `GradientDescentOptimizer` does these calculations automatically for you based on the graph structure.\n",
    "\n",
    "The third line is equivalent to `weights -= learning_rate * gradient`, so it subtracts a constant the gradient after scaling by the learning rate (to avoid jumping too far each time, which risks moving in the wrong direction). It's also the same thing that `GradientDescentOptimizer(learning_rate).minimize(loss)` does in the earlier code. Gradient descent updates its first parameter based on the values in the second after scaling by the third, so it's equivalent to the `assign_sub(weights, learning_rate * gradient)`.\n",
    "\n",
    "Hopefully, this other code gives you a better understanding of what the operations we used previously are actually doing. In practice, you'll want to use those high level operators most of the time rather than calculating things yourself. For this toy example and simple network, it's not too bad to compute and apply the gradients yourself from scratch, but things get more complicated with larger networks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "cellView": null,
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "output_extras": []
    },
    "colab_type": "code",
    "collapsed": true,
    "executionInfo": {
     "elapsed": 164,
     "status": "ok",
     "timestamp": 1474671842705,
     "user": {
      "displayName": "",
      "photoUrl": "",
      "userId": ""
     },
     "user_tz": 420
    },
    "id": "ty5-b_nYSYWR",
    "outputId": "311b7bff-5c8b-43ee-da0f-439a879636d1"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "default_view": {},
   "name": "Untitled",
   "provenance": [],
   "version": "0.3.2",
   "views": {}
  },
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}