12-Custom_Models_&_Training_with_Tensorflow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b6d7733e-1d62-47a2-94be-c97129d44625",
   "metadata": {},
   "source": [
    "# Custom Models and Training with Tensorflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "bc661171-782a-4de4-8365-ef523a5c10f7",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-01-07 19:23:09.585092: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.\n",
      "2024-01-07 19:23:09.722478: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.\n",
      "2024-01-07 19:23:09.723557: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
      "To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2024-01-07 19:23:10.875653: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "from tensorflow import keras"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8e3ecb7-16da-43ae-b8f2-2163a77ad4d1",
   "metadata": {},
   "source": [
    "## Using TensorFlow like Numpy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5509d0b0-9b89-448e-892d-4ae8b0ca5d41",
   "metadata": {},
   "source": [
    "TensorFlow's API revolves around *tensors*, which flow from operation to operation, hence the name Tensor*flow*. A tensor is usually a multidimensional array (exactly like Numpy `ndarray`), but it can also hold a scalar (a simple, value such as 42).\n",
    "\n",
    "These tensors will be important when we create custom cost functions, custom metrics, custom layers , and more. Let's see how to create and manipulate them."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8eaf6667-8ea2-4bdd-b8da-1138562191fd",
   "metadata": {},
   "source": [
    "### Tensor and Operations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d29922ad-1cca-4b8c-8760-fc6cb6367809",
   "metadata": {},
   "source": [
    "We can create a tensor with `tf.constant()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "a27f630b-d45b-479d-a58a-33145f446ac7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 3), dtype=float32, numpy=\n",
       "array([[1., 2., 3.],\n",
       "       [4., 5., 6.]], dtype=float32)>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t = tf.constant([[1., 2., 3.], [4., 5., 6.]]) # Here's a tensor representing matrix with two rows and three columns of floats\n",
    "t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "fb68ddd3-c0e6-48d2-bbba-c7ccaaa14ee8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=int32, numpy=42>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.constant(42) #scalar"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a1ca170-3fb8-42fc-8609-b1b753319ba8",
   "metadata": {},
   "source": [
    "Just like `ndarray`, a `tf.Tensor` has a shape and a datatype (dtype):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "7a8131c1-b39f-4bbe-8882-9268015c8624",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([2, 3])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "1b414dc4-0af0-48df-b37d-e459be81bbb2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tf.float32"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21235964-6f3d-4905-9337-76bea4c2fe49",
   "metadata": {},
   "source": [
    "Indexing works much like NumPy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "ad38bdbb-aeaa-40ac-b70e-981e1bb66984",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 2), dtype=float32, numpy=\n",
       "array([[2., 3.],\n",
       "       [5., 6.]], dtype=float32)>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t[:, 1:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "e295d6bb-590c-44e7-871f-7250031121d9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 1), dtype=float32, numpy=\n",
       "array([[2.],\n",
       "       [5.]], dtype=float32)>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t[..., 1, tf.newaxis] # ... is same as :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "992a0c56-d9fa-4988-a536-74009a210f89",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 1), dtype=float32, numpy=\n",
       "array([[2.],\n",
       "       [5.]], dtype=float32)>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t[:, 1, tf.newaxis]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e78f99d9-8d9a-4671-94b5-8c369595c2e4",
   "metadata": {},
   "source": [
    "Most importantly, all sorts of tensor operations are available:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "e1bd30c9-2d6d-4127-98cf-d0bdc0338dd6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 3), dtype=float32, numpy=\n",
       "array([[11., 12., 13.],\n",
       "       [14., 15., 16.]], dtype=float32)>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t + 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "abbfa763-e2ed-4381-9d69-4e305aa5b079",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 3), dtype=float32, numpy=\n",
       "array([[ 1.,  4.,  9.],\n",
       "       [16., 25., 36.]], dtype=float32)>"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.square(t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "8c39d033-a591-4f14-88df-397b54c55d28",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 2), dtype=float32, numpy=\n",
       "array([[14., 32.],\n",
       "       [32., 77.]], dtype=float32)>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t @ tf.transpose(t) # @ is for multiplication, equivalent to tf.matmul()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c06670d0-8887-4b30-aa38-f107cd390240",
   "metadata": {},
   "source": [
    "Note that writing `t + 10` is equivalent to calling `tf.add(t, 10)`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6ac31c2-ebe3-4f62-9e1b-ee508c929c40",
   "metadata": {},
   "source": [
    "### Tensor and Numpy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e623753-7373-4fe7-9f37-e39843aa6302",
   "metadata": {},
   "source": [
    "Tensors play nice with Numpy: we can create a tensor from Numpy array and vice versa. We can even apply TensorFlow operations to NumPy arrays and NumPy operations to tensors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "52ec891b-3ea3-42cc-b1b4-e0be42c154b2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3,), dtype=float64, numpy=array([2., 4., 5.])>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = np.array([2., 4., 5.])\n",
    "tf.constant(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "f7e8fc57-0ec0-4481-b591-7c6a63291f5a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1., 2., 3.],\n",
       "       [4., 5., 6.]], dtype=float32)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.numpy() # or np.array(t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "25817c6d-2ee9-4f07-b29a-b6b5d2027cd3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3,), dtype=float64, numpy=array([ 4., 16., 25.])>"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.square(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "2ef5e306-a612-4216-b6c9-d1e42f4f30be",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.,  4.,  9.],\n",
       "       [16., 25., 36.]], dtype=float32)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.square(t)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9d45ff5-10dc-4ab3-a3e8-9704f02d98b4",
   "metadata": {},
   "source": [
    "**WARNING:**\n",
    "\n",
    "Notice that Numpy uses a 64-bit precision by default, while TensorFlow uses 32-bit. This is because 32-bit precision is generally more than enough for neural networks, plus it runs faster and uses less RAM. So when we create a tensor from Numpy array, make sure to set `dtype=tf.float32`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80e20e58-b119-4c49-b025-286bf2a1ad0d",
   "metadata": {},
   "source": [
    "### Type Conversions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8fc714d-801e-4f06-98e5-74ec5363705c",
   "metadata": {},
   "source": [
    "Type conversions can significantly hurt performance, and they can easily gets unnoticed when they are done automatically. To avoid this, TensorFlow does not perform any type conversion automatically: it just raises an exception if we try to execute an operation on tensors with compatible types. For example: we cannot add a float tensor and an integer tensor, and we cannot even add a 32-bit float and a 64-bit float:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "afa7eeba-2950-4964-a804-b71eff2833cb",
   "metadata": {},
   "outputs": [
    {
     "ename": "InvalidArgumentError",
     "evalue": "cannot compute AddV2 as input #1(zero-based) was expected to be a float tensor but is a int32 tensor [Op:AddV2] name: ",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mInvalidArgumentError\u001b[0m                      Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[32], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m tf\u001b[38;5;241m.\u001b[39mconstant(\u001b[38;5;241m2.\u001b[39m) \u001b[38;5;241m+\u001b[39m tf\u001b[38;5;241m.\u001b[39mconstant(\u001b[38;5;241m40\u001b[39m)\n",
      "File \u001b[0;32m~/anaconda3/envs/latest/lib/python3.11/site-packages/tensorflow/python/util/traceback_utils.py:153\u001b[0m, in \u001b[0;36mfilter_traceback.<locals>.error_handler\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m    151\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m    152\u001b[0m   filtered_tb \u001b[38;5;241m=\u001b[39m _process_traceback_frames(e\u001b[38;5;241m.\u001b[39m__traceback__)\n\u001b[0;32m--> 153\u001b[0m   \u001b[38;5;28;01mraise\u001b[39;00m e\u001b[38;5;241m.\u001b[39mwith_traceback(filtered_tb) \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m    154\u001b[0m \u001b[38;5;28;01mfinally\u001b[39;00m:\n\u001b[1;32m    155\u001b[0m   \u001b[38;5;28;01mdel\u001b[39;00m filtered_tb\n",
      "File \u001b[0;32m~/anaconda3/envs/latest/lib/python3.11/site-packages/tensorflow/python/framework/ops.py:6656\u001b[0m, in \u001b[0;36mraise_from_not_ok_status\u001b[0;34m(e, name)\u001b[0m\n\u001b[1;32m   6654\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mraise_from_not_ok_status\u001b[39m(e, name):\n\u001b[1;32m   6655\u001b[0m   e\u001b[38;5;241m.\u001b[39mmessage \u001b[38;5;241m+\u001b[39m\u001b[38;5;241m=\u001b[39m (\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m name: \u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;241m+\u001b[39m \u001b[38;5;28mstr\u001b[39m(name \u001b[38;5;28;01mif\u001b[39;00m name \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;28;01melse\u001b[39;00m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m\"\u001b[39m))\n\u001b[0;32m-> 6656\u001b[0m   \u001b[38;5;28;01mraise\u001b[39;00m core\u001b[38;5;241m.\u001b[39m_status_to_exception(e) \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m\n",
      "\u001b[0;31mInvalidArgumentError\u001b[0m: cannot compute AddV2 as input #1(zero-based) was expected to be a float tensor but is a int32 tensor [Op:AddV2] name: "
     ]
    }
   ],
   "source": [
    "tf.constant(2.) + tf.constant(40)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "9ff62a6c-9346-4834-afc5-b5f93a060079",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cannot compute AddV2 as input #1(zero-based) was expected to be a float tensor but is a int32 tensor [Op:AddV2] name: \n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    tf.constant(2.) + tf.constant(40)\n",
    "except Exception as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "434f8bd2-0941-4998-be8a-d6c2e1f314d7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cannot compute AddV2 as input #1(zero-based) was expected to be a float tensor but is a double tensor [Op:AddV2] name: \n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    tf.constant(2.) + tf.constant(40., dtype=tf.float64)\n",
    "except Exception as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "203724d7-afc7-4dd6-b0a1-87fa729ac21d",
   "metadata": {},
   "source": [
    "This may be a bit annoying at first, but remember that it's for good cause! And ofcourse we can use `tf.cast()` when we really need to convert types:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "61644c9b-fbfa-451f-89c4-a82d6b4eb4fa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=42.0>"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t2 = tf.constant(40, dtype=tf.float64)\n",
    "tf.constant(2.0) + tf.cast(t2, tf.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75252a75-a74d-49af-822e-a5143e3932ee",
   "metadata": {},
   "source": [
    "### Variables"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15a6ab8d-04c3-4257-8033-d168508a97fb",
   "metadata": {},
   "source": [
    "The `tf.Tensor` values we've seen so far are immutables: we cannot modify them. This means that we cannot use regular tensors to implement weights in neural networks, since they needs to be tweaked by backpropogation. What we need is `tf.Variable`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "b8a5e4ea-d114-4671-ae02-af742ddaa00c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'Variable:0' shape=(2, 3) dtype=int32, numpy=\n",
       "array([[1, 2, 3],\n",
       "       [4, 5, 6]], dtype=int32)>"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v = tf.Variable([[1, 2, 3], [4, 5, 6]])\n",
    "v"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5d4dee4-cec5-4dc1-905d-452f48f73519",
   "metadata": {},
   "source": [
    "A `tf.Variable` acts much like a `tf.Tensor`: we can perform the same operations with it, it plays nicely with Numpy as well, and it is just as picky with types. But it can also be modified in place using `assign()` method (direct assignment will not work)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "1a83a2b3-f123-4116-be35-d5723dd75eff",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'Variable:0' shape=(2, 3) dtype=int32, numpy=\n",
       "array([[ 2,  4,  6],\n",
       "       [ 8, 10, 12]], dtype=int32)>"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v.assign(2*v)\n",
    "v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "dc48dccb-e7c5-4300-b64b-9758da83b0f5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'UnreadVariable' shape=(2, 3) dtype=int32, numpy=\n",
       "array([[ 2, 42,  6],\n",
       "       [ 8, 10, 12]], dtype=int32)>"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v[0,1].assign(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "1a51303d-5133-4d13-9b0c-01e52140ac7a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'UnreadVariable' shape=(2, 3) dtype=int32, numpy=\n",
       "array([[ 2, 42,  0],\n",
       "       [ 8, 10,  1]], dtype=int32)>"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v[:, 2].assign([0, 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "65ae1315-b355-435c-85d1-127e09dba302",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'UnreadVariable' shape=(2, 3) dtype=int32, numpy=\n",
       "array([[100,  42,   0],\n",
       "       [  8,  10, 200]], dtype=int32)>"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v.scatter_nd_update(indices=[[0,0], [1,2]], updates=[100, 200])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b821ba52-257a-44b0-a543-6f2c3834241d",
   "metadata": {},
   "source": [
    "**NOTE:**\n",
    "\n",
    "In practice we will rarely have to create variables manually."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e11ec91d-7f8b-4b82-8b43-466e42ef7dfa",
   "metadata": {},
   "source": [
    "### Other Data Structures"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3e291bb-6ab7-49e9-ae45-24a77d2c7baf",
   "metadata": {},
   "source": [
    "- Sparse tensors (`tf.SparseTensor`) => Tensors containing mostly zeros\n",
    "- Tensor Arrays (`tf.TensorArray`) => List of tensors\n",
    "- Ragged Tensors (`tf.RaggedTensor`) => static list of lists of tensors. Every tensor has same shape and data type\n",
    "- String Tensors => Regular tensor of type `tf.string`\n",
    "- Sets\n",
    "- Queues"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26c2669a-325b-4fec-8178-b0a8670f5c90",
   "metadata": {},
   "source": [
    "## Customizing Model and Training Algorithms"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f869e06e-e792-428e-8131-22fb8075ec70",
   "metadata": {},
   "source": [
    "Let's start by creating a custom loss function, which is simple and common use case."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87a41765-ab61-4503-a8b2-0c0a6d3ecea3",
   "metadata": {},
   "source": [
    "Let's start by loading and preparing the California housing dataset. We first load it, then split it into a training set, a validation set and a test set, and finally we scale it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "392ecb46-e772-4cae-8c74-02e27276ddef",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import fetch_california_housing\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "housing = fetch_california_housing()\n",
    "X_train_full, X_test, y_train_full, y_test = train_test_split(\n",
    "    housing.data, housing.target.reshape(-1, 1), random_state=42)\n",
    "X_train, X_valid, y_train, y_valid = train_test_split(\n",
    "    X_train_full, y_train_full, random_state=42)\n",
    "\n",
    "scaler = StandardScaler()\n",
    "X_train_scaled = scaler.fit_transform(X_train)\n",
    "X_valid_scaled = scaler.transform(X_valid)\n",
    "X_test_scaled = scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74f62235-2056-4e62-956c-3445dc39fcc6",
   "metadata": {},
   "source": [
    "### Custom Loss Function"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bb6ff31-88b6-4e4a-94e6-cf2ae3702990",
   "metadata": {},
   "source": [
    "Suppose we want to train a linear regression model, but our training set is a bit noisy. Ofcourse, we start by trying to clean up the dataset by removing or fixing the outliers, but that turns out to be insufficient; the dataset is still noisy. \n",
    "\n",
    "Which loss function should we use? The mean squared error might penalize large errors too much and cause model to be imprecise. The mean absolute error would not penalize outliers as much, but training might take a while to converge, and trained model not be very precise. This is probably good time to use Huber loss instead of the good old MSE. \n",
    "\n",
    "The Huber loss is not currently part of the official Keras API, but it is available in tf.keras (just use an instance of the `keras.losses.Huber` class). But let's pretend it's not there: implementing it is easy as pie!.\n",
    "\n",
    "Just create a function that takes the labels and parameters as arguments, and use TensorFlow operations to compute every instance's loss:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47b7c548-2b0b-4c7a-8b7e-46c33dfbd253",
   "metadata": {},
   "source": [
    "Hubber Loss:\n",
    "$$\n",
    "L_{\\delta}(a) =\n",
    "\\begin{cases}\n",
    "    \\frac{1}{2} a^2 & \\text{for |a|} \\le \\delta \\\\\n",
    "    \\delta \\cdot (|a| - \\frac{1}{2} \\cdot \\delta) , & otherwise\n",
    "\\end{cases}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "354c0c5d-28af-424e-b99d-eb865830efd4",
   "metadata": {},
   "outputs": [],
   "source": [
    "def huber_loss_fn(y_true, y_pred):\n",
    "    error = y_true - y_pred\n",
    "    is_small_error = tf.abs(error) < 1 # here the delta = 1\n",
    "    squared_loss = tf.square(error) / 2\n",
    "    linear_loss = 1 * (tf.abs(error) - (0.5 * 1))\n",
    "    return tf.where(is_small_error, squared_loss, linear_loss) # returns squared loss when error is small, else returns linear loss"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fd0e5eb-bbaf-4431-86cb-62a5526345e2",
   "metadata": {},
   "source": [
    "It is also preferable to return a tensor containing one loss per instance, rather than returning the mean loss. This way, Keras can apply class weights or sample weights when requested."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f801980-abc8-4184-a249-3daeefe6620e",
   "metadata": {},
   "source": [
    "Now we can use this loss when we compile the Keras model, then train our model. When compiling the model, just pass this function to `loss` argument like this:\n",
    "```\n",
    "model.compile(loss=huber_loss_fn, optimizer=\"nadam\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "1b8b150b-bd44-4343-bc78-5e80b0a81529",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 3s 5ms/step - loss: 0.5731 - mae: 0.9348 - val_loss: 0.3381 - val_mae: 0.6284\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 2s 4ms/step - loss: 0.2123 - mae: 0.5053 - val_loss: 0.2907 - val_mae: 0.5726\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.src.callbacks.History at 0x7f90e4c42090>"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_shape = X_train.shape[1:] # (8,1)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"selu\", kernel_initializer=\"lecun_normal\",\n",
    "                      input_shape=input_shape),\n",
    "    keras.layers.Dense(1)\n",
    "])\n",
    "\n",
    "model.compile(loss=huber_loss_fn, optimizer=\"nadam\", metrics=[\"mae\"])\n",
    "\n",
    "model.fit(X_train_scaled, y_train, epochs=2, validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b08252c3-1f63-4a5d-b896-a884efca25ca",
   "metadata": {},
   "source": [
    "And that's it! For each batch during training, Keras will call the `huber_fn()` to compute the loss and use it to perform a Gradient Descent step. Moreover, it will keep track of the total loss since the beginning of the epoch, and it will display the mean loss. \n",
    "\n",
    "But what happens to this custom loss when we save the model?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11461e44-c527-4369-9d68-4379c51fa841",
   "metadata": {},
   "source": [
    "### Saving and Loading Models That Contain Custom Components"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "b1dc2c02-3720-4a9a-8c76-209c1eee0ef8",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save(\"my_model_with_a_custom_loss.keras\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4e05d26-2703-4d6a-88aa-638675aef256",
   "metadata": {},
   "source": [
    "Saving a model containing custom loss function works fine, as Keras saves the name of the function. Whenever we load it, we we'll need to provide a dictionary that maps the function name to the actual function.\n",
    "\n",
    "More generally, when we load a model containing custom objects, we need to map the names of the objects:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "8ea4bb4a-fdd6-43e1-80a0-3f1353f737a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.load_model(\"my_model_with_a_custom_loss.keras\", \n",
    "                               custom_objects={\"huber_loss_fn\": huber_loss_fn})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "e25377ac-2da3-49be-96f1-cb6491b5c31d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 2s 4ms/step - loss: 0.2017 - mae: 0.4893 - val_loss: 0.2442 - val_mae: 0.5253\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.1952 - mae: 0.4809 - val_loss: 0.2016 - val_mae: 0.4813\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.src.callbacks.History at 0x7f90e49d1890>"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train_scaled, y_train, epochs=2, validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f9784c7-3497-4500-8450-c5759a6938f1",
   "metadata": {},
   "source": [
    "With the current implementation, any error between -1 and 1 is considered \"small\". But what if we want a different threshold? One solution is to create a function that creates a configured loss function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "368d2005-b9c4-4c2b-999c-2caf10e4fd60",
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_huber(threshold=1.0):\n",
    "    def huber_fn(y_true, y_pred):\n",
    "        error = y_true - y_pred\n",
    "        is_small_error = tf.abs(error) < threshold \n",
    "        squared_loss = tf.square(error) / 2\n",
    "        linear_loss = threshold * tf.abs(error) - threshold**2 / 2\n",
    "        return tf.where(is_small_error, squared_loss, linear_loss)\n",
    "    return huber_fn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "f16b4ee9-585f-42cc-abb5-47ebe05a3d66",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=create_huber(2.0), optimizer=\"nadam\", metrics=[\"mae\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "3504ad52-2c0d-451e-96bb-223aa50b8b16",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.2145 - mae: 0.4774 - val_loss: 0.2259 - val_mae: 0.4760\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.2099 - mae: 0.4727 - val_loss: 0.2013 - val_mae: 0.4587\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.src.callbacks.History at 0x7f9104e0efd0>"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train_scaled, y_train, epochs=2, validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "74ac1112-6823-4c59-ae61-5c4e648131c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save(\"my_model_with_custom_loss_threshold_2.keras\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff1a6ea2-5d5b-4a2e-bd3f-08ab601f009c",
   "metadata": {},
   "source": [
    "Unfortunately, when we save the model, the `threshold` will not be saved. This means that we will have to specify the `threshold` value when loading the model (note that the name to use is \"`huber_fn`\", which is the name of the function we gave Keras, not the name of the function that created it):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "4cff4ab6-dca3-468a-82aa-1b3533101688",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.load_model(\"my_model_with_custom_loss_threshold_2.keras\",\n",
    "                               custom_objects={\"huber_fn\": create_huber(2.0)})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "984399e6-8af9-453a-86c1-29a6b2d7fed4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 2s 2ms/step - loss: 0.2052 - mae: 0.4665 - val_loss: 0.2369 - val_mae: 0.4675\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.2043 - mae: 0.4639 - val_loss: 0.2005 - val_mae: 0.4569\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.src.callbacks.History at 0x7f9104ce6fd0>"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train_scaled, y_train, epochs=2, validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e846abc-3dc6-4fdd-9264-6976081e0c23",
   "metadata": {},
   "source": [
    "We can solve this by creating a subclass of `keras.losses.Loss` class, and then implementing its `get_config()` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "00e5b199-791f-41a2-98fa-59bbad978757",
   "metadata": {},
   "outputs": [],
   "source": [
    "class HuberLoss(keras.losses.Loss):\n",
    "    def __init__(self, threshold=1.0, **kwargs):\n",
    "        self.threshold = threshold\n",
    "        super().__init__(**kwargs)\n",
    "    def call(self, y_true, y_pred):\n",
    "        error = y_true - y_pred\n",
    "        is_small_error = tf.abs(error) < self.threshold \n",
    "        squared_loss = tf.square(error) / 2\n",
    "        linear_loss = self.threshold * tf.abs(error) - self.threshold**2 / 2\n",
    "        return tf.where(is_small_error, squared_loss, linear_loss)\n",
    "    def get_config(self):\n",
    "        base_config = super().get_config()\n",
    "        return {**base_config, \"threshold\": self.threshold}\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7af2a3e2-3a84-4455-9022-5e5b8524535e",
   "metadata": {},
   "source": [
    "Explanation of above code:\n",
    "\n",
    "- The constructor accepts ``**kwargs` and passes them to the parent constructor, which handles standard hyperparameters: the `name` of the loss and the `reduction` algorithm to use to aggregate the individual instance losses. By default, it is ``\"sum_over_batch_size\"`, which means that the loss will be the sum of the instance losses, weighted by the sample weights, if any, and divided by the batch size (not by the sum of weights, so this is not the weighted mean). Other possible values are `\"sum\"` and `None`.\n",
    "\n",
    "- The `call()` method takes the labels and predictions, computes all the instance losses, and returns them.\n",
    "\n",
    "- The `get_config()` method returns a dictionary mapping each hyperparameter name to its value. It first calls the parent class’s `get_config()` method, then adds the new hyperparameters to this dictionary."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9105008-0de1-4900-8dc5-27f4ec976cc9",
   "metadata": {},
   "source": [
    "We can than use any instance of the class when we compile the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "1d778339-c07d-49b6-935b-2864b29cdb55",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 2s 2ms/step - loss: 0.9749 - mae: 1.0742 - val_loss: 0.4231 - val_mae: 0.6038\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.2504 - mae: 0.5184 - val_loss: 0.3407 - val_mae: 0.5439\n"
     ]
    }
   ],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"selu\", kernel_initializer=\"lecun_normal\",\n",
    "                       input_shape=input_shape),\n",
    "    keras.layers.Dense(1),\n",
    "])\n",
    "\n",
    "model.compile(loss=HuberLoss(2.0), optimizer=\"nadam\", metrics=[\"mae\"])\n",
    "\n",
    "h = model.fit(X_train_scaled, y_train, epochs=2, validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fae8044-8153-44c7-9e20-f60124559325",
   "metadata": {},
   "source": [
    "When we save the model, the threshold will get saved along with it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "657a029e-4abf-4aff-81a1-a39107acdbb4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: my_model_with_custom_loss_class.kears/assets\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: my_model_with_custom_loss_class.kears/assets\n"
     ]
    }
   ],
   "source": [
    "model.save(\"my_model_with_custom_loss_class.kears\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b197d884-88b6-4897-98d6-d569bbd4a54b",
   "metadata": {},
   "source": [
    "And when we load the model, we just need to map the class name to class itself:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "3dd1c95e-7cac-4691-b546-b38b36d19a82",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.load_model(\"my_model_with_custom_loss_class.keras\",\n",
    "                               custom_objects={\"HuberLoss\": HuberLoss})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd06ba15-a708-4617-99ca-2867a0ddfbba",
   "metadata": {},
   "source": [
    "When we save a model, Keras calls the loss instance's `get_config()` method and saves the config as the JSON. When we load the model, it calls the `from_config()` class method on `HuberLoss` class: this method is implemented by the base class (Loss) and creates an instance of the class, passing `**config` to the constructor. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52d0423b-7983-41c8-a3b8-52126e5b14f8",
   "metadata": {},
   "source": [
    "That's it for losses! Similar for activation functions, initializers, regularizers and constraints. Let's look at these now."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d64c0242-be57-4ace-9ceb-4806ae51990a",
   "metadata": {},
   "source": [
    "### Custom Activation Functions, Initializers, Regularizers and Constraints"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be01ae6b-8779-4e94-b025-d64377ff6bc7",
   "metadata": {},
   "source": [
    "Most of the Keras functionalities, such as losses, regularizers, constraints, initializers, metrics, activation functions, layers, and even full models, can be customized in very much same way. Most of the time, we just need to write a simple function with appropriate inputs and outputs.\n",
    "\n",
    "Here are some examples:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d25c391a-5f03-4a4b-a301-7e1a71fcf8fa",
   "metadata": {},
   "source": [
    "Custom activation function `softplus`:\n",
    "\n",
    "$$\n",
    " Y = log(1 + e^x)\n",
    "$$\n",
    "\n",
    "`softplus` is smooth continuous version of ReLU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "84a0f86b-e0ef-4cef-9233-bcadb8102169",
   "metadata": {},
   "outputs": [],
   "source": [
    "def my_softplus(z):\n",
    "    return tf.math.log(tf.exp(z) + 1.0) # same as tf.nn.softplus(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9abd608f-acb2-4895-bee1-5d6d5eefcaf8",
   "metadata": {},
   "source": [
    "A custom Glorot Initializer:\n",
    "\n",
    "$$\n",
    " \\text{Normal distribution with mean 0 and variance } {\\sigma}^2 = \\frac{1}{fan_{avg}} \\\\\n",
    "  \\text{where, } fan_{avg} = \\frac{fan_{in} + {fan_{out}}}{2}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "c93b9ece-a3af-4865-b837-032e6983c604",
   "metadata": {},
   "outputs": [],
   "source": [
    "def my_glorot_initializer(shape, dtype=tf.float32):\n",
    "    stddev = tf.sqrt( 2 / (shape[0]  + shape[1]))\n",
    "    #tf.random.normal outputs random values from a normal distribution\n",
    "    return tf.random.normal(shape=shape, mean=0.0, stddev=stddev, dtype=dtype) # same as keras.initializers.glorot_normal()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89f52539-e33d-4124-a9a8-6e6f00d33a0e",
   "metadata": {},
   "source": [
    "A custom $l_1$ regularizer:\n",
    "$$\n",
    "    Penalty = \\lambda \\cdot \\sum_{i=1}^N |w_i|\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "801bda08-cffd-4449-a817-5570cae1c539",
   "metadata": {},
   "outputs": [],
   "source": [
    "def my_l1_regularizer(weights):\n",
    "    # here lambda = 0.01\n",
    "    \"\"\"\n",
    "    this tf.reduce_sum() will calculate the sum of all the values in the tensor.\n",
    "    For example:\n",
    "    t = [[1,1,1], [1,1,1]]\n",
    "    tf.reduce_sum(t) ==> 6 \n",
    "    \n",
    "    So here, this tf.reduce_sum() acts as sigma (summation)\n",
    "    \"\"\"\n",
    "    return tf.reduce_sum(tf.abs(0.01 * weights)) # eqv to keras.regularizer.l1(0.01)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a9b988a-4b63-40f9-91b9-cdb2f4100d1c",
   "metadata": {},
   "source": [
    "A custom constraint that ensures weights are all positive:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "80af57fc-5307-4721-a8e4-1d7e07ba5a6b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def my_positive_weights(weights):\n",
    "    \"\"\"\n",
    "    tf.zeros_like creates a tensor with all elements set to zero\n",
    "    \"\"\"\n",
    "    # return value is same as tf.nn.relu()\n",
    "    return tf.where(weights < 0., tf.zeros_like(weights), weights) # eqv to keras.constraints.nonneg() or tf.nn.relu()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "729085ac-f372-4e27-9379-7fb16d5529d4",
   "metadata": {},
   "source": [
    "These custom function then can be used normally."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "3993f82a-75e9-45e9-800b-0c82694d8c1b",
   "metadata": {},
   "outputs": [],
   "source": [
    "layer = keras.layers.Dense(1, activation=my_softplus, kernel_initializer=my_glorot_initializer,\n",
    "                          kernel_regularizer=my_l1_regularizer, \n",
    "                          kernel_constraint=my_positive_weights)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a28c685-150e-4835-9a56-cf0a8ce26099",
   "metadata": {},
   "source": [
    "The activation function will be applied to the output of this `Dense` layer, and its result will be passed on to the next layer. The layer's weights will be initialized using the value returned by the initializer. At each training step the weights will be passed to the regularization function to compute the regularization loss, which will be added to the main loss to get the final loss used for training. Finally, the constraint function will be called after each training step, and the layer's weights will be replaced by the constrained weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "bbdb60b8-1748-4ec6-9eb2-789a00c93ba7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 1.7512 - mae: 0.9273 - val_loss: inf - val_mae: inf\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.6521 - mae: 0.5370 - val_loss: inf - val_mae: inf\n"
     ]
    }
   ],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"selu\", kernel_initializer=\"lecun_normal\",\n",
    "                      input_shape=input_shape),\n",
    "    keras.layers.Dense(1, activation=my_softplus, kernel_initializer=my_glorot_initializer,\n",
    "                          kernel_regularizer=my_l1_regularizer, \n",
    "                          kernel_constraint=my_positive_weights)\n",
    "])\n",
    "\n",
    "model.compile(loss=\"mse\", optimizer=\"nadam\", metrics=[\"mae\"])\n",
    "\n",
    "h = model.fit(X_train_scaled, y_train, epochs=2, validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "c1d5f2e9-7366-4623-b2cd-cbf654326b79",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save(\"my_model_with_many_custom_parts.keras\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "0e25d0a8-ebb8-423f-9a7c-5bde9d489274",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.load_model(\"my_model_with_many_custom_parts.keras\",\n",
    "                               custom_objects={\n",
    "                                   \"my_l1_regularizer\": my_l1_regularizer,\n",
    "                                   \"my_positive_weights\": my_positive_weights,\n",
    "                                   \"my_glorot_initializer\": my_glorot_initializer,\n",
    "                                   \"my_softplus\": my_softplus\n",
    "                               })"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "549ad60c-9407-4949-9cb0-edae78f1d4a7",
   "metadata": {},
   "source": [
    "If the function has hyperparameters that need to be saved along with a model, then we will want to subclass the appropriate class, such as `keras.regularizers.Regulizer`, `keras.constraints.Constraint`, `keras.initializers.Initializer` or `keras.layers.Layer` (for any layer including activation functions). Much like we did for custom loss, here's a simple class for $l_1$ regularization that saves its `factor` hyperparameter (this time we do not need to call the parent constructor or the `get_config()` method, as they are not defined by the parent class):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "82a08e70-669b-4575-9d71-8c8eb79e3ab7",
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyL1Regularizer(keras.regularizers.Regularizer):\n",
    "    def __init__(self, factor):\n",
    "        self.factor = factor\n",
    "    def __call__(self, weights):\n",
    "        return tf.reduce_sum(self.factor * tf.abs(weights))\n",
    "    def get_config(self):\n",
    "        return {\"factor\": self.factor}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "391be500-004a-48bd-a86e-4750fb8c33c7",
   "metadata": {},
   "source": [
    "Note that we must implement the `call()` method for losses, layers (including activation functions), and models, or the `__call__()` method for regularizers, initializers, and constraints. For metrics, things are bit different, we will see them in a while."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "8e3dd638-844b-4f24-b508-6afdc41e711a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 1.7632 - mae: 0.9487 - val_loss: inf - val_mae: inf\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.6212 - mae: 0.5393 - val_loss: inf - val_mae: inf\n"
     ]
    }
   ],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"selu\", kernel_initializer=\"lecun_normal\",\n",
    "                      input_shape=input_shape),\n",
    "    keras.layers.Dense(1, activation=my_softplus, \n",
    "                       kernel_initializer=my_glorot_initializer,\n",
    "                       kernel_regularizer=MyL1Regularizer(0.01),\n",
    "                       kernel_constraint=my_positive_weights)\n",
    "])\n",
    "\n",
    "model.compile(loss=\"mse\", optimizer=\"nadam\", metrics=[\"mae\"])\n",
    "\n",
    "h = model.fit(X_train_scaled, y_train, epochs=2, validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "792e9712-a2a7-400d-8bc8-082e9e72f600",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save(\"my_model_with_many_custom_parts.keras\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "399ca198-1576-4833-8532-1f66c16ae0f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.load_model(\"my_model_with_many_custom_parts.keras\",\n",
    "                               custom_objects={\n",
    "                                   \"MyL1Regularizer\": MyL1Regularizer,\n",
    "                                   \"my_positive_weights\": my_positive_weights,\n",
    "                                   \"my_glorot_initializer\": my_glorot_initializer,\n",
    "                                   \"my_softplus\": my_softplus\n",
    "                               })"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ceff654-abda-495f-8964-1dcbfd52d0ad",
   "metadata": {},
   "source": [
    "### Custom Metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "e9029cc8-f8bb-43d7-89d4-09dcda4ce26a",
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"selu\", kernel_initializer=\"lecun_normal\",\n",
    "                      input_shape=input_shape),\n",
    "    keras.layers.Dense(1)\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a84043d-b1e9-41d0-833e-5de1fb220824",
   "metadata": {},
   "source": [
    "In most cases, defining a custom metric function is exactly the same as defining a custom loss function. In fact, we could even use the Huber loss function we created earlier as a metric; (however, Huber loss is seldom used as metric, MSE or MAE is preferred) it would just work fine:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "3ef435ce-b303-4fea-9983-fc378a6057c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"mse\", optimizer=\"nadam\", metrics=[create_huber(2.0)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "5ad01943-2c4c-4111-b1e3-615d7c7dd86f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 1s 876us/step - loss: 2.8613 - huber_fn: 1.0737\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 0s 924us/step - loss: 0.8171 - huber_fn: 0.2885\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.src.callbacks.History at 0x7f90d5e48b10>"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train_scaled, y_train, epochs=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11cca561-1407-4233-99e9-3be494f6c02d",
   "metadata": {},
   "source": [
    "For each batch during training, Keras will compute this metric and keep track of its mean since the beginning of the epoch. Most of the time, this is exactly what we want. But not always!\n",
    "\n",
    "Consider a binary classifier's precision, for example. As we saw in Chapter 3, precision is the number of true positives, divided by the number of positive predictions (including both true and false positives). Suppose the model made five positive predictions in first batch, four of which were correct: that's 80% prediction. \n",
    "\n",
    "Then suppose model made three positive predictions in the second batch, but they all were incorrect: that's 0% precision for the second batch. If we compute the mean of these two precisions, we get 40%. \n",
    "\n",
    "But wait a second - that's *not* the model's precision over these two batches! Indeed, there were total of 4 true positives (4 + 0) out of 8 positive predictions (5 + 3), so the overall precision is 50% and not 40%. \n",
    "\n",
    "What we need is an object that can keep track of the number of true positives and the number of false positives and that can compute their ratios when requested. \n",
    "\n",
    "This is precisely what the `keras.metrics.Precision` class does:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "24e31f61-d817-4556-9c15-dafdae535f73",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=0.8>"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "precision = keras.metrics.Precision()\n",
    "precision([0, 1, 1, 1, 0, 1, 0, 1], [1, 1, 0, 1, 0, 1, 0, 1]) # labels and predictions for 1st batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "522d1ee7-daf9-4a35-8b99-532f4df6aa33",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=0.5>"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "precision([0, 1, 0, 0, 1, 0, 1, 1], [1, 0, 1, 1, 0, 0, 0, 0]) # labels and predictions for 2nd batch"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55f9984c-6dfd-43e2-83e4-34c330c7e751",
   "metadata": {},
   "source": [
    "In this example, we create a `Precision` object, then we used it like a function, passing it a label and predictions for the first batch, then for the second batch (we could also have passed sample weights). We used the same number of true and false positives as in the example we just discussed. \n",
    "\n",
    "After the first batch, it returns a precision of 80%; then after the second batch, it returns 50% (which is overall precision so far, not the second batch's precision). \n",
    "\n",
    "This is called `streaming metric` or (`stateful metric`), as it is gradually updated, batch after batch."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94ea9169-3513-4669-9e0c-bce300c5c42d",
   "metadata": {},
   "source": [
    "At any point, we can call the `result()` method to get the current value of the metric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "73ae1945-f252-4900-9991-f96c724608ba",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=0.5>"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "precision.result()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8b90c44-8604-4555-a360-8316121df517",
   "metadata": {},
   "source": [
    "We can also look at its variables (tracking the number of true and false positives) by using the `variables` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "id": "03880b57-64e5-44d9-9de4-940e11bb43cf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Variable 'true_positives:0' shape=(1,) dtype=float32, numpy=array([4.], dtype=float32)>,\n",
       " <tf.Variable 'false_positives:0' shape=(1,) dtype=float32, numpy=array([4.], dtype=float32)>]"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "precision.variables"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23f1c566-c798-4387-bac2-744a21b6b13c",
   "metadata": {},
   "source": [
    "We can reset these variables using the `rest_states()` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "id": "66b88617-e94e-4bef-89ca-a51625e17936",
   "metadata": {},
   "outputs": [],
   "source": [
    "precision.reset_states() # both variables will get reset to 0.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "id": "a2777101-2122-463f-a997-558d142837b3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Variable 'true_positives:0' shape=(1,) dtype=float32, numpy=array([0.], dtype=float32)>,\n",
       " <tf.Variable 'false_positives:0' shape=(1,) dtype=float32, numpy=array([0.], dtype=float32)>]"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "precision.variables"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96f1e5e1-17ac-486e-bd34-1fad47813c39",
   "metadata": {},
   "source": [
    "If we need to create such a streaming metric, create a subclass of `keras.metrics.Metric` class. \n",
    "\n",
    "Here's the simple example that keeps track of the total Huber loss and the number of instances seen so far. When asked for result, it returns the ratio, which is simply the mean Huber loss:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "id": "fa467254-56d5-4cf8-9c70-581c8f88da70",
   "metadata": {},
   "outputs": [],
   "source": [
    "class HuberMetric(keras.metrics.Metric):\n",
    "    def __init__(self, threshold=1.0, **kwargs):\n",
    "        super().__init__(**kwargs) # handles base args (e.g dtype)\n",
    "        self.threshold = threshold\n",
    "        self.huber_fn = create_huber(threshold)\n",
    "        # add_weight adds a new variable to the layer\n",
    "        self.total = self.add_weight(\"total\", initializer=\"zeros\")\n",
    "        self.count = self.add_weight(\"count\", initializer=\"zeros\")\n",
    "    def update_state(self, y_true, y_pred, sample_weight=None):\n",
    "        \"\"\"\n",
    "        This method is called when we use an instance of this class as function (as we did with\n",
    "        `Precision` object). It updates the variables, given the labels and predictions for one batch\n",
    "        (and sample weights, but in this case we ignore them). \n",
    "        \"\"\"\n",
    "        metric = self.huber_fn(y_true, y_pred)\n",
    "        # assign_add() is to update the variable. Since in tf, direct updation is not allowed\n",
    "        self.total.assign_add(tf.reduce_sum(metric))\n",
    "        # tf.size() => Returns a 0-D Tensor representing the number of elements\n",
    "        self.count.assign_add(tf.cast((tf.size(y_true)), tf.float32))\n",
    "    def result(self):\n",
    "        \"\"\"\n",
    "        Computes and returns the final result, in this case the mean Huber metric over all instances.\n",
    "        When we use the metric as a function, the `update_state()` method gets called first, then the\n",
    "        `result()` method is called, and its output is returned\n",
    "        \"\"\"\n",
    "        return self.total / self.count\n",
    "    def get_config(self):\n",
    "        \"\"\"\n",
    "        Implemented this method to ensure that `threshold` gets saved along with the model. \n",
    "        \"\"\"\n",
    "        base_config = super().get_config()\n",
    "        return {**base_config, \"threshold\": self.threshold}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "id": "cc3ff8e4-6c37-4175-b806-2d4ff9d47ff7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=14.0>"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m = HuberMetric(2.0)\n",
    "\n",
    "\"\"\"\n",
    "error (10 - 2 = 8) > threshold (2) . Therefore it will act linearly\n",
    " total = 2 * (|10-2| - 2/2) = 14\n",
    " count = 1\n",
    " result = 14 / 1 = 14\n",
    "\"\"\"\n",
    "\n",
    "# y_true, y_pred\n",
    "m(tf.constant([[2.]]), tf.constant([[10.]]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "id": "1f59607c-921e-47a0-b335-269945b47102",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=7.0>"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# total = total + (|1-0|^2 / 2) + (2 * (|9.25 - 5| - 2/2)) = 14 + 0.5 + 6.5 = 21\n",
    "# count = count + 2 = 1 + 2 = 3\n",
    "# result = total / count = 21 / 3 = 7\n",
    "\n",
    "m(tf.constant([[0.], [5.]]) , tf.constant([[1.], [9.25]]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "id": "4af63c46-86fb-4f49-a583-3750a273b36e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=7.0>"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m.result()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "id": "c0a63238-c0b6-4cba-bcab-4d5c669bf750",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Variable 'total:0' shape=() dtype=float32, numpy=21.0>,\n",
       " <tf.Variable 'count:0' shape=() dtype=float32, numpy=3.0>]"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m.variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "id": "67897748-f5cc-4276-b0af-ad944367394c",
   "metadata": {},
   "outputs": [],
   "source": [
    "m.reset_states()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "id": "bd4eb83f-0720-47da-96c7-58d585bb4b13",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Variable 'total:0' shape=() dtype=float32, numpy=0.0>,\n",
       " <tf.Variable 'count:0' shape=() dtype=float32, numpy=0.0>]"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m.variables"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9f2c356-ffc3-4b8b-9a8a-d48a8c89f56e",
   "metadata": {},
   "source": [
    "Let's check that the `HuberMetric` class works well:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "id": "dff75c82-3d14-4f71-9c13-9fa129072883",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 1s 1ms/step - loss: 0.9816 - huber_metric: 0.9816\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2520 - huber_metric: 0.2520\n"
     ]
    }
   ],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"selu\", kernel_initializer=\"lecun_normal\", \n",
    "                      input_shape=input_shape),\n",
    "    keras.layers.Dense(1)\n",
    "])\n",
    "\n",
    "model.compile(loss=create_huber(2.0), optimizer=\"nadam\", metrics=[HuberMetric(2.0)])\n",
    "\n",
    "h = model.fit(X_train_scaled.astype(np.float32), y_train.astype(np.float32), epochs=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "id": "d2309be4-2fe5-4926-933d-ae8897f87093",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save(\"my_model_with_a_custom_metric.keras\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "id": "d06529c1-ba79-4236-a14d-ee0a6c70acf5",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.load_model(\"my_model_with_a_custom_metric.keras\",\n",
    "                               custom_objects={\n",
    "                                   \"huber_fn\": create_huber(2.0),\n",
    "                                   \"HuberMetric\": HuberMetric\n",
    "                               })"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "id": "b7bc4b01-f721-4e1c-89ad-f5677359d858",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 1s 1ms/step - loss: 0.2299 - huber_metric: 0.2299\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2219 - huber_metric: 0.2219\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.src.callbacks.History at 0x7f90d450fd90>"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train_scaled.astype(np.float32), y_train.astype(np.float32), epochs=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "id": "2b9606df-c035-42e2-ae06-c08aeeb9f8bc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<keras.src.metrics.base_metric.Mean at 0x7f90d6a81dd0>,\n",
       " <__main__.HuberMetric at 0x7f90d46a1ed0>]"
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "id": "6d409c52-15d9-4d6a-8fd0-2f05f831091c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.0"
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.metrics[-1].threshold"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c85924c-de34-4e06-9469-3f5eb610d076",
   "metadata": {},
   "source": [
    "When we define a metric using a simple function, Keras automatically calls it for each batch, and it keeps track of the mean durin each epoch, just like we did manually. So the only benefit of our `HuberMetric` class is that the `threshold` will be saved. But of course, some metrics like precision, cannot simply be averaged over batches, in those cases, there's no other option than to implement a streaming metric. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fb5e07f-65a7-40f2-af00-511ccec16229",
   "metadata": {},
   "source": [
    "More simply, we could have the created the class like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "id": "a7024caf-2444-4242-b302-18a472e47dac",
   "metadata": {},
   "outputs": [],
   "source": [
    "class HuberMetric(keras.metrics.Mean):\n",
    "    def __init__(self, threshold=1.0, name=\"HuberMetric\", dtype=None):\n",
    "        self.threshold = threshold\n",
    "        self.huber_fn = create_huber(threshold)\n",
    "        super().__init__(name=name, dtype=dtype)\n",
    "    def update_state(self, y_true, y_pred, sample_weight=None):\n",
    "        metric = self.huber_fn(y_true, y_pred)\n",
    "        super(HuberMetric, self).update_state(metric, sample_weight)\n",
    "    def get_config(self):\n",
    "        base_config = super().get_config()\n",
    "        return {**base_config, \"threshold\": self.threshold}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a30be7a0-ff08-46e3-9390-8e77c3db0c7b",
   "metadata": {},
   "source": [
    "This class handles shapes better, and it also supports sample weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "id": "caa091d1-84df-4bee-badf-bce199f807c4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 1s 1ms/step - loss: 0.3758 - HuberMetric: 0.7602\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.1223 - HuberMetric: 0.2494\n"
     ]
    }
   ],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"selu\", kernel_initializer=\"lecun_normal\",\n",
    "                      input_shape=input_shape),\n",
    "    keras.layers.Dense(1)\n",
    "])\n",
    "\n",
    "model.compile(loss=keras.losses.Huber(2.0), optimizer=\"nadam\", metrics=[HuberMetric(2.0)])\n",
    "\n",
    "sample_weight = np.random.rand(len(y_train))\n",
    "h = model.fit(X_train_scaled.astype(np.float32), y_train.astype(np.float32),\n",
    "             epochs=2, sample_weight=sample_weight)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "id": "9749e280-a208-4828-bd45-f95c4d7abce7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.37584829330444336, 0.37724936285833965)"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "h.history[\"loss\"][0] , h.history[\"HuberMetric\"][0] * sample_weight.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "id": "9ede3c20-f350-4087-8c4f-b42976ac404a",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save(\"my_model_with_a_custom_metric_v2.keras\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "id": "45512559-3285-4eb6-acca-2bc42d4db09e",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.load_model(\"my_model_with_a_custom_metric_v2.keras\",\n",
    "                               custom_objects={\n",
    "                                   \"HuberMetric\": HuberMetric\n",
    "                               })"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "id": "8f8536fc-9cc7-4f29-9eaa-36c0c0a4bd17",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 1s 1ms/step - loss: 0.2344 - HuberMetric: 0.2344\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.2235 - HuberMetric: 0.2235\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.src.callbacks.History at 0x7f908c6e7890>"
      ]
     },
     "execution_count": 132,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train_scaled.astype(np.float32), y_train.astype(np.float32), epochs=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "id": "18a83c1f-e63a-412d-a65a-b2bb74588f7e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.0"
      ]
     },
     "execution_count": 137,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.metrics[-1].threshold"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a6e0a0c-9fbf-4aec-9d94-12da0ee365cb",
   "metadata": {},
   "source": [
    "Now we have built a streaming metric, building a custom layer will seem like a walk in park. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18332ce1-db47-4379-9c76-e2cd2aad694e",
   "metadata": {},
   "source": [
    "### Custom Layers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d5302db-6861-4520-978a-9451a8699345",
   "metadata": {},
   "source": [
    "Let's see how to build custom layers. \n",
    "\n",
    "First, some layers have no weights, such as `keras.layers.Flatten` or `keras.layers.ReLU`. If we want to create a custom layer without any weights, the simplest option is to write a function and wrap it in a `keras.layers.Lambda` layer. For eg: the following layer will apply the exponential function to its input:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "id": "fe1eb911-6975-4ef0-8557-bf792be4af3c",
   "metadata": {},
   "outputs": [],
   "source": [
    "exponential_layer = keras.layers.Lambda(lambda x: tf.exp(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "id": "8e9ab8af-d18e-4bef-97e3-46bf03bd8c56",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3,), dtype=float32, numpy=array([0.36787945, 1.        , 2.7182817 ], dtype=float32)>"
      ]
     },
     "execution_count": 140,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "exponential_layer([-1., 0., 1.])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88dd23a9-d5c5-4073-8dbb-f0c6bcf6d21b",
   "metadata": {},
   "source": [
    "This custom layer can then be used like any other layer, using the Sequential API, the Functional API or the Subclassing API. We can also use it as activation function (or we could use `activation=tf.exp, activation=keras.activations.exponential`, or simply `activation=\"exponential\"`).\n",
    "Adding an exponential layer at the output of a regression model can be useful if the values to predict are positive and with a very different scales (e.g 0.001, 10, 10000)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "id": "81a63c02-b12c-47e6-9405-4eba7152b9d7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "363/363 [==============================] - 1s 1ms/step - loss: 0.6991 - val_loss: 0.4308\n",
      "Epoch 2/5\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.4440 - val_loss: 0.3812\n",
      "Epoch 3/5\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.5031 - val_loss: 0.3808\n",
      "Epoch 4/5\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.4053 - val_loss: 0.3649\n",
      "Epoch 5/5\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.3915 - val_loss: 0.3584\n",
      "162/162 [==============================] - 0s 745us/step - loss: 0.3731\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.37313514947891235"
      ]
     },
     "execution_count": 142,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"relu\", input_shape=input_shape),\n",
    "    keras.layers.Dense(1),\n",
    "    exponential_layer\n",
    "])\n",
    "\n",
    "model.compile(loss=\"mse\", optimizer=\"SGD\")\n",
    "\n",
    "h = model.fit(X_train_scaled, y_train, epochs=5, validation_data=(X_valid_scaled, y_valid))\n",
    "model.evaluate(X_test_scaled, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbce0d4b-030e-4ab2-bd4b-372e9ed2173c",
   "metadata": {},
   "source": [
    "To build a custom stateful layer (i.e., layer with weights), we need to create a subclass of the `keras.layers.Layer` class. \n",
    "\n",
    "For example, the following class implements a simplified version of the `Dense` Layer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "id": "4130d1bd-09a1-4594-9206-0224f8348126",
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyDense(keras.layers.Layer):\n",
    "    def __init__(self, units, activation=None, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.units = units\n",
    "        \n",
    "        \"\"\"\n",
    "        Convers the activation argument to appropriate activation function using\n",
    "        `keras.activations.get()` function (it accepts function strings like \"relu\" or \"selu\" or\n",
    "        simple None). [This function is specifig to tf.keras . We could also use\n",
    "        keras.activations.Activation()]\n",
    "        \"\"\"\n",
    "        self.activation = keras.activations.get(activation)\n",
    "        \n",
    "    def build(self, batch_input_shape):\n",
    "        \"\"\"\n",
    "        Role is to create layer's variables by calling `add_weight()` method for each weight. \n",
    "        \"\"\"\n",
    "        self.kernel = self.add_weight(\n",
    "            name=\"kernel\", shape=[batch_input_shape[-1], self.units],\n",
    "            initializer=\"glorot_normal\")\n",
    "        self.bias = self.add_weight(\n",
    "            name=\"bias\", shape=[self.units], initializer=\"zeros\")\n",
    "        super().build(batch_input_shape) # must be at end\n",
    "        \n",
    "    def call(self, X):\n",
    "        \"\"\"\n",
    "        This method performs the desired operations. In this case, we compute the matrix\n",
    "        multiplication of the inputs X and the layer's kernel, we add the bias vector, and we \n",
    "        apply the activation function to the result, and this gives the output of the layer\n",
    "        \"\"\"\n",
    "        return self.activation(X @ self.kernel + self.bias)\n",
    "    \n",
    "    def compute_output_shape(self, batch_input_shape):\n",
    "        \"\"\"\n",
    "        Simply returns the shape of this layer's outputs. In this case, it is same shape as the\n",
    "        inputs, except the last dimension is replaced with the number of neurons in the layer. Note\n",
    "        that in tf.keras, shapes are instances of the `tf.TensorShape` class, which we can convert\n",
    "        to Python lists using `as_list()`\n",
    "        \"\"\"\n",
    "        return tf.TensorShape(batch_input_shape.as_list()[:-1] + [self.units])\n",
    "    \n",
    "    def get_config(self):\n",
    "        \"\"\"\n",
    "        Just like previous custom classes. We save the activation function's full configuration\n",
    "        by calling `keras.activations.serialize()`.\n",
    "        \"\"\"\n",
    "        base_config = super().get_config()\n",
    "        return {**base_config, \n",
    "                \"units\": self.units,\n",
    "                \"activation\": keras.activations.serialize(self.activation)}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e51264e9-6b45-4c8a-828b-57e2837d98fc",
   "metadata": {},
   "source": [
    "Code explanation:\n",
    "\n",
    "- The constructor takes all the hyperparameters as arguments (in this example, units and activation), and importantly it also takes `**kwargs` argument. It calls the parent constructor, passing it `kwargs`: this takes care of standard arguments such as `input_shape`, `trainable` and `name`. Then it saves the hyperparameters as attributes. \n",
    "\n",
    "- The `build()` method is called the first time layer is used. At that point, Keras will know the shape of this layer's inputs, and it will pass it to the `build()` method, which is often necessary to create some of the weights. For ex: we need to know the number of neurons in the previous layer in order to create the connection weights matrix (i.e the \"kernal\"): this corresponds to the size of the last dimension of the inputs. At the end of the `build()` method (and only at the end), we must call the parent's `build()` method: this tells Keras that the layer is built (it just sets `self.built=True`)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5254a2a-d46d-4f35-8682-b8de126309b2",
   "metadata": {},
   "source": [
    "We can now use a `MyDense` layer just like any other layer:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43aa545e-b9c2-4566-9831-789f44864f41",
   "metadata": {},
   "source": [
    "**NOTE:**\n",
    "\n",
    "We can generally omit the `compute_output_shape()` method, as tf.keras automatically infers the output shape, except where the layer is dynamic (we will see shortly)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "id": "802007f3-1818-4a5e-9152-c5ff410203b0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 1.3340 - val_loss: 0.6066\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.5447 - val_loss: 0.5085\n",
      "162/162 [==============================] - 0s 851us/step - loss: 0.4712\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.47121259570121765"
      ]
     },
     "execution_count": 146,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    MyDense(30, activation=\"relu\", input_shape=input_shape),\n",
    "    MyDense(1)\n",
    "])\n",
    "\n",
    "model.compile(loss=\"mse\", optimizer=\"nadam\")\n",
    "\n",
    "h = model.fit(X_train_scaled, y_train, epochs=2, validation_data=(X_valid_scaled, y_valid))\n",
    "\n",
    "model.evaluate(X_test_scaled, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "id": "f82a6e0b-3423-4f3a-b57c-87272d0fccd8",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save(\"my_model_with_a_custom_layer.keras\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "id": "e5971142-ccbe-4817-9452-85953f76e53f",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.load_model(\"my_model_with_a_custom_layer.keras\",\n",
    "                               custom_objects={\n",
    "                                   \"MyDense\": MyDense\n",
    "                               })"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6834f9ed-c41b-4b53-acdf-74d4ece56abe",
   "metadata": {},
   "source": [
    "To create a layer with multiple inputs (e.g: `Concatenate`), the argument to `call()` method should be tuple containing all the inputs, and similarly the argument to the `compute_output_shape()` method should be a tuple containing each input's batch shape. \n",
    "\n",
    "To create a layer with multiple outputs, the `call()` method should return a list of outputs, and `compute_output_shape()` should return the list of batch output shapes (one per output). \n",
    "\n",
    "For ex: the following toy layer takes two inputs and returns three outputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "id": "3b3354b3-9e17-44c6-a45d-f450941561d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyMultiLayer(keras.layers.Layer):\n",
    "    def call(self, X):\n",
    "        X1, X2 = X\n",
    "        print(\"X1.shape: \", X1.shape, \" X2.shape: \", X2.shape) \n",
    "        return X1 + X2, X1 * X2\n",
    "    \n",
    "    def compute_output_shape(self, batch_input_shape):\n",
    "        b1, b2 = batch_input_shape\n",
    "        return [b1, b1, b1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04112425-09cd-49d7-9f9e-399e62b5695c",
   "metadata": {},
   "source": [
    "This layer may now be used like any other layer, but of course only using the Functional or Subclassing APIs, not the Sequential API (which accepts layers with one input and one output). "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41aa8d6f-2b9b-4687-be2a-b758ea858697",
   "metadata": {},
   "source": [
    "Our custom layer can be called using the functional API like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "id": "34fd6c92-cda2-4692-9325-b69661131006",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X1.shape:  (None, 2)  X2.shape:  (None, 2)\n"
     ]
    }
   ],
   "source": [
    "inputs1 = keras.layers.Input(shape=[2])\n",
    "inputs2 = keras.layers.Input(shape=[2])\n",
    "outputs1, outputs2 = MyMultiLayer()((inputs1, inputs2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16b1d583-4fee-4ca1-9442-6d7d18a21a55",
   "metadata": {},
   "source": [
    "Note that the `call()` method receives symbolic inputs, whose shape is only partially specified (at this stage, we don't know the batch size, which is why the first dimension is `None`):"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1523e487-89aa-4e20-83ab-5216bbe776d8",
   "metadata": {},
   "source": [
    "We can also pass the actual data to the custom layer. To test this, let's split each dataset's input into two parts, with four features each:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "id": "27897683-61d7-4bc4-b9a4-01de42044d93",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((11610, 4), (11610, 4))"
      ]
     },
     "execution_count": 155,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def split_data(data):\n",
    "    columns_count = data.shape[-1]\n",
    "    half = columns_count // 2\n",
    "    return data[:, :half], data[:, half:]\n",
    "\n",
    "X_train_scaled_A, X_train_scaled_B = split_data(X_train_scaled)\n",
    "X_valid_scaled_A, X_valid_scaled_B = split_data(X_valid_scaled)\n",
    "X_test_scaled_A, X_test_scaled_B = split_data(X_test_scaled)\n",
    "\n",
    "X_train_scaled_A.shape, X_train_scaled_B.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "id": "82488ff1-a8ae-418f-a64b-8698a65efca8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X1.shape:  (11610, 4)  X2.shape:  (11610, 4)\n"
     ]
    }
   ],
   "source": [
    "outputs1, outputs2 = MyMultiLayer()((X_train_scaled_A, X_train_scaled_B))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c31f6f19-9c6f-40bf-8505-ecc97ddd1f57",
   "metadata": {},
   "source": [
    "Now notice above, shapes are fully specified."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef4049b8-f900-4881-a304-23844722be3f",
   "metadata": {},
   "source": [
    "Let's build a more complete model using the functional API (this is just a toy model):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "id": "2383b3de-842d-4b3f-942f-0fed94d1135d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X1.shape:  (None, 4)  X2.shape:  (None, 4)\n"
     ]
    }
   ],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "input_A = keras.layers.Input(shape=X_train_scaled_A.shape[-1])\n",
    "input_B = keras.layers.Input(shape=X_train_scaled_B.shape[-1])\n",
    "hidden_A, hidden_B = MyMultiLayer()((input_A, input_B))\n",
    "hidden_A = keras.layers.Dense(30, activation=\"selu\")(hidden_A)\n",
    "hidden_B = keras.layers.Dense(30, activation=\"selu\")(hidden_B)\n",
    "concat = keras.layers.Concatenate()((hidden_A, hidden_B))\n",
    "output = keras.layers.Dense(1)(concat)\n",
    "model = keras.models.Model(inputs=[input_A, input_B], outputs=[output])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "id": "217a5ed6-5282-463d-8351-76b68ab54e32",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"mse\", optimizer=\"nadam\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "id": "0c2f800a-bd8e-47fb-abe4-3580c71516d7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "X1.shape:  (None, 4)  X2.shape:  (None, 4)\n",
      "X1.shape:  (None, 4)  X2.shape:  (None, 4)\n",
      "336/363 [==========================>...] - ETA: 0s - loss: 1.9787X1.shape:  (None, 4)  X2.shape:  (None, 4)\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 1.9091 - val_loss: 1.4686\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 1s 1ms/step - loss: 0.9841 - val_loss: 1.1028\n"
     ]
    }
   ],
   "source": [
    "h = model.fit((X_train_scaled_A, X_train_scaled_B), y_train, epochs=2, \n",
    "             validation_data=((X_valid_scaled_A, X_valid_scaled_B), y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "095be38b-d921-4d0e-8ffe-1b2e13e95df1",
   "metadata": {},
   "source": [
    "If our layer needs to have a different behaviour during training and during testing (e.g., if it uses `Dropout` or `BatchNormalization` layers), then we must add a `training` argument to `call()` method and to use this argument to decide what to do.\n",
    "\n",
    "For ex., let's create a layer that adds Gaussian noise during training (for regularization) but does nothing during testing (Keras has a layer that does the same thing, `keras.layers.GaussianNoise`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "id": "05e8eec8-905b-4a7c-b625-1c8830d56f0f",
   "metadata": {},
   "outputs": [],
   "source": [
    "class AddGaussianNoise(keras.layers.Layer):\n",
    "    def __init__(self, stddev, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.stddev = stddev\n",
    "    \n",
    "    def call(self, X, training=None):\n",
    "        if training:\n",
    "            noise = tf.random.normal(tf.shape(X), stddev=self.stddev)\n",
    "            return X + noise\n",
    "        else:\n",
    "            return X\n",
    "    \n",
    "    def compute_output_shape(self, batch_input_shape):\n",
    "        return batch_input_shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27cca08a-c6b1-493a-b6a2-3d1dc76ece8a",
   "metadata": {},
   "source": [
    "Here's a simple model that uses this custom layer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "id": "4d109420-3049-4d36-b560-a827e28f806e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 1s 1ms/step - loss: 2.2817 - val_loss: 1.2067\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 1.0056 - val_loss: 0.8844\n",
      "162/162 [==============================] - 0s 709us/step - loss: 0.7399\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.7398827075958252"
      ]
     },
     "execution_count": 165,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    AddGaussianNoise(stddev=1.0),\n",
    "    keras.layers.Dense(30, activation=\"selu\"),\n",
    "    keras.layers.Dense(1)\n",
    "])\n",
    "\n",
    "model.compile(loss=\"mse\", optimizer=\"nadam\")\n",
    "h = model.fit(X_train_scaled, y_train, epochs=2, \n",
    "             validation_data=(X_valid_scaled, y_valid))\n",
    "model.evaluate(X_test_scaled, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb2c7f46-3e34-4452-b4c2-2f13f1bf2af5",
   "metadata": {},
   "source": [
    "Now let's create custom models:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "800d29e5-7659-4899-b916-1e64dc9dd813",
   "metadata": {},
   "source": [
    "### Custom Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "id": "4892eba4-9f8f-4dcb-8f2f-87739df022b0",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_new_scaled = X_test_scaled"
   ]
  },
  {
   "attachments": {
    "46d04e3a-f78a-43ed-bf64-dd272c823242.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAAqkAAAG6CAYAAAArjN9vAAAABHNCSVQICAgIfAhkiAAAABl0RVh0U29mdHdhcmUAZ25vbWUtc2NyZWVuc2hvdO8Dvz4AAAAtdEVYdENyZWF0aW9uIFRpbWUAVGh1IDA0IEphbiAyMDI0IDA3OjU1OjQ0IFBNIElTVNLUHkgAACAASURBVHic7N13dBTlwgbwZ7ald0I6JJBeSaX3DqIURUW9Nq6i194+9dru1Wu9YlesiHoVlCJNOqElQAIpEJIQEiAJ6T2btsnuzPcHshqTQGiZTXh+53AOmZ3dfTYn2Tz7zjvvCJIkSSAiIiIiMiEKuQMQEREREf0VSyoRERERmRyWVCIiIiIyOSypRERERGRyWFKJiIiIyOSwpBIRERGRyWFJJSIiIiKTw5JKRERERCaHJZWIiIiITA5LKhERERGZHJZUIhOg1WrljkBERGRSWFKJZKbX67F06VKIoih3FCIiIpPBkkoks/T0dKSkpODYsWNyRyEiIjIZLKlEMisrK0Nrayv27NkDSZLkjkNERGQSWFKJZCSKItLS0gAAaWlpqK+vlzkRERGRaWBJJZJRUVER8vLyAACNjY1ISUmROREREZFpYEklklFKSgqam5uNXyckJPCQPxEREVhSiWTT2NiIjRs3ttt2/PhxpKeny5SIiIjIdLCkEskkPT0dDQ0N7bYZDAYcPHiQo6lERHTNY0klkoEkSdi7d2+nt6Wnp/MEKiIiuuaxpBLJoKysDCdOnOj0ttraWrS0tPRwIiIiItPCkkokgz179qCpqQkKhQIODg4AABsbG2g0GgBAdna2nPGIiIhkx5JK1MO0Wi327t0LQRAwffp0jBo1CgDg6emJe++9F+bm5ti0aRPnpRIR0TWNJZWoh2VnZ6OqqgpTpkzBzTffDJVKZbxt+PDhuOeee1BRUYFTp07JmJKIiEheLKlEPezUqVOYOXMmbr/99nYFFQAEQcCoUaMQFRWFxMREmRISERHJjyWVqIeFhYVh/vz5UCqVXe5zzz33wNHRETqdrgeTERERmQ7VhXchoispMDAQgiCcdx8LCwtMnz69hxIRERGZHpZUoh52oYJ6sfsRERH1RTzcT0REREQmhyWViIiIiEwOSyoRERERmRyWVCIiIiIyOSypRERERGRyWFKJiIiIyOSwpBIRERGRyWFJJSIiIiKTw5JKRERERCaHJZWIiIiITA5LKhERERGZHJZUIiIiIjI5LKlEREREZHJYUomIiIjI5LCkEhEREZHJYUklIiIiIpPDkkpEREREJocllYiIiIhMDksqEREREZkcllQiIiIiMjksqURERERkclhSiYiIiMjksKQSERERkclRyR2Aeh+dToeioiKUlZWhtrZW7ji9Xl5eHgCguroamzdvhkLBz46Xw9LSEl5eXhg4cCCUSqXccYiI6BIJkiRJcocg0yeKIjIzM5GZmYn0tFRIeh36O9rC0sJc7mhE7bS2taGwpApQqhEcEgofHx8MHz4carVa7mhERHQROJJKF9TW1ob169fj119/haerE+ZPH41gvwFQcsSPTJReb8CJ/GJs25eC+J07UFFRgTlz5nCUmoioF2FJpQtas2YNNmzYgGmjo3D9xGGwMNfIHYnovFQqJYIGe8Hf2wMpx3Kx7NctsLCwwIwZM+SORkRE3cSSSudVU1OD/fv3Y97UEbhufJzccYguilKpQGy4PwpKKrBixQp4enoiPDxc7lhERNQNPPZFXZIkCStWrEDIIFdMHR0tdxyiSzZzXCxC/Qbgxx9/RF1dndxxiIioG1hSqUsnT57EgQP7MTQiAGoVz5Km3svcTIO75k1CWWkJdu/eLXccIiLqBpZU6lJ6ejoszTTwcu0ndxSiy+Zga41g3wFIS0uDKIpyxyEiogtgSaUunT59Gh6u/WBjbSl3FKLLJggCAnw8UFhYyJJKRNQLsKRSl+rr69HPwVbuGERXTD9HOzQ2NoLLQxMRmT6WVDovMw0XgKC+Q6XkWx4RUW/Bd2wiIiIiMjksqURERERkclhSiYiIiMjksKQSERERkclhSSUiIiIik8OSSkREREQmhyWViIiIiEwOSyoRERERmRyu1E70J9/9vB7fr9zYYbuLsyMsLSzg5e6CmZNHY0hIABQKfsYjIiK6WlhSif7kVGEx4hOSz7vPW598i8ljhuHj15+Fh1v/HkpGRER0bWFJJepEwOCBeO3Zh9ptKyotx67EQ9gcn4j1W3ejXtuA7b98DkEQZEpJRETUd7GkEnXC0cEOc2ZM6LD9gTvnY81vO3Df068iOe0Y9h5MxZhhUTIkJCIi6ttYUokuglKpwI2zJsPMTIP59z2Dl97+FJt+/AQW5mbt9svOPYWde5NxLCcP9rY2iB0SgkljhsLayrLdfilHs6BtaMKYYVEoLC7DlvhE5J8pxqCBnhg/Mg4+A9w7zZF5PA9rNscjv7AEDnY2CAvyw6wpY2Fna91hX21DI/YcSMGBw0dQVVOHID8fTB4zDAG+3hwFJiIik8WSSnQJZk0Zi7BAPyQmp+N0YTGC/HwAADpdK95d8j3e+/wH1NTVG/dXCAJGD4vCR68/i2D/QcbtH3z5IzKyc/HcI/fiqX8txpniMuNtHq798fZLj+PmG6YYtxkMBrz50VK898UPqK3TGrcLAKZPHIWvFr+M/v0cjdvTMo7j4effRFJqBgyiaNzu5GCHZx66C4/cuwBqNd8GiIjI9PD0ZKJLFBcZAlEUkZZx3Ljt42+W4+V3PoMgAC89eT8S1n+Ldd99gHEjY7HnQApm3/U4Kqpq2j1OfmEJ7nnsZfgPGoClH/wL65a9j9HDolBUWo4H/+8/KCmrNO67ftsevPb+l7A0N8f/Pn0DGbtXIn71V5h/w1Rs2pmA19770rhvVU0t5tz9BA6kHMX4UbFY8cXbOLT1R7zy1CIYDCKe+89H+GTpiqv/jSIiIroEHEIhukTeAzwAAAVFJQAASZKwY18S1CoVvvvoNUybMNK477gRMZh/3zPYvDMBP6/bin/cfbPxttp6LW6+YSq+XvwyzH+fNhAdEYyYqQtQUlaJA4ePGOfHbtmZiLY2PZ7+x53GEdZAXyAqLBBZJ05h9/7DxsfNOnEKhcWlmDZhJH7+4m1YWpgDAIaEBCA6Ihjz7n0Sb3/yLR6777ar+F0iIiK6NBxJJbpEKqUSwNlD/ABQVlGNnfuSMXnsMEweN7zdvpYW5nhk4a1QCAKKSso7PNYjC281FlQAcHZyRExECACg8E9TAPQGA4Czh/Erq2rQ3KKDJEmwsrTA9p+XYMfKL4z7Lv91CxQKBV5+4n5jQT1n4qg4DIsKQ31DI+q1jZfzbSAiIroqOJJKdImqa+sAAP0cHQAAO/YdhF6vR3FZBe58+IUO+zc166DRaHDg8JEOt8VEBLf7WqEQ4GBvCwAoLvuj1D5+/+3Yums/lv1+0YGAwQMxdng0po4fifEjY9qdmLUlPhGWFuZ457Nl0HQy77SyuhZtbXocPpKJ8SNjL+E7QEREdPWwpBJdoqycUwCAAZ6uAIDqmrOlNfVoNlKPZnd5vza94aKeRzT8ccJTsP8g/LrsPbz67hfYvCsRWSdOIevEKXzxw2oMjQzFp289j7AgP2OehsYmrN64o8vHVioU0F9kHiIiop7Akkp0CZpbWpCQnAY7W2vjKKhadfbXafG/n8LcTtZYPUejVl/Wc0eGBmL10sU4knUCG7ftRWJyGg6kHMX+w0cw5oZ7kLbzZwz0dINarUKw/yCs+vq/sPjL4f5zBAhwcrS/rDxERERXA0sq0SVYsXYrKqtqcOOsyXB2Orvkk8/vJ1JVVFbD082lw31aW9tw+kwxHOxsL/l5W9vaUFhUigGebggP8kN4kB8kScKJUwX4+xP/RkJyGtZujscjCxfAZ4AHThUUwdbWBi5/WpbqnKKSckiQYG6mueQ8REREVwtPnCLqJr3BgOLScrzzybd44c2PYW9ng38+thBK5dlfo9ghIfDycMUPq35DypGsdveVJAmfLF2B6MkL8OriLzp7+AtqbtFhys0PIGryre0O4QuCAP9BA+Hh1h8AoFCczTNnxgRUVtfinU++RZte3+6xCotKMeO2hxA16dYOS2IRERGZAo6kEnUiO/c0Zt/1uPFrRwc7lJRVIDntGGrrtFAoFHj07wsQEjC43T7zZ03Gu0u+x/z7nsE7Lz2OodFh0Ola8fO6rfjXu5/DYBAxPCbikjJZmJth9vTxSExOx5OvLIZrfyeEBvhCbzAg5Wg2DqZmwM7WGjdeNwkAcPu8GXjjw6/x4Vc/QadrxaI7b4KzkwMyc07i2f98gGPH8xA7JATOTg6X980iIiK6ClhSiTpRU1uPDdv2tNsmCAI0ahUiQvzx2N9vwy1zpnW439P/uAu5pwuxYdteLHjgOdjb2UBvMKCuTguNmQbPPXIPZk4efcm5Fi6Yg+TUY1i5fhtm3/k4rCwtIEoSauu1sLa0xNL3/wXX/v0AAK4uzvjyvy/jH8+/gSXfrcT/Vv0GCwsz1NU3QNfahogQfyz94N+XnIWIiOhqYkkl+pPZ08dj0EDPTm8z06gRHuSHwN8vgdqZfo72+OXLd7A5PhHrt+5Bdu4pqJRKhAb64pbZUzE0Kqzd/vfdPg+TxgwzHqL/s4UL5mD8yNh2l1G1trLED5/8B7fMnoqf123DqYIiaNQqBPh6Y9GdNyEi2N+4r0IQcOOsSRgeE47vftmA5LRjKK+shpe7C8aOiMGts6fBztb6Yr9FREREPUKQJEmSOwSZpldeeQXe/a1wx+yJckchuiIOZ5zA+9+uxbJly6C+zFUWiIjo6uKJU0RERERkclhSiYiIiMjksKQSERERkclhSSUiIiIik8OSSkREREQmhyWViIiIiEwOSyoRERERmRyWVDovg8hldKnvEPnzTETUa7CkUpdsbGxQWlEjdwyiK6asshYWFhZyxyAiom5gSaUuBQUFIb+oDBXVdXJHIbpsoigh/fgpBAQEQKXiFaGJiEwdSyp1KTIyEm0GCSXl1XJHIbpsZ0orceJ0MeLi4iAIgtxxiIjoAlhSqUuurq4IDw/HkeOnIUmcy0e9lyRJ+HVbIpycnBAZGSl3HCIi6gaWVOqSIAi4+eabkZSRh72HjqGltU3uSEQXTRRFbNyVjOzTJbjxxhthZ2cndyQiIuoGQeIQGV1AfHw8li1bhhBfLyy4fhxc+znIHYmoW5p1rVi/4yB2H8rE/fcv4igqEVEvwpJKFyRJElJSUvD111+jVdeM6ycMxdAhgbC2NIdSqZQ7HlE7oiiivqEJhzNysT0xFXUNLVi0aBFiY2M5F5WIqBdhSaVuq6ysxKZNm7Bnzx7o21rRz8EWFuZmcsciaqetTY+SimooVWoEBwdj1qxZ8Pf3lzsWERFdJJZUuiiSJKG8vBzHjx9HSUkJDAaD3JF6PUmSsH//fgwbNgwKBaeJXy5BEODh4YHg4GDY29tzuSkiol6K7950UQRBgIuLC1xcXOSO0mccPXoU27dvR2RkJEJCQuSOQ0REZBI4bEMks7KyMuh0OiQkJHCpLyIiot+xpBLJSJIkHDt2DACQmpqKhoYGmRMRERGZBpZUIhlVVVUhKysLAFBXV4ejR4/KnIiIiMg0sKQSySg9PR319fXGr/ft2wdRFGVMREREZBpYUolkotPpsG7dunbbMjIykJ2dLVMiIiIi08GSSiSTnJwcVFRUtNum1+uRnJwsUyIiIiLTwZJKJJM9e/Z0uv3w4cPQarU9nIaIiMi0sKQSyaCurs54Vv9fVVVVobm5uYcTERERmRaWVCIZJCQkoK6uDgBgZnb20rLm5uZQqVSQJAk5OTlyxiMiIpIdSypRD2tpacGuXbsgSRJCQ0MxevRoAIC3tzfmzp0LpVKJ7du3y5ySiIhIXiypRD3s5MmTKCoqQlBQEB5++GHY2NgAOHvJ2RkzZmD+/PkoKCjAmTNnZE5KREQkH5ZUoh6WmZmJkJAQPPbYY8aCeo5SqcR1110Hf39/HDhwQKaERERE8mNJJephrq6ueOihhzoU1HMEQcADDzwAlUqFpqamHk5HRERkGlRyByC61owYMQIKxfk/H9rZ2eGGG27ooURERKalvLwcVlZWsLKykjsKyYgjqUQ97EIF9RxBECAIwlVOQ0RkehISEnDo0CG5Y5DMWFKJiIjIZOh0OiQlJSEhIQGSJMkdh2TEw/1ERERkMs6tgKJWq1FUVARPT0+5I5FMOJJKREREJkEURWzcuBF6vR7Nzc3YsmULR1OvYSypREREZBKqqqpw5MgR49dJSUloaWmRMRHJiSWViIiITEJ6ejr0er3xa61WC4PBIGMikhNLKhEREZmE+vr6DtuOHz8uQxIyBSypREREJDudTof09PQO2+Pj4yGKogyJSG4sqURERCS7Y8eO4eTJkx22Z2RkdLqd+j6WVCIiIpKVJEnYv39/p/NPW1tbkZycLEMqkhtLKhEREcmqvr4eGRkZAACNRmPcbmZmBgBISUlBU1OTLNlIPiypREREJKukpCTU1dXBz88PU6dONW6/++674eLigqKiImRlZcmYkOTAkkpERESy0ev12LhxI3x9ffHoo4/C1tbWeJuzszOeeeYZeHh4YO3atTKmJDmwpBIREZFsMjMz4ejoiCeeeAKOjo4dbndzc8NTTz0Fg8GAM2fOyJCQ5MKSSkRERLKpqKjAI488Ant7+y73cXFxwSOPPIKCgoIeTEZyU8kdgIiIiK5dY8eOhUp14Tri4uICJyenHkhEpoIjqURERCSb7hTUS9mXej+WVCIiIiIyOSypRERERGRyWFKJiIiIyOSwpBIRERGRyWFJJSIiIiKTw5JKRERERCaHJZWIiIiITA4XHKOLJkkSSkpKUFpaivr6ernj9Hr5+fkAgNraWuzevRsKBT87Xg4rKyt4enrCzc1N7ihERHQZWFKpW0RRRE5ODlJSUpCVlY1abQMc+rlApdbIHa0PUCIwIhYAsC85XeYsvZ9Br0dVWTHsbKzg6zsYQUFBiImJ4SLgRES9DN+16YIMBgM2btyI1atXw9ahH8ZMm4OBvoFQa8zkjkbUqZamRpzJz0Pq/l3YuTMeCxbciqlTp3KUmoioF2FJpQtau3Yt1q5dh4hhYzBy0iyYmVvIHYnovMwtreAbFA4f/2BkpSVj9ZqVUKlUmDx5stzRiIiom1hS6bxqamqwd+9exI2bhhETZ0IQBLkjEXWbUqlCaPRwVJWX4Mcff8TAgQPh7+8vdywiIuoGQZIkSe4QZJpEUcSSJUvQoFdiwqz5UKnUckciuiQtzU3YsPxrKA0teOqpp2BjYyN3JPrduT9Bf/4ArNVq0djYCFEUodPpIAgCBg4caNIfktva2pCbm4uCggLja6qurkZFRQWamprAP7TdV1lRgdLSUgDAoEGDYGllJXOi7lMIAlxcXODi4mKcXmRpaYmYmBhYWFiY9M+wKeJIKnUpNzcXBw8exNy7HmZBpV7N3MISk2ffim/efQX79u3D9OnT5Y50zRJFESUlJaitrUVDQwOqq6thZmaGsWPHQqlUAgC2bNmCmpoaqFQqCIIAtVoNFxcXWFiY1lSj5uZmZGVlITk5GampqWhoaICNnQPsHZ1h6+AEBydnuAwOh4ZTpC5KztEUY0n1CoyEs6uHzIkuTm1VBTJPFqG+thr1NVXQ1tfi66+/RmhoKCIjIxESEgJnZ2eezNkN/A5Rl9LT02FmYYn+bp5yRyG6bLb2jnAbMAhHjhzhSVQ9QKfToaqqCjU1NVAqlQgMDDRu37VrF3Q6HaytrWFraws7O7t2950wYQIMBoOxpKpUKpibm8vxMjpVXFyMHTt24MiRIygvL4edozMCI4dhUEAYnF09YGFlLXfEXq2qrNj4f3cvH3gN6r1TdETRgNqqSuRmpiE3Mx3LvvsOGrUawcHBmDZtGkJCQuSOaNJYUqlL+fn5cHb15Bsu9QmCoIC3fzBS922DKIosqVdJS0sLtm/fjsrKSkiSBJVKBTc3N2NJNTc3x+zZs2FmZgZBEKBQKDocAnV0dJQj+gVptVqsWbMGu3btgtrMAt7+wZgw+2/o7+EFpZJ/TqkjhUIJR2cXxI2dipjRk9DU0IDTJzKRuj8eb739NiLCw3HDDTdg8ODBnArQCf5WUZcaGhpg5+AqdwyiK8bWzgFarRacin/5JElCbW0tiouLIQgCgoODAQAqlQo2NjYYOHAgXFxcYGNj0+4wvSAIsOpFcwyBs1MUMjIysHr1ahQUnkHkiImIGjkeVta2ckejXkShUMLa1g6h0cPhHxqJ40cP40D8Jrz66qu4/vrrMXPmTJM6YmAKWFLpvBQcHaA+RM2LT1y2lpYWbNmyBU1NTRAEAc7OzvDx8YEkScZD82PGjOkTo0Ln1ojesWMH9CIQO3YqJt98H8spXTaNmTnCYkbCPzQKDfW12LtlLeLjn4Kvry/uuusu2Nvbyx3RJLCBEBFRl9ra2mAwGIwjPObm5vDy8oKrqyucnZ2hVnc8qbIvFFRRFPHDDz9g+44diBw2DsMmzIAlpz7RFWZmbgEzcwvMWrAQWalJ2L15DV599VUsXLgQQUFBcseTHSdlERFRB62trcjIyMC6deuQlJTU7raoqCi4u7t3WlD7AlEUsXbtWuzdtw+Trr8F42beyIJKV5VSqUJozAjcdM8jEBVqvPfeeyguLr7wHfs4llQiIjJqa2tDUlISfvrpJxw6dAjW1tbX1BnIWq0Wn376Kdas+RVT5t6BiKFjeJId9Zj+7l645b4n4RMUgVdffRXZ2dlyR5IVD/cTEZGRIAgwGAwIDAyEn58fHBwc+sTh++4oLCzEt99+i+r6RkyZezt8g4fIHYmuQWbmFpgy53ZsXrkMn3zyKRYsuBVxcXHGdYSvJSypRETXsKqqKqSkpCA2Nhb29vZQqVQYPny43LF6nMFgwCeffAJRaYb5Cx+DhSUP75N8lEolZsy/G2kHdmPJks9RVFSEefPmXTMfGM9hSSUiugZJkoTCwkLs27cPKpUKOp1O7kiykSQJW7ZsgbaxGTctvJ8FlUyCIAiIHD4O9bXV2LBhA/z9/REeHi53rB7FkkpEdI1pbW1FUlIScnJy4Ofnh9jY2Gt6fcb09HT88ssvuP72RbB3dJY7Tt8jSRBFsdu7i6L0p/+LEA2Gbt9X0QcPiQ+fMAPlxYXYvXs3wsLCrqnRVJZUIqJrUEtLC2JiYhASEnJNznU7Jzc3F1999RV8AsLg6e0rd5w+pbaqAsUFJ3Eq5xgqS0sgdPNc7abGBigVZ9c03r5mOdQas+49oSChv4cXfPxD4DXIv8+syKAxM8fUubdj1bcf47fffsO0adOumd9ZllSSRcKubdixaW2Xt5uZWcDNwxNjJk2H9+CevW7zhlU/4cdvPsMbH32DgYMu/Edry7pV2LzuF4yZOB1zbr2zBxKe9f7rL6JBW48X3vgAAFBeWowXH78f2vq6dvsJAOwdndCvvytGjpuECdOvh0rVfumgg/visfzbzxEeFYe7H3yiR/K/+Pj9cPccgAee/GePPN+1TpIk6PV6qNVqaDQaTJo0Se5IsqutrcVHH32Efu7euO6We/vkKJxcivLzsHH519A1iOhvF4p+ZkOhUHSzctj9/u8iGcRW1OaVYHPK/2DbzxbTbrwTbl7eF/9AJsjWwQnDJszA8uXfwN3dHZGRkXJH6hEsqSSL48eOYM1Pyy6435L33sCDT72ABfc82GPLwFRVlCPraCpaWpq6tf/pkznYtXUj3DwHXOVk7aUkJaCmqtL4dVtrK3KyjqK2uqrL+6xf+SNGjJ2Ify/+HI79/jisWXKmELu2buzRT+cJu7bCPyi0x57vWmYwGHDgwAE0NzdjwoQJXFLpd1u3bkVdfT2u/9sNLKhXUG11Jdb97wuYS64Y7DEKCkXPradrZdYf/WyDkFu6Gb/+sAR/e+h5WNn0jSuE+YUMgfvAwVi1ahXCw8OvidFUllSS1W0L/4HRE6Z12F5RVoI9OzZj19YN+OitVzDYPwjDx0zskUwDB/li4vTrYWNzCR/lTcTTr7yFwf7Bxq8btHUoLTqDX374Ggm7tmHxa8/jtfe/lDEh9RSDwYCDBw8iOzsb0dHRcscxGdXV1di6dSvCY0fB0dlV7jh9hsGgx57Nq9HSoIOPR3SPFtRzlAoNBvQbiezitTgQ/xsmzJoPQej9H8xUKjVix0zBuv99jgMHDmDkyJFyR7rqWFJJVj6+ARg6alynt02ffRMe+ttcHNgbj9U/LsXQkeN6ZLRj3JSZGDdl5lV/nqspOCwKQ2KHddjez8UVzz10N/Zs34TC03nw8h4sQzrqKZIkISkpCVlZWYiKikJERMQ1ddLF+aSnp0OhUiN61CR+T66gqrIS5GSkYqDjGJip5RvBtNA4wcUuHJmpBxE7ejJsHZxky3IlDfIPgae3H9avX48hQ4bAyspK7khXFUsqmSylUoVJM2bjwN54ZKQdRn19HewdHNvtk52Rho2rV6CspAj2Do4IGRKN8VNnwdbOvsPjtTQ3I2HXNqQd2o/y0hK4uLojPDoOoydMhZm5hXG/vJwsZB1NxZiJ02Fr72Dc3tqqQ3LCbuyL34bK8lIM9g/CLXcv6jR7UsJuHE1NRkT0UMQMH93utkP79yD9cBIGBwRh3OT2ZVhbX4tD+/ciOXEvKspKYGFpCW/fAIydNB2D/S//Os5RcSPh5OyC6spyFJw62a2SWlleis1rVyIn6yh0LS3w8fXH9Nk3dzlft7amCvGbNyApcTcEAL6BIRg1YWq3Du3r29qwL34rGrR1sLWzx4ixk6Hqo5fe7Al1dXXIz8/H+PHj4ePjwzL2u/r6esTHx2PCrFtg10fKi6nIyUiBSrCAg/UguaOgn00ASmvTkJ+XjbCYvjHqqFAqMXn2Avz42VtYs2YNbrvttj79e82SSiattLgIAGBuYQmV6o8f11ZdC7748G0sX7oETY0NUCgUkCQJK//3DQ7s2YkX3vwQVtY2xv1Ligrx4mN/R9qhAxAlCQpBgChKEL7+BJFxI/DyO5/Aa+DZN9V9O7fi/ddfwPLNCcaSKkkSfl3+Hf77r/+DXq+HQqFA/Jb12Be/FYP9AzvkTty9DcuWgkeYugAAIABJREFUfIB7H3qqQ0lN3L0DSz9djOmz57crqZlHUvDKUw8iNycLAs6ukSf9vnTL6v99g4+WrYaP7+WdRNbU2IDmpgaYW1jC1cPzgvsfTU3GK08/iJM52VAoFBAEAaIoYtX/luKJl97A9OtvBP70BnlwXzxef/5xFJzOg0KhBARg09pfsPTTxXj7s+8uOGVjzfJleOvlp2Fra4+3P/sOShXfoi6Hvb09Zs2a1edHWy5WRkYGWvRAQOi1cfJJT6oqL4VGZQOFIP+HS5XSAmqlJZoaG+SOckU59OuPmNFTsG3bOtx0000wM+vm6ge9UO+fpEF9lq6lGYm7twMAIuOGtyudn3/wFr7+6B1YWVvj/a9XYP2+o/hm5RZMmjEbWzesxrv/fg6G39fWkyQJi197HocPJmD0xGn4cvlGrN93BB99uxKBoUNw+MA+LP108XmzpBxMwOJXn4edvSNeeOND/LorFR9+uxIVZSVY98v/Lvu1lhQV4sn7b0deThZmzrkZ36+Lx8bEY/hixUZMnH49is8UYMni/1zy44uiiNqaanz/5UdobGjAhGmzLjgyeyo3Bw/fOQ8lZwpx1wOP44f1u/HrrlQ8/crbaGlpxr+efhA7t6w37p+TeRRP3X87CvNPYubcW/Dz1v1Ys/Mw7nv0/9DW1oavP/4vmpsau3y+jLRDWPLe63Bw7If3v16BmOGj+/QIQU9hQW1PkiSkpKTAsb8bBJ5AdlXwt/bq8w2OgKBQGv/O9VUcpiBZHU1NhvmfDrUDgGgwoLioEEn7duH4sSNw8/DCHX9/2FhYigrz8dM3n0Gt1uDltz/FiHFnl9Jx8/BCUHgk2tpasXH1ctx4x70IDouEJIo4sGcnVCoVXnzzI+NZ7W4eA+Ds6oZbpo2AcIG31W+XvA9BIeDV977AsDETAACeA33w1iff4oHbZ6OluXsrAXRl24bVKC0qRHhUHF5480OYmZ1dWN3FzQOD/YOQk5mBlKSEbj/ef55/tF2pF0UDaqurUFpShAlTZ+GJF1+/4GN89u6rqKutwQ3zb8cjz/7L+P2/9e5FqCgrwdJPF+Pjt/+NUeOnQmNmhm8+eRcN2nqMn3odXnnnU+Nh+kVP/BOV5WVY+/P3OLR/L0ZP7HiiXG72Mfzfg3cBAN78eCnCo+O6/VqpI61WC5VKBQsLiwvvfI1pbW1FRkYGQmJHX3hnIhNl5+gEp/5uOHnyJEJD++4qKSypJKuNq5Zj86+/ADg72tfW1goAUKs16NffBXEjx+Dpl9+Gj2+A8T6b1/6C5qZGjJ4wFUNHj2/3eBqNGbwH+2PX1o3YuOonBIdFAgKgUqlgMBiwL34LYkeMhZ2DIywtreAfFIqtySdga99xDus5JUWFSE1KRHBYJGJHjGl325DYYYgbORZ7tm+6rO/D6InT0K+/K1zdPY0FFYDxU7KTc3+cys1BU2MjLLsxMlZTXYXGvxzi0rW0QKFQoL6+FqdOHIfVEBtozh0m+ktHLy7MR/L+vVCr1bhu3oIOI5p3PfA4Viz7AqdyjyPt0H4M9g9GUuIeqDUa3P3A4+3mkQqCgNv//hCcXdzg6t5+ioHBYEB2Rjqeuv82lJcV46W3Pkb0sFEXfH3UNVEUkZCQAFEUMXXq1GtimZqLkZmZicbGJgwODJM7CtElU6nUGBwUhuTkZJZUoqvlb/c/gvFTrwMA6HQ6bNuwGut+/gFKlQp3LnoMc269E5q/XG0kKWEXAKDg9Ek8du/NHR6zrPgMACDt0EEAgEKhxCPP/htvvPgEXn7yAQiCAL+gUAwbPR4jx05GeHRch+f4s8wjqWhqbIDGzLyTOZICRoyddNkl1cc3AD6+AWjQ1uNIShKOpR/GmYLTOJV7HCdzslFeWgxrG1u0NDd1q6T+d8kPHc7ur62uwrYNq/He6y9i4fzpePjZf+HuBx4/e6PU/v6F+adQW10FjwHe8AsM6fD4tnZ2GBIzDIm7t+PA3ngIgoD62hp4DBiIgZ1cfMHHNwCLnni+w/bM9BT8bfYEtLW2wsrKGkFhQy742uj8ioqKcObMGcycOZMFtRNbt26Fx8BBcPUYKHcUossSGj0CP3/xX8ydOxd2dr13ycTzYUklWbl7DUTokBjj11FxIxAePRSvPfsI3nnlGXgNHGQ8nH9OfV0tAKDkTD7KS4s6fVwLy/YnWl0//3Y49nPG0k8XIzsjHTmZGcjJPIqfvvkMXt6D8dxr7yJm+JhOH0s06CFJUqe3AUB/V/duv96uiAYD1q/6CV999DbKS4vRqmuFWq2GWq1GUPjZkzsud/K/vaMT5t1+L8pKi/H1x//F2uXfYd6CuztdCeEcpVLVxclLAlzcPAAAzU2NaGluhsGgh1qjafd9v5Cmpka4untikF8gEndvx9b1q+EbEHzhO1KXCgsLYWlpCRcXF7mjmJza2lqcPn0aEcMncPF+6vVs7R3R1NiIvLw8REVFyR3nqmBJJZMiCAJmzJ6PUyey8dVH7+Ctl5/CNyu3wsm5v3EftebsNZ0fePIFXHfjgi4f689lSalUYuzkGRgzaTryT+Zi+29rcGBvPI6lp+DkiWx8/Pa/8eXPv0Gt1nR8HLXmvCfw6HQtl/JS29m8biVee/ZhQBAwfMxETJx+PQb7B8PL2wd29o5YtGAWMo+kXvbzKBQKxI4Yg2+XvI8zBaegra/rtKQaz+Q3GCB2MTG/vv7shwU7e0eoNRooFEro29pg0Ou7nae/qzs+/HYlLK2ssPCm6Vj941LMmDO/xy+F21fo9XoUFRVhwIABvKpUJ8rLy9Hc0gJPHz+5oxBdEUq1BidOnOizJZXvYmSS7n3oKQz2D0LBqTx8+ObL7W7z+P1azOVlJejn7NLhHyQJjVotrKz/WEi6vrYGhfknAQDeg/2w8OFn8NkPa7Hs1+1w6tcfGWmHUFle1mkWF3cPqM8zHSA/70SXt+lauldgv/v8A+j1esy/YyEWf/kjbph/B0KHRMPO3hE6XQsatPXdepzu0NbXQRJFqNUaKBWdjybZOzjCwtIKtdWVqK6q6HC7JEnIzToGAPDyHgRHJ2eYmZujtroK2vq6Dvu36nR44bG/49fly/DnuQU+vv7wCwyBh5c3/v7IM6iprsRHb75inJtMF0cQBAQEBMDPjyWsM9XV1RANImzsHC68M8kmOLI/7vhHJG65PwIOTjz573zsHJ1QWVl54R17KZZUMknmFpb4euUWDPAZjHW//ID3X38RoigCAOb/7e9QKpX45fuvsC9+a7v7SZKED954GXPGR2HFss8BnD05Z0KkD26dPgrFhfnGfdUaDfwCQ+EXHAqVSt1hlYFzgsMiMXbyDKQmJeLA3p3tbjt14jh+/ObTDvfp73J2CkBC/NZ2UwUqykqQ8JfMAHD696IbGTsCKlX79QV/W7MCWUfTOv9GXQSDwYADe3fijX8+AVEUMWH69ejv6tbpvn5BoYgbORYNDVosW/JBh+kOX3/8X+SfyoWruyfGTpoO/+AwDIkZhrraGny2+PV2o6mSJOHLD9/Cpl9//n3Fgc5HpecuuBvzbrsHO7esx4uP3QeDofsjsnSWUqlEeHg4D/V34fTp07CwtuEC/ibOwlKNfq5W6NffEko1a8r5uLh5oaioCK2tffODPQ/3k8mys3dAeFQcCk7l4efvvsSkGTcgdEgMhsQMxfTZ87Fh1U/4z/OP4Y2PvoHnAG80NGjx25oV2LphFQYO8sPMObcAOHvoOmroSCQn7sE/H12IR5/7N7y8B6OhoR5b1q5EcsJuDB01vt3Vpf7q3oeexP7d2/Gvp/+Bh555GTHDR6OmqgIvPbGo01HO2JFjYWFphYL8k3jrpacQHhWHosJ8bF77M7T1HfcPi4rFof178c2n78LT2wd+gaEoPH0S2zauwbefvQfF7+vhNTU2GJfQOp83X3oS1udGkoWzh+RrqytxPOsoGrVauLp74r5Hnz3vvLwHn3wBhw/sw7aNa+AXFIoxE6fB3MICR1OTzy4BptFg0RP/hLXt2Qn79z32LNIPH8TG1cvh2M8ZC+5ehLa2NuzcvA4/LV2C0CExGD52UpfPBwATp9+AtT//gB2b1mL8xusw9fobL/haibqroqICdvaOF96RqJewc+yHghMZaGtrg0bTcbpab8eSSibtwadeRMKubaipqsSbLz6Jb1ZuhcbMDM+9thgajRk2rl6O+26ZCbVaA1EUoWtpxsBBvnjj42/g4NQPwNlDoM+9thjPP7IQR1OTcf+CWdBoNDAYzu7v1K8/br7zvvOeCe0fFIanXn4Trz//OF56YhHMLSwgigbY2jlg3oJ78MsPX7Xb3zcgGC+88QHeffV5rFj2BVYs+wIKhQJe3oMxb8Fd+OaTxe3mzD7+z9fwzIN3IjsjHXfNngQzcwu0trZAEBSYOfcWWFha4qelS1B8Jh+eA30u+H07fuxIh22CIMDG1g7jpszEw//3ry4va3qOX1Ao/vv5//Dq/z2M//77WXz89isQBAV0LS2wsrbG0y+/hVl/mhMcET0Ur3/4Nf7z/GP4/osPsfKHryBJElqam2FhaYVFTzwPa5vzX8s7dsQYzF1wF1Z8+zk+eOMlhEXGwt2LZ2F3l06nQ11dHZycnHhm/1/o9XpUV1fzUD/1Kda29qirq0NbW5vcUa4KllSSxfCxE2FlbYMhMcPOu5+bhxfe/HgpTuXmQKFQoK62Gs4ubrC0ssYLb3yAmfNuwYaVP6GstAgajTl8A4Jxw823w3NA+yLn4xuAL5ZvwK6tG5F++CBKi8/A2sYWQaERmDxzTrsiNGHaLAwc5Guc+wqcLXiz5i2Aq7sntq5fjZLiQri6e+LO+x+FwWDA4ICgdtemFwQBM+bcjOhho/DbmhVobdXBNyAYQ0eOg96gx7AxE+Hg+Mchx+DwKCxdtRUbVv2EY+mHIUkS/AJDMXbyDASFRaDg9El4eHnD3MLSeJ/Hnnu13dxNJ+f+eO29L7ucz6lUqeAbEAw3D68Otw0dPR7vfbUczi6u7bbHjRyLz39aj9U/fYu8nCy0turg6uaJW+5e1GFpKkEQMHbyDAQEh2HTupVIS0qEKIrwCwrFhGnXI3RIdLv9X33viw6lVaFQYNHjz2P46AkQRZEn/1yk0tJSbNmyBfPnz4f9edb+vRadu8SwgtdDoj5EEATo9XrjdLi+RpDOt7YOXdNeeeUVWPfzxKQbbpE7CtEVcSIjFWu+/wzLli2DWi3/tcWvtPz8fJbULrS1teG1116DytIBsxYslDtOn7X2h89RfKIYAe43QBAu/CEzeqRHh22ePnYYFOAIg0FC2oFiNGrbf/AuOFmLipKuL7F8jijpcazwZwydPBFDx07t/ovoRbLSkrD+p6/w8ccfw9Gx701l4UgqERER9ThBAMZM63r6klIpdFpid67P61ZJpd6PJZWIiIhkUVHasWxaWKphbauBJEmorW6Bvq39oezmpr45/5I6YkklIuojrKysMHjw4D55li/1PZIE/PhZx+X1okZ4YPRUb+jbRPy2IhuVZU1/uR9nKV4rWFKJiPqIfv36YeLEiXLHIOo2UexYOP9cQkWp833o2sBTZ4mIiIjI5LCkEhH1EaIoQqfT8XAoEfUJLKlERH1ESUkJVq1aBa1WK3cUIhPCD229Feek0vlxRIb6kL664PU5er0eDQ0Nff51Ut9WUqjFwV2FMBhENDVciTP5eQGH3oollbqk0WhQW10hdwyiK6a2ugJWVlYQBP7RIjJVxQX1KC6olzsGmQAe7qcuhYaGorQoHw31tXJHIbpsoigiN+sIAgMDeV17IqJegCWVuhQREYE2XQsqS4vljkJ02aorSlFaeAqRkZF9diRVrVbD1tYWCgXf2omo9+M7GXVpwIABCAsLw5nTuZybSr2aKBqwb+taODs7Izo6Wu44V42rqyvmzp0LGxsbuaMQEV02llTqkiAIuPHGG5GVuh/HUg9C38ZL0VHvI0kiDu3bgZKCPMyePRt2dnZyR7pqFAoFNBpNnx0pJqJrC0sqnZe3tzeun3Udtqz6DptXfYfmpga5IxF1m0Gvx/4dvyF512b8feFCjBkzRu5IRETUTTy7ny5o6tSpsLKywvfff48v33kRY6bOhm9wBMwtraBSqeWOR9SOaDCgQVuHU8eP4XDCDmhrq7Bo0SLExMTIHe2qKy0tRWJiIiZPnsxD/iQLM3MLGKQ2SBAhyDwOJkkiDGIbFALH43orllTqllGjRsHb2xubN2/Gnk2rkLB9A6xt7aAxM5c7GlE7Bn0bairL0aprQWhoKK5beDeCgoLkjtUjdDodKisrYTAY5I5C1yhPH19kHDoAvaEFGpW1rFl0bXVoMzTC2c1D1hx06VhSqds8PT1xzz33YMaMGThy5AiKiopQXV0td6xer6SkBKWlpbC2toavry/nE14mGxsb+E4aj4iICDg6OnK5KaIe5OHtC7WZGmV1R+DlNEK2HJIkorQ2DXYOTujv5iVbDro8LKl0URQKBdzd3eHu7i53lD7jl19+wZo1a+Dl5YUnn3ySpYqIei17R2dMnXcHNv/yPazNXOBgPQg9fcUnCRLOVB9Ek1iCWfMWwtKaU196K5ZUIqI+QhAEKJVKjsaTbARBQEBYNKrKSpC0azsadOVwtQuHSmkBXO2fS0lCm6ERRdXJ0LYWYNx18+DtF3x1n5OuKpZUIqI+wsPDA7feeivMzTlXnOQjCAJGTLoOao0GB+I3oawgHRYaJ6iVV/fnslXfhJbWalja2OCGO+5jQe0DWFKJiPoIpVIJS0tLuWMQQRAExI6ZAt+QISjIPY6y4nyIBrHb968oPQONmTnsHPp1+z5qtQauXgPh5eMPO8fu349MF0sqERERXRSDwYDWlmYYDHoAgCiKxv//mSAIGOgXiIF+gX/ct60N+rY2OLm4Qa0x63AfUTTg1++XwNrGDpPn3MbpK9cwllQioj6itrYWx48fR0REBA/501UlGvQ4kZmG9KS9qK2sgF7fhrZWXbfua2llg2k33QlXL+9Oby8uOIXTOZlQqdWIGT0Zjs4uVzA59SZc4ZaIqI+oq6tDeno6Wlpa5I5CfZxaY4bw2FGYf8+jmHTDrXDxGNCtEU8bOwfMvmMRfIPCu9hDwvEjh6DXt6GluQm5mWlXNjj1KiypREREdEnMLCwRNCQWN937KCbPuQ0WVl0v4O/U3w1z7/wHPH38utynVadDTkaq8evs9EMw6NuuaGbqPVhSiYiI6JJIkgTRYEBe1hGk7t8FXXNTp/upNRqMnT4HLh4Dzvt4hSdzoK2rMX5dVlyIttbWK5qZeg/OSSUiIqKLUl9bjay0JBxJ3oeaynLjdqVSBY2ZBq26P6ac+IUMwbR5fzvvKOs5pUUF7b6WJBHFBScxKDDsyoWnXoMllYioj3B2dsakSZNgZWUldxTqoxq0dTiatA9pB/e0G/FUqTXwD41EUEQsEravR+mZfABAYEQMps37GzRmFz6RT5IkFORmd9iemXaQJfUaxZJKRNRHWFpaYtCgQXLHoD7GYNCjKP8k0g/uwcnso9C1NBtvs7C0grd/CKJGjIf7gEE4czoX5SVnAADhsSMx8fpbodZouvU8FaVFKCvK77D9VE4mGurrYG1rd2VeEPUaLKlERETUgcFgQOHJHKQd2I0TmWmQxD8W43d0dkFYzCgEhke3Wzj/aHICRIMBbl7eGDFpVrcLKgCcOJaK1k6WsWppasSpnAyExYy8vBdEvQ5LKhFRH9HY2Iji4mIMHDgQmosoB0TnGPR6nD6RiZLCUziZnYGK0iIYDHooFAo4OLtiUEAo7J36ISA8BlbWtu3uW1FahJPHMxA9ciKGTZje4fbzaW5qRGbqQQCASqWG/vcz+s/9/+ihRASGx15U6aXejyWViKiPqKysRHx8PObPn8+SShdFW1eDE8fScCRpHypKz0CSJACAICjg6e2HuLFT4BMQAqWy89ogSRLSD+7BlNkL4B8WddHPn5eZjprKcjj1d4O3XzAOJ+wAAIy/7ibs3rQKRadzUVyQh4G+QZf+IqnXYUklIiK6RrU0NWLPll9x4lgqGrX1xu1KpQo+ASGIGDoGXj5+FzzxSd/WipCoYXDz8rmkHJlpB6FUqjBq8vWor60ybu/n4o5hE2Zg7+ZfkZWWzJJ6jWFJJSIiuobo9W0oLy7E0UMJyE4/1O5EKHMLS/iHRWPExJmwtXfs9mOqNWaXXFBrqspRVlSIKXNvQ0BYFJL3bmt3+9Cx06BQKJG8ZxtadS3dWimA+gaWVGpHkiTU19ejuroaNTU10Gq10Gq1qK2thV6vR1NTk/Ew0Dnm5uawtLSEra0tzMzMYGdnB0dHR7i4uMDa+sLr4hER0dUnSRLyso4gec9WlBSeNs77BAB7R2cERcYheEgcHJ1du3WJ0yvlZPZRjJ0xF6HRI4BOnlcQBMSMmgQAOHX8GALCo3ssG8mLJfUaI4oiWlpaUFdXh7y8PKSnp+PEiRNoavrjKiEGgwF6vR56vb5DIe0OQRCgVCqhVquhVCqN221tbTFo0CD4+/tj8ODBcHV1hbm5eY++GRL1Zfb29oiNjYW5OUea6KymxgacOp6B1P27UF1RhrbWFhgMBggKBeydnBEcORRRI8ZDrdZArTGTJWNE3GgoVerz/i1QKBSIHT0ZBoO+B5OR3FhSrwGSJKGsrAyZmZlITU3FmTNnUFFRAfFPy4n8lUKhgI2NDSwtLWFhYQELCwuo1WrY2tpCoWh/Nd36+nq0tLSgpaUFOp3OWIKbm5vb7dfQ0IDi4mLs27cPAODm5gYfHx8MHjwYgYGB8PLygkrFH0miS2VnZ4fIyEi5Y5AJaG5swIljqdi3fT0a6mqN25Wqs3NNQ6OGY3BQuEkcOlepu3eSnyAIUKnUVzkNmRI2gj5KkiRUVVUhMTER6enpKCgoQGNjY4f9lEol7O3t4e7ujv79+8PNzQ3u7u7GUU61Wm38d75PuaIoGkdf9Xo9tFotysrKUFlZibKyMlRUVCAnJweNjY0QRRGSJKGkpAQlJSVITEyERqOBl5cXxo0bh4iICDg6OnYow0RE1DVJklBTVY60A7uRczQF2roa49EwpUoFjwGDMWzCDHgN8m93lIvIVLGk9jE1NTVIS0vD4cOHcezYMeh07RdGtra2hqOjI7y8vBAQEIDg4GC4u7tf9vMqFApoNBrjsje2trbw8PBot48oiiguLkZ+fj5ycnKQnZ2NyspKNDc3o7W1FXl5ecjLy4NSqYS3tzeGDh2KESNGwNGx+5P3iYiuNaLBgMryEqQf3IP0pL0QDQbjbbYOTvALjkDE0DHo53L57/VEPYkltQ+or6/HgQMHsGfPHtTU1KCm5o/rKVtbWyMsLAwxMTHGE5ns7e1lWUNRoVDA09MTnp6eGDlyJJqbm1FbW4vm5mbU19fj0KFD2L9/P5qbm42FdevWrbC3t0dQUBDGjRsHNze3Hs9N1FuUlpZi//79mDRpEmxsbOSOQ1fZyd/nmmrralBTWY6236/WZGVti6AhcfAPjYSDs8tFLapPZEpYUnspg8GA06dPIz4+HgcPHmx31r2lpSX8/f0xcuRIREdHw8zMzCRPTjo31/WciIgILFiwABkZGTh8+DDS09NRVVWFyspK5ObmYtOmTYiOjsa8efPg4eFhkq+JSE46nQ4VFRUw/GkkjfqWluYm5OdmIe3AHhTkHYck/X5ugSDAoV9/DB07DYERMSYx15TocrGk9kKNjY1YvXo1tm/fjra2P5YQcXFxwfDhwzFixIheWeIEQYClpSXi4uIQFxeH6upqbNmyBbt370Z9fT30ej0OHjyII0eOYNSoUZgyZUqHKQVERH1RU6MWqYm7kJWehJrKcuOghEKhgIe3L0KihsEveAgsrLjsH/UdLKm9SHNzM3bu3Ilt27ahoqICkiRBoVDA0dERo0aNwqxZs9qNTPZ2jo6OuOWWWzBjxgwcOnQIu3btQn5+Ppqbm7Ft2zYkJCRg8uTJmDNnDi8BSUR9jiSKqKupQsbh/cg4nIj62mrjbUqVCn7BQxA7ejL6u3tByZVRqA/iT3UvoNVqcfDgQWzevBnFxcUAzi414+3tjeHDhyMuLq7ProsoCALs7OwwceJEjBs3DikpKdi2bRvy8/Oh1Wqxbt06HD16FFOnToWvry/nrNI1zdzcHO7u7lzKrZerKi9BVXkJstKSkJedAX1bK4Czc02d+rvBsb8rQqOGw22AT687YkZ0MfhOZsJEUURKSgpWrFiBoqIiAGeXjAoPD8dNN90Eb29veQP2MKVSidjYWMTGxqKsrAwbN27Enj17cPLkSXz22WewsbHBzJkzMWPGDP6RpmuSi4sLrrvuOrlj0CUQRQPyT2Qh7eBeFORlt7tUqcbMHBFxoxE9auJFXaqUqLfjX3ITVVZWhuXLlyMpKQmSJEGpVMLf3x833XQTAgMD5Y4nOxcXF9xzzz2YOnUqPv30U5w+fRparRbLly9Hfn4+7rjjDtjb28sdk4jovHQtzSg6nYv4jStRVV5i3K5UquDs5oHQ6BEIHhIHc0srGVMSyYMl1QTl5ubivffeMy4lpVarceedd2LUqFGce/kXHh4e+Oc//4ktW7Zg/fr1aGlpwf79+1FQUICbbroJcXFxckckIuqgvrYax48eRsahRFSWl0D60xUAXT0HYsTE6+DtF9TtqzER9UUsqSbEYDAgMTERy5YtQ1NTEwRBgKurK26//XZe6vA8LC0tMWfOHERHR+Onn37C0aNHUVRUhI8//hjTpk3DnDlz+tQJZURdqampQU5ODiIiIvrsPPXeTJL+n737jo+qygI4/puaSe+NVJIQIIEUIHQEAQHpILLYBUFsKHZdhFVRLOC6VhBxFRDpSpHeAhh6SEInBUIgvSeTNpmZt39ERrMBQRIySbjfz4eP8N6b986MycyZ+849V6KkqIC4g9EkHN5X65a+pZU1/sGhhHfrg5cNzLx7AAAgAElEQVRfIAqF+HgWBPFb0EScPHmSDRs2cO7cOQwGA23btmXEiBEEBwdjYyNaitwMX19fXnrpJeLi4lizZg1Xrlzh119/5dSpU0ydOhU/Pz9zhygIt1VJSQkJCQm0bdtWJKlNyKXks5yNP0pJUT5ZVy5RWVEOgIXGkjahEQS2D8fDyxd7JxczRyoITYtIUs1MkiR27drFTz/9RGVlJQBt27bl1VdfxcrKyszRNT8qlYquXbsSFBTEDz/8QGxsLKmpqXz44Yc8+uijdOvWDblcbu4wBUFo4Qx6PRlpF4g/vI/EU8cx6PWmfXYOzkT27Eu7sCjsHJzEDH1BuA6RpJrZkSNHWL58OZWVlcjlcrp27cqkSZNEglpPTk5OvPDCC2zcuJGNGzdSXFzMl19+SVpaGvfdd5+Y/S8Iwm2h01Vx6tgBTh8/SOblVNN2mUyOi7snIZHdCO3UHRs7MbFTEG5EfFKbUUxMDIsWLaKqqgqVSsW4ceMYMmQIKpXK3KG1CAqFgpEjRxIQEMCHH36IJEls3LgRnU7HAw88IBJVQRAaTFVlBWfiDnHiaAzZ6Wm19gWFhBMW1RufgGAsNKI+XhBulviUNoOysjKio6NZvXo1Op0Od3d3Jk+eTGhoqLlDa3HkcjlhYWFMmTKFdevWkZuby5YtWwgICKBz586ibk9oUVxdXRk0aBDW1qJdUWPQlhRTVJDL2fgjnIk7bJoIZaGxxMbWHic3Dzp26UVg+zBxS18QboFIUhtZQUEBX3zxBefPnwdqPlTeeOMN3N3dzRxZy3b33XfTqVMnvvrqK06dOsXXX39Nly5dmDp1qiitEFoMKyurO26Rj8YmSRKZly9yKvYgSafjKSstNu1TKlUEhYTTY8AwXNxbicRUEOpJJKmNyGg0sn37dlOC2qpVK6ZPny4S1EZib2/P9OnTWbBgAbGxsRw7dozq6mqmT58u+s8KgvCXrk6E+m37ejIvp6LXVwMgk8uxsbWnQ5eedOjUA3snl9s2OdNg0JNx6QLJZ09w5WIixYX5SJJ0W65lTvrqatPff178FXKFwozR3B5yuRx7JxdatwklsH1H3Fr5IJe3vOdZXyJJbUR79+5l06ZNAPj7+zNt2jSx1nwjs7Ky4umnn2bmzJlkZGQQHx/P4sWLefzxx0UtsNDsVVZWkpeXh4eHh6i5biAGg55zCUc5cTSG9NQUjEaDaZ+jiztdeg8gJLLbba81LS7MZ8cvy0hNOoOLox1Bfq3wDfdDLf4/N0ul5RWkZ+Vx7lg0h/ZsJqrPPXTvPwy1hYW5Q2tSxE93I5Akibi4OH788UeMRqNpBNXNzc3cod2RLC0tefbZZ/noo48oKSlh7969uLq6Mnr0aHOHJgj1kp2dzbZt2xg/frxYFriBnDjyGzvXL0eSJGQyGUqlila+AYR364N/cCiWjbBcaXFBHptW/he9Np+nHriXsLatsbYS9fTNnSRJlJZVEHsqiV92xJCdeZmh9z+Ota29uUNrMkSSeptJksSaNWtYt24dkiQRHBzMjBkzxKidmbVu3Zq5c+fyySefkJiYyOrVq3F2dqZPnz7mDk0QhCYkJ/OKKUEdOn4ioZ26N+r1U86dZMOyb7i7awceEV+kWxSZTIadjRV3dw+nW3hb5v+0mUXzZnL/pOm08gswd3hNguhqfpvFxMSwYcMGJEnCwsKCUaNGiQS1ibC1teXZZ5/F2dkZSZJYsmQJ586dM3dYgiA0UX6B7Rr1euXaUnZtWEGgjzvjh4ov0C2ZlaWGJyfci5ujDXs2raZaV2XukJoEkaTeRnq9no0bN2IwGFAqlTz66KNERkaaOyzhT1xdXXnppZdwdHSkrKyM7777jqKiInOHJQiCQNyhaCpLi3hszAAs1GJwo6Wztbbk4VH9ybqSyrkTseYOp0kQSepttG/fPjIyMpDJZAwePJi+ffuaOyThGlq3bs3IkSMBSE9P56effmqRM2aFls/W1pbQ0FAsxOSLZk+SJE4fP0RUWDCt3J3NHY7QSNq29iYk0IdTsQfMHUqTIJLU2yQ2NpYVK1ZgMBjo1asXY8aMEWvGN2FRUVGmxRQOHjzIunXrMBgMN3iUIDQtTk5O9OrVC0tLsapRc5ednkZFaTHdI9ohF/1W7xgKhZxuEW3JunLJ3KE0CSJrug0uXbrEwoUL0Wq1BAQEMHnyZNEwvolzcnLiqaeews3NDYPBwM8//8zly5fNHZYgCHeorCupqFUKvD1dzB2K0MgCfDywUIl57SCS1AZXXV3NqlWrKC0tBeCBBx4QjeKbCWdnZx566CFUKhUGg4G9e/eaOyRB+Fv0ej1arRaj0WjuUIR6qiwvQyGXiz6odyCVSolCIdIzEElqg4uPjyc+Ph6ANm3aEBgYaOaIhL8jKiqKAQMGALBr1y7S0tLMHJEg3LyrNdUlJSXmDkUQBKHeRJLawNauXYskSSgUCsaOHYtGIxouNzcjR47E3t4evV7PmjVrRG2qIAiCIJiBSFIbUHx8POnp6QBMmDCBsLAwM0ck3AoHBwemTZuGhYUFsbGxpi8egiAIgiA0HlHs0kCqqqpYv349BoMBHx8f+vXrh6yFzsi8dOkSJ06cIDMzs8X2FL26wowkSWzbto0LFy7ctu4MWVlZAFy+fJl58+a12J+bxmJjY4Ovry9hYWF4eXmhUCjMHZIgCIJwC0SS2kBOnTpFUlISCoWCkSNHYm19+9dzbkySJJGRkcH69euJjT2OUQJHF1dUqpbbj9HV09f098LSytt2HZWlHT6t7QAoKKm4bde5U2QXFHP0WCy//PILAQEBjBkzhvbt298Ryb9Go8HT0xOlmGwjCEILIN7JGsj27dsxGo0EBATQtWtXc4fToIxGI3v37mXp0qWoNVb0G34/wR0isdCItlpC01RWWkzKuZMkHN7PBx9+yBOTJtG3b98Wn6i6u7szYsQIc4chCILQIESS2gBiY2M5ffo0AP/4xz9QqVrO8nWSJLF48WIOHTnKoPsexScgGGsbO3OHJQh/ydrWnrCo3rQLjyIvK51f1y8nMTGRyZMni0U1BEEQmgnxbl1Per2ezZs3YzQaUSgUBAUFmTukBpWSksKRI0cZOn4i7cK6iARVaFbUagta+QYQ0bM/v/0Ww+7du80d0m0lSRJ6vV5M9BMEoUUQSWo9XbhwgZSUFADatWvXohr3S5LEmjVr6BDVB9/AduYORxBuWUhkN4JCI9i0aVOLnewHNZPw1q9fj1arNXcogiAI9SZu99fTsWPH0Ol0yOVyBgwY0KJmEqempnL27DnGdO3f4mv5hJZNLpfTa8Bwfvh8NrGxsaYFG1qakpISysvLRUmD0GBKSrUM+sfTXLycUWu7k4M91lYaHOxs6di+DRMnjCIspI2ZohRaKvFOVg+VlZUcP34cADc3N0JCQswcUcM6fvw4GitrXD28zB2KINSbk6s7PgHBHDt2rMUuG5qXl4eVlVWLuqMjmJdRkigsLqW4uJTIDu1Mf/y8PbGy1JB6OYOvvl/JgPuf5L/L11FVpTN3yEILIkZS6yExMZGMjJpvl507d8bOrmXVa54/fx5PH3+sbe3NHYog1JtMLiewbUdidtT0M26Jo41paWkEBAS0qMmbQtNgaalh6/Kvam2rrtZToi3jUOwJnn7tfaa++h5p6Vn86+WnEDffhIbQ8t6lG9HFixdNf4+KijJjJLeHTqfDxs7R3GEIQoOxtXeksvL29bw1N2tra/z8/MwdhnCHUKmUODvaM2xgH7799ywAvli0nFPnkswcmdBSiJHUeqiurgbA1taWwMBAM0cjCMKNtPTa6iFDhohG/oJZRIWHEujnTXLqZdZu2kXH9rXrU/MLi/hmyVp27T9MRWUV7YJaM3Rgb8YNH1jruANHE9gdc4RhA/oQHOjHinVb+XXHPqytrOjYPojJD43F2bHu3b0jcadYuvpXziZdpKKyivDQYB4aO5ReXSOuGW/0gWN8/cMqLqdn4eRgT1REKFMeHouXp1vDvShCvbW4d7Pq6mry8/PRarVoNBrs7e2xsbG5LR9OZ8+eBSA8PLxFTZgSBKF5ErWogrk4OdpzT78eJP9wmeiYY+heqEb9e9nJjn2HmPrKbK5k5uBgb4tapSL+1Hl+XLuJ4pJSHp8wCsXv5Tf7Dx/n7bkLkIwSm17fT9ypc9hYW1GqLWfFuq38d/k6dqz6Bl8vD9O1N27fx4Spr2GUJOxtbVCplBw/eZbvV6znqUfv56O3XkCtromlpFTLi7Pm8dPPW1AqFTjY2ZJ0MY3tew+yJ+Yoy76eg3cr98Z/AYVrajG3+7VaLbNmzeLuu+8mIiKCNm3a0KFDB3r27MmwYcPYvHlzg16vpKSE9PR0ACIjIxv03IIgCH9HYmIi5eXl5g5DuMP16BKGTCbjbNJFystrymp+3bGPh55+k5LSMpZ88R6HNy/l+I7l7FqzkFefeYwX3prL19+v5P9b+86bv4ROYe3ZuvxrYjYuZvvK+YwbcQ8pqVfYsfeQ6ThtWTmzPv6KrpEd2LV6IUe2LiNu50q2r1xAax8vFixezdpNuwCorNLx0LMzWLxqI2OHDSD650Uc37Gcg78uYc2ieeTkFTD2iZfJzi1otNdM+GstYiS1tLSUSZMmsXbtWmQyGe7u7vTv3x+DwUB8fDynTp1i//79fPXVVzzyyCMNMqqak5NDRUUFSqVS1IAJgmA2ycnJ7N+/n549e9K+fXtzhyPcwZwc7JHJam7tV+v1GI0SnyxYSkFRCR/MeJ4JowebjnVzcSIqIoSPvvyej79azMQJo7GxtjTtt7O1Zt6/XsLKUgNAuyB/HO3tWLNxB6fPp5iOK6+o5PT5Czzx4GhTkgzQp1sk7735LEkXLhMcWPMZHR1zlO3RB2nt68V/Zr+Kq3PNnAtXFyfaBPhy+PhJPv7qB1au38bzkx+47a+XcGMtYiT1iy++YO3atQDcf//9nD9/nl27dhEdHU1sbCwdOnRAq9Xy/PPPc+HChQa5Znp6OtXV1Tg4OGBjY9Mg5xQEQfg78vLyOHjwIH5+fgQHB5s7HEEA/qj9Tr2czulzKVhbWTJ8YJ86xykUCpwd7cnMziXu1Nla+7pFdjQlqFcF+nsDUFRSatpmZamhTWtfFq/ayKvvfsqBownkF9Ys2DF26ABef+5xOofVfHnbsjsGg8HA3b2jTAnqn7m61GzbvHP/rT51oYE1+5FUrVbLxx9/jCRJeHt7M2/ePGxtbU37AwMDefnll5k8eTLFxcUsX76ct956q97XvXz5MgDOzs4iSRUEodGVlJSwa9culEolPXv2FHXxgtnl5BVgNEo4OzqgUirJysmnVFuGJEmMmfQSqv+b1CdRc7seICMrt9Y+N1enOue/2jYuL78QSZKQyWRYW1vx9OPjmT7zY/6zcBlf/7AKDzcXRg7qy0P3DaVLeIgpaU66mAbAhq3RHDyaUOf8xaU1K7WlZ+fW2SeYR7MfSY2Pj8fS0hK1Ws2kSZPw9vauc8yoUaOwsrICYMuWLfW+ptFoNCWpvr6+LbLfoiAITVt2djYajYZBgwaZ3t8EwVwkSeLI8ZNAzYinRqOmorKKar0euUKOWqVCqVTW+qNSKgkO9Kdj+zbY/Y3BHulPBawyYOKEkXz23mt07xxG20B/jAYDC5aspuewRxn+yPOm8oBSbRkAanXdWJRKJc6ODnRs34Y2/j4N98II9dIkRlKTkpJ47733qKqqIjAwkNmzZ9dJ/CRJYv78+ezbtw+1Ws3bb79NQEAA3bt35+jRo2RlZeHjc+0frD//QDdEQllZWUl+fj7Ada8pCIJwO7Vp04agoKAW31ZLaB4ys/PYsG0vAA+PG4bGwgIHOxssNRbY29qwadmX+NymWfPWVpY8O/EfPDvxHwBk5+azcfs+vvhuOdv2HODk2SRSj27Gw9UFgIkTRvHOq0/flliEhtUkhgADAwOxtbVl5cqVfPjhh6xcubLOMTt37uT1119n1apV2Nvb4+vrC4BSqcTb25suXbrg7n7tX4ANGzaYZr727t273vEaDAYqKyuRyWQtbpUpQRCaJqPRSHJyMqWlf9TjiQRVaCpiT5wlIzsXn1bujB3aH4BWnm442NuRk19I7IkzdR4jSRIjHnmeKS+/y8W0jFu6bsLpRCa+MIuYI/Gmbe6uzkx+aAzfzJ0JQHZOPjl5BYS2q+lnvvdALBWVVXXOtXT1r4x74hUWLFl9S7EIDa9JjKTK5XJmzZrFN998g16v56233qJ3796mUcqcnByee+45tFot3bt3Z/bs2TfdsDopKYm5c+diMBjw9PRkypQp9Y5Xr9dTVVWFXC7H2tq63ue7E8UdOcDBfbuvu99Co8HTy4dO3Xrh0apuCcfttGvLBtavWsrr78zFy9f/hsfv37WVfbu2EtXzLgYNH3v7A/zdD/P/Q3lZKc+8UvNGnJ+bwyfvvoFWW1rrOBng4OSCi5s7XXv1pUuPu+rUL8YfO8Smn1fQLjSM+x6a1Cjxf/r+W7h7tOLBJ55plOs1Z5IkceLECWJjY4mMjKRTp07mDkkQMEoSaVcy2bHvELM++hqAV555DA+3mhFLTzcXBvTuyo9rN/HuJ9/Qu2skLk4OpsdfTMtgy+4YfLw8mPevl24pBl11NT/9spVqvYEeXcJq3S0t+31wyt7OFhcnR+4bNoBP5i/hYGwCy9ZuYvJDf7xfG40SO/YdYt3WPfTt2fmWYhEaXpNIUgFcXFyYPHky3377LRcuXGDmzJksXLgQo9HIK6+8QmJiIgqFgvnz5+Pg4PCX5zpw4AC7du0iPj6ePXv2UFRUhJeXFz/88AOtW7eud6x6vZ7y8nKUSqVIUm9R3NGDfPv5R6gtLK65zrjBYEBXVYWzqzsv/PNdhoy8v9Emhly5dJH9u7by7Kszb+r4MyfjWPPjd6jU6kZNUndv20Bhfp4pSa2sKOfg/t0UFeSjsbRCofjjzdpolDAY9Cxb9BWDR97Hy7M+xM7+j9+j1JRE1vz4HQPuHdloSerGNcsIbt9BJKk3UF5eztGjR0lLS6NTp0507NjR3CEJd6CysnICuw03/dvezoay8koKioopLtGisVDz/OQHefLh+2o97u1Xn+LQ8ZOcOpfC8299xPNPPICdrQ2xCWf44Iv/Ym1lyT+ffwJ7u1ubgBzWvg39e0ex5tcdONjb8uj9w7G1sSb+1Hlmffw1apWKqY+OQ6lU0KFdEC899SifzF/M67M/w97WlpC2ARSXaPlh5QZ+3rSLrpEdeOi+ofV6rYSG02SSVLlczpdffgnAggULWLx4MVFRUcTExLB8+XLkcjnvvPMO4eHhNzzX559/XqtkQKlUEhwcTEVFBQaDoUGWDbxa5yput9XPa+/M5b4HJ9bZXpCXQ0z0Tv4z5y1mTn+S0uJiJjw+tVFi6jNgMC5u7nh6+TbK9W6H+T+uJyKqu+nfZdpScrMz+eyDWWxYvYyUxLN8//MOVKrfVyiSrnMiwawkSSI+Ph6DwcC4ceOwtLS88YMEoQEpFQoiO7TFzaVuyyYbays6h7Wnd7dIhg/sg6+3Z51j/H1acWLPKpau/pUvFi2n/31Poquuxs7WhgBfLxJ2r8Lfp5XpeC8PN3p0CSPAr+4dNLlcTo8uYbRr88dgk4WFmh+/ep8ZH3zFph37+GHFeiqrdDg62NE5LITvP3uH3l1rFtyRyWS88+pTPDBmCHM+W8Qzb8yhqKQUhUKOh6sL/3xhMm9MmygmQzchTSZJhZqeaR9++CG7d+8mMTGRefPmceXKFQAGDhzI9OnTbyopfPjhh3nhhRdQq9XExcXxr3/9iz179hATE8Obb77J22+/fZufiVBfTi5ujBj3IId/28Omn1ewY9MvjH3wcdRqi9t+7YA27Qho0+62X6cxWdvYYm1jy8SnXyImegfJ58+SdPYUIWHitnFTU11djV6vx9LSEplMRteuXZHL5eKDUzALG2srVnzzUb3OoVapeOLBMYwbfg/pWTlUV+uxsbbEw80Fa6vaX7weHjeMh8cNu+Z5NBZq9q//vs52Z0cHFnw8g+zcfAqKStDpqrGztca7lXudtldQszDA4s9nk3o5gxJtGQq5Ahcne9xdncXAUxPTpJJUAHt7e/773/8ydOhQUlNTAfD29mbRokU3fWt9+PA/bkl07tyZLl26cO+995KVlcUHH3zA4MGD6dGjx+0IX2hg3Xr3Y9PPK7hy6SJl2lLUTrWT1PTLqezctI7szHQcHJ0JDe9Ep+69sbSs25KnulrH6fhYEo4fIScrA3cPLzp2iqJDRJdaJQcZV9JIu5hMWKeuWFn/cQtKr9eTeOYkB/ftIi8ni4Dg9gwb849rxn064TjJ508T3L4D7TvWXjb3zIk4ks6dwtuvNZ271Z7IV1lRTuLZUxw7uJ/c7EwsrazxDwymW6++uDdAba6Xnz/OLm5kZ6aTm5N1U48pLSkmZs92Es+eoqqygtZBbel7z1Bc3euOmgCUl2mJPfQbRw7sRYaMoHYhRPXsi6fXjTthGAwGTsfHUl6uxdLKmtDwTiiVdctBWqKKigqSkpI4ffo0np6e9OvXD6BB7vwIQlNgb2dzy7f1b4a7qzPurs43daxMJqO1r9dti0VoGE3y3a9Hjx6MGTOGxYsXAzBhwoR6tXqKiIjgrbfe4rnnnkOn0/Htt9+KJLWZyMqoGUnXWFrVSlb01dWsWvot330xj4L8Pxovy2Ryxj08iZdmzkGj+eMben5uDh/MfIm9Ozajr642bVcoFNw9eASvvv0Rbh41t5x2/PoL/5nzFiu2xtA2JOz3IyW2bVzL7NenUVVZUXMtuZzobRtp5VN3Wdwdm35m8YLPeOK5V+okqTs3r+P7r//NvaPH10pSLyafZ86MFzl+OAaj0VjrMYFtQ/j8+9W08q5fCUJFeRll2lIsNJY3NSEtJfEss1+fRkLs4Vrbl377Ja+98zG9+t1Ta/up+GN8NOsVTsXH1tru7OrG3AU/Ehn11793W9ev5t3XnkNtYcHb875GLm/5DeqNRiNxcXGcP3+eyspKfH19CQ0NNXdYgiAIZtckk9Rff/2VLVu2IJPJkCSJRYsW0bdv31ojpH/XyJEjeeONN9BqtZw/f75e8SmVSqysrKiqqkKr1dbrXEJdkiSh01WRcfkSm39ZhUwmo+/Ae7GxqVlJzGg08u/3ZrByyUJCOkby/Jvv0KZdKDlZGSTEHmbNj/+loryMGXM+Q2NpiSRJfDjzZXZtXs/4R6fQuXtvvH39SU1JYteWDezcvA5nN3feeHfedWOK3r6Zj2a+QvuOEQwePpaQ8E7kZmex4odvWL9yab2fc9K5U7z85EPkZWfxyJRpdO7RB1c3Dy6nXiD+2CFWLf2Wf8/+J/O++fGmzldcVEBeTrbp33p9NZdTU1i1ZBHlZWU8PPlZ2rTr8JfnOHZwP7NemopcLufZV2fRvkM4tvaOnIo/xpJvPuPN5yby4oz3GT3hUWQyGYf272bmi1PRlhTz1Iv/pFO3XqjUauKPHmTZd1/z5cfvMHfBUpycXetcy2g0sOPXX/hq7mzah0Xy7Csziep51997EZsJo9GIXq9HrVabtikUCkJCQvD398fe3l7cchQEQaAJJqlJSUlMmjSJ/Px8hg4dyqlTp0hLS2PKlCls3769zszWM2fOsH79etLT05k7d+51JxZYW1vj4ODQIEmlSqXC2tqa8vJyiouL632+O9nqpYuI2bOj1jaj0UB+bg4piWepKC9jwL2jmPriP5H9XpOXcOwQKxd/g4OTMx/PX2K6jRwS1ol+g4ZTptWyeuki7h40nP73jsRoNBK9/VeUKhXPvToL299ntYeEdaLX3fdQXFSAf0Cb68ZoMBj4et57aCwtmfPZIjz/NJoZ2bUHE4b0Ijc7s16vw4HonRQXFXLXwHuZPuM90/Z2HcK5a+C9xB87xPEjMTd9vulPXLsMQalU8fjT03n65Rl/WeNYXa3jP3PeIjP9Mk+9+E8mT3vVtC+sUxSu7h689vSjfPr+DPoNHo69gwPz/z2HvJwsnnrpn0yd/qbp+Igu3VEqlXz6/lsc3LuLYWMn1LqWJEmsX/kjH816hZCwSOYtXHbNRLY5kySJ0tJSsrKySE5ORqVSMXDgQGQyGXK5nIiICHOHKAiC0OQ0qSS1rKyM5557jvz8fGQyGVOnTqWwsJApU6aQlZXFCy+8wPbt22vVaO3atYt//vOfADz00EPXvY2fkZFBZmZNIlHfJQSVSiUajQaoWT9buHWJZ06SdPYUUPNB/ueuCY7OLjz14puMe/iJWrWhe7b9itFopGffgde8ZW3vULPm887N6+h/70hkMhmu7p5kpl/mw1mv8PRLM/D09kWhUGDv4MQ3P21E8Rd1fxeTz3P5Ugrhnbvh8X91lU7OrnTv05+Na5bV63UY/+gU7h09HgtN7S9ZkiShUCqwtrFBr9dTVFiAg2PdNa3/n0qtRi6rnYQajUYMBj1rlv0XtYUFTzz7ynWfd9rFFFJTkrCytqb/vSPr7L970HBcXN3Jy83m2IF9hIZ3IjX5PNY2tgweMa7O8SPHP0JUz754+9VuASdJElvWreKjWa9godHw8qwPWlyCWllZye7du8nNzUWv1+Pi4kLHjh3FaKkgCMINNJkkVZIkPv30U3bsqBlVe/jhhxk6tKZX2datW1mxYgV79uwhOTmZdu3+mHnt7f1HkvLJJ5+wYsWKOhMNJEnis88+w2AwIJPJGDRoUL1iValUppWmsrOzb3C08FcmTHyK3nfX/P+orKhg64bV7N2xGUsra157ey4Dho6sM3Em6dxpoKZe8p1X6/bYvHQhGYC01AtATduSh554lv/MeYvNv6xk6/rVRHTpTt97htKt9920btMWxV/8KqRdTKGqshKZTH7NxKJ9x4h6J6mWVtZYWllTWVFOakoSiWdOcCUtlZTEs5yKj+XypQvY2NhiNBhu6nwLl2+q1fdsfVUAACAASURBVIIKIDc7kw2rfmTh5x/xzacfYGvnwAMTn7rm4/NysinTltLKxw9XN486+5UqJUHtQsjLzeb8mRM4OrtQXqbF09sXR2eXOsfb2TvU6st61ZmT8STEHqaqqhILjQY7+7ptbpoLSZIoLy8nKyuLyspKQkJCkMlkaDQanJyc8Pf3x8vLC3t7e3OHKgiC0Cw0mSRVq9UyZ84cJEnCz8+Pd99915Rszp07l507d5KXl8fzzz/P+vXrTbf1Bw0ahL+/P6mpqWzcuJGFCxfyzDO1E5cFCxawZMkSoCapfeyxx+oVq0KhoFWrVpw+fZq0tDQkSRKjIrcoMLg9PfsONP377kHDWLF4If95fwZvvTgFVw8POnXtVesxRYX5AJw9Gc/Zk/Fcz58nSI1/bArWNrYs/uYzUlMSOX7kAMePHEBjaUVIWCQvz/qAkP+b4HSVQV9tGuG9FnePVtfdd7MkSWLfzi18+/nHpKYkUvanVaOC2oXi6ORMtU5Xr2u4unvy+DMvkZ+Xw/LvF/DzT98zYtyD2Nhef2lfpVKF8hqLLYDM1G2gTFtKRXlZTZ3ldRZnuB5tSTEOjk60DQnjxPEjbNu4tlZpQXOg1+s5fvw4mZmZlJaWUlFRgaenJ23atDHVnXbv3v0GZxEEQRD+X5NIUisqKhgzZgwVFRW4uLiwZ88e/P39Tfu9vb3Zvn07gwcPZufOnYwePZqff/4Za2trrK2tOXz4MI8++ijbt2/n+eef59133yUgIACAc+fOUVxcjCRJDBo0iGXLluHiUnek5+8KCQlh586dZGRkUFRUhKNj8x0BakpkcjkTHp+Km7snrz37GNMeG8f8ZesJ69TVdIydXc2I3PNvvPOXqxX9+Xa3SqVm9IRHGT7uAQwGA5dTL3AgeifbNq4h7sgBXnv6UVZvP4SlVd02Z1Y2tn+52lVJSf3rkj+bM5Ol336Jk7MLL771PgPvHYXm97IUhVzBM4+M5typhHpfR6FQ0G/QMFYvXURK4lmKCguumaSq1GoUCgW6qkqqqiqvcYxEdkY6AJ5ePljb2KJUqagoL6eqqrJWeQbUJOHFhQVoLK3Q/KluvG1oGN+u3IwMeGHSeL77ch5BbUPoN+jafRIbglwux2AwmJLpmol6OoxG4+8lEQYkSao14rlr1y60Wi3FxcVUV1fj4ODAfffVrKyjUChwdXXFx8cHZ2dnFAqFqdZUEARBuHVmfxe9eis+OjoapVLJe++9d82lSyMjI3njjTcA2L59O/Pnzzftc3NzY9WqVbz++us4OjqSnZ3NwYMHOXjwIIWFhTg6OvLmm2+ycuXKBklQATw9PVGr1ZSWlorJUw1MJpMxYOgohowcR3mZlo//9Splf5rw5hsQBMC5UwmoVGosLDS1/hw/HMNP331tapskSRLrVi5h3YolyOUKLCw0BLUN4dGpz/PDLztp3yGC9LRUsrMyrhmPj19r1BYaqqt117zdfjoh9hqPqlFSVHhTz3nzLysxGg1MeeF17ntwIvaOTqbnU1lZQX5uzk2d52ZcTD6PwWDA3sHJVFv9/1zdPbCxtaMgL5crly7W2a+rqjKNYrcOaouLmwcaSysK8nJqdRW4qqggnweH9eGbT+fU2u7g6IStnT02dva8+Nb7IEl89sEsCvIa7vn+mUZjQXh4OJs3b2bz5s1s27aNrKwsNmzYwNq1a1m9ejUrVqxg9+7dXLp0iZycHFNzfVtbWzp27Ejfvn3p3fuP1mEymYzWrVub3hMUCoVIUIV601hZYzAaqdbrzR2K0Miqqw0YDMYbH3gHMPs7qUwmY8qUKaSlpZGWlsYTTzxx3WOnTZvGlStXSE9PZ+LE2ktp2tnZMWfOHA4dOsSiRYuYMWMGs2bNYsmSJRw7dozZs2fj4FC3Ju5Wubq6otFoMBqNplWxhIb1wpvv4unty5kTcaxc/I3plvvdg4Yhl8vZt2sLJ+OO1nncrs3r+fzDfxF76DegZsLQu689x38+mElBXm6tY1UqNa7uniiVquve9vbxC6BT156cPRnP6RPHa+3LyrjCnm2/1nmMo1PN5J8Tx4/U2q4tKb7mLP3C30sY3K7RIP/wb9GkXUy+Zmx/hyRJnDuVwJKFXyBJEhFR3XFwunbjay8ff9qFhqPTVbF1/Zo6+9ev+pHiogIcnV3oEBmFt19rgtqGUFlRwfqVS5D+r8/rxjXLyMq4Qus2ba8bX2h4Z+4ZPpbUlEQWfPoBktTwb9JGo0RZWRkODg5YWVkhk8mwtLQkMDCQ8PBwXF1dUavVFBUVsWvXLjZt2kR+fj733HMP/fv3x9PTk4KCAoqLi8nNzaWqquovS0EE4VY5urih0xsoKBZtDu80OflFVNazvKulaBK3+52db26FCJVKRatW16//k8lkBAYGEhgY2FChXZelpSUuLi4UFxdz/vz5WiMrQsNw82hFx4jOZF5J4/v5/6ZP/8G0ad+B7ncNoM+AIezdsZlZLz3Fv7/9CVc3Dyoqyone/iubflmJp5cPYx98HAC5TEZoeGdOxR/j7VeeZvo/Z+Pm0YqKigr27dxMTPQOIrv2xNHx2j+HMrmcJ6e/wVMPjGDWi1N59e2PCQmLpLiokH+9/FStxQSuiup5FxYWGi4knePbzz8mvHM30i+n8svyxVxJqzsyGRrWiYTYw/z43VcEtg3By9ef3OxMYqJ38NmcmUiShNFopLKi/KZeu/n/fh/HqwmorKbjQUFeLscO7qeoMB8nFzeefmnGdVdzksvlPPPqTE7EHWXr+tWEd+lGl+69UastSE48y7eff4xSqWTK86/h6OgEMhlPvvAGL07+B8u/X4CLmwej/vEIBr2eQ/v38N+vPiG4fUf63XP92/gymYwB945k6/pVrFuxhN79B3HXgHtv6vneLJ1OR2JiIjNmzKhVO9upU83ysB07dkSv16PT6aisrKSyshJnZ2fTyGhJSQmpqalUVFRQVVWFRqOhT58+te7+iBp1oSG4efqATMGpxEsE+l57dTeh5ZGAE+cvYmV9/bkCd5ImkaQ2V2FhYaSkpHDw4EEefPDB6/ZoFeqSyWQ1f25w3OvvfkJqShJJ507z8tSH+H7tDpxd3Xj/s0VE79jE/Hnv8fjYe7C1tUOn01FUkEf/e0fx0lvvm5btlMnl/Pvbn1jyzees+fE7Jo4bXOv4bn3688rMD0ztmGSympWr+FN0HSK68J/vVzFz+pO8OHkCTi6uVFfraBsSxpTnX+Pbzz7iz8+mXWgYi1bXTISa/0lN31NLKyuGjpnAmAceY+F/PqzVSmrO59/xxUdvs2PTLzw6uj9Ozi6UFBfh6OTCzI++IDUliQWfziEh9rBphas63QZkV19XOUcP7KvzWqpUKiKiejB09Hiiet5Ve6Ws3x/35/OFRUbx48ZoFvx7DjOnP4mjswtKpZLC/Dzah0Xy8fwlhIZ3qnnBgO597mbxL7uY/8l7fDX3XVb88A1Go5GC/Fz8A9ow+9NvsHP4o3Zbfo1uCb37D2L6jPf493szmP3aNOYtXEZ45243+ClpWEql0rRgx/9r06YN/v7+6HQ6qn+fmGdj80f9bVlZGTt37kSn06FUKvHw8CAwMBA3N7dGi19oGSytbQjr2psdMTF0DQ/G0/XGreeE5u9CWhYHYs/Q+966rfzuRDJJ3Ku6ZVu2bGHp0prVhqZPn07Xrl1v8Ijm5e2338bGxZuBoybc+OC/qbxMi7a0BFs7+2tOVvozbWkJ5WU1t7zs7B3QWP6RPOirqzm4bxc5WRmoLSwIDA6hfceI645kFebnceZkHNkZV7C2taNdh3D8WgfddNy6qiqOxOwlK+My7p5e9Lp7EFWVlZSWFGFpZY2tXe32QpIkkZqSSHW1Dh+/gBs+1wtJ5zh7Mh5JkghqF0pw+1DkcgVVlRUUFxWi0VjWSvQag9Fo5OzJOFISz1Gtq8Ld05uoXjUjxdeTmpJI/LFDGI1G2rQNpX1YZJNYgz7pVBy/LJ3P4sWL/1YXgr+jsLCQzMxMsrOzyc7OxsPDg379+pn2V1VVYWFhcVuuLVxfdXU17733HkorR0Y8OPmmH7ft5x9JOLwPmUzG029+hM01WqndLmXaEn788gP83O144bFRqFXm/x0Sbh9teSUfLVxNJRomPPky6r94j73qbPwRNi5fxJdffomTU8v7IiN+4uuhbds/6uvi4uJaXJJ6O1lZ29SZAX49NrZ2160XVapU9Bkw5Kav6+jsUme9+b9DbWFB7/61++xaWllheZ0FImQyGa2Drl+H+f8C2rQjoE27OtstNJa4eZhnpF4ulxMa3pnQ8M43/Rj/wGD8A4NvY1RNl6OjI46OjoSEhKDT6dD/aeJLRUUFmzZtok2bNgQEBGBra2vGSIWmztrGjr5D72Pzqu9Zt+MA44e2zKWCBSivrOKHtTvIyi9l7OOP3VSCeicQSWo9eHl54e7uTnZ2NufOnaO8vLzeq1kJgtByqNVqU69UqBlZ9/T0JCEhgePHj+Pm5kaHDh3w8/P7i7MId7K2HTujLSli166NWFlqGNgzEo3F7bkLIJhHaVkF/12znZNJlxl836N4+9/83b2WTiSp9aBWq+ncuTObN28mLy+PhISE6y7LKgiCYGVlRa9evQgLC+PSpUtkZGSYaluhZmKXXC5vEqURQtMgk8no3GsAVjZ27PttJ4fiz9E9oh3eHi43rOkXmjYJuHA5i4NxZ1HbuTH6kafxC2pv7rCaFPFOWA8ymYwBAwawb98+tFotP//8M126dLlttW6CILQMtra2dOjQgQ4dOtTafuLECS5dukSnTp3w9/cXXQIEoOazJiSiK8GhkVxKOcvZ0/FsP/wb2pIiWuqsEsloME3obInkMhn2Ti74BARz99hJtPILRKEQKdn/E69IPXl6ehIeHk5MTAzp6elUVVWJJFUQhFvi6elJdnY2e/bsoV27dnTo0AE7O9GKRqihVKkIbBdGYLswACoryjHoq2/wqOZp/7Z12Ng5ENmjn7lDuS00ltamjjLC9YlXqAEMGjSImJiaBu1nzpwRE6gEQbglXl5eeHh4cPHiRX777TfS09MZO3bsXy7LK9y5/tzppCWpqijn8sVkLK2s6TlgOHLx83/HEklqAwgKCqJNmzYkJSWxdetWunTpIpZFFAThligUCoKCgvD09ESr1YoEVbjjXLmUQlF+LtriQrIzLuPp42/ukAQzEZlUA5DJZIwfPx6VSkVycjJbt241d0iCIDRz1tbWuLu7m/59/vx5YmJi0InlEoUWzGgwEPvbTiTJSHW1jtiYXXWWWRbuHCJJbSDt2rWjS5cu6PV6NmzYQE5OjrlDahBGg/7GBwlCM6HTVZk7hFumVCpJSkpi06ZNFBUVmTscQbgtCvKyuZR8zvTv5DMJzfr3VqgfkaQ2EIVCwdixY1GpVJSUlLB+/Xpzh1RvNjY2FBbUXZdeEJqrksJ87O3tm+Ws+cDAQIYMGUJVVRWbNm0iPT3d3CEJQoNLTTzDnxfC1FVVoq9umZPDhBsTSWoD8vLyMi1LFhMTQ3Jyspkjqh9/f3/yszKo+H1JUkFoziTJSNqFRPz8/JptnaeHhwfDhg3Dzs6OtLQ0c4cjCA2uvKy0zra0lHPXOFK4E4gktYFNnDgRCwsLdDodixYtata35fr06QOSgczLF80diiDU26Wkc2RevkC/fv2a5UjqVba2tgwePJiIiAhzhyIIDapMW8L5k7F1th/Zt12Mpt6hRJLawDp06EDfvn0BSEtL4/Tp02aO6Na5u7vTt29fTh47gEEvalOF5quyopw9m9cQFBhIp06dzB1OvanVaiwtLYGapVbFZCqhJbiUdJai/LolZnlZGWSkpZghIsHcRJLawORyOffddx8+Pj4ALFu2jLy8PDNHdetGjBhBUc4VDu7ehNFgMHc4gvC3SZLE3s1r0Rbl88ADD6BWq80dUoM6fPgwW7ZsoapKTC4Rmi+j0cjZhKO16lGvMhj0nI0/Ss1CosKdRCSpt4GtrS2TJk1CrVZTVFTE999/32xHOuzs7OjXrx9H9m5jz6Y1VFVWmDskQbhpRoOB+EN7ST4Tz2OPPUZgYKC5Q2pwvr6+5Ofnc/z48Wt+wAtCc1BSlM/lC4mADGvbP1ZZs3OsmedxMekMFeVlZopOMBfRzP82CQ4OZvTo0axZs4b4+Hi2bt3K0KFDUTbDZdCGDx+OUqlkw8aNpKWcpfeg0fi0bmPusAThL125lELsb7vIz7rCP8bfX1Nj3QK1atWKiIgIEhISCA8Px8qqZa5CJLRsyacTqNZVEdQ+DA8fP37bvgGAASP+we6NqyguLCAt+TxtwzqbOVKhMTW/jKmZkMlkjB49mtOnT3P69GlWrFhBTk4Ojz/+eLNLVBUKBcOGDWPIkCHExcWxZcsWfl3+LdWikF1ooiwtLfHx8aFfrx706fNyi0/cIiIiyMvLIykpifDwcHOHIwh/i66qksN7t9EuPIoh9z1C/KG9pn0aS2see+EtNi5fRMzOjQR3iEQmVnS8YzSvbKkZeuihh5g9ezYVFRVER0fj7u7OiBEjzB3WLVEoFHTp0oWOHTtSWFhIdnY2BQUF5g6r2auurmbjxo2MGjWq2bZGakpsbW3x8vLC1dW12X0hvFVyuZyoqCjx8yM0S6lJZ/ANCGbw2IdRqS3q7NdYWjPsH0+waeV3ZKVfwtOntRmiFMzhzngHNyN/f3+mTZvG559/TmVlJTt37qR79+64urqaO7RbZmFhgYeHBx4eHuYOpV7Kyso4e/Ys+/btY/z48Xh7e5sljuPHj1NaWoqfnx9t2ogyCuHWODo6mjsEQbglRqORoeMnoviLL5VW1jaMemjq73Wrwp1CjJk3gvDwcB588EEUCgW5ubl8/fXXVFSICUjmotPp2LBhA//617/49NNP0Wg0eHl5mS2egoICdDodR48eNVsMgiAI5tK2Y+e/TFCvUltoCGjXsREiEpoKMZLaCGQyGQMGDECtVvPTTz9x/vx55s6dy6hRo0T9WCPJyckhNjaW3Nxc4uLiyMnJQZIkXF1dGTdunNmau+v1eo4fPw7UrFI2dOhQHBwczBKL0PxlZGSQk5NDaGgoKpXK3OEIwk35O++/zXkhDuHvE0lqI5HJZNx1111IksTChQs5d+4cqampTJ06la5du4pfvNskJyeHgwcPsnHjRsrLy2vtuzq5zc3NzUzRwZUrVzh79iwAhYWFHD58mMGDB5stHqF5KysrIyEhgaCgIJGkCoLQ7IkktZH16NGDpKQk9uzZQ2VlJd988w329va0a9fO3KG1GHq9nuTkZHbv3k1cXBxlZdfureft7U3Xrl0bObraYmNjazVhP3jwIIMGDRJfWoRbolarqaqqQi9WiBMEoQUQSWojU6vVPPHEE9jY2LBp0yYqKytZuHAhzz33HAEBAeYOr1nT6XQkJyezYcMGTp48+ZeNzeVyOaNHj8ba2roRI6xNp9Nx4MCBWtsuXbrEpUuX8Pf3N09QgiAIgtBEiIlTZiCTyRg3bhxjx45FJpORlZXFvHnzOHz4sFgx5hZJksT58+c5d+4crq6u9O7dm8jISNq2bXvNtjze3t5ERUWZIdI/XLx4kaysrFrbqqqqOHbsmJkiEgRBEISmQ4ykmolSqWTMmDF069aNTz75hMzMTD7//HOcnJyYNm0awcHB5g6xWZHJZHTs2JGOHWtmfmq1Wn766Sfi4+ORJAmFQoHBYDAd/+yzz5q1h6bRaGT16tXX/FKyfft2BgwYIFoKCX+bh4cHI0eOxMbGxtyhCIIg1JsYSTWzVq1aMXXqVDw9PZEkifz8fL788kvS0tLMHVqzlZqaygcffEB0dLQpQR05cqRpv5OTk9kTwMzMTC5evHjNfVqtFp1O18gRCS3B1R7Gd8oiBoIgtGwiSW0CgoODmTlzJlFRUcjlcvLy8njvvffYvn17rUk1wl+TJIm0tDTmz59vSgAdHR2ZNm0aQUFBQM2I68CBA80+0nTkyBFTr9yr5Qhyudw0YSolJcVssQmCIAhCUyCS1CbCwcGB5557jnHjxqFWq9FqtSxevJi5c+eKhOUmGI1Gdu7cyfvvv8/ly5cB6NChAzNnzqRr166m2c5ubm4MHDjQnKHWqjv19fWlR48eAAQEBJjqZH/77TezxSc0X0ajkaqqKlHbLghCiyDuCTUhKpWKUaNGERkZybJlyzh9+jRnzpxhzpw5dOrUiQceeAAnJydzh9mkGI1GDhw4wIYNG8jIyMBoNNK6dWsee+wx/Pz8sLCoWQe6pKQEmUzGpEmTzD6KGhcXR1paGkOHDmXUqFFs3boVqPn//+STT9KpUyeWLl1KYmKiqE0W/pbMzEyio6MZMWIEdnZ25g5HEAShXkSS2sTIZDL8/Px444032LNnDz///DOFhYXExMQQHx/PkCFD6N+/v9lrKpuCiooKdu/ezbJly0zbAgMDeeaZZ/D09Kx1rF6vp2vXroSGhjZ2mHVcvnyZoUOHMn78+DqdBywsLLjrrrs4c+YMMTExIkkV/ha9Xk9ZWRlGo9HcoQi3kSRJVJaXUVVVgdQC/1+Xl2lNfy8tLqQwL9uM0dweMrkcC40Vllbma4PYHIgktYmSy+UMGDCA0NBQVq9ezcGDBykrK2Pt2rXs27ePsWPHctddd92RTd+NRiOHDh1i06ZNpKamAjV1nYMHD2b06NHXHCnV6/UMGTIEudz8FS6tW7cmIiLimq2xrnr88cf55ZdfKC0txdbWthGjEwShqarW6TibcISUsyfJybxMVWXLTFINf1qMYtvPPzaJ9+2GJpPL0WisaOXbmqCQCII7dmqRz7O+RJLaxHl4ePDUU0/h4uLC/v37KSoqIjc3l2+//Zb9+/czbtw4AgMD74glEI1GIxcvXuT7778nNTXVNFrk4uLCxIkTiYiIuG7SHh4eTqtWrRoz3OsKDQ294exrjUbD2LFjxZuWIAgA5GReYePyb9EW5uHj6cKALsEE+HigUomP8eZIW1bBxSvZnEq8xJZVxzl/MpYBIydgY2dv7tCaFPHT3QyoVCoeeOABBg0axJYtW0hLS+P8+fOmetXg4GB69uyJn58f/v7+fzlC19xUV1eTnJxMYWEhx48fJy4ujoqKCmQyGa1bt8bf358RI0bg4eHxl+fx9vZupIhvzNLS8qaOu1pPKwg3y8rKCn9//zviS+udQpIkks7Ecyx6M2087blnXD+C/Fohl995d9Famq7hbRk3xMjJxFQ27TnCmu8/p++9Y2kdbP6ytKZCJKnNiLOzMw8//DBGo5Hjx4+zdOlS8vPzOXPmDGfOnEGtVuPr68uAAQPo3LkzVlZWzXIkTpIkdDod586dY82aNVy6dMk0O18ul+Pn58eYMWOIiIhArVabOVpBaDpcXV0ZNGiQucMQGtDJYwfYtf4nRvSPYvTAntyBFV4tmkIhJ6J9AO0DfVi+cS/rly5g+AOTCQoJN3doTYJIUpshuVxOly5d6NChA/Hx8ezZs4fz58+b1q5PTk7G1taWzp0706dPH/z8/LCysjJ32DdUWVnJ2bNniY+P58SJE2Rn/1EsL5fLTQl43759RbNyQRBavOLCPPZvW0fn0EBG9O8mEtQWzEKt4sGR/cjKKyB68xq8/IPEpCpEktqsaTQaunfvTpcuXUhPT2fJkiUkJiZiMBgoLS0lOjqa/fv34+TkRGhoKD179iQkJKRJja4aDAYuXLjA0aNHSUhIID09vdbMZKVSSZcuXejfvz9BQUFoNBozRisIgtB4jh+IBkMV99/bG2ULKuMSrk2tUnL/vX2YM38VZ+IO0bnXAHOHZHYiSW0BlEolfn5+zJw5k8zMTPbu3Ut8fDzZ2dlUVVWRm5tLdHQ00dHR2NnZ0bp1a4KDgwkMDMTHxwcHB4dG6RJwNXnOzc0lOTmZuLg4UlJSTCsvXaVQKHBycqJjx44MGzasTjspQRCuLTMzk4MHD3LPPfeIrhDNnMFg4FzCUXpGtsfN2cHc4QiNpLWPB+Ht/DkVe1AkqYgktcXx9PRkwoQJ3HfffWRmZqLVaklMTOTYsWNcunSJkpISEhISSEhIAGqWDXVxcTGNrmo0Gry8vHBzc8PFxQUnJyesrKywsLDA2tr6mrfZJUmiuLiYiooKqqqq0Gq1ZGZmkp6eTn5+PlptTc87vV5PYWEhhYWFdVbE8fDwICoqisjISFQqFe7u7mZvui8IzY1OpyMvLw+DwWDuUIR6unIxierKcnpEtjN3KEIjkstk9O4Sylc/bTF3KE2CSFJbKJVKha+vLwAhISEMHz6c3NxcTp48SWJiIqmpqRQVFVFUVERhYWGtx15NYGUyGQqFwrSm/NX/Xl31SpIkSktLqa6uxmg0IkkSRqPR9Od6LCwssLW1JSQkhLZt2xIUFIS7u7uYBCUIgvC7gtwsVCoFLk6iJdGdxs3ZAbVoLQaIJPWOoVQq8fT0xNPTk0GDBmE0GsnJySExMZGUlBSys7MpLi6moKCA0tJSoCYJ1f+pqfJV5eXlN31dGxsb7OzscHBwMLWM8vb2xtPTUySlgiAI11FVUY5CJkMlalHvOEqlAkUTmjtiTiJJvUPJ5XI8PDzw8PDgrrvuMrV9qqyspLS0lOzsbAoKCsjPz6ekpASdTkdRURFGo9GUxEJNz0+VSoWTk5OpJMDGxgZvb2/c3d2xtrZGrVajVqtbVP9WQWiKZDIZSqXyjlyJThCElkckqQJQ8+FmYWGBhYUF9vb2Tar5vSAIN8fLy4sJEyaILhiCILQIIkkVBEFoIRQKRbPoiSwIgnAzRNGDIAiCIAiC0OSIJFUQBKGFKCws5NChQ3V6DwuCIDRHIkkVBEFoIUpKSjhx4gRVVVXmDkUQBKHeRJIqCIIgCIIgNDkiSRUEQRAEQRCaHDG7XxAEQRCEazIYDJxLTqWqSldru0ZjgUqlRGOhxsXJEUuNhZkiFFoykaQKgiC0EK6urtxzzz1YEdRMKAAAIABJREFUW1ubOxShhSgtK2fspJdJSb1ca7ulRoNapUSjsaCVuytTHh7LlIfHioUkhAYlklRBEIQWwsrKitatW5s7DKEFsrBQM/v1Z2ptKy7Rci45lQNHE5g24yOiDxzjk7dfxtPdxUxRCi2NSFIFQRAEQfhLapWKl6Y+cs19VzKzeeLFd1i1YTtVOh0rFnyISqVq5AiFlkhMnBIEQWghtFotiYmJ6HS6Gx8sCA3E29Od915/BpVKydbdB9h3KO6ax/2vvfuOr6q+/zj+uiM3497svUMSskgCSQhhyZLhQKyKA0TbOorFOn611modWLVS7dBWq7ZurVqwDBmCbAgkEJIQIGSRhJAAIXuvu35/IJEYEGTdm5vP8/HgAfecc+9930PuvZ98z3f06PU0NbfS2NRCe0cnJpO53zEmkwm9wdC7z2g00dLWTlt7JwaD8QdzdHZ109Ry8vE7u7p+8Fiz+WRXhoamFppb2+jR68/z1YorSYpUIYSwEfX19WzZsoWOjg5LRxGDTGJcFCMT4+ju6eHLVesxm78rQE0mExu2ZTJz3sMEJ8/AN34KydPu4FdPvYxeb+jzOP9d8Q3X3LGArzemk7O/gLsfepqoMTeSOPlWZt71EJvSd2Mymfrcp7WtnVfe/JDR199FSPI1+CZczahr7+LFv/2b+sbmfllr6xt57tW3iLvqZnzjpxA+aibTb/slW3buuTwnR1wwudwvhBBCiItib6/hqtHJZGTvIzuvgO6eHhzs7TGbzbz6z4956bV/YzAYSUqIwcPNhd25B/j3p0vp6dHzjz/+rnd2gCNHj7M1I5vwkEBWbdhOc2sb0RFhlJQdofLocTKz95Ox6iNio8J7n/uTL1fz9Mtv4OrqTEpiLC7OOnZm7eX5v7zDpvQsln3wV1xddACUHTnK7HsfY3/BIQL8vLlm8liqa+rJyN7H3b96ms/efpnxo5Iscg5Ff1KkCiGEEOKijRk5HIVCQUFJGe0dXTjY2/PKmx/y+5ffIDYqnA2L38bX27P3+I3bdzPjjl9iNJl476/PoVR+d3H3w8Ur+eSNl7jjJzN6t736z4948qW/s3ztlt4itaGpmYWvvsWds6/ng9ee7z3WZDJx7dwH2ZqRw0eLV/LwfXOoa2hiwo33UFvXwN/+8Di/uuf23uPb2ju45d7fMHPeQ6R/9SHxMZGX81SJ8yRFqrhgXV1ddJ2j3484t1NLWBoMBpqamlCpVBZONLCp1Wp0Op2lYwgx6NjZqVEoTvYNNZlMdHX38NnSrwF4+tH7+hSoAJPHpQKwdPVGXnnmUbw93Xv3xUWFc+usaX2Onz1zKk++9HeOVtf0bjMaTbS1d1JX30h3Tw/2Gg0ASqWSv/3hcdo7OgkLDgBg9YbtnKitJzkhhnvn3NjnsXVaJ1JHDGPj9l18tvRr/vjUQ5forIiLIUWqOG9ms5mmpibWr1/P7t27aWxs7Nc3SPx4BsPJPlmlpaX85je/sXCagU+hUODj40NKSgppaWkEBQX1aaGxZW5ubowaNQpHR0dLRxGDkNFoBDPYqdUoFArKKqqoPFqNq4uOQD8fKo9W9znebDbj7+vN8RO1ZOcd5Jop43r3JcZFofre+/ZUkdvU3IrZDAoFuOi0pCUnsHbzTsbO/Cm3XD+VCaOTSUqIIe60LgEA2zKyMZvNDB8WTV1DU7/8Ou3J9822zJxLcj7ExZMiVZyX7u5u1q5dy7p161BiZFhkCKGp0Xi6uaBSDY4CQFg/sxmaWtooLKti2+YNrFmzhrS0NObMmYOrq6ul4112rq6ujBgxwtIxxCB1pOo4JrMZPy8PNHZqausbaevoxGQyMenm+37wvo3NLX1uO+ucznpsc0srYAYU2NtreOJXPyP7FwfJyy8mL78YhUJBUkIMN107hTtvvpaQIH8ADlcdA+D9z5fz/ufLzzuLsBwpUsU55eTk8PHHH6O1V/Lg3GsI9PVE5+SILCwirNXk0Ym0dXTR0tbBqk27eP75hTzwwC+JioqydDQhbJLJZGL1xnQAxqQkoNU6YTQaMZlMBPh5884rT+OsO/tKaNERoRf83NMmjubg9qXU1TexLTObDdt3sXN3Hs/86U1e+9en/HzOT1j0+4d7ZxJ45P653HTtlLM+nqOjLPFqLaRIFT+op6eH5cuX4+vuxIN3zkTr5GDpSEKck0KhwFnriLPWkftvv4Z/fLKS119/nRdeeAEPDw9Lx7ts9Ho97e3tODs7S99mcUWVlB9hZ9ZeVCoVc2++DpVSiY+XB85aJzo6uxgaHkLkkJB+96uuqUOn06JzuvAuKu0dXXh7uBPk78uI+Ggevm8uBcVlvPT6e3yxfC1vfbiY5x6bT3hoEDuz8jCZTIxP6z+Cv629g86ubjzcXC44i7i05Dqt+EEZGRl0tTdz320zpEAVA5JSqeSWGeMwGXpYvHhxn/kbbU11dTXLli2jra3N0lHEIGI2m1mzIZ2W1nbiYyKZOiENgJBAP3y8PWhuaWPt5p397mcym0m77i7SrrmTg8VlF/Tc67dmknbtPNZ824p7SmxUOM/8+hcAdHV109rWzsjhcSgUCtZtyaChqf8l/bc+WkLadXfx2MK/XlAWcelJkSrOqquri2XLlnH9pFQ8XJ0tHUeICxbk58XMyaPYtWsXpaWllo5z2ZhMJvR6vU0X4sIyTCYTRaWHe/+UHzlK0aHDLFm5njkP/I7fvvAajg72/GXhr9F+2yrq4qzjrtkzMZvN/OEv77C/oKT38YwmE6s3bOfo8Rrc3VyJDAu+oFzhoYEcra5h0Rsf9OlLemoBAYDwsCB8vDyYPXMqPl4eFJdW8PuX3+jzPmlobGbNxnSOHj/B1VelXVAWcenJ5X5xVvv27aO1pZkgX89zHyyElZuUlsA36Tmkp6cTERGBQjpVC3He2js6GTbhlt7bTo4OdPfoMRqNqFUqRqck8OsH7mLimJQ+93v0F3dyrLqWL1as484Hn2L29VNxcdaRe6CQFWs3MzolkVeefRSNxu6CcoUFB/DgPbfz17c/Ycbtv2TWjEk467TsLyhh8VffMHRICH986mEA/Hy8+PdfnuW5V97iwy9WYDQaiB0aTnNrGyvWbqby2Al+s+CnTJ80+sJPlLikpEgVZ5WdnY2vpxuhgT6WjiLERXOw1zA2OZb07GzmzZuHWi0ff0Kci4O9PQ/deweNZ7g8rlAqCPL3ZUxKImEhAb1zlJ5Op3XizUVP8tB9c/jbO5/yyZer6ejswt/Xm5/eNotXnnkUe/vv7jd+VBLPPTaflMS4fo9lZ6fmucfmEznku1ZXlUrFC79dQEpiLJ8sWc1/lq6hpbWNsOAAnn70fn4+58Y+869ed/V4Jo5J4X+rN/LWh4tZs3EHGo0d8dER/GXhY0wZP+piT5m4hORTWpxVbW0tAb4e0uIkbIavlzsNDQ02ezncwcGBwMBAKcDFJeNgr+FX99xxUY+hUCiIHTqEf/35mXMeO27UCMaNOvM0anZqdW8/09Op1Wpmz5zG7JnTznCv/rROjtx960zuvnXmeR0vLEc+ycRZmUwmdE5nnzJEiIHGyUFjswUqgK+vL9dff72lYwghxCUhA6eEEEIIIYTVkSJVCCGEEEJYHSlShRDCRhw5coT333+f5uZmS0cRQoiLJkWqEELYCLPZjMFgsOl+t0KIwUOKVCGEEEIIYXWkSBVCCCGEEFZHilQhhLARKpUKnU6HUikf7UKIgU/mSRVCCBsRFBTE3LlzLR1DXAJaF1cMJhNdPXp0WkdLxxFXUHe3HoPBaOkYVkF+3RZCCCGsjKu7Fz09BurPsBypsG0n6pvo0RssHcMqSEuqEKf50xsf8Oo/P+q33dfLE62TA0EBfsyeOZXxaUkE+HqjVqsskFIIYev8g8Owc3Bi78EyosICZXnqQcJoMrFnfzHuXj6WjmIVpCVViNN0dffQ1NwKwOjkhN4/Q0ICUKlUbE7fzd0PPU3k6Bt48bV/WzitEH3V1NSwbt062tvbLR1FXCQ7jT1pE2fwTXoOZZXVlo4jrpC9B8vYlVdM2uRrLR3FKkhLqhBnEDt0CKs+/UefbV1d3RytruWZP73JkpXrefc/y3jg7tn4+XhZKKUQfXV2dlJRUUFaWpqlo4hLID5lDPv27OCjpRt4Yv6taB0dLB1JXEaNzW18vmoLgWGRRMYNt3QcqyAtqUKcJwcHeyLCgnjvbwu5ZeZUqmvqeP3dz2TidCHEZeHgpGX6T+ZS3djOJ8s2YTKZLB1JXCZ1Dc38/eOv6NDD9JvuxM5OY+lIVkFaUoX4kRwd7Hn5qYdY9c02vly5nicfvhcXnbbPMWs2bGfZ15s5WFyGq4uOtKR45p+h1XXZmk3U1DVw/7yb2ZVzgE++XEVF5XHCQ4O4ddY0JoxO7vf8ZrOZVeu38fHiVRyuOoa7qwuJsUN55P65BAf69Tv+WHUtHy7+ip2786hvaiZ26BBmz5zKNZPHylRFQpwnsxkyt3zNyKum4eZx5a6eBA0ZyrSfzGXj8s/4ZMUmbpw6Bjdn7bnvKAaM+qZW3vnvWo7XtzFzzn14+vhbOpLVkCJViAswJCSQqRPSWLV+G7n7C5k4JgWAjD37WPjnt9masYeo8FDGjEykoamFv7/3Oe/+Zxm//NltPL7gp9jZnXzrLV+7mYw9+9i+K5cN2zIZFh2Bu5sLny/7mvc+X8YdP7mGtxY9hb39yd+qS8qO8KunFrE9M4epE9KYPC6VuvpGvlixltUbt/PKM49yw/SJwMnuCYve+IC3P1qC3mBg7MjhpI4YxraMbD5buoapE0bzh98uIDkhxjInUVxyCoUCtVotg2wuIf/gIRTk7qKnp5ucnZvJzdxKUGgEPgEh6FxciU5MwdXd67Ke85jEkXh4+7F7y1qe+/tnxIYHonWSS/+2oKm5jaKKEwxNTGXuzRNx9/K1dCSrIkWqEBcoZXgsq9ZvY39BSW+R+s4nX7Jx+y4W/Ow2XnryVzjrtJjNZnbl7OeO+U+w8NW3SIwbysxpE3ofp6yiCrVaxVcfv07qiGEoFAq+3rSDW+59jI8Xr+Su2dczeVwqAJ/+bzUbt+/i/nk38/qLv0VjZwfAgcJDTL7lfp7/8zu9ReqRY9W89Nq7+Pt4seKj1xiVFI9CoaClrZ2nF73JPz/4Lw2Nzexc1X82AzEw+fn5ceONN6LT6SwdxWYkjByLb2AwJQdyKT6QS92JY1SWl1BZXgJA9o6NBIdHE5eURvCQSDT2l2dOUx//IGbOuY8TR49QcaiA0vISWpoaMBpsc6qi7q5OVCoVahu97G2n0eDq7klIRAq3To/Dw1uK0zORIlWIC+SsPXnJrb6xGYDunh7Wb8kgNiqcRU8/gtO3gxwUCgWjUxJ58uF7efDJl9lfUNKnSAV49te/YFRSfO/t6RNHk5acwPbMHEoPV/UWqSVlRwCIiQzDTv3d23dYdARv/vFJunt6eret3rAds9nMH596iLTkhN7tLjotf3j8l6zfksGBwkOYTCa57G8jNBoNnp6elo5hUxQKBb4BIfgGhJA6YTplRQc4mLOLoxWldHV10NbSTMHe3RTt24OT1pmIuOGMmjAdF3cPVKpL/xXrGxiCb+DJLEaD3mb7xK9f/hnOru6MttFR7gqFEpVc9TgnKVKFuEjKbz9k9uw9yIm6Bu6+7YbeAvV0k8aOROvkyL6DJf32TRid0ue2SqUiNMif7UBTS2vv9uHDoln81Te8+Nq7aJ2cmDxuJKHBAahVKm67cXqfx1i9IR13NxemThjd7/lcXZwZN2oE/1m6hoPFZcTHRF7ISxdiULF3cCR2eCqxw1Npa2ni0MF95OdkcLyyHJPJRFtrM3m7tpGfnYFfcBhxI0YRGhlzWS7hKhQKm21l7GhroeJQIY5aHWOmXGezr1OcmxSpQlygQ+UnWzXDQgIBKPn29of//Yrlazf3O95gMNLV3cOxE7X99vl6e/Tbdqp1s7auoXfbXbOvp6yiis+Xfc0vn3gJN1dnoiPCmDwulbiocG6+bspp/VcraGvvYOJN95yxpbS+oQmTycSJ2nopUm1ES0sLFRUVREVFYW9vb+k4Nk3n4saI0ROIS0rjxLEjlBcdoKuzg4pDhTTV11BVXkJVeQkOjk4MSx6Nq4cXbp4+hEXGSNF1DsUHcmlraaKjvZXDJQUyHdMgJkWqEBfAbDaTvnsvKpWKpPhoAFrbTk6gPmFMCvHREWe9b1DAj2tVOf1yXoCfN++8+jS/f/Q+3nz/C9Zu3snB4jIys/cB8J/J4/jyvVdxsLenra0Db093br9xRp+uAadTKBS9RbYY+BobG8nIyCA4OFiK1CtEY29P8JChBA8ZCoDRaKTkQA4FeXs4WnGIjrZWsnds6j3eSedCXNIoho+agJunNyqVrFp3OqPBwN7MrZjN5t5/h8fEo1TKeRqMpEgV4gLk5RdTeKicpPhoIocEA/ROLzV2ZCKP3H/nZX3+kEA//vTMoyx8/AEOVx5n5TdbeeO9L1i7eQer1m9n9syp+Pl4oTfo+b/583B3dbmseYQQJ6lUKmKGpxIzPJXmxnoqS4vI3PI1DbUngJOXsvds38D+rB14+wcRmziSyGEjcHZ1t3By61BbXUVt9dHe25Vlxei7u7F3dLJgKmEpMlpCiB+pqaWV377wGiaTmfvn3YLDty1Ww6IjUCoVfLMlE4PB2O9++UWlXDv3Qd7+aMkFPW93dw/PvfoWs+5+hIqq4wA4OjgQO3QIv33wZ4xPSzr5PIWHAEiIjaSi6jj5haX9HstoMvG7l/7OHfOf4ERt/QXlEUL8MFd3T+JHjuXnjz7HvAVPkDx2Mm6e3iiVKrq7OqkqL2H9ii/496tPs27pp1QdLqG7q9PSsS2q4lBhn6tHen0P3d2D+5wMZlKkCnGeOjq72JS+m1vve5zNO7II9PPmtlnTevfHRIaRnBDLhu27WLpmY5/7mkwmXv77+6zfmkldQ9MFPb9CqWDH7r2s2ZjOx4tX9hvVa/r2duSQEADm3HQNRqOJ3y96o7crwilZufm8/dESVm9Ix8PN9YLyCCHOj0qtJiA0gqk3zuFnjzzD9XfcQ9jQODQae8CMQa8nb9c2Pn/7L3z65stsWfM/GmqrLR3bIjraWvttKy3Yb4EkwhrI5X4hziB7XwFhI/tOfWIwmmj4drqpyeNSeeOPv8P5tJWmlEolf3rmUeY88Dvu+/XzvPTau6QMj6Ojo5M9eQepPFZNcmIsv7jrlgvKpLGz47UXHmfmXQ+z6I0PSN+9l8ghwfTo9eTuK6Sk/Ag/v+NGZt9wsnCePmksc2++lv8uX0fcVTeTOCyK4ABf8vKLKS6rQG8w8NfnH+tdWEAMfN7e3kybNg2tVlYkslYaewdih6cSFZ9MV0c7Bn0PR8qK2bVlLc0NtdTXVFNfU03erm1o7B1w9fAiOiGZ6ISR6JxdwIanLGptbqRg7+5+2/ekb2BY0mg0DrKAwWAj305CnCYk0K/3svn3OWg0pAyP4/ppV5E6YtgZByNNHJPCui/e4o+vv8vOrDz+8781KJVKAny9+MW82Sx8/AE83b9ruYyKCD35fGf44okKD2F8WhJDThvYFB8Tybov/snTi95ke2YOG7fvQq1S4ePtwa/nz+OJh+7BXnNygn9HB3v+9eozpI4Yxvufr2BnVh5t7R24uTozLCqCJx76OdddPf5iT5mwIk5OTgwZMsTSMcR5UKlUaJ1P9hVP8PAiLimN40fKKNy3h/LifJrq6+ju6qS1uZGq8hJ2rF9JzLfTX/kGhqCxt72CrbRgP22tzf22NzfUc6SskMi4ERZIJSxJYbbVmYDFRVu4cCFhPlru+snVlo4y4JhMJmrqGmhsakGhVOLp7oq356UbGGE0mjhWXUNzazsqlRI3V2d8vTzOOil/a1sHtfUNdHZ1o9M64evtiYP94JsGJ/tACa99uIKPPvoIu29X6xKDg16v58UXX0Tt5M4Nc++zdJwf1NXRTnF+Lvk5mRw9fHLBjVNUKjVunl5Exg4nacwkXNxtY/EGo9HAl+//g4pDBWfcPyx5NNfd9nOZ/P57CvbuZuXn7/LGG2/g4dF/KsOBTlpShbgMlEolfj5evSP+LzWVSklwoB/B53m8s84JZ52MjrV1ZrMZg8GAWlayGdAcnLQkpo4nMXX8t5fAsyg+kMPxqsMYjYbeLgF70jcSEhFNfMoYgsIi0bm6D9j/98a6Go4ePoRCocDZ1Z2WppPzQ3v6+NNQW015UT5tLU0yC8IgI0WqEELYiOrqanbs2MGMGTNwdna2dBxxCTi7ujNq4nSSx07m8KECygr20dnRTnXVYZob6ykvzqe8OB+tzoWwqDjCo+PRubgREBoxoOZgzc/OwGDQk5A6Djd3L7Z/swKASdfPZs/29VQcKqRofw4jx8uVvcFEilQhhLARPT09NDQ0YDT2nwJtsFOr1Wi1WloG6BRPajs7ImMTiYxNBKCnu5uSg7ns372DE8eO0N7WQn5OJvk5mShVKlxcPRgaP4KR46eidXax6snwOzva2ZeVzvBRV3H1rNvJzdjSu0+jseemux9k7ZcfkbNzM0mjJ6I6y+Ikg1FPTzdarRa1jZ4T23xVQgghxGkUCgU+Pj5U7cu3dJRLQmNvz7Ck0QxLGk1j3QlKC/dTsDeL6qrDmIxGmhpqydq2nn270wkMiyQmcSRhQ2PRubhZOno/pQV5RCekMOWG21Gfoa+4xt6ea2bfzTfL/kPV4RJCI2MtkNI6NdXX4urqikZjm2MMpEgVQggxKHh7e9PSWI/RaEClsp2vP3cvX0aO92VE2gQqyw9xMDeT8qJ8Ojva6O7qpKxwP+VFB3B00hGdmMLwtAl4+vhZzTlwcHRiyg23nbFAPUVj78D0m+dxtPzQFUxm/Zrqa3B3d7fZgaDW8RMqhBDiojk6OqLT6Swdw2r5+PhgMhppbWrAzdPH0nEuObWdhiFRcQyJiqOnu4uywv3sSd9AzfEqDHo9He2t5GZsYW/mVrx8A4hLSmNI1DC8/YMsOuDqfKeW0mjsGRI97DKnGViaGuqIixpY/Y9/DClShRDCRnh5eXHTTTfhIJOen5Gvry92dmpqjlXZZJF6Oo29AzHDU4kcNoLmhjr0Pd0cO1JObsYW6muOU1t9lK1fL2XnxtW4e/ng7OJOdGIykXEjcHCUmUAGiuaGOoKCJlo6xmUjRaoQQtgIpVKJo6OjpWNYrcDAQHx8fCg6kENUQrKl41wRarUdnj7+APgFhTE87Soqy4op2Lub8uKDtLe1UHOskppjlZQW7kNj70BUQjIpY6fg4e2Lncbewq9AnI1Br8dOrWbYMNttXZYiVZyVWq2mpW1gjoQV4kzaOroG7DySP8apNVoGw2v9MVQqFcOHD2dr+k4629tw1A6+rhEqlZqwoXGEDY2jo72VqvJD7N66juqqCkwmIz3dXRzYs5OCvbvx8g1gaNwIohNH4unjZ+no4nuOVx3Gy8uTwMDAcx88QEmRKs7K29ubsuKDmM1m+bITNqGqug4vLy+b/nlubW1l48aNjBw5kqCgIEvHsTpJSUmsXLmSE8cqCRs6uEeJO2mdiYpPIjwmnqb6Wor27aH4QC4NdScwGgycOHqEE0ePkLV9PUOi44lLSiMwNAIHRyebfg8NFKUH84iPj7fZ/qggRar4AampqWRmZlB65DiRoQGWjiPERWnv7GLX3iLGT5xs0x/qTk5OODs7s2vXLtzd3dFqtZaOZFUiIiIIDQ2lvOjAoC9ST1Gr7fDyDcBr2izGXD2T45XltDQ2UHGogMMlB2lpaqAwL4vCvCy0zq74B4fh6eNHZNwI/AJDZd5SC+hob6Ukfy93/P53lo5yWclPljirhIQEfHx8OXKsVopUMeB9sz2HLr2RMWPG2HQrkEqlIjU1lSVLllBUVERy8uDoe3m+1Go1aWlp7C8+bOkoVkmpVBIYGkFgaASxI0bS0dZGeUk+e7ZvoPZ4Fe2tzRw6mMehg3lk79iEh7cfkbGJDEsZg5uHt02/t6zJ8SPlNDXU4uZmffPeXkpSpIqzsrOzY/bs2Xz2yYfERgbj7+1h6UhCXJCi8qOsS89h4sRJhIWFWTrOZefi4kJ8fDxHjx4lKSlJCofvGTFiBGu/WU97azNaZ1dLx7FiCpx0zr2LBtTXHKdwXzYl+bnUnziOQa/vHXSVtW09EbGJxI5IJSA0Aq3OxdLhbZbZZCJn52YCAgJsdqWpU1QLFy5caOkQwnr5+vqSnbOXrNwDJEQPwclBRnqKgaVHb+C9Jd+gttcyf/58m12Z5fs8PT3R6/X4+flJkfo9Li4uVB6poOJwhcy7+SM4aZ0JCY8iPmUMPgFBmIxG2lqbMRoMmExG6k4co2hfNoX7sqk9XoWLmwc6Fxfgx/38Haso5XDJQQASRo7F1d3zMryagaviUCEZG1cxZ84cwsPDLR3nspIiVfwgpVKJv78/u7P3sm1XHh6uOrRODtjZqVHKF5+wUkajiea2do6dqOeD/62nrQceeughvLy8LB3tirGzs8PX11cK1DNQKBTo9XrWrllJSEQ0zq7ulo40oChVKjx9/IlOHMnwUVfh6eP3baFqwmQ00NneRs2xSg7s2UFpwX7MJhN2GgfMZhN2dppz/kxKkXp23V0dfL3kI4ID/bn99tttviVVYT41V4kQP8BoNLJt2zZWr15NS3MjEcF+BPt74+HmLMWqsCotbR0UHz5K2ZFqFCo1U6ZMYdasWYN6AFFPTw9FRUXExsba/Jfa+TKbzbz44ovUNDRx14NPorGXBRAulslkouZ4JWUF+ynJ30ttdRUmk6l3v5POmdDIWKLikwiNjD3rogFZ275h8+ovAZgz/zcEh0ddkfzWzmw2s2nlYgr3ZrJo0SI8PW2/eJdPK3FeVCoVkydPZvTo0aSnp5OVlUXm/jI6O2Ue1Uuhs7NTJmG/RNRqNSEhIcy5cyoJCQl4e8tgjurYX9NKAAAJ5klEQVTqarKysqirq+Oqq66SQpWTram33norixYtIj8nk6QxkywdacBTKpX4BYbiFxhK6oTpVJYVkbc7nSOlhfR0d9HR1krB3t0U5u1B5+JCSEQM46begLObh03PuHGpNNRUsy8rnZnXXzcoClSQIlX8SI6OjkybNo1p06ZZOorNqK+v58UXX+T555/HxUUGG4hLLyQkhLFjx7Jz507c3NwYPnw4SqXS0rEsLioqilGjRpGxaTWBYZH4+Mu8speKnUZDeEwC4TEJtLe1UFa4nwN7dnLsSBlGo5HW5ibyczIpyMvCPziMmMRUwqOH2fxytRfjQPZO3N1cB9X3rxSpQlhYfn4+tbW1FBYWMmrUKEvHETYqOjoajUbD9u3baW9vZ9y4cYO+hVmpVHLbbbdRUlLC10s+5Pb7fy3r1l8GWp0LCSPHEZM4kprjVRTm7aH4QA6tzY2YjEaOHi7l6OFSdjg6ERoZi8ZeBuh+X1X5IfJzMnlg/v02P+3U6WTglBAWtmPHDoqKimhubmbs2LFy2UtcFgqFAnd3dzw9PfHy8sLZ2dnSkayCk5MTI0aMIH9/HjUnThASGSOtzJeJSqXGxc2D8Oh4UsZdTUhkNL6BoWg0Gjo72unsaKe+5hg1xyp77xMQGo5fUKgFU1te4b49rFv6CbNmXsfEiRMtHeeKkpZUISyoo6OD7OxsAIqKiqisrLT5KUWEZQUHB/e5XV5ejoeHB66ug3e+UD8/PxYsWMCzzz6Lk86ZtEnXDPpW5stNqVQSEh5NSHg0KeOm0NLUQGVZMXm7t1NVXtJ73GCfb/VYRSlr//cJScMTmTFjhqXjXHHy66IQFlRUVMSJEyeAkyNj09PTLZxIDDalpaUsX76crKws2tvbLR3HYry9vZk0aRI7N65mf9aOPqPSxeXn4ubBsOTR3H7//5E26dre7YO5+0X10QpWfvEeYSHB3HPPPdjZ2Vk60hUnRaoQFmI2m8nIyOjzZbhnzx66urosmEoMNmPGjCE6Opr8/HyWLl3KwYMHLR3JYm666SZGp41i41dfkP7NCgz6HktHGnRUKjVOg3i6uFPaWppY8t7rODs5sGDBgkHbPUcu9wthIdXV1eTk5PTZ1tDQQFZWFldddZWFUonBRqvVkpaWRnR0NGVlZXR0dPTZbzKZBk0fTY1Gw3333UdUVBRr166jp7uL0ZOvRecyeAaqCMsyGPTk52SyN2MrU6dMZsaMGXh4DN4lyQfHJ48QVmjr1q1nLAg2btyIXq+3UCoxGJ0aVJWSksLIkSN7tzc2NrJq1SoOHTpEd3e3BRNeORqNhqlTp/Lss8/QcKycz9/5M8cryy0dSwwC7a0trFn8IZu++oKxaSnMmTNnUBeoIEWqEBbR09NDVlbWGfeVl5fT3Nx8hRMJ0Z9SqcTe3p6tW7eyfPlyMjIyaGxstHSsK8LFxYX58+fjZG/Hlx/8gy2rv6SzY/D22RWXV2nBPr7411+oKM7n3nvv5ZZbbrF0JKsgl/uFsICCggJqamrOuE+v18tKXsIquLq6Mn36dOrq6igqKqK8vBwvLy/c3U+udW8wGABsdgWr0NBQnnvuOZYvX862bdsozs/lmtl3ExQ2dNB0gRCXV2PdCXZvW09R3m5iYmKYO/f/CAqSRSVOsc1PFiGsmNlsJjMzE6PRiFKpxMvLi5qaGtzc3Ojq6qKrq4ucnJx+UwUJYQkKhQJvb2+8vb3p7u7uM49vYWEhJSUlBAQEEBERgYeHh80Vbzqdjnnz5jFp0iSWLVvGkvdeJyo+mbSJM/AJkPeouDDtbS3k7txCbsYWVEq45557GDNmjMyT/T1SpApxhR0/fpzMzEzc3d155JFHyM3NZcWKFfj7+3P//ffzr3/9i7Vr1zJt2jScnAbv9CvC+th/byWg6OhoPDw8qKysZNOmTRiNRqZOnYq3tzcALS0tGAwGtFotarV6QH8BBwUFsWDBAm6vr2f9+vV89elb9OgNuHv5EJ2QQnhMPM4u7qhstFVZXDiz2UxXRzsVpYWUHtzHkbIiFAoYGhHO75/6HT4+Puh0OkvHtErybhLiCsvOzkan0/Hoo48SERFBXl5e7z5vb28eeeQRnn32WXJzcxk3bpwFkwrxw+zs7AgICCAgIIBRo0bR1NSEi8t3k68XFhayd+9eHBwccHFxwdvbm/j4+N6FA05NvzZQWl9VKhU+Pj7ceeedzJo1i8LCQrZu3crO9SvYvHoJPv7BhEfH4xMQjLOrO05aHY5aZ+w0GktHF1dQR1sL7W0ttLU001RfS82xSor2Z2PQ9xAUFMSkCeMZN24cwcHBsmjEOUiRKsQVplKpeOSRR4iIiDjjfldXV+6++2527NjB2LFj5UNMDAinZgg4XWJiIgEBATQ1NVFfX091dTWRkZG9RWplZSWZmZk4OTnh6OiInZ0dMTEx+Pr6AidboBoaGtDr9ajVauzt7VEqlWitYB5NZ2dnUlNTSU5OpqmpiQMHDpCbm8uBrG20traiUquxs7PHzt4elUq+an+Mrs7vBqit/u/7qO0GVpHf091FT3cXBn0PCoUCf39/bpx1A8nJyfj4+PS7IiHOTt45Qlxh8fHxhISE/OAxKSkpaLVaWltb+7RMCTGQODg4EBQUdNaBIM7OzoSEhNDR0UFPTw/Nzc20tbX1FqkGg4EtW7ZQX1/fex8XFxfuuOOOK5L/fKhUKjw9PZk4cSITJ07EbDZTV1dHXV0dNTU11NfXYzabz/k4ZrO5zy+k3799Ic70mMA5t12O5znTcWd7nOLiYvbv3w/AiMT43u4jp/af/vfFPM/5vq4fe450Oh1+fn54enri6+s7KFeKulSkSBXiCjtXgXpKTEzMZU4ihGV5eHgwZsyY3ttms7lPQadWq5k6dWqfOVqtvV/r6QPNYmNjLR1nQFq9enVvkTpx4kQ5j4OYFKlCCCGsgkKh6NNapVAoersGCCEGn4HRW10IIYQQQgwqUqQKIYQQQgirI0WqEEIIIYSwOlKkCiGEEEIIqyNFqhBCCCGEsDpSpAohhBBCCKsjRaoQQgghhLA6UqQKIYQQQgirI0WqEEIIIYSwOlKkCiGEEEIIqyNFqhBCCCGEsDpSpAohhBBCCKsjRaoQQgghhLA6UqQKIYQQQgirI0WqEEIIIYSwOlKkCiGEEEIIqyNFqhBCCCGEsDpqSwcQYrCLjY3FZDLh4+ODQqGwdBwhhBDCKijMZrPZ0iGEEEIIIQDq6+uprq4GICwsDK1Wa+FEwlKkSBVCCCGEEFZH+qQKIYQQQgirI0WqEEIIIYSwOlKkCiGEEEIIqyNFqhBCCCGEsDpSpAohhBBCCKsjRaoQQgghhLA6UqQKIYQQQgirI0WqEEIIIYSwOlKkCiGEEEIIq/P/wFjRMzOMs/gAAAAASUVORK5CYII="
    }
   },
   "cell_type": "markdown",
   "id": "a476778b-5bb8-4583-b9f7-4e00c1f285a5",
   "metadata": {},
   "source": [
    "![image.png](attachment:46d04e3a-f78a-43ed-bf64-dd272c823242.png)\n",
    "\n",
    "Note that this model does not make much sense; it's just an example to illustrate the fact that we can easily build any kind of model we want, even one that contains loop and skip connections.\n",
    "\n",
    "To implement this model, it is best to first create a `ResidualBlock` layer, since we are going to create a couple of identical blocks (and we might want to use it in another model):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "id": "02d1b403-c797-427a-b6c9-912b61c0ef80",
   "metadata": {},
   "outputs": [],
   "source": [
    "class ResidualBlock(keras.layers.Layer):\n",
    "    def __init__(self, n_layers, n_neurons, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.hidden = [keras.layers.Dense(n_neurons, activation=\"elu\", \n",
    "                                          kernel_initializer=\"he_normal\")\n",
    "                      for _ in range(n_layers)]\n",
    "    \n",
    "    def call(self, inputs):\n",
    "        Z = inputs\n",
    "        for layer in self.hidden:\n",
    "            Z = layer(Z)\n",
    "            return inputs + Z"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f67df91-1cda-463a-91b5-11efe59ab266",
   "metadata": {},
   "source": [
    "This layer is bit special since it contains other layers. This is handeled transparent by Keras: it automatically detects the `hidden` attribute contains trackable objects (layers in this case), so their variables are automatically added to this layer's list of variables. \n",
    "\n",
    "Next, let's use the Subclassing API to define the model itself:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "id": "b772b658-ad91-47b1-a787-cdfb748fcdb4",
   "metadata": {},
   "outputs": [],
   "source": [
    "class ResidualRegressor(keras.Model):\n",
    "    def __init__(self, output_dim, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.hidden1 = keras.layers.Dense(30, activation=\"elu\",\n",
    "                                         kernel_initializer=\"he_normal\")\n",
    "        self.block1 = ResidualBlock(2, 30)\n",
    "        self.block2 = ResidualBlock(2, 30)\n",
    "        self.out = keras.layers.Dense(output_dim)\n",
    "        \n",
    "    def call(self, inputs):\n",
    "        Z = self.hidden1(inputs)\n",
    "        for _ in range (1 + 3):\n",
    "            Z = self.block1(Z)\n",
    "        Z = self.block2(Z)\n",
    "        return self.out(Z)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c591fc9b-ca5c-4505-b6c3-0476491b7db5",
   "metadata": {},
   "source": [
    "We create the layers in the constructor and use them in the `call()` method. This model then can be used like any other model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "id": "18e12ef0-e1be-4f0f-90f2-daeaffde3c65",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "363/363 [==============================] - 2s 1ms/step - loss: 57.0355\n",
      "Epoch 2/5\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 8.3522\n",
      "Epoch 3/5\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 1.6286\n",
      "Epoch 4/5\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 1.0257\n",
      "Epoch 5/5\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.9373\n",
      "162/162 [==============================] - 0s 809us/step - loss: 1.0312\n",
      "162/162 [==============================] - 0s 691us/step\n"
     ]
    }
   ],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = ResidualRegressor(1)\n",
    "model.compile(loss=\"mse\", optimizer=\"nadam\")\n",
    "h = model.fit(X_train_scaled, y_train, epochs=5)\n",
    "score = model.evaluate(X_test_scaled, y_test)\n",
    "y_pred = model.predict(X_new_scaled)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97cd6a38-152e-42e3-9ebf-c72aa47b5fa1",
   "metadata": {},
   "source": [
    "If we also want to be able to save the model using the `save()` method and load it using the `keras.models.load_model()` function, we must implement the `get_config()` method (as we did earlier), in both the `ResidualBlock` and `ResidualRegressor` class. Alternatively we can save and load the weights using `save_weights()` and `load_weights()` methods. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "126dc52e-0d4d-4be9-a557-0cc1b8f1ab3a",
   "metadata": {},
   "source": [
    "We could have defined the same model using the Sequential API instead like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 171,
   "id": "e3186fb0-56db-4ec3-a800-3d5101bed3e4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "363/363 [==============================] - 2s 2ms/step - loss: 1.7440\n",
      "Epoch 2/5\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.7228\n",
      "Epoch 3/5\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.5986\n",
      "Epoch 4/5\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.4572\n",
      "Epoch 5/5\n",
      "363/363 [==============================] - 0s 1ms/step - loss: 0.5240\n",
      "162/162 [==============================] - 0s 807us/step - loss: 0.9162\n",
      "162/162 [==============================] - 0s 737us/step\n"
     ]
    }
   ],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "block1 = ResidualBlock(2, 30)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"elu\", \n",
    "                      kernel_initializer=\"he_normal\"),\n",
    "    block1, block1, block1, block1,\n",
    "    ResidualBlock(2, 30),\n",
    "    keras.layers.Dense(1)\n",
    "])\n",
    "\n",
    "model.compile(loss=\"mse\", optimizer=\"nadam\")\n",
    "h = model.fit(X_train_scaled, y_train, epochs=5)\n",
    "score = model.evaluate(X_test_scaled, y_test)\n",
    "y_pred = model.predict(X_new_scaled)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c8b5d2a-8da2-4efe-b38c-7a593e010169",
   "metadata": {},
   "source": [
    "With that, we can naturally and concisely build almost any model that\n",
    "we find in a paper, using the Sequential API, the Functional API, the\n",
    "Subclassing API, or even a mix of these. “Almost” any model? Yes, there\n",
    "are still a few things that we need to look at: first, how to define losses or\n",
    "metrics based on model internals, and second, how to build a custom\n",
    "training loop."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4d74681-7e87-4d34-9fd1-308b1a6bf004",
   "metadata": {},
   "source": [
    "### Losses and Metrics Based on Model Internals"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28a0b76c-85c4-4f5d-8ccb-dbec6bea98cc",
   "metadata": {},
   "source": [
    "To define a custom loss based on models internals, compute it based on any part of the model we want, then pass the result to the `add_loss()` method. \n",
    "\n",
    "For example: Let's build a custom regression MLP model composed of a stack of five hidden layers plus an output layer. This custom model will also have an auxiliary output on the top of the upper hidden layer. The loss associated to this auxiliary output will be called the *reconstruction* loss (more in Ch-17): it is the mean squared difference between the reconstruction and the inputs. \n",
    "\n",
    "By adding this reconstruction loss to the main loss, we will encourage the model to preserve as much information as possible through the hidden layers - even information that is not directly useful for the regression task itself. In practice, this loss sometimes improves generalization (it is a regularization loss).\n",
    "\n",
    "Here is the code for this custom model with custom reconstruction loss:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "d4809c5b-5294-4a2c-aaa7-72b26f1d9db5",
   "metadata": {},
   "outputs": [],
   "source": [
    "class ReconstructingRegressor(keras.Model):\n",
    "    def __init__(self, output_dim, **kwargs):\n",
    "        \"\"\"\n",
    "        Constructor creates the DNN with 5 dense hidden layers and one dense output layer\n",
    "        \"\"\"\n",
    "        super().__init__(**kwargs)\n",
    "        self.hidden = [keras.layers.Dense(30, activation=\"selu\", kernel_initializer=\"lecun_normal\")\n",
    "                      for _ in range(5)]\n",
    "        self.out = keras.layers.Dense(output_dim)\n",
    "        \n",
    "    def build(self, batch_input_shape):\n",
    "        \"\"\"\n",
    "        This method creates an extra dense layer which will be used to reconstruct the inputs to the\n",
    "        model. \n",
    "        \"\"\"\n",
    "        n_inputs = batch_input_shape[-1]\n",
    "        self.reconstruct = keras.layers.Dense(n_inputs)\n",
    "        # super().build(batch_input_shape) #Due to an issue introduced in TF 2.2 (#46858), we must not call super().build() inside the build() method.\n",
    "    \n",
    "    def call(self, inputs):\n",
    "        \"\"\"\n",
    "        This method processes the inputs through all five hidden layers, then passes the result to\n",
    "        the reconstruction layer, which produces the reconstruction. \n",
    "        \"\"\"\n",
    "        Z = inputs\n",
    "        for layer in self.hidden:\n",
    "            Z = layer(Z)\n",
    "        reconstruction = self.reconstruct(Z)\n",
    "        \"\"\"\n",
    "        Then call method computes the reconstruction loss and adds it to the model's list of losses\n",
    "        using `add_loss()` method (We can also add add_loss on any layer inside the model, as the\n",
    "        model recursively gathers losses from all its layers. )\n",
    "        \"\"\"\n",
    "        recon_loss = tf.reduce_mean(tf.square(reconstruction - inputs))\n",
    "        \"\"\"\n",
    "        We scaled down the reconstruction loss by multiplying it by 0.05 (this is hyperparameter we\n",
    "        can tune). This ensures that the reconstruction loss does not dominate the main loss. \n",
    "        \"\"\"\n",
    "        self.add_loss(0.05 * recon_loss)\n",
    "        return self.out(Z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "89acaa14-8eea-4a1b-a1e6-bc1df0ac4b03",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 3s 2ms/step - loss: 0.6997\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 1s 2ms/step - loss: 0.4098\n",
      "162/162 [==============================] - 0s 891us/step\n"
     ]
    }
   ],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = ReconstructingRegressor(1)\n",
    "model.compile(loss=\"mse\", optimizer=\"nadam\")\n",
    "h = model.fit(X_train_scaled, y_train, epochs=2)\n",
    "y_pred = model.predict(X_test_scaled)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93a60ee9-a730-4caa-b3ae-bac6a11f5198",
   "metadata": {},
   "source": [
    "Similarly, we can add a custom metric model on the models internals by computing it in any way we want, as long as the result is the output of a metric object. For example: we can create a `keras.metric.Mean` object in the constructor, then call it in the `call()` method, passing it the `recon_loss`, and finally add it to the model by calling the model's `add_metric()` method. This way, when we train the model, Keras will display both the mean loss over each epoch and the mean reconstruction error over each epoch. Both will go down during training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "84efed01-c8c0-49f4-9359-167ce2fbbb4b",
   "metadata": {},
   "outputs": [],
   "source": [
    "class ReconstructingRegressor(keras.Model):\n",
    "    def __init__(self, output_dim, **kwargs):\n",
    "        \"\"\"\n",
    "        Constructor creates the DNN with 5 dense hidden layers and one dense output layer\n",
    "        \"\"\"\n",
    "        super().__init__(**kwargs)\n",
    "        self.hidden = [keras.layers.Dense(30, activation=\"selu\", kernel_initializer=\"lecun_normal\")\n",
    "                      for _ in range(5)]\n",
    "        self.out = keras.layers.Dense(output_dim)\n",
    "        self.reconstruction_mean = keras.metrics.Mean(name=\"reconstruction_error\")\n",
    "        \n",
    "    def build(self, batch_input_shape):\n",
    "        \"\"\"\n",
    "        This method creates an extra dense layer which will be used to reconstruct the inputs to the\n",
    "        model. \n",
    "        \"\"\"\n",
    "        n_inputs = batch_input_shape[-1]\n",
    "        self.reconstruct = keras.layers.Dense(n_inputs)\n",
    "        # super().build(batch_input_shape) #Due to an issue introduced in TF 2.2 (#46858), we must not call super().build() inside the build() method.\n",
    "    \n",
    "    def call(self, inputs, training=None):\n",
    "        \"\"\"\n",
    "        This method processes the inputs through all five hidden layers, then passes the result to\n",
    "        the reconstruction layer, which produces the reconstruction. \n",
    "        \"\"\"\n",
    "        Z = inputs\n",
    "        for layer in self.hidden:\n",
    "            Z = layer(Z)\n",
    "        reconstruction = self.reconstruct(Z)\n",
    "        \"\"\"\n",
    "        Then call method computes the reconstruction loss and adds it to the model's list of losses\n",
    "        using `add_loss()` method (We can also add add_loss on any layer inside the model, as the\n",
    "        model recursively gathers losses from all its layers. )\n",
    "        \"\"\"\n",
    "        recon_loss = tf.reduce_mean(tf.square(reconstruction - inputs))\n",
    "        \"\"\"\n",
    "        We scaled down the reconstruction loss by multiplying it by 0.05 (this is hyperparameter we\n",
    "        can tune). This ensures that the reconstruction loss does not dominate the main loss. \n",
    "        \"\"\"\n",
    "        self.add_loss(0.05 * recon_loss)\n",
    "        if training:\n",
    "            result = self.reconstruction_mean(recon_loss)\n",
    "            self.add_metric(result)\n",
    "        return self.out(Z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "d38f6e53-d39e-4beb-8fc9-9e9114e1fefc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "363/363 [==============================] - 2s 1ms/step - loss: 0.8096 - reconstruction_error: 0.7123\n",
      "Epoch 2/2\n",
      "363/363 [==============================] - 1s 1ms/step - loss: 0.3985 - reconstruction_error: 0.3447\n",
      "162/162 [==============================] - 0s 760us/step\n"
     ]
    }
   ],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "model = ReconstructingRegressor(1)\n",
    "model.compile(loss=\"mse\", optimizer=\"nadam\")\n",
    "h = model.fit(X_train_scaled, y_train, epochs=2)\n",
    "y_pred = model.predict(X_test_scaled)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4421916b-95e6-42e0-9219-322c10018744",
   "metadata": {},
   "source": [
    "### Computing Gradients Using Autodiff"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d0487e5-97ce-44a1-a4a3-aff14ad3f634",
   "metadata": {},
   "source": [
    "To compute gradients automatically, let's consider a simple toy function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "4e388945-b5d7-4e64-b48a-317c77da59b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(w1, w2):\n",
    "    return 3 * w1 ** 2 + 2 * w1 * w2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c8c5960-0343-4066-b1f6-1e1b2830408d",
   "metadata": {},
   "source": [
    "The partial derivative of this function wrt to `w1` is `6*w1 + 2*w2` and wrt to `w2` is `2 * w1`. For example, at point `(w1, w2)` = `(5, 3)`, these partial derivatives are equal to `36` and `10` respectively, so the gradient vector at this point is `(36, 10)`. But if this were a neural network, the function would be much more complex, typically with tens of thousands of parameters, and finding the partial derivatives analytically by hand would be an almost impossible task, \n",
    "\n",
    "One solution could be to compute and approximation of each partial derivative by measuring how much the function's output changes when we tweak the corresponding parameter:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "5c5d3d5e-5ac6-43fb-baae-c006d04a6a3d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "36.000003007075065"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w1, w2 = 5, 3\n",
    "eps = 1e-6\n",
    "(f(w1+eps, w2) - f(w1,w2)) / eps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "1aecfa80-0bf7-4b1b-bd47-c53188f352a3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10.000000003174137"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(f(w1, w2+eps) - f(w1, w2)) / eps"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aae9c2ed-ca82-4369-bdf7-f1603a4396fc",
   "metadata": {},
   "source": [
    "Looks about right! This works rather well and is easy to implement, but it is just an approximation, and importantly we need to call `f()` at least once per parameter (not twice, as we could compute `f(w1,w2)` just once). Needing to call `f()` at least once per parameter makes this approach intractable for large neural networks.\n",
    "\n",
    "So instead, we should use autodiff. TensorFlow makes this pretty simple:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "b05713f0-f851-421b-ad9e-ee99b3753605",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Tensor: shape=(), dtype=float32, numpy=36.0>,\n",
       " <tf.Tensor: shape=(), dtype=float32, numpy=10.0>]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w1, w2 = tf.Variable(5.), tf.Variable(3.)\n",
    "with tf.GradientTape() as tape:\n",
    "    z = f(w1, w2)\n",
    "    \n",
    "gradients = tape.gradient(z, [w1, w2])\n",
    "gradients"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df6d046b-ebbd-49f7-a81d-d4c8a3158796",
   "metadata": {},
   "source": [
    "We first define two variables `w1` and `w2`, then we create a `tf.GradientTape` context that will automatically record every operation that involves a variable, and finally we ask this tape to compute the gradient of the result `z` wrt to both variables `[w1, w2]`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15fa0305-af27-4f60-a541-05004f3bd8cd",
   "metadata": {},
   "source": [
    "The tape is automatically erased immediately after we call its `gradient()` method, so we will get an exception if we try to call `gradient()` twice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "c019d985-e9ee-47f7-8a20-76f14a8f94a5",
   "metadata": {},
   "outputs": [
    {
     "ename": "RuntimeError",
     "evalue": "A non-persistent GradientTape can only be used to compute one set of gradients (or jacobians)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[20], line 5\u001b[0m\n\u001b[1;32m      2\u001b[0m     z \u001b[38;5;241m=\u001b[39m f(w1, w2)\n\u001b[1;32m      4\u001b[0m dz_dw1 \u001b[38;5;241m=\u001b[39m tape\u001b[38;5;241m.\u001b[39mgradient(z, [w1])\n\u001b[0;32m----> 5\u001b[0m dz_dw2 \u001b[38;5;241m=\u001b[39m tape\u001b[38;5;241m.\u001b[39mgradient(z, [w2])\n",
      "File \u001b[0;32m~/anaconda3/envs/latest/lib/python3.11/site-packages/tensorflow/python/eager/backprop.py:1003\u001b[0m, in \u001b[0;36mGradientTape.gradient\u001b[0;34m(self, target, sources, output_gradients, unconnected_gradients)\u001b[0m\n\u001b[1;32m    963\u001b[0m \u001b[38;5;250m\u001b[39m\u001b[38;5;124;03m\"\"\"Computes the gradient using operations recorded in context of this tape.\u001b[39;00m\n\u001b[1;32m    964\u001b[0m \n\u001b[1;32m    965\u001b[0m \u001b[38;5;124;03mNote: Unless you set `persistent=True` a GradientTape can only be used to\u001b[39;00m\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m   1000\u001b[0m \u001b[38;5;124;03m   called with an unknown value.\u001b[39;00m\n\u001b[1;32m   1001\u001b[0m \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m   1002\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_tape \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m-> 1003\u001b[0m   \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mRuntimeError\u001b[39;00m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mA non-persistent GradientTape can only be used to \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m   1004\u001b[0m                      \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mcompute one set of gradients (or jacobians)\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m   1005\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_recording:\n\u001b[1;32m   1006\u001b[0m   \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_persistent:\n",
      "\u001b[0;31mRuntimeError\u001b[0m: A non-persistent GradientTape can only be used to compute one set of gradients (or jacobians)"
     ]
    }
   ],
   "source": [
    "with tf.GradientTape() as tape:\n",
    "    z = f(w1, w2)\n",
    "    \n",
    "dz_dw1 = tape.gradient(z, [w1])\n",
    "dz_dw2 = tape.gradient(z, [w2])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59f1d77a-eb3f-40d7-aa09-0a83cbc37db6",
   "metadata": {},
   "source": [
    "If we need to call `gradient()` more than once, we must make the tape persistant and delete it each time we are done with it to free resources. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "2764f080-f191-42fe-beff-a412f9d13800",
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.GradientTape(persistent=True) as tape:\n",
    "    z = f(w1, w2)\n",
    "\n",
    "dz_dw1 = tape.gradient(z, [w1])\n",
    "dz_dw2 = tape.gradient(z, [w2])\n",
    "del tape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "2a208e44-96aa-40a8-96b8-048cbb2b966f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([<tf.Tensor: shape=(), dtype=float32, numpy=36.0>],\n",
       " [<tf.Tensor: shape=(), dtype=float32, numpy=10.0>])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dz_dw1, dz_dw2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "537961b2-6f73-49f9-acd8-e89b2504c839",
   "metadata": {},
   "source": [
    "By default, the tape will only track operations involving variables, so if we try to compute the gradient of `z` wrt anything other than variable, the result will be `None`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "7a567700-bcca-4001-8173-ae5b986b8350",
   "metadata": {},
   "outputs": [],
   "source": [
    "c1, c2 = tf.Variable(5.) , tf.Variable(3.)\n",
    "with tf.GradientTape() as tape:\n",
    "    z = f(c1, c2)\n",
    "\n",
    "gradients = tape.gradient(z, [w1, w2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "f86d90c8-fada-44da-b40a-22e681223fa1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[None, None]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gradients"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4c9facd-be41-4639-bc53-3d01ff2e3185",
   "metadata": {},
   "source": [
    "However, we can force the tape to watch any tensors we like, to record every operation that involves them. We can then compute gradients wrt these tensors, as if they were variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "70ca2ed3-7d24-48d1-be37-0419fe3dd69b",
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.GradientTape() as tape:\n",
    "    tape.watch(c1)\n",
    "    tape.watch(c2)\n",
    "    z = f(c1, c2)\n",
    "\n",
    "gradients = tape.gradient(z, [c1, c2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "a7e03407-8d66-4c76-9a9e-f4f9d321cbc2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Tensor: shape=(), dtype=float32, numpy=36.0>,\n",
       " <tf.Tensor: shape=(), dtype=float32, numpy=10.0>]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gradients"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae1d6c21-cdf0-4f3b-8a86-b7671bbcccb7",
   "metadata": {},
   "source": [
    "In some cases, we may need to stop gradients from backpropogating through some part of our neural network. To do this, we must use the `tf.stop_gradient()` function. The function returns its inputs during the forward pass (like `tf.identity()`), but it does not let gradients through backpropogation (it acts like a constant):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "769e22ba-4c56-4f79-afca-646ecbc60441",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Tensor: shape=(), dtype=float32, numpy=30.0>, None]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def f(w1, w2):\n",
    "    return 3 * w1 ** 2 + tf.stop_gradient(2 * w1 * w2)\n",
    "\n",
    "with tf.GradientTape() as tape:\n",
    "    z = f(w1, w2)\n",
    "    \n",
    "gradients = tape.gradient(z, [w1, w2])\n",
    "gradients"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0912f8a-44f2-42c9-b988-ddf5346c2867",
   "metadata": {},
   "source": [
    "Finally, we may occasionally run into some numerical issues when computing gradients. For example, if we compute the gradients of the `my_softplus()` function for large inputs, the result will be `NaN`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "78a3ddad-978a-4620-b874-a3302c306690",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Tensor: shape=(1,), dtype=float32, numpy=array([nan], dtype=float32)>]"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = tf.Variable([100.])\n",
    "with tf.GradientTape() as tape:\n",
    "    z = my_softplus(x)\n",
    "\n",
    "tape.gradient(z, [x])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c831bfa8-2740-4800-8b91-c46f6cf1599c",
   "metadata": {},
   "source": [
    "This is because computing the gradient of this function using autodiff leads to some numerical difficulties: due to floating-point precision errors, autodiff ends up computing infinity divided by infinity (which return NaN). \n",
    "\n",
    "Analytically, the derivative of the softplus function is just\n",
    "$$\n",
    "\\frac{e^x}{1+e^x} = \\frac{1}{1 + \\frac{1}{e^x}}\n",
    "$$\n",
    ", which is numerically stable. Next, we can tell TensorFlow to use this stable function when computing the gradients of the `my_softplus()` function by decorating it with `@tf.custom_gradient` and making it return both its normal output and the function that computes the derivatives (note that it will receive as inputs the gradients that were backpropogated so far, down to the softplus function; and according to the chain rule, we should multiply them with this function's gradients):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "bf0a8a73-85d7-4e30-b035-b1c7f09b1b18",
   "metadata": {},
   "outputs": [],
   "source": [
    "@tf.custom_gradient\n",
    "def my_better_softplus(z):\n",
    "    exp = tf.exp(z)\n",
    "    def my_softplus_gradients(grad):\n",
    "        return grad / (1 + 1/exp)\n",
    "    return tf.math.log(exp + 1), my_softplus_gradients"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9c06064-98c3-45df-83bc-0a4960a9faec",
   "metadata": {},
   "source": [
    "Now when we compute the gradients of the `my_better_softplus()`\n",
    "function, we get the proper result, even for large input values (however,\n",
    "the main output still explodes because of the exponential; one workaround\n",
    "is to use `tf.where()` to return the inputs when they are large)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "2afd10b5-5002-41ae-8976-96ee97aca4dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "def my_better_softplus(z):\n",
    "    return tf.where(z>30, z, tf.math.log(tf.exp(z) + 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "e02b21b9-c70d-4c0d-a4c9-21d8276a594c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<tf.Tensor: shape=(1,), dtype=float32, numpy=array([1000.], dtype=float32)>,\n",
       " [<tf.Tensor: shape=(1,), dtype=float32, numpy=array([nan], dtype=float32)>])"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = tf.Variable([1000.])\n",
    "with tf.GradientTape() as tape:\n",
    "    z = my_better_softplus(x)\n",
    "    \n",
    "z, tape.gradient(z, [x])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2a1ab33-4770-4f06-a19f-04ff90d4858e",
   "metadata": {},
   "source": [
    "### Custom Training Loops"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bcfdfe9-a7a9-452f-84ef-0a3a3f1b2b41",
   "metadata": {},
   "source": [
    "In some rare case, the `fit()` method may not be flexible enough for what we need to do. For example, the Wide and Deep paper , uses two different optimizers: one for wide path and the other for deep path. Since the `fit()` method only uses one optimizer (the one that we specify when compiling the model), implementing this paper requires writing our custom loop. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d4897fe-a1e9-48fd-82a1-275c09f74538",
   "metadata": {},
   "source": [
    "First, let's build a simple model. No need to compile it, since we will handle the training loop manually:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "ec2f9225-f28b-431c-bab9-d17286248de7",
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)\n",
    "\n",
    "l2_reg = keras.regularizers.l2(0.05)\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"relu\", kernel_initializer=\"he_normal\",\n",
    "                      kernel_regularizer=l2_reg),\n",
    "    keras.layers.Dense(1, kernel_regularizer=l2_reg)\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1f576a4-28b0-4cee-9bed-ec5c949369e0",
   "metadata": {},
   "source": [
    "Next let's create a tiny function that will randomly sample a batch of instances from the training set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "53c46782-c1de-48a0-8a05-699f1413ea8f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def random_batch(X, y, batch_size=32):\n",
    "    idx = np.random.randint(len(X), size=batch_size)\n",
    "    return X[idx], y[idx]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3867ac18-f40a-4eae-94c3-f2d02c85aa03",
   "metadata": {},
   "source": [
    "Let's also define a function that will display the training status, including the number of steps, the total number of steps, the mean loss since the start of epoch (i.e., we will use `Mean` metric to compute it), and other metrics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "53bc9a93-8406-4745-856a-176b2eeed867",
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_status_bar(iteration, total, loss, metrics=None):\n",
    "    metrics = \" - \".join([\"{}: {:.4f}\".format(m.name, m.result()) \n",
    "                          for m in [loss] + (metrics or None)])\n",
    "    end = \"\" if iteration < total else \"\\n\"\n",
    "    print(\"\\r{}/{} - \".format(iteration, total) + metrics, end=end)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "e73a1a0b-dcf2-4739-972c-b7f60293fa93",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50/50 - loss: 0.0900 - mean_square: 858.5000\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "mean_loss = keras.metrics.Mean(name=\"loss\")\n",
    "mean_square = keras.metrics.Mean(name=\"mean_square\")\n",
    "\n",
    "for i in range(1, 50 + 1):\n",
    "    loss = 1 / i\n",
    "    mean_loss(loss)\n",
    "    mean_square(i**2)\n",
    "    print_status_bar(i, 50, mean_loss, [mean_square])\n",
    "    time.sleep(0.05)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c349903e-4e46-4618-a4a3-57c175928d43",
   "metadata": {},
   "source": [
    "With that, let's get down to business! First, we need to define some hyperparameters and choose the optimizer, the loss function, and the metrics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "9f0e513f-0a62-478f-bb6b-24dac0ee8393",
   "metadata": {},
   "outputs": [],
   "source": [
    "n_epochs = 5\n",
    "batch_size = 32\n",
    "n_steps = len(X_train) // batch_size\n",
    "optimizer = keras.optimizers.Nadam(learning_rate=0.01)\n",
    "loss_fn = keras.losses.mean_squared_error\n",
    "mean_loss = keras.metrics.Mean()\n",
    "metrics = [keras.metrics.MeanAbsoluteError()]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41151e69-9b75-40cf-ab63-71a4030b9845",
   "metadata": {},
   "source": [
    "Now we are ready to build the custom loop!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "9ed56db2-ac1b-4812-b3d2-da36afe365b2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "11610/11610 - mean: 0.6380 - mean_absolute_error: 0.5147\n",
      "Epoch 2/5\n",
      "11610/11610 - mean: 0.6239 - mean_absolute_error: 0.5083\n",
      "Epoch 3/5\n",
      "11610/11610 - mean: 0.6331 - mean_absolute_error: 0.5147\n",
      "Epoch 4/5\n",
      "11610/11610 - mean: 0.6270 - mean_absolute_error: 0.5126\n",
      "Epoch 5/5\n",
      "11610/11610 - mean: 0.6552 - mean_absolute_error: 0.5212\n"
     ]
    }
   ],
   "source": [
    "\"\"\"\n",
    "We created two nested loops: one for epochs, the other for the batches within the epochs.\n",
    "Then we sample a random batch from the training set.\n",
    "\"\"\"\n",
    "\n",
    "for epoch in range(1, n_epochs + 1):\n",
    "    print(\"Epoch {}/{}\".format(epoch, n_epochs))\n",
    "    for step in range(1, n_steps + 1):\n",
    "        X_batch, y_batch = random_batch(X_train_scaled, y_train)\n",
    "        \"\"\"\n",
    "        Inside tf.GradientTape() we make a prediction for one batch (using model as a function),\n",
    "        and we compute the loss: it is equal to main loss plus other losses\n",
    "        \"\"\"\n",
    "        with tf.GradientTape() as tape:\n",
    "            y_pred = model(X_batch, training=True)\n",
    "            \"\"\"\n",
    "            Since the `mean_squared_error()` function returns one loss per instance, we compute\n",
    "            the mean over the batch using tf.reduce_mean() (if we wanted to apply different weights\n",
    "            to each instance, this is where we would do it). \n",
    "            \"\"\"\n",
    "            main_loss = tf.reduce_mean(loss_fn(y_batch, y_pred))\n",
    "            \"\"\"\n",
    "            add_n => returns element wise sum of list of tensors\n",
    "            For ex:\n",
    "            >>> a = tf.constant([[3, 5], [4, 8]])\n",
    "            >>> b = tf.constant([[1, 6], [2, 9]])\n",
    "            >>> tf.math.add_n([a, b, a]).numpy()\n",
    "            output: [[7, 16],\n",
    "                     [10,25]]\n",
    "            \"\"\"\n",
    "            loss = tf.add_n([main_loss] + model.losses)\n",
    "            \n",
    "        \"\"\"\n",
    "        Next we ask the tape to compute the gradient of the loss wrt to each trainable variable\n",
    "        (not all variables!), and we apply them to optimizer to perform a GD step.\n",
    "        \"\"\"\n",
    "        gradients = tape.gradient(loss, model.trainable_variables)\n",
    "        optimizer.apply_gradients(zip(gradients, model.trainable_variables))\n",
    "        \"\"\"\n",
    "        Then we update the mean loss and the metrics (over the current epoch) and we display the\n",
    "        status bar. \n",
    "        \"\"\"\n",
    "        mean_loss(loss)\n",
    "        for metric in metrics:\n",
    "            metric(y_batch, y_pred)\n",
    "        print_status_bar(step * batch_size, len(y_train), mean_loss, metrics)\n",
    "    \n",
    "    \"\"\"\n",
    "    At the end of each epoch, we display the status bar again to make it look complete and to\n",
    "    print a line feed, and we reset the stats of the mean loss and metrics\n",
    "    \"\"\"\n",
    "    print_status_bar(len(y_train), len(y_train), mean_loss, metrics)\n",
    "    for metric in [mean_loss] + metrics:\n",
    "        metric.reset_states()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34f3a5b3-674d-4d95-815a-3a864e7379ff",
   "metadata": {},
   "source": [
    "If we want to apply any other transformation to the gradients, simply do so before calling the `apply_gradients()` method.\n",
    "\n",
    "If we add weight constraints to our model (e.g., by setting `kernel_constraint` or `bias_constraint` when creating a layer), we should update the training loop to apply these constraints just after `apply_gradients()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "19e8e25b-7b8b-4b99-996e-edd9a2298f47",
   "metadata": {},
   "outputs": [],
   "source": [
    "for variable in model.variables:\n",
    "    if variable.constraint is not None:\n",
    "        variable.assign(variable.constraint(variable))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5dd37b55-9064-4612-94dc-c249e3798bb0",
   "metadata": {},
   "source": [
    "## TensorFlow Functions and Graphs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff45342a-35e0-4465-a22d-684dcd1ef323",
   "metadata": {},
   "source": [
    "Let's start with a trivial function that computes the cube of its input:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "865088e7-900a-49e0-89e4-af64feacf996",
   "metadata": {},
   "outputs": [],
   "source": [
    "def cube(x):\n",
    "    return x**3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "2dcbc0e8-8180-49d4-9979-9a4c2a5bf440",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cube(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "194ab597-26e7-4f6a-8984-20a0d3485d12",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=int32, numpy=8>"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cube(tf.constant(2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdf7ba6f-a05b-47c7-95ba-0bfd3a4fa5b3",
   "metadata": {},
   "source": [
    "Now let's use `tf.function()` to convert this Python function to a `TensorFlow Function`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "789cc354-6971-415a-aff3-8ddad2140b52",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.eager.polymorphic_function.polymorphic_function.Function at 0x7fad30d2bc50>"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf_cube = tf.function(cube)\n",
    "tf_cube"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0cf08bde-26fc-47dd-89be-17a4791776b3",
   "metadata": {},
   "source": [
    "This TF Function can then be used exactly like the original Python function, and it will return the same results (but as tensors):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "a8e39f19-46ea-467b-b74c-a83ed5e09e8c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=int32, numpy=8>"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf_cube(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "8d5b8148-692f-4a14-9527-26857b206ee3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=27.0>"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf_cube(tf.constant(3.0))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be718a35-2b3f-48f0-aac0-cd5471734d81",
   "metadata": {},
   "source": [
    "Under the hood, `tf.function()` analyzed the computations performed by the `cube()` function and generated an equivalent computation graph!\n",
    "\n",
    "Alternatively, we could have used `tf.function` as a decorator; this is actually more common:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "ccd1bf0e-5ea5-4c03-9f9a-9b4ac2c98f44",
   "metadata": {},
   "outputs": [],
   "source": [
    "@tf.function\n",
    "def tf_cube(x):\n",
    "    return x**3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c157451-1c31-49ee-9d7a-8acc3ed38451",
   "metadata": {},
   "source": [
    "The original Python function is still available via the TF Function's `python_function` attribute, in case we ever need it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "27d746c2-1e3a-4a4b-82b2-7b9af49b6307",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf_cube.python_function(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6ae838a-e43e-43b8-8cdc-118d06d08225",
   "metadata": {},
   "source": [
    "**TIP:**\n",
    "\n",
    "We can tell Keras *not* to convert our Python Functions to TF Functions by setting `dynamic=True` when creating a custom layer or a custom model. Alternatively, we can set `run_eagerly=True` when calling the model's `compile()` method."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a54c7510-b968-4677-a7d3-a7c31caf3956",
   "metadata": {},
   "source": [
    "By default, a TF Function generates a new graph for every unique set of input shapes and data types and caches it for subsequent calls. \n",
    "\n",
    "For ex., if we call `tf_cube(tf.constant(10))`, a graph will be generated for int32 tensors of shape []. Then if we call `tf_cube(tf.constant(20))`, the same graph will be reused. But if we call `tf_cube(tf.constant([10,20]))`, a new graph will be generated for int32 tensors of shape [2]. This is how TF Functions handle polymorphism (i.e., varying arguments types and shapes). However, this is only true for tensor arguments: if we pass numerical Python values to a TF Function, a new graph will be generated for every distinct value: for eg., calling `tf_cube(10)` and `tf_cube(20)` will generate two graphs."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2839df0e-8769-4b05-b113-78535cd0f911",
   "metadata": {},
   "source": [
    "**Warning:**\n",
    "\n",
    "If we call a TF Function many times with different numerical Python values, then many graphs will be generated, slowing down our program and using up a lot of `RAM` (we must delete the TF Function to release it). Python values should be reserved for arguments that will have few unique values, such as hyperparameters like the number of neurons per layer. This allows TF to better optimize each variant of our model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96744fbf-9b80-4060-8b6f-ef495e2cb2a8",
   "metadata": {},
   "source": [
    "**TIP:**\n",
    "\n",
    "To view the generated function's source code, we will call `tf.autograph.to_code(sum_squares.python_function)`. The code is not meant to be pretty, but it can sometimes help for debugging."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e89c8b00-5b16-477c-9e5d-a88efdb95e48",
   "metadata": {},
   "source": [
    "## Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6f379e8-6328-457f-994d-497bcf758e02",
   "metadata": {},
   "source": [
    "#### 12.\n",
    "\n",
    "Implement a custom layer that performs *Layer Normalization*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f1b4829-4d63-44ed-a06f-29cc60940ed9",
   "metadata": {},
   "source": [
    "##### a. \n",
    "\n",
    "The `build()` method should define two trainable weights $\\alpha$ and $\\beta$, both of shape `input_shape[-1:]` and data type `tf.f`\n",
    "\n",
    "##### b. \n",
    "\n",
    "The `call()` method should compute the mean $\\mu$ and\n",
    "standard deviation $\\sigma$ of each instance’s features. For this,\n",
    "you can use `tf.nn.moments(inputs, axes=-1,\n",
    "keepdims=True)`, which returns the mean $\\mu$ and the\n",
    "variance $\\sigma^2$ of all instances (compute the square root of\n",
    "the variance to get the standard deviation). Then the\n",
    "function should compute and return $\\alpha \\cdot \\frac{(X - \\mu)}{(\\sigma + \\epsilon)} + \\beta$, where $\\cdot$ represents itemwise multiplication (*) and $\\epsilon$\n",
    "is a smoothing term (small constant to avoid division by\n",
    "zero, e.g., 0.001)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "4293afd2-90c8-4420-8ba4-84cb70a23503",
   "metadata": {},
   "outputs": [],
   "source": [
    "class LayerNormalization(keras.layers.Layer):\n",
    "    def __init__(self, eps=0.001, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.eps = eps\n",
    "    \n",
    "    def build(self, batch_input_shape):\n",
    "        self.alpha = self.add_weight(name=\"alpha\", shape=batch_input_shape[-1:], initializer=\"ones\")\n",
    "        self.beta = self.add_weight(name=\"alpha\", shape=batch_input_shape[-1:], initializer=\"zeros\")\n",
    "        super().build(batch_input_shape) # must be at end\n",
    "    \n",
    "    def call(self, X):\n",
    "        mean, variance = tf.nn.moments(X, axes=-1, keepdims=True)\n",
    "        return self.alpha * (X - mean) / (tf.sqrt(variance + self.eps)) + self.beta\n",
    "    \n",
    "    def compute_output_shape(self, batch_input_shape):\n",
    "        return batch_input_shape\n",
    "    \n",
    "    def get_config(self):\n",
    "        base_config = super().get_config()\n",
    "        return {**base_config, \"eps\": self.eps}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fcf7b720-13c4-4f34-9e85-ffde2a4d7839",
   "metadata": {},
   "source": [
    "**NOTE:**\n",
    "\n",
    "That making $\\epsilon$ was not compulsory. Also it is preferable to compute `tf.sqrt(variance + self.eps)` rather than `tf.sqrt(variance) + self.eps` because square root of x is undefined at x=0, so training will bomb whenever the variance vector has atleast one component equal to 0. Adding $\\epsilon$ within the square root guarantees that this will never happen."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f470c871-7587-4ea7-bbef-cb55d3a39501",
   "metadata": {},
   "source": [
    "##### c. \n",
    "Ensure that your custom layer produces the same (or very\n",
    "nearly the same) output as the\n",
    "`keras.layers.LayerNormalization` layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "7a8d102b-af8c-421e-9726-591d18722e9d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=3.9782837e-08>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = X_train.astype(np.float32)\n",
    "\n",
    "custom_layer_norm = LayerNormalization()\n",
    "keras_layer_norm = keras.layers.LayerNormalization()\n",
    "\n",
    "tf.reduce_mean(keras.losses.mean_absolute_error(keras_layer_norm(X), custom_layer_norm(X)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c027757d-e891-4ee6-97c2-7a6e55ef7d51",
   "metadata": {},
   "source": [
    "Yep, that's close enough. To be extra sure, let's make alpha and beta completely random and compare again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "0ebfa5d8-2e3d-46d1-b691-283552b0d985",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=3.046251e-08>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random_alpha = np.random.rand(X.shape[-1])\n",
    "random_beta = np.random.rand(X.shape[-1])\n",
    "\n",
    "custom_layer_norm.set_weights([random_alpha, random_beta])\n",
    "keras_layer_norm.set_weights([random_alpha, random_beta])\n",
    "\n",
    "tf.reduce_mean(keras.losses.mean_absolute_error(keras_layer_norm(X), custom_layer_norm(X)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5de8539-5496-47fd-b1e9-8e03973db380",
   "metadata": {},
   "source": [
    "Still negligible difference. Our custom layer works fine!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9ad8ccc-4b35-42ac-9eb5-5136aa46c899",
   "metadata": {},
   "source": [
    "#### 13.\n",
    "\n",
    "Train a model using a custom training loop to tackle the Fashion\n",
    "MNIST dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e66bd5c8-a190-4161-874c-5e798332c91f",
   "metadata": {},
   "source": [
    "##### a.\n",
    "\n",
    "Display the epoch, iteration, mean training loss, and\n",
    "mean accuracy over each epoch (updated at each\n",
    "iteration), as well as the validation loss and accuracy at\n",
    "the end of each epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "a4166f1d-8f55-4517-a2fc-20254ed2c233",
   "metadata": {},
   "outputs": [],
   "source": [
    "(X_train_full, y_train_full), (X_test, y_test) = keras.datasets.fashion_mnist.load_data()\n",
    "X_train_full = X_train_full.astype(np.float32) / 255\n",
    "X_valid, X_train = X_train_full[:5000], X_train_full[5000: ]\n",
    "y_valid, y_train = y_train_full[:5000], y_train_full[5000:]\n",
    "X_test = X_test.astype(np.float32) / 255"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "7010dd8e-9ffe-4dde-beda-542170832442",
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "tf.random.set_seed(42)\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "adfe1e86-5e8d-4f43-9953-9a5a01189f48",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28,28]),\n",
    "    keras.layers.Dense(100, activation=\"relu\"),\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "150dc870-6dd6-42fc-940c-b8d3fe418297",
   "metadata": {},
   "outputs": [],
   "source": [
    "n_epochs = 5\n",
    "batch_size = 32\n",
    "n_steps = len(X_train) // batch_size\n",
    "optimizer = keras.optimizers.Nadam(learning_rate=0.01)\n",
    "loss_fn = keras.losses.sparse_categorical_crossentropy\n",
    "mean_loss = keras.metrics.Mean()\n",
    "metrics = [keras.metrics.SparseCategoricalAccuracy()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "23b77414-6287-4b86-b529-5071b74122ba",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Epoch 1/5: 100%|█| 1718/1718 [00:55<00:00, 30.84it/s, loss=0.469, sparse_categor\n",
      "2024-01-07 21:27:22.944152: W tensorflow/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 15680000 exceeds 10% of free system memory.\n",
      "Epoch 2/5: 100%|█| 1718/1718 [00:57<00:00, 29.82it/s, loss=0.398, sparse_categor\n",
      "2024-01-07 21:28:20.582380: W tensorflow/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 15680000 exceeds 10% of free system memory.\n",
      "Epoch 3/5: 100%|█| 1718/1718 [00:57<00:00, 30.14it/s, loss=0.374, sparse_categor\n",
      "2024-01-07 21:29:17.612594: W tensorflow/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 15680000 exceeds 10% of free system memory.\n",
      "Epoch 4/5: 100%|█| 1718/1718 [00:57<00:00, 30.05it/s, loss=0.375, sparse_categor\n",
      "2024-01-07 21:30:14.812017: W tensorflow/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 15680000 exceeds 10% of free system memory.\n",
      "Epoch 5/5: 100%|█| 1718/1718 [00:57<00:00, 29.78it/s, loss=0.36, sparse_categori\n",
      "2024-01-07 21:31:12.515550: W tensorflow/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 15680000 exceeds 10% of free system memory.\n",
      "All epochs: 100%|█████████████████████████████████| 5/5 [04:45<00:00, 57.06s/it]\n"
     ]
    }
   ],
   "source": [
    "from tqdm import trange\n",
    "from collections import OrderedDict\n",
    "\n",
    "with trange(1, n_epochs + 1, desc=\"All epochs\", position=0, leave=True) as epochs:\n",
    "    for epoch in epochs:\n",
    "        with trange(1, n_steps + 1, desc=\"Epoch {}/{}\".format(epoch, n_epochs), position=0, \n",
    "                    leave=True) as steps:\n",
    "            for step in steps:\n",
    "                X_batch, y_batch = random_batch(X_train, y_train)\n",
    "                with tf.GradientTape() as tape:\n",
    "                    y_pred = model(X_batch)\n",
    "                    main_loss = tf.reduce_mean(loss_fn(y_batch, y_pred))\n",
    "                    loss = tf.add_n([main_loss] + model.losses)\n",
    "                gradients = tape.gradient(loss, model.trainable_variables)\n",
    "                optimizer.apply_gradients(zip(gradients, model.trainable_variables))\n",
    "                for variable in model.variables:\n",
    "                    if variable.constraint is not None:\n",
    "                        variable.assign(variable.constraint(variable))                    \n",
    "                status = OrderedDict()\n",
    "                mean_loss(loss)\n",
    "                status[\"loss\"] = mean_loss.result().numpy()\n",
    "                for metric in metrics:\n",
    "                    metric(y_batch, y_pred)\n",
    "                    status[metric.name] = metric.result().numpy()\n",
    "                steps.set_postfix(status)\n",
    "            y_pred = model(X_valid)\n",
    "            status[\"val_loss\"] = np.mean(loss_fn(y_valid, y_pred))\n",
    "            status[\"val_accuracy\"] = np.mean(keras.metrics.sparse_categorical_accuracy(\n",
    "                tf.constant(y_valid, dtype=np.float32), y_pred))\n",
    "            steps.set_postfix(status)\n",
    "        for metric in [mean_loss] + metrics:\n",
    "            metric.reset_states()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "1ae245db-d3ad-4acd-9a1e-44f5d8b048bb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('loss', 0.35977197),\n",
       "             ('sparse_categorical_accuracy', 0.86867),\n",
       "             ('val_loss', 0.41471246),\n",
       "             ('val_accuracy', 0.8698)])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "status"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7df09ea7-ee21-43f3-9153-468f0301c293",
   "metadata": {},
   "source": [
    "##### b. \n",
    "\n",
    "Try using a different optimizer with a different learning\n",
    "rate for the upper layers and the lower layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "04e76f77-34e5-4a49-8b91-f8afcee721e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "lower_layers = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[28, 28]),\n",
    "    keras.layers.Dense(100, activation=\"relu\")\n",
    "])\n",
    "\n",
    "upper_layers = keras.models.Sequential([\n",
    "    keras.layers.Dense(10, activation=\"softmax\")\n",
    "])\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    lower_layers, upper_layers\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "029fc819-5227-4923-b454-cc57ff828875",
   "metadata": {},
   "outputs": [],
   "source": [
    "lower_optimizer = keras.optimizers.SGD(learning_rate=1e-4)\n",
    "upper_optimizer = keras.optimizers.Nadam(learning_rate=1e-3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "cdf66384-e7b2-4303-8819-d83239db893a",
   "metadata": {},
   "outputs": [],
   "source": [
    "n_epochs = 5\n",
    "batch_size = 16\n",
    "n_steps = len(X_train) // batch_size\n",
    "loss_fn = keras.losses.sparse_categorical_crossentropy\n",
    "mean_loss = keras.metrics.Mean()\n",
    "metrics = [keras.metrics.SparseCategoricalAccuracy()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "1116f3c9-b624-4f72-9914-f8148d3235e8",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Epoch 1/5: 100%|█| 3437/3437 [01:31<00:00, 37.64it/s, loss=0.841, sparse_categor\n",
      "Epoch 2/5: 100%|█| 3437/3437 [01:35<00:00, 35.80it/s, loss=0.562, sparse_categor\n",
      "Epoch 3/5: 100%|█| 3437/3437 [01:35<00:00, 35.82it/s, loss=0.521, sparse_categor\n",
      "Epoch 4/5: 100%|█| 3437/3437 [01:36<00:00, 35.58it/s, loss=0.499, sparse_categor\n",
      "Epoch 5/5: 100%|█| 3437/3437 [01:37<00:00, 35.37it/s, loss=0.476, sparse_categor\n",
      "All epochs: 100%|█████████████████████████████████| 5/5 [07:57<00:00, 95.43s/it]\n"
     ]
    }
   ],
   "source": [
    "from tqdm import trange\n",
    "from collections import OrderedDict\n",
    "\n",
    "with trange(1, n_epochs + 1, desc=\"All epochs\", position=0, leave=True) as epochs:\n",
    "    for epoch in epochs:\n",
    "        with trange(1, n_steps + 1, desc=f\"Epoch {epoch}/{n_epochs}\", position=0, \n",
    "                    leave=True) as steps:\n",
    "            for step in steps:\n",
    "                X_batch, y_batch = random_batch(X_train, y_train)\n",
    "                with tf.GradientTape(persistent=True) as tape:\n",
    "                    y_pred = model(X_batch)\n",
    "                    main_loss = tf.reduce_mean(loss_fn(y_batch, y_pred))\n",
    "                    loss = tf.add_n([main_loss] + model.losses)\n",
    "                for layers, optimizer in ((lower_layers, lower_optimizer),\n",
    "                                         (upper_layers, upper_optimizer)):\n",
    "                    gradients = tape.gradient(loss, layers.trainable_variables)\n",
    "                    optimizer.apply_gradients(zip(gradients, layers.trainable_variables))\n",
    "                del tape\n",
    "                for variable in model.variables:\n",
    "                    if variable.constraint is not None:\n",
    "                        variable.assign(variable.constraint(variable))                    \n",
    "                status = OrderedDict()\n",
    "                mean_loss(loss)\n",
    "                status[\"loss\"] = mean_loss.result().numpy()\n",
    "                for metric in metrics:\n",
    "                    metric(y_batch, y_pred)\n",
    "                    status[metric.name] = metric.result().numpy()\n",
    "                steps.set_postfix(status)\n",
    "            y_pred = model(X_valid)\n",
    "            status[\"val_loss\"] = np.mean(loss_fn(y_valid, y_pred))\n",
    "            status[\"val_accuracy\"] = np.mean(keras.metrics.sparse_categorical_accuracy(\n",
    "                tf.constant(y_valid, dtype=np.float32), y_pred))\n",
    "            steps.set_postfix(status)\n",
    "        for metric in [mean_loss] + metrics:\n",
    "            metric.reset_states()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "3342a886-b793-4579-8462-70c79fe9ced5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('loss', 0.47624412),\n",
       "             ('sparse_categorical_accuracy', 0.8332303),\n",
       "             ('val_loss', 0.47591075),\n",
       "             ('val_accuracy', 0.8346)])"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "status"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (latest)",
   "language": "python",
   "name": "latest"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}