courses/machine_learning/deepdive/05_artandscience/a_handtuning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4f3CKqFUqL2-",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Hand tuning hyperparameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Learning Objectives:**\n",
    "  * Use the `LinearRegressor` class in TensorFlow to predict median housing price, at the granularity of city blocks, based on one input feature\n",
    "  * Evaluate the accuracy of a model's predictions using Root Mean Squared Error (RMSE)\n",
    "  * Improve the accuracy of a model by hand-tuning its hyperparameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data is based on 1990 census data from California. This data is at the city block level, so these features reflect the total number of rooms in that block, or the total number of people who live on that block, respectively.  Using only one input feature -- the number of rooms -- predict house value."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6TjLjL9IU80G"
   },
   "source": [
    "## Set Up\n",
    "In this first cell, we'll load the necessary libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "          <script src=\"/static/components/requirejs/require.js\"></script>\n",
       "          <script>\n",
       "            requirejs.config({\n",
       "              paths: {\n",
       "                base: '/static/base',\n",
       "              },\n",
       "            });\n",
       "          </script>\n",
       "          "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import math\n",
    "import shutil\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "\n",
    "tf.logging.set_verbosity(tf.logging.INFO)\n",
    "pd.options.display.max_rows = 10\n",
    "pd.options.display.float_format = '{:.1f}'.format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ipRyUHjhU80Q"
   },
   "source": [
    "Next, we'll load our data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "          <script src=\"/static/components/requirejs/require.js\"></script>\n",
       "          <script>\n",
       "            requirejs.config({\n",
       "              paths: {\n",
       "                base: '/static/base',\n",
       "              },\n",
       "            });\n",
       "          </script>\n",
       "          "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df = pd.read_csv(\"https://storage.googleapis.com/ml_universities/california_housing_train.csv\", sep=\",\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HzzlSs3PtTmt",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Examine the data\n",
    "\n",
    "It's a good idea to get to know your data a little bit before you work with it.\n",
    "\n",
    "We'll print out a quick summary of a few useful statistics on each column.\n",
    "\n",
    "This will include things like mean, standard deviation, max, min, and various quantiles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "          <script src=\"/static/components/requirejs/require.js\"></script>\n",
       "          <script>\n",
       "            requirejs.config({\n",
       "              paths: {\n",
       "                base: '/static/base',\n",
       "              },\n",
       "            });\n",
       "          </script>\n",
       "          "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>longitude</th>\n",
       "      <th>latitude</th>\n",
       "      <th>housing_median_age</th>\n",
       "      <th>total_rooms</th>\n",
       "      <th>total_bedrooms</th>\n",
       "      <th>population</th>\n",
       "      <th>households</th>\n",
       "      <th>median_income</th>\n",
       "      <th>median_house_value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-114.3</td>\n",
       "      <td>34.2</td>\n",
       "      <td>15.0</td>\n",
       "      <td>5612.0</td>\n",
       "      <td>1283.0</td>\n",
       "      <td>1015.0</td>\n",
       "      <td>472.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>66900.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-114.5</td>\n",
       "      <td>34.4</td>\n",
       "      <td>19.0</td>\n",
       "      <td>7650.0</td>\n",
       "      <td>1901.0</td>\n",
       "      <td>1129.0</td>\n",
       "      <td>463.0</td>\n",
       "      <td>1.8</td>\n",
       "      <td>80100.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-114.6</td>\n",
       "      <td>33.7</td>\n",
       "      <td>17.0</td>\n",
       "      <td>720.0</td>\n",
       "      <td>174.0</td>\n",
       "      <td>333.0</td>\n",
       "      <td>117.0</td>\n",
       "      <td>1.7</td>\n",
       "      <td>85700.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-114.6</td>\n",
       "      <td>33.6</td>\n",
       "      <td>14.0</td>\n",
       "      <td>1501.0</td>\n",
       "      <td>337.0</td>\n",
       "      <td>515.0</td>\n",
       "      <td>226.0</td>\n",
       "      <td>3.2</td>\n",
       "      <td>73400.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-114.6</td>\n",
       "      <td>33.6</td>\n",
       "      <td>20.0</td>\n",
       "      <td>1454.0</td>\n",
       "      <td>326.0</td>\n",
       "      <td>624.0</td>\n",
       "      <td>262.0</td>\n",
       "      <td>1.9</td>\n",
       "      <td>65500.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   longitude  latitude  housing_median_age  total_rooms  total_bedrooms  \\\n",
       "0     -114.3      34.2                15.0       5612.0          1283.0   \n",
       "1     -114.5      34.4                19.0       7650.0          1901.0   \n",
       "2     -114.6      33.7                17.0        720.0           174.0   \n",
       "3     -114.6      33.6                14.0       1501.0           337.0   \n",
       "4     -114.6      33.6                20.0       1454.0           326.0   \n",
       "\n",
       "   population  households  median_income  median_house_value  \n",
       "0      1015.0       472.0            1.5             66900.0  \n",
       "1      1129.0       463.0            1.8             80100.0  \n",
       "2       333.0       117.0            1.7             85700.0  \n",
       "3       515.0       226.0            3.2             73400.0  \n",
       "4       624.0       262.0            1.9             65500.0  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "cellView": "both",
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "test": {
      "output": "ignore",
      "timeout": 600
     }
    },
    "colab_type": "code",
    "collapsed": false,
    "id": "gzb10yoVrydW",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "          <script src=\"/static/components/requirejs/require.js\"></script>\n",
       "          <script>\n",
       "            requirejs.config({\n",
       "              paths: {\n",
       "                base: '/static/base',\n",
       "              },\n",
       "            });\n",
       "          </script>\n",
       "          "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>longitude</th>\n",
       "      <th>latitude</th>\n",
       "      <th>housing_median_age</th>\n",
       "      <th>total_rooms</th>\n",
       "      <th>total_bedrooms</th>\n",
       "      <th>population</th>\n",
       "      <th>households</th>\n",
       "      <th>median_income</th>\n",
       "      <th>median_house_value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>17000.0</td>\n",
       "      <td>17000.0</td>\n",
       "      <td>17000.0</td>\n",
       "      <td>17000.0</td>\n",
       "      <td>17000.0</td>\n",
       "      <td>17000.0</td>\n",
       "      <td>17000.0</td>\n",
       "      <td>17000.0</td>\n",
       "      <td>17000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>-119.6</td>\n",
       "      <td>35.6</td>\n",
       "      <td>28.6</td>\n",
       "      <td>2643.7</td>\n",
       "      <td>539.4</td>\n",
       "      <td>1429.6</td>\n",
       "      <td>501.2</td>\n",
       "      <td>3.9</td>\n",
       "      <td>207300.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>2.0</td>\n",
       "      <td>2.1</td>\n",
       "      <td>12.6</td>\n",
       "      <td>2179.9</td>\n",
       "      <td>421.5</td>\n",
       "      <td>1147.9</td>\n",
       "      <td>384.5</td>\n",
       "      <td>1.9</td>\n",
       "      <td>115983.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>-124.3</td>\n",
       "      <td>32.5</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>14999.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>-121.8</td>\n",
       "      <td>33.9</td>\n",
       "      <td>18.0</td>\n",
       "      <td>1462.0</td>\n",
       "      <td>297.0</td>\n",
       "      <td>790.0</td>\n",
       "      <td>282.0</td>\n",
       "      <td>2.6</td>\n",
       "      <td>119400.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>-118.5</td>\n",
       "      <td>34.2</td>\n",
       "      <td>29.0</td>\n",
       "      <td>2127.0</td>\n",
       "      <td>434.0</td>\n",
       "      <td>1167.0</td>\n",
       "      <td>409.0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>180400.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>-118.0</td>\n",
       "      <td>37.7</td>\n",
       "      <td>37.0</td>\n",
       "      <td>3151.2</td>\n",
       "      <td>648.2</td>\n",
       "      <td>1721.0</td>\n",
       "      <td>605.2</td>\n",
       "      <td>4.8</td>\n",
       "      <td>265000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>-114.3</td>\n",
       "      <td>42.0</td>\n",
       "      <td>52.0</td>\n",
       "      <td>37937.0</td>\n",
       "      <td>6445.0</td>\n",
       "      <td>35682.0</td>\n",
       "      <td>6082.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>500001.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       longitude  latitude  housing_median_age  total_rooms  total_bedrooms  \\\n",
       "count    17000.0   17000.0             17000.0      17000.0         17000.0   \n",
       "mean      -119.6      35.6                28.6       2643.7           539.4   \n",
       "std          2.0       2.1                12.6       2179.9           421.5   \n",
       "min       -124.3      32.5                 1.0          2.0             1.0   \n",
       "25%       -121.8      33.9                18.0       1462.0           297.0   \n",
       "50%       -118.5      34.2                29.0       2127.0           434.0   \n",
       "75%       -118.0      37.7                37.0       3151.2           648.2   \n",
       "max       -114.3      42.0                52.0      37937.0          6445.0   \n",
       "\n",
       "       population  households  median_income  median_house_value  \n",
       "count     17000.0     17000.0        17000.0             17000.0  \n",
       "mean       1429.6       501.2            3.9            207300.9  \n",
       "std        1147.9       384.5            1.9            115983.8  \n",
       "min           3.0         1.0            0.5             14999.0  \n",
       "25%         790.0       282.0            2.6            119400.0  \n",
       "50%        1167.0       409.0            3.5            180400.0  \n",
       "75%        1721.0       605.2            4.8            265000.0  \n",
       "max       35682.0      6082.0           15.0            500001.0  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this exercise, we'll be trying to predict median_house_value. It will be our label (sometimes also called a target). Can we use total_rooms as our input feature?  What's going on with the values for that feature?\n",
    "\n",
    "This data is at the city block level, so these features reflect the total number of rooms in that block, or the total number of people who live on that block, respectively.  Let's create a different, more appropriate feature.  Because we are predicing the price of a single house, we should try to make all our features correspond to a single house as well"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "          <script src=\"/static/components/requirejs/require.js\"></script>\n",
       "          <script>\n",
       "            requirejs.config({\n",
       "              paths: {\n",
       "                base: '/static/base',\n",
       "              },\n",
       "            });\n",
       "          </script>\n",
       "          "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>longitude</th>\n",
       "      <th>latitude</th>\n",
       "      <th>housing_median_age</th>\n",
       "      <th>total_rooms</th>\n",
       "      <th>total_bedrooms</th>\n",
       "      <th>population</th>\n",
       "      <th>households</th>\n",
       "      <th>median_income</th>\n",
       "      <th>median_house_value</th>\n",
       "      <th>num_rooms</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>17000.0</td>\n",
       "      <td>17000.0</td>\n",
       "      <td>17000.0</td>\n",
       "      <td>17000.0</td>\n",
       "      <td>17000.0</td>\n",
       "      <td>17000.0</td>\n",
       "      <td>17000.0</td>\n",
       "      <td>17000.0</td>\n",
       "      <td>17000.0</td>\n",
       "      <td>17000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>-119.6</td>\n",
       "      <td>35.6</td>\n",
       "      <td>28.6</td>\n",
       "      <td>2643.7</td>\n",
       "      <td>539.4</td>\n",
       "      <td>1429.6</td>\n",
       "      <td>501.2</td>\n",
       "      <td>3.9</td>\n",
       "      <td>207300.9</td>\n",
       "      <td>5.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>2.0</td>\n",
       "      <td>2.1</td>\n",
       "      <td>12.6</td>\n",
       "      <td>2179.9</td>\n",
       "      <td>421.5</td>\n",
       "      <td>1147.9</td>\n",
       "      <td>384.5</td>\n",
       "      <td>1.9</td>\n",
       "      <td>115983.8</td>\n",
       "      <td>2.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>-124.3</td>\n",
       "      <td>32.5</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>14999.0</td>\n",
       "      <td>0.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>-121.8</td>\n",
       "      <td>33.9</td>\n",
       "      <td>18.0</td>\n",
       "      <td>1462.0</td>\n",
       "      <td>297.0</td>\n",
       "      <td>790.0</td>\n",
       "      <td>282.0</td>\n",
       "      <td>2.6</td>\n",
       "      <td>119400.0</td>\n",
       "      <td>4.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>-118.5</td>\n",
       "      <td>34.2</td>\n",
       "      <td>29.0</td>\n",
       "      <td>2127.0</td>\n",
       "      <td>434.0</td>\n",
       "      <td>1167.0</td>\n",
       "      <td>409.0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>180400.0</td>\n",
       "      <td>5.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>-118.0</td>\n",
       "      <td>37.7</td>\n",
       "      <td>37.0</td>\n",
       "      <td>3151.2</td>\n",
       "      <td>648.2</td>\n",
       "      <td>1721.0</td>\n",
       "      <td>605.2</td>\n",
       "      <td>4.8</td>\n",
       "      <td>265000.0</td>\n",
       "      <td>6.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>-114.3</td>\n",
       "      <td>42.0</td>\n",
       "      <td>52.0</td>\n",
       "      <td>37937.0</td>\n",
       "      <td>6445.0</td>\n",
       "      <td>35682.0</td>\n",
       "      <td>6082.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>500001.0</td>\n",
       "      <td>141.9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       longitude  latitude  housing_median_age  total_rooms  total_bedrooms  \\\n",
       "count    17000.0   17000.0             17000.0      17000.0         17000.0   \n",
       "mean      -119.6      35.6                28.6       2643.7           539.4   \n",
       "std          2.0       2.1                12.6       2179.9           421.5   \n",
       "min       -124.3      32.5                 1.0          2.0             1.0   \n",
       "25%       -121.8      33.9                18.0       1462.0           297.0   \n",
       "50%       -118.5      34.2                29.0       2127.0           434.0   \n",
       "75%       -118.0      37.7                37.0       3151.2           648.2   \n",
       "max       -114.3      42.0                52.0      37937.0          6445.0   \n",
       "\n",
       "       population  households  median_income  median_house_value  num_rooms  \n",
       "count     17000.0     17000.0        17000.0             17000.0    17000.0  \n",
       "mean       1429.6       501.2            3.9            207300.9        5.4  \n",
       "std        1147.9       384.5            1.9            115983.8        2.5  \n",
       "min           3.0         1.0            0.5             14999.0        0.8  \n",
       "25%         790.0       282.0            2.6            119400.0        4.4  \n",
       "50%        1167.0       409.0            3.5            180400.0        5.2  \n",
       "75%        1721.0       605.2            4.8            265000.0        6.1  \n",
       "max       35682.0      6082.0           15.0            500001.0      141.9  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['num_rooms'] = df['total_rooms'] / df['households']\n",
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Lr6wYl2bt2Ep",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Build the first model\n",
    "\n",
    "In this exercise, we'll be trying to predict `median_house_value`. It will be our label (sometimes also called a target). We'll use `num_rooms` as our input feature.\n",
    "\n",
    "To train our model, we'll use the [LinearRegressor](https://www.tensorflow.org/api_docs/python/tf/contrib/learn/LinearRegressor) interface. The Estimator takes care of a lot of the plumbing, and exposes a convenient way to interact with data, training, and evaluation.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "          <script src=\"/static/components/requirejs/require.js\"></script>\n",
       "          <script>\n",
       "            requirejs.config({\n",
       "              paths: {\n",
       "                base: '/static/base',\n",
       "              },\n",
       "            });\n",
       "          </script>\n",
       "          "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using default config.\n",
      "INFO:tensorflow:Using config: {'_save_checkpoints_secs': 600, '_session_config': None, '_keep_checkpoint_max': 5, '_task_type': 'worker', '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f21487f3f10>, '_save_checkpoints_steps': None, '_keep_checkpoint_every_n_hours': 10000, '_service': None, '_num_ps_replicas': 0, '_tf_random_seed': None, '_master': '', '_num_worker_replicas': 1, '_task_id': 0, '_log_step_count_steps': 100, '_model_dir': './housing_trained', '_save_summary_steps': 100}\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Saving checkpoints for 1 into ./housing_trained/model.ckpt.\n",
      "INFO:tensorflow:loss = 10836183000000.0, step = 1\n",
      "INFO:tensorflow:Saving checkpoints for 100 into ./housing_trained/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 5202707000000.0.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-06-06:02:37\n",
      "INFO:tensorflow:Restoring parameters from ./housing_trained/model.ckpt-100\n",
      "INFO:tensorflow:Evaluation [1/1]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-06-06:02:37\n",
      "INFO:tensorflow:Saving dict for global step 100: average_loss = 26268346000.0, global_step = 100, loss = 3362348300000.0\n",
      "RMSE on training dataset = 1833670.75\n"
     ]
    }
   ],
   "source": [
    "train_fn = tf.estimator.inputs.pandas_input_fn(x = df[[\"num_rooms\"]],\n",
    "                                              y = df[\"median_house_value\"],\n",
    "                                              num_epochs = 1,\n",
    "                                              shuffle = True)\n",
    "\n",
    "features = [tf.feature_column.numeric_column('num_rooms')]\n",
    "outdir = './housing_trained'\n",
    "shutil.rmtree(outdir, ignore_errors = True) # start fresh each time\n",
    "model = tf.estimator.LinearRegressor(model_dir = outdir, feature_columns = features)\n",
    "model.train(input_fn = train_fn, steps = 100)\n",
    "def print_rmse(model, name, input_fn):\n",
    "  metrics = model.evaluate(input_fn = input_fn, steps = 1)\n",
    "  print 'RMSE on {} dataset = {}'.format(name, np.sqrt(metrics['loss']))\n",
    "print_rmse(model, 'training', train_fn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Scale the output\n",
    "Let's scale the target values so that the default parameters are more appropriate.  Note that the RMSE here is now in 100000s so if you get RMSE=0.9, it really means RMSE=90000."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "          <script src=\"/static/components/requirejs/require.js\"></script>\n",
       "          <script>\n",
       "            requirejs.config({\n",
       "              paths: {\n",
       "                base: '/static/base',\n",
       "              },\n",
       "            });\n",
       "          </script>\n",
       "          "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using default config.\n",
      "INFO:tensorflow:Using config: {'_save_checkpoints_secs': 600, '_session_config': None, '_keep_checkpoint_max': 5, '_task_type': 'worker', '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f21487f3bd0>, '_save_checkpoints_steps': None, '_keep_checkpoint_every_n_hours': 10000, '_service': None, '_num_ps_replicas': 0, '_tf_random_seed': None, '_master': '', '_num_worker_replicas': 1, '_task_id': 0, '_log_step_count_steps': 100, '_model_dir': './housing_trained', '_save_summary_steps': 100}\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Saving checkpoints for 1 into ./housing_trained/model.ckpt.\n",
      "INFO:tensorflow:loss = 357.06677, step = 1\n",
      "INFO:tensorflow:Saving checkpoints for 100 into ./housing_trained/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 73.85011.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-06-06:02:50\n",
      "INFO:tensorflow:Restoring parameters from ./housing_trained/model.ckpt-100\n",
      "INFO:tensorflow:Evaluation [1/1]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-06-06:02:51\n",
      "INFO:tensorflow:Saving dict for global step 100: average_loss = 2.0942805, global_step = 100, loss = 268.0679\n",
      "RMSE on training dataset = 16.3727798462\n"
     ]
    }
   ],
   "source": [
    "SCALE = 100000\n",
    "train_fn = tf.estimator.inputs.pandas_input_fn(x = df[[\"num_rooms\"]],\n",
    "                                              y = df[\"median_house_value\"] / SCALE,  # note the scaling\n",
    "                                              num_epochs = 1,\n",
    "                                              shuffle = True)\n",
    "\n",
    "features = [tf.feature_column.numeric_column('num_rooms')]\n",
    "outdir = './housing_trained'\n",
    "shutil.rmtree(outdir, ignore_errors = True) # start fresh each time\n",
    "model = tf.estimator.LinearRegressor(model_dir = outdir, feature_columns = features)\n",
    "model.train(input_fn = train_fn, steps = 100)\n",
    "print_rmse(model, 'training', train_fn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Change learning rate and batch size\n",
    "Can you come up with better parameters?  (of course, we are doing this only on the training dataset -- normally, you evaluate on a separate validation dataset)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "          <script src=\"/static/components/requirejs/require.js\"></script>\n",
       "          <script>\n",
       "            requirejs.config({\n",
       "              paths: {\n",
       "                base: '/static/base',\n",
       "              },\n",
       "            });\n",
       "          </script>\n",
       "          "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using default config.\n",
      "INFO:tensorflow:Using config: {'_save_checkpoints_secs': 600, '_session_config': None, '_keep_checkpoint_max': 5, '_task_type': 'worker', '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f21347b9ad0>, '_save_checkpoints_steps': None, '_keep_checkpoint_every_n_hours': 10000, '_service': None, '_num_ps_replicas': 0, '_tf_random_seed': None, '_master': '', '_num_worker_replicas': 1, '_task_id': 0, '_log_step_count_steps': 100, '_model_dir': './housing_trained', '_save_summary_steps': 100}\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Saving checkpoints for 1 into ./housing_trained/model.ckpt.\n",
      "INFO:tensorflow:loss = 97.87912, step = 1\n",
      "INFO:tensorflow:global_step/sec: 431.079\n",
      "INFO:tensorflow:loss = 9.867794, step = 101 (0.234 sec)\n",
      "INFO:tensorflow:global_step/sec: 450.944\n",
      "INFO:tensorflow:loss = 16.647133, step = 201 (0.221 sec)\n",
      "INFO:tensorflow:global_step/sec: 487.467\n",
      "INFO:tensorflow:loss = 35.201576, step = 301 (0.205 sec)\n",
      "INFO:tensorflow:global_step/sec: 509.583\n",
      "INFO:tensorflow:loss = 14.139624, step = 401 (0.196 sec)\n",
      "INFO:tensorflow:global_step/sec: 481.705\n",
      "INFO:tensorflow:loss = 47.339302, step = 501 (0.208 sec)\n",
      "INFO:tensorflow:global_step/sec: 498.758\n",
      "INFO:tensorflow:loss = 3.9687383, step = 601 (0.200 sec)\n",
      "INFO:tensorflow:global_step/sec: 480.291\n",
      "INFO:tensorflow:loss = 16.979471, step = 701 (0.208 sec)\n",
      "INFO:tensorflow:global_step/sec: 500.147\n",
      "INFO:tensorflow:loss = 6.827774, step = 801 (0.200 sec)\n",
      "INFO:tensorflow:global_step/sec: 499.194\n",
      "INFO:tensorflow:loss = 13.607988, step = 901 (0.200 sec)\n",
      "INFO:tensorflow:global_step/sec: 465.97\n",
      "INFO:tensorflow:loss = 17.162128, step = 1001 (0.215 sec)\n",
      "INFO:tensorflow:global_step/sec: 541.351\n",
      "INFO:tensorflow:loss = 42.739685, step = 1101 (0.185 sec)\n",
      "INFO:tensorflow:global_step/sec: 509.364\n",
      "INFO:tensorflow:loss = 23.57767, step = 1201 (0.196 sec)\n",
      "INFO:tensorflow:global_step/sec: 507.066\n",
      "INFO:tensorflow:loss = 13.194671, step = 1301 (0.197 sec)\n",
      "INFO:tensorflow:global_step/sec: 516.935\n",
      "INFO:tensorflow:loss = 4.54657, step = 1401 (0.194 sec)\n",
      "INFO:tensorflow:global_step/sec: 514.334\n",
      "INFO:tensorflow:loss = 12.8494835, step = 1501 (0.194 sec)\n",
      "INFO:tensorflow:global_step/sec: 548.619\n",
      "INFO:tensorflow:loss = 9.2962055, step = 1601 (0.182 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 1700 into ./housing_trained/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 8.325928.\n",
      "INFO:tensorflow:Starting evaluation at 2018-02-06-06:03:43\n",
      "INFO:tensorflow:Restoring parameters from ./housing_trained/model.ckpt-1700\n",
      "INFO:tensorflow:Evaluation [1/1]\n",
      "INFO:tensorflow:Finished evaluation at 2018-02-06-06:03:44\n",
      "INFO:tensorflow:Saving dict for global step 1700: average_loss = 1.641695, global_step = 1700, loss = 16.41695\n",
      "RMSE on training dataset = 4.05178356171\n"
     ]
    }
   ],
   "source": [
    "# I got RMSE = 0.528. Did you do better?\n",
    "SCALE = 100000\n",
    "train_fn = tf.estimator.inputs.pandas_input_fn(x = df[[\"num_rooms\"]],\n",
    "                                              y = df[\"median_house_value\"] / SCALE,  # note the scaling\n",
    "                                              num_epochs = 1,\n",
    "                                              batch_size = 10, # note the batch size\n",
    "                                              shuffle = True)\n",
    "\n",
    "features = [tf.feature_column.numeric_column('num_rooms')]\n",
    "outdir = './housing_trained'\n",
    "shutil.rmtree(outdir, ignore_errors = True) # start fresh each time\n",
    "myopt = tf.train.FtrlOptimizer(learning_rate = 0.01) # note the learning rate\n",
    "model = tf.estimator.LinearRegressor(model_dir = outdir,\n",
    "                                   feature_columns = features,\n",
    "                                   optimizer = myopt)\n",
    "model.train(input_fn = train_fn, steps = 10000)  # note: more steps, since batchsize is smaller\n",
    "print_rmse(model, 'training', train_fn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QU5sLyYTqzqL",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Is there a standard method for tuning the model?\n",
    "\n",
    "This is a commonly asked question. The short answer is that the effects of different hyperparameters is data dependent.  So there are no hard and fast rules; you'll need to run tests on your data.\n",
    "\n",
    "Here are a few rules of thumb that may help guide you:\n",
    "\n",
    " * Training error should steadily decrease, steeply at first, and should eventually plateau as training converges.\n",
    " * If the training has not converged, try running it for longer.\n",
    " * If the training error decreases too slowly, increasing the learning rate may help it decrease faster.\n",
    "   * But sometimes the exact opposite may happen if the learning rate is too high.\n",
    " * If the training error varies wildly, try decreasing the learning rate.\n",
    "   * Lower learning rate plus larger number of steps or larger batch size is often a good combination.\n",
    " * Very small batch sizes can also cause instability.  First try larger values like 100 or 1000, and decrease until you see degradation.\n",
    "\n",
    "Again, never go strictly by these rules of thumb, because the effects are data dependent.  Always experiment and verify."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GpV-uF_cBCBU",
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 3: Try adding more features\n",
    "\n",
    "See if you can do any better by adding more features.\n",
    "\n",
    "Don't take more than 5 minutes on this portion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "default_view": {},
   "name": "first_steps_with_tensor_flow.ipynb",
   "provenance": [],
   "version": "0.3.2",
   "views": {}
  },
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}