Lightgbm.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lightgbm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import time\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "import lightgbm\n",
    "\n",
    "from sklearn.externals import joblib\n",
    "from sklearn.model_selection import StratifiedKFold, train_test_split\n",
    "from sklearn.model_selection import ParameterGrid, RandomizedSearchCV, ParameterSampler\n",
    "\n",
    "from IPython.display import display\n",
    "\n",
    "from utils import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.set_option('display.max_columns', 500)\n",
    "pd.set_option('display.max_rows', 500)\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load data\n",
    "train.csv and test.csv are obtained using Data_Merging.ipynb from all the provided files. To load files pandas [read_csv](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html) method used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(307511, 199)\n"
     ]
    }
   ],
   "source": [
    "train = pd.read_csv('input/train.csv')\n",
    "print(train.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "#display(train.head(2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(48744, 198)\n"
     ]
    }
   ],
   "source": [
    "test = pd.read_csv('input/test.csv')\n",
    "print(test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load possible outliers (Optional)\n",
    "Outliers are found using [isolation forest](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.IsolationForest.html), indexes of outliers are saved into file 'isolation_forest_outliers.npy'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "outliers_idx = np.load('isolation_forest_outliers.npy', mmap_mode='r')\n",
    "outliers_idx = np.array(outliers_idx)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Outliers removal (optional step)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train = train.drop(index=train.index[outliers_idx])\n",
    "train = train.reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Removing service and target columns\n",
    "Let's split target (Y) column from train set and leave there only feature vectors (X). Another column to drop is SK_ID_CURR, it is different for each row thus it is not useful because does not add any information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "targets = train[['TARGET']]\n",
    "\n",
    "train_ids = train['SK_ID_CURR']\n",
    "train = train.drop(columns=['SK_ID_CURR', 'TARGET'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_ids = test['SK_ID_CURR']\n",
    "test = test.drop(columns=['SK_ID_CURR'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(307511, 1)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "targets.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Drop redundant columns\n",
    "There are number of columns with high correlation almost or equial to 1. See Data_Exploration.ipynb notebook Correlations section for details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "cols_drop = appartment_mode_cols + appartment_medi_cols\n",
    "train.drop(columns=cols_drop, inplace=True)\n",
    "test.drop(columns=cols_drop, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(307511, 177)\n",
      "(48744, 177)\n"
     ]
    }
   ],
   "source": [
    "print(train.shape)\n",
    "print(test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### List Categorical Features\n",
    "Lightgbm algorithm nateively works with categorical features. To process those it needs a parameter 'categorical_feature' set , see [lightgbm parameters](https://lightgbm.readthedocs.io/en/latest/Parameters.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_features = train.select_dtypes('object').columns.tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "for col in cat_features: \n",
    "    train[col] = train[col].astype('category') \n",
    "    test[col] = test[col].astype('category') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parameter Tuning\n",
    "Stratified K-Fold algorithm is used to split data set into train and validation sets to best use all the data in the data set. To find appropriate parameters Random Search technique is used. Params to find through were changed several times each closer to best parameters found on previous step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration 0, params: {'max_depth': 7, 'learning_rate': 0.05, 'num_leaves': 32, 'reg_lambda': 2.0, 'n_estimators': 10000, 'max_bin': 255, 'reg_alpha': 4.0, 'min_child_samples': 80}\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.799395\tvalid's auc: 0.765144\n",
      "[200]\ttrain's auc: 0.825386\tvalid's auc: 0.772813\n",
      "[300]\ttrain's auc: 0.844485\tvalid's auc: 0.774921\n",
      "[400]\ttrain's auc: 0.860025\tvalid's auc: 0.775358\n",
      "[500]\ttrain's auc: 0.873535\tvalid's auc: 0.775464\n",
      "[600]\ttrain's auc: 0.885917\tvalid's auc: 0.775596\n",
      "Early stopping, best iteration is:\n",
      "[555]\ttrain's auc: 0.880612\tvalid's auc: 0.77574\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.798488\tvalid's auc: 0.767914\n",
      "[200]\ttrain's auc: 0.824301\tvalid's auc: 0.774598\n",
      "[300]\ttrain's auc: 0.843218\tvalid's auc: 0.776464\n",
      "[400]\ttrain's auc: 0.859115\tvalid's auc: 0.776663\n",
      "Early stopping, best iteration is:\n",
      "[376]\ttrain's auc: 0.855431\tvalid's auc: 0.776956\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.798317\tvalid's auc: 0.769387\n",
      "[200]\ttrain's auc: 0.824561\tvalid's auc: 0.775791\n",
      "[300]\ttrain's auc: 0.843714\tvalid's auc: 0.777128\n",
      "[400]\ttrain's auc: 0.859198\tvalid's auc: 0.776806\n",
      "Early stopping, best iteration is:\n",
      "[310]\ttrain's auc: 0.845372\tvalid's auc: 0.777227\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.798944\tvalid's auc: 0.767734\n",
      "[200]\ttrain's auc: 0.825167\tvalid's auc: 0.773421\n",
      "[300]\ttrain's auc: 0.844314\tvalid's auc: 0.77482\n",
      "[400]\ttrain's auc: 0.860237\tvalid's auc: 0.774821\n",
      "Early stopping, best iteration is:\n",
      "[304]\ttrain's auc: 0.845025\tvalid's auc: 0.774894\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.798521\tvalid's auc: 0.769744\n",
      "[200]\ttrain's auc: 0.82471\tvalid's auc: 0.775444\n",
      "[300]\ttrain's auc: 0.843791\tvalid's auc: 0.777089\n",
      "[400]\ttrain's auc: 0.859538\tvalid's auc: 0.777007\n",
      "Early stopping, best iteration is:\n",
      "[360]\ttrain's auc: 0.853547\tvalid's auc: 0.777196\n",
      "Iteration 1, params: {'max_depth': 7, 'learning_rate': 0.05, 'num_leaves': 32, 'reg_lambda': 2.0, 'n_estimators': 10000, 'max_bin': 255, 'reg_alpha': 2.0, 'min_child_samples': 160}\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.799766\tvalid's auc: 0.765029\n",
      "[200]\ttrain's auc: 0.825567\tvalid's auc: 0.772815\n",
      "[300]\ttrain's auc: 0.84418\tvalid's auc: 0.774743\n",
      "[400]\ttrain's auc: 0.859706\tvalid's auc: 0.775322\n",
      "Early stopping, best iteration is:\n",
      "[342]\ttrain's auc: 0.851097\tvalid's auc: 0.775473\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.798989\tvalid's auc: 0.768233\n",
      "[200]\ttrain's auc: 0.824648\tvalid's auc: 0.774945\n",
      "[300]\ttrain's auc: 0.843661\tvalid's auc: 0.776819\n",
      "[400]\ttrain's auc: 0.859046\tvalid's auc: 0.777222\n",
      "[500]\ttrain's auc: 0.872478\tvalid's auc: 0.777449\n",
      "[600]\ttrain's auc: 0.884671\tvalid's auc: 0.777086\n",
      "Early stopping, best iteration is:\n",
      "[506]\ttrain's auc: 0.873188\tvalid's auc: 0.777569\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.798779\tvalid's auc: 0.769209\n",
      "[200]\ttrain's auc: 0.824267\tvalid's auc: 0.774916\n",
      "[300]\ttrain's auc: 0.843193\tvalid's auc: 0.776164\n",
      "[400]\ttrain's auc: 0.8588\tvalid's auc: 0.77639\n",
      "[500]\ttrain's auc: 0.872315\tvalid's auc: 0.776427\n",
      "Early stopping, best iteration is:\n",
      "[452]\ttrain's auc: 0.866143\tvalid's auc: 0.776632\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.799181\tvalid's auc: 0.767575\n",
      "[200]\ttrain's auc: 0.825185\tvalid's auc: 0.773416\n",
      "[300]\ttrain's auc: 0.844412\tvalid's auc: 0.77493\n",
      "Early stopping, best iteration is:\n",
      "[295]\ttrain's auc: 0.843619\tvalid's auc: 0.774995\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.798618\tvalid's auc: 0.769116\n",
      "[200]\ttrain's auc: 0.824846\tvalid's auc: 0.775283\n",
      "[300]\ttrain's auc: 0.844208\tvalid's auc: 0.776896\n",
      "[400]\ttrain's auc: 0.860067\tvalid's auc: 0.777207\n",
      "[500]\ttrain's auc: 0.873482\tvalid's auc: 0.776807\n",
      "Early stopping, best iteration is:\n",
      "[420]\ttrain's auc: 0.862705\tvalid's auc: 0.777378\n",
      "Iteration 2, params: {'max_depth': 7, 'learning_rate': 0.05, 'num_leaves': 64, 'reg_lambda': 1.0, 'n_estimators': 10000, 'max_bin': 255, 'reg_alpha': 4.0, 'min_child_samples': 160}\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.823156\tvalid's auc: 0.76659\n",
      "[200]\ttrain's auc: 0.855915\tvalid's auc: 0.772671\n",
      "[300]\ttrain's auc: 0.879467\tvalid's auc: 0.773757\n",
      "[400]\ttrain's auc: 0.897899\tvalid's auc: 0.773474\n",
      "Early stopping, best iteration is:\n",
      "[327]\ttrain's auc: 0.88481\tvalid's auc: 0.773982\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.822245\tvalid's auc: 0.769152\n",
      "[200]\ttrain's auc: 0.854037\tvalid's auc: 0.77473\n",
      "[300]\ttrain's auc: 0.876165\tvalid's auc: 0.77584\n",
      "[400]\ttrain's auc: 0.895621\tvalid's auc: 0.775705\n",
      "Early stopping, best iteration is:\n",
      "[313]\ttrain's auc: 0.878929\tvalid's auc: 0.775994\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.821588\tvalid's auc: 0.770656\n",
      "[200]\ttrain's auc: 0.852843\tvalid's auc: 0.774954\n",
      "[300]\ttrain's auc: 0.875965\tvalid's auc: 0.775756\n",
      "Early stopping, best iteration is:\n",
      "[274]\ttrain's auc: 0.870778\tvalid's auc: 0.775934\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.822281\tvalid's auc: 0.768498\n",
      "[200]\ttrain's auc: 0.855284\tvalid's auc: 0.772604\n",
      "[300]\ttrain's auc: 0.879193\tvalid's auc: 0.773119\n",
      "[400]\ttrain's auc: 0.897627\tvalid's auc: 0.772899\n",
      "Early stopping, best iteration is:\n",
      "[321]\ttrain's auc: 0.883375\tvalid's auc: 0.773367\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.821658\tvalid's auc: 0.768626\n",
      "[200]\ttrain's auc: 0.854254\tvalid's auc: 0.773784\n",
      "[300]\ttrain's auc: 0.877866\tvalid's auc: 0.774842\n",
      "[400]\ttrain's auc: 0.897863\tvalid's auc: 0.773724\n",
      "Early stopping, best iteration is:\n",
      "[311]\ttrain's auc: 0.880324\tvalid's auc: 0.77508\n",
      "Iteration 3, params: {'max_depth': -1, 'learning_rate': 0.05, 'num_leaves': 64, 'reg_lambda': 2.0, 'n_estimators': 10000, 'max_bin': 255, 'reg_alpha': 4.0, 'min_child_samples': 80}\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.828122\tvalid's auc: 0.767702\n",
      "[200]\ttrain's auc: 0.867957\tvalid's auc: 0.77394\n",
      "[300]\ttrain's auc: 0.895714\tvalid's auc: 0.775211\n",
      "[400]\ttrain's auc: 0.916894\tvalid's auc: 0.77451\n",
      "Early stopping, best iteration is:\n",
      "[333]\ttrain's auc: 0.903517\tvalid's auc: 0.775331\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.827962\tvalid's auc: 0.771089\n",
      "[200]\ttrain's auc: 0.867515\tvalid's auc: 0.776115\n",
      "[300]\ttrain's auc: 0.895433\tvalid's auc: 0.776433\n",
      "[400]\ttrain's auc: 0.916642\tvalid's auc: 0.776026\n",
      "Early stopping, best iteration is:\n",
      "[320]\ttrain's auc: 0.900325\tvalid's auc: 0.77665\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.827797\tvalid's auc: 0.771835\n",
      "[200]\ttrain's auc: 0.867363\tvalid's auc: 0.775921\n",
      "[300]\ttrain's auc: 0.894961\tvalid's auc: 0.775508\n",
      "Early stopping, best iteration is:\n",
      "[206]\ttrain's auc: 0.86933\tvalid's auc: 0.776021\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.828329\tvalid's auc: 0.770236\n",
      "[200]\ttrain's auc: 0.868229\tvalid's auc: 0.774349\n",
      "[300]\ttrain's auc: 0.896118\tvalid's auc: 0.774542\n",
      "Early stopping, best iteration is:\n",
      "[284]\ttrain's auc: 0.89209\tvalid's auc: 0.77495\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.827878\tvalid's auc: 0.771899\n",
      "[200]\ttrain's auc: 0.867949\tvalid's auc: 0.777361\n",
      "[300]\ttrain's auc: 0.895597\tvalid's auc: 0.777308\n",
      "Early stopping, best iteration is:\n",
      "[234]\ttrain's auc: 0.878368\tvalid's auc: 0.77752\n",
      "Iteration 4, params: {'max_depth': -1, 'learning_rate': 0.05, 'num_leaves': 64, 'reg_lambda': 1.0, 'n_estimators': 10000, 'max_bin': 255, 'reg_alpha': 2.0, 'min_child_samples': 80}\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.82922\tvalid's auc: 0.768016\n",
      "[200]\ttrain's auc: 0.869757\tvalid's auc: 0.773804\n",
      "[300]\ttrain's auc: 0.897908\tvalid's auc: 0.774406\n",
      "Early stopping, best iteration is:\n",
      "[274]\ttrain's auc: 0.891202\tvalid's auc: 0.774554\n",
      "Training until validation scores don't improve for 100 rounds.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[100]\ttrain's auc: 0.828543\tvalid's auc: 0.77064\n",
      "[200]\ttrain's auc: 0.86904\tvalid's auc: 0.776138\n",
      "[300]\ttrain's auc: 0.897393\tvalid's auc: 0.776528\n",
      "Early stopping, best iteration is:\n",
      "[292]\ttrain's auc: 0.895344\tvalid's auc: 0.776776\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.828881\tvalid's auc: 0.771527\n",
      "[200]\ttrain's auc: 0.869424\tvalid's auc: 0.776055\n",
      "[300]\ttrain's auc: 0.897456\tvalid's auc: 0.775831\n",
      "Early stopping, best iteration is:\n",
      "[211]\ttrain's auc: 0.873181\tvalid's auc: 0.77617\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.828623\tvalid's auc: 0.769925\n",
      "[200]\ttrain's auc: 0.869602\tvalid's auc: 0.774022\n",
      "[300]\ttrain's auc: 0.897497\tvalid's auc: 0.77358\n",
      "Early stopping, best iteration is:\n",
      "[223]\ttrain's auc: 0.876793\tvalid's auc: 0.774418\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.828713\tvalid's auc: 0.771929\n",
      "[200]\ttrain's auc: 0.869343\tvalid's auc: 0.776336\n",
      "[300]\ttrain's auc: 0.897352\tvalid's auc: 0.776383\n",
      "Early stopping, best iteration is:\n",
      "[252]\ttrain's auc: 0.88479\tvalid's auc: 0.776569\n",
      "Iteration 5, params: {'max_depth': -1, 'learning_rate': 0.05, 'num_leaves': 64, 'reg_lambda': 1.0, 'n_estimators': 10000, 'max_bin': 255, 'reg_alpha': 4.0, 'min_child_samples': 80}\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.828619\tvalid's auc: 0.768405\n",
      "[200]\ttrain's auc: 0.868457\tvalid's auc: 0.774384\n",
      "[300]\ttrain's auc: 0.896124\tvalid's auc: 0.77512\n",
      "Early stopping, best iteration is:\n",
      "[287]\ttrain's auc: 0.892885\tvalid's auc: 0.775252\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.82785\tvalid's auc: 0.770926\n",
      "[200]\ttrain's auc: 0.867902\tvalid's auc: 0.776136\n",
      "[300]\ttrain's auc: 0.896313\tvalid's auc: 0.776803\n",
      "Early stopping, best iteration is:\n",
      "[249]\ttrain's auc: 0.882636\tvalid's auc: 0.776981\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.828235\tvalid's auc: 0.77192\n",
      "[200]\ttrain's auc: 0.868095\tvalid's auc: 0.776092\n",
      "[300]\ttrain's auc: 0.895645\tvalid's auc: 0.775527\n",
      "Early stopping, best iteration is:\n",
      "[213]\ttrain's auc: 0.872181\tvalid's auc: 0.776253\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.828315\tvalid's auc: 0.770064\n",
      "[200]\ttrain's auc: 0.868671\tvalid's auc: 0.774311\n",
      "[300]\ttrain's auc: 0.896558\tvalid's auc: 0.774491\n",
      "Early stopping, best iteration is:\n",
      "[252]\ttrain's auc: 0.884327\tvalid's auc: 0.774913\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.828132\tvalid's auc: 0.771612\n",
      "[200]\ttrain's auc: 0.867863\tvalid's auc: 0.776236\n",
      "[300]\ttrain's auc: 0.895964\tvalid's auc: 0.77616\n",
      "Early stopping, best iteration is:\n",
      "[219]\ttrain's auc: 0.873883\tvalid's auc: 0.776378\n",
      "Iteration 6, params: {'max_depth': -1, 'learning_rate': 0.05, 'num_leaves': 32, 'reg_lambda': 1.0, 'n_estimators': 10000, 'max_bin': 255, 'reg_alpha': 2.0, 'min_child_samples': 160}\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.800762\tvalid's auc: 0.766025\n",
      "[200]\ttrain's auc: 0.828148\tvalid's auc: 0.773142\n",
      "[300]\ttrain's auc: 0.847819\tvalid's auc: 0.774443\n",
      "[400]\ttrain's auc: 0.864188\tvalid's auc: 0.77489\n",
      "[500]\ttrain's auc: 0.878685\tvalid's auc: 0.774872\n",
      "Early stopping, best iteration is:\n",
      "[463]\ttrain's auc: 0.873431\tvalid's auc: 0.775149\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.800189\tvalid's auc: 0.769211\n",
      "[200]\ttrain's auc: 0.827388\tvalid's auc: 0.775695\n",
      "[300]\ttrain's auc: 0.846632\tvalid's auc: 0.776639\n",
      "[400]\ttrain's auc: 0.863727\tvalid's auc: 0.77683\n",
      "Early stopping, best iteration is:\n",
      "[372]\ttrain's auc: 0.859316\tvalid's auc: 0.777162\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.799784\tvalid's auc: 0.770085\n",
      "[200]\ttrain's auc: 0.827416\tvalid's auc: 0.776106\n",
      "[300]\ttrain's auc: 0.847575\tvalid's auc: 0.776649\n",
      "[400]\ttrain's auc: 0.864037\tvalid's auc: 0.776669\n",
      "[500]\ttrain's auc: 0.878619\tvalid's auc: 0.777118\n",
      "[600]\ttrain's auc: 0.891178\tvalid's auc: 0.776578\n",
      "Early stopping, best iteration is:\n",
      "[511]\ttrain's auc: 0.880181\tvalid's auc: 0.777237\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.800303\tvalid's auc: 0.768325\n",
      "[200]\ttrain's auc: 0.827931\tvalid's auc: 0.774122\n",
      "[300]\ttrain's auc: 0.848041\tvalid's auc: 0.774659\n",
      "[400]\ttrain's auc: 0.864882\tvalid's auc: 0.774519\n",
      "Early stopping, best iteration is:\n",
      "[318]\ttrain's auc: 0.851305\tvalid's auc: 0.774826\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.799984\tvalid's auc: 0.769755\n",
      "[200]\ttrain's auc: 0.827311\tvalid's auc: 0.775503\n",
      "[300]\ttrain's auc: 0.847248\tvalid's auc: 0.776314\n",
      "Early stopping, best iteration is:\n",
      "[297]\ttrain's auc: 0.846737\tvalid's auc: 0.776359\n",
      "Iteration 7, params: {'max_depth': -1, 'learning_rate': 0.05, 'num_leaves': 64, 'reg_lambda': 1.0, 'n_estimators': 10000, 'max_bin': 255, 'reg_alpha': 4.0, 'min_child_samples': 160}\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.82845\tvalid's auc: 0.768258\n",
      "[200]\ttrain's auc: 0.868312\tvalid's auc: 0.774626\n",
      "[300]\ttrain's auc: 0.896128\tvalid's auc: 0.775298\n",
      "Early stopping, best iteration is:\n",
      "[260]\ttrain's auc: 0.885867\tvalid's auc: 0.775475\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.827969\tvalid's auc: 0.770881\n",
      "[200]\ttrain's auc: 0.867487\tvalid's auc: 0.775765\n",
      "[300]\ttrain's auc: 0.89542\tvalid's auc: 0.775659\n",
      "Early stopping, best iteration is:\n",
      "[243]\ttrain's auc: 0.880709\tvalid's auc: 0.776003\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.828235\tvalid's auc: 0.771877\n",
      "[200]\ttrain's auc: 0.868193\tvalid's auc: 0.776048\n",
      "[300]\ttrain's auc: 0.895831\tvalid's auc: 0.776591\n",
      "[400]\ttrain's auc: 0.916888\tvalid's auc: 0.776072\n",
      "Early stopping, best iteration is:\n",
      "[321]\ttrain's auc: 0.9007\tvalid's auc: 0.776658\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.828451\tvalid's auc: 0.769953\n",
      "[200]\ttrain's auc: 0.8685\tvalid's auc: 0.774653\n",
      "[300]\ttrain's auc: 0.896286\tvalid's auc: 0.774052\n",
      "Early stopping, best iteration is:\n",
      "[205]\ttrain's auc: 0.87025\tvalid's auc: 0.774745\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.827624\tvalid's auc: 0.771189\n",
      "[200]\ttrain's auc: 0.868087\tvalid's auc: 0.775834\n",
      "[300]\ttrain's auc: 0.895975\tvalid's auc: 0.776569\n",
      "Early stopping, best iteration is:\n",
      "[286]\ttrain's auc: 0.892515\tvalid's auc: 0.776636\n",
      "Iteration 8, params: {'max_depth': -1, 'learning_rate': 0.05, 'num_leaves': 32, 'reg_lambda': 2.0, 'n_estimators': 10000, 'max_bin': 255, 'reg_alpha': 2.0, 'min_child_samples': 160}\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.800634\tvalid's auc: 0.765994\n",
      "[200]\ttrain's auc: 0.827676\tvalid's auc: 0.773927\n",
      "[300]\ttrain's auc: 0.847095\tvalid's auc: 0.775486\n",
      "[400]\ttrain's auc: 0.863762\tvalid's auc: 0.77584\n",
      "Early stopping, best iteration is:\n",
      "[382]\ttrain's auc: 0.861142\tvalid's auc: 0.776083\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.80009\tvalid's auc: 0.76966\n",
      "[200]\ttrain's auc: 0.827427\tvalid's auc: 0.776111\n",
      "[300]\ttrain's auc: 0.847014\tvalid's auc: 0.77746\n",
      "[400]\ttrain's auc: 0.863565\tvalid's auc: 0.777465\n",
      "Early stopping, best iteration is:\n",
      "[302]\ttrain's auc: 0.847452\tvalid's auc: 0.777526\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.800152\tvalid's auc: 0.769727\n",
      "[200]\ttrain's auc: 0.827453\tvalid's auc: 0.77552\n",
      "[300]\ttrain's auc: 0.847145\tvalid's auc: 0.776159\n",
      "[400]\ttrain's auc: 0.863828\tvalid's auc: 0.776352\n",
      "Early stopping, best iteration is:\n",
      "[390]\ttrain's auc: 0.862335\tvalid's auc: 0.776533\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.799915\tvalid's auc: 0.767545\n",
      "[200]\ttrain's auc: 0.827691\tvalid's auc: 0.773173\n",
      "[300]\ttrain's auc: 0.847515\tvalid's auc: 0.773918\n",
      "Early stopping, best iteration is:\n",
      "[276]\ttrain's auc: 0.843271\tvalid's auc: 0.774035\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.800195\tvalid's auc: 0.769982\n",
      "[200]\ttrain's auc: 0.827442\tvalid's auc: 0.776728\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[300]\ttrain's auc: 0.847184\tvalid's auc: 0.777371\n",
      "Early stopping, best iteration is:\n",
      "[261]\ttrain's auc: 0.83992\tvalid's auc: 0.777561\n",
      "Iteration 9, params: {'max_depth': 7, 'learning_rate': 0.05, 'num_leaves': 32, 'reg_lambda': 1.0, 'n_estimators': 10000, 'max_bin': 255, 'reg_alpha': 4.0, 'min_child_samples': 80}\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.799299\tvalid's auc: 0.765324\n",
      "[200]\ttrain's auc: 0.825003\tvalid's auc: 0.772993\n",
      "[300]\ttrain's auc: 0.843875\tvalid's auc: 0.775043\n",
      "[400]\ttrain's auc: 0.859915\tvalid's auc: 0.77536\n",
      "[500]\ttrain's auc: 0.873677\tvalid's auc: 0.774751\n",
      "Early stopping, best iteration is:\n",
      "[427]\ttrain's auc: 0.864019\tvalid's auc: 0.775476\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.799251\tvalid's auc: 0.768471\n",
      "[200]\ttrain's auc: 0.82481\tvalid's auc: 0.775481\n",
      "[300]\ttrain's auc: 0.843495\tvalid's auc: 0.776877\n",
      "[400]\ttrain's auc: 0.859298\tvalid's auc: 0.777335\n",
      "[500]\ttrain's auc: 0.873271\tvalid's auc: 0.777052\n",
      "Early stopping, best iteration is:\n",
      "[402]\ttrain's auc: 0.859661\tvalid's auc: 0.777402\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.798356\tvalid's auc: 0.769025\n",
      "[200]\ttrain's auc: 0.824609\tvalid's auc: 0.775168\n",
      "[300]\ttrain's auc: 0.843736\tvalid's auc: 0.776328\n",
      "Early stopping, best iteration is:\n",
      "[294]\ttrain's auc: 0.84267\tvalid's auc: 0.776402\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.798909\tvalid's auc: 0.767429\n",
      "[200]\ttrain's auc: 0.825085\tvalid's auc: 0.773024\n",
      "[300]\ttrain's auc: 0.844549\tvalid's auc: 0.774568\n",
      "[400]\ttrain's auc: 0.860399\tvalid's auc: 0.774661\n",
      "Early stopping, best iteration is:\n",
      "[371]\ttrain's auc: 0.856077\tvalid's auc: 0.774785\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.798702\tvalid's auc: 0.769011\n",
      "[200]\ttrain's auc: 0.824916\tvalid's auc: 0.775167\n",
      "[300]\ttrain's auc: 0.844023\tvalid's auc: 0.776619\n",
      "[400]\ttrain's auc: 0.859385\tvalid's auc: 0.77625\n",
      "Early stopping, best iteration is:\n",
      "[318]\ttrain's auc: 0.847123\tvalid's auc: 0.776862\n"
     ]
    }
   ],
   "source": [
    "# Lightbm parameters\n",
    "params = {\n",
    "    'learning_rate': [0.05],\n",
    "    'n_estimators': [10000],\n",
    "    'num_leaves': [32, 64],\n",
    "    'max_depth': [-1, 7],\n",
    "    'min_child_samples': [80, 160],\n",
    "    'reg_alpha': [2.0, 4.0],\n",
    "    'reg_lambda': [1.0, 2.0],\n",
    "    'max_bin': [255]\n",
    "}\n",
    "\n",
    "# to store results for each instance of parameter sampler\n",
    "cv_results = []\n",
    "\n",
    "# Iterate over random combonations of parameters\n",
    "for i, param_smple in enumerate(list(ParameterSampler(params, 10))):\n",
    "    print('Iteration %i, params: %s' % (i, param_smple))\n",
    "    k_fold = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)\n",
    "    valid_scores = []\n",
    "    train_scores = []\n",
    "    params_ = dict(param_smple)\n",
    "    params_['params'] = param_smple\n",
    "    start = time.time()\n",
    "    fold_i = 0\n",
    "    model = lightgbm.LGBMClassifier(n_estimators=param_smple['n_estimators'], objective='binary', \n",
    "                                    class_weight='balanced', learning_rate=param_smple['learning_rate'], \n",
    "                                    num_leaves=param_smple['num_leaves'], max_depth=param_smple['max_depth'],\n",
    "                                    reg_alpha=param_smple['reg_alpha'], reg_lambda=param_smple['reg_lambda'], \n",
    "                                    min_child_samples=param_smple['min_child_samples'],\n",
    "                                    subsample=0.8, n_jobs=6, random_state=4242, max_bin=param_smple['max_bin'])\n",
    "\n",
    "    # use k-fold to use train and validation sets to test how algorithm learned\n",
    "    for train_indices, valid_indices in k_fold.split(train, targets):\n",
    "        train_features, train_labels = train.iloc[train_indices], targets.iloc[train_indices]\n",
    "        valid_features, valid_labels = train.iloc[valid_indices], targets.iloc[valid_indices]\n",
    "    \n",
    "        model.fit(train_features, train_labels, eval_metric='auc', categorical_feature=cat_features,\n",
    "                      eval_set=[(valid_features, valid_labels), (train_features, train_labels)],\n",
    "                      eval_names=['valid', 'train'], early_stopping_rounds=100, verbose=100)\n",
    "\n",
    "        valid_scores.append(model.best_score_['valid']['auc'])\n",
    "        train_scores.append(model.best_score_['train']['auc'])\n",
    "        params_['train_score_%s' % fold_i] = model.best_score_['train']['auc']\n",
    "        params_['valid_score_%s' % fold_i] = model.best_score_['valid']['auc']\n",
    "        fold_i += 1\n",
    "    \n",
    "    end = time.time()\n",
    "    params_['fit_time'] = end - start\n",
    "    params_['mean_train_score'] = np.mean(train_scores)\n",
    "    params_['mean_valid_score'] = np.mean(valid_scores)\n",
    "    cv_results.append(params_)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create DataFrame from stored results of each parameter sample to benefit from visualization of DataFrames."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>learning_rate</th>\n",
       "      <th>max_bin</th>\n",
       "      <th>max_depth</th>\n",
       "      <th>mean_train_score</th>\n",
       "      <th>mean_valid_score</th>\n",
       "      <th>min_child_samples</th>\n",
       "      <th>n_estimators</th>\n",
       "      <th>num_leaves</th>\n",
       "      <th>params</th>\n",
       "      <th>reg_alpha</th>\n",
       "      <th>reg_lambda</th>\n",
       "      <th>train_score_0</th>\n",
       "      <th>train_score_1</th>\n",
       "      <th>train_score_2</th>\n",
       "      <th>train_score_3</th>\n",
       "      <th>train_score_4</th>\n",
       "      <th>valid_score_0</th>\n",
       "      <th>valid_score_1</th>\n",
       "      <th>valid_score_2</th>\n",
       "      <th>valid_score_3</th>\n",
       "      <th>valid_score_4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>209.345991</td>\n",
       "      <td>0.05</td>\n",
       "      <td>255</td>\n",
       "      <td>7</td>\n",
       "      <td>0.879643</td>\n",
       "      <td>0.774872</td>\n",
       "      <td>160</td>\n",
       "      <td>10000</td>\n",
       "      <td>64</td>\n",
       "      <td>{'max_depth': 7, 'learning_rate': 0.05, 'num_l...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.884810</td>\n",
       "      <td>0.878929</td>\n",
       "      <td>0.870778</td>\n",
       "      <td>0.883375</td>\n",
       "      <td>0.880324</td>\n",
       "      <td>0.773982</td>\n",
       "      <td>0.775994</td>\n",
       "      <td>0.775934</td>\n",
       "      <td>0.773367</td>\n",
       "      <td>0.775080</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>221.092630</td>\n",
       "      <td>0.05</td>\n",
       "      <td>255</td>\n",
       "      <td>-1</td>\n",
       "      <td>0.884262</td>\n",
       "      <td>0.775698</td>\n",
       "      <td>80</td>\n",
       "      <td>10000</td>\n",
       "      <td>64</td>\n",
       "      <td>{'max_depth': -1, 'learning_rate': 0.05, 'num_...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.891202</td>\n",
       "      <td>0.895344</td>\n",
       "      <td>0.873181</td>\n",
       "      <td>0.876793</td>\n",
       "      <td>0.884790</td>\n",
       "      <td>0.774554</td>\n",
       "      <td>0.776776</td>\n",
       "      <td>0.776170</td>\n",
       "      <td>0.774418</td>\n",
       "      <td>0.776569</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>230.559262</td>\n",
       "      <td>0.05</td>\n",
       "      <td>255</td>\n",
       "      <td>-1</td>\n",
       "      <td>0.886008</td>\n",
       "      <td>0.775903</td>\n",
       "      <td>160</td>\n",
       "      <td>10000</td>\n",
       "      <td>64</td>\n",
       "      <td>{'max_depth': -1, 'learning_rate': 0.05, 'num_...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.885867</td>\n",
       "      <td>0.880709</td>\n",
       "      <td>0.900700</td>\n",
       "      <td>0.870250</td>\n",
       "      <td>0.892515</td>\n",
       "      <td>0.775475</td>\n",
       "      <td>0.776003</td>\n",
       "      <td>0.776658</td>\n",
       "      <td>0.774745</td>\n",
       "      <td>0.776636</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>219.593719</td>\n",
       "      <td>0.05</td>\n",
       "      <td>255</td>\n",
       "      <td>-1</td>\n",
       "      <td>0.881182</td>\n",
       "      <td>0.775955</td>\n",
       "      <td>80</td>\n",
       "      <td>10000</td>\n",
       "      <td>64</td>\n",
       "      <td>{'max_depth': -1, 'learning_rate': 0.05, 'num_...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.892885</td>\n",
       "      <td>0.882636</td>\n",
       "      <td>0.872181</td>\n",
       "      <td>0.884327</td>\n",
       "      <td>0.873883</td>\n",
       "      <td>0.775252</td>\n",
       "      <td>0.776981</td>\n",
       "      <td>0.776253</td>\n",
       "      <td>0.774913</td>\n",
       "      <td>0.776378</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>243.125517</td>\n",
       "      <td>0.05</td>\n",
       "      <td>255</td>\n",
       "      <td>-1</td>\n",
       "      <td>0.888726</td>\n",
       "      <td>0.776094</td>\n",
       "      <td>80</td>\n",
       "      <td>10000</td>\n",
       "      <td>64</td>\n",
       "      <td>{'max_depth': -1, 'learning_rate': 0.05, 'num_...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.903517</td>\n",
       "      <td>0.900325</td>\n",
       "      <td>0.869330</td>\n",
       "      <td>0.892090</td>\n",
       "      <td>0.878368</td>\n",
       "      <td>0.775331</td>\n",
       "      <td>0.776650</td>\n",
       "      <td>0.776021</td>\n",
       "      <td>0.774950</td>\n",
       "      <td>0.777520</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>227.609989</td>\n",
       "      <td>0.05</td>\n",
       "      <td>255</td>\n",
       "      <td>-1</td>\n",
       "      <td>0.862194</td>\n",
       "      <td>0.776147</td>\n",
       "      <td>160</td>\n",
       "      <td>10000</td>\n",
       "      <td>32</td>\n",
       "      <td>{'max_depth': -1, 'learning_rate': 0.05, 'num_...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.873431</td>\n",
       "      <td>0.859316</td>\n",
       "      <td>0.880181</td>\n",
       "      <td>0.851305</td>\n",
       "      <td>0.846737</td>\n",
       "      <td>0.775149</td>\n",
       "      <td>0.777162</td>\n",
       "      <td>0.777237</td>\n",
       "      <td>0.774826</td>\n",
       "      <td>0.776359</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>220.591223</td>\n",
       "      <td>0.05</td>\n",
       "      <td>255</td>\n",
       "      <td>7</td>\n",
       "      <td>0.853910</td>\n",
       "      <td>0.776186</td>\n",
       "      <td>80</td>\n",
       "      <td>10000</td>\n",
       "      <td>32</td>\n",
       "      <td>{'max_depth': 7, 'learning_rate': 0.05, 'num_l...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.864019</td>\n",
       "      <td>0.859661</td>\n",
       "      <td>0.842670</td>\n",
       "      <td>0.856077</td>\n",
       "      <td>0.847123</td>\n",
       "      <td>0.775476</td>\n",
       "      <td>0.777402</td>\n",
       "      <td>0.776402</td>\n",
       "      <td>0.774785</td>\n",
       "      <td>0.776862</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>205.577855</td>\n",
       "      <td>0.05</td>\n",
       "      <td>255</td>\n",
       "      <td>-1</td>\n",
       "      <td>0.850824</td>\n",
       "      <td>0.776347</td>\n",
       "      <td>160</td>\n",
       "      <td>10000</td>\n",
       "      <td>32</td>\n",
       "      <td>{'max_depth': -1, 'learning_rate': 0.05, 'num_...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.861142</td>\n",
       "      <td>0.847452</td>\n",
       "      <td>0.862335</td>\n",
       "      <td>0.843271</td>\n",
       "      <td>0.839920</td>\n",
       "      <td>0.776083</td>\n",
       "      <td>0.777526</td>\n",
       "      <td>0.776533</td>\n",
       "      <td>0.774035</td>\n",
       "      <td>0.777561</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>209.558289</td>\n",
       "      <td>0.05</td>\n",
       "      <td>255</td>\n",
       "      <td>7</td>\n",
       "      <td>0.855997</td>\n",
       "      <td>0.776403</td>\n",
       "      <td>80</td>\n",
       "      <td>10000</td>\n",
       "      <td>32</td>\n",
       "      <td>{'max_depth': 7, 'learning_rate': 0.05, 'num_l...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.880612</td>\n",
       "      <td>0.855431</td>\n",
       "      <td>0.845372</td>\n",
       "      <td>0.845025</td>\n",
       "      <td>0.853547</td>\n",
       "      <td>0.775740</td>\n",
       "      <td>0.776956</td>\n",
       "      <td>0.777227</td>\n",
       "      <td>0.774894</td>\n",
       "      <td>0.777196</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>226.068312</td>\n",
       "      <td>0.05</td>\n",
       "      <td>255</td>\n",
       "      <td>7</td>\n",
       "      <td>0.859351</td>\n",
       "      <td>0.776409</td>\n",
       "      <td>160</td>\n",
       "      <td>10000</td>\n",
       "      <td>32</td>\n",
       "      <td>{'max_depth': 7, 'learning_rate': 0.05, 'num_l...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.851097</td>\n",
       "      <td>0.873188</td>\n",
       "      <td>0.866143</td>\n",
       "      <td>0.843619</td>\n",
       "      <td>0.862705</td>\n",
       "      <td>0.775473</td>\n",
       "      <td>0.777569</td>\n",
       "      <td>0.776632</td>\n",
       "      <td>0.774995</td>\n",
       "      <td>0.777378</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     fit_time  learning_rate  max_bin  max_depth  mean_train_score  \\\n",
       "2  209.345991           0.05      255          7          0.879643   \n",
       "4  221.092630           0.05      255         -1          0.884262   \n",
       "7  230.559262           0.05      255         -1          0.886008   \n",
       "5  219.593719           0.05      255         -1          0.881182   \n",
       "3  243.125517           0.05      255         -1          0.888726   \n",
       "6  227.609989           0.05      255         -1          0.862194   \n",
       "9  220.591223           0.05      255          7          0.853910   \n",
       "8  205.577855           0.05      255         -1          0.850824   \n",
       "0  209.558289           0.05      255          7          0.855997   \n",
       "1  226.068312           0.05      255          7          0.859351   \n",
       "\n",
       "   mean_valid_score  min_child_samples  n_estimators  num_leaves  \\\n",
       "2          0.774872                160         10000          64   \n",
       "4          0.775698                 80         10000          64   \n",
       "7          0.775903                160         10000          64   \n",
       "5          0.775955                 80         10000          64   \n",
       "3          0.776094                 80         10000          64   \n",
       "6          0.776147                160         10000          32   \n",
       "9          0.776186                 80         10000          32   \n",
       "8          0.776347                160         10000          32   \n",
       "0          0.776403                 80         10000          32   \n",
       "1          0.776409                160         10000          32   \n",
       "\n",
       "                                              params  reg_alpha  reg_lambda  \\\n",
       "2  {'max_depth': 7, 'learning_rate': 0.05, 'num_l...        4.0         1.0   \n",
       "4  {'max_depth': -1, 'learning_rate': 0.05, 'num_...        2.0         1.0   \n",
       "7  {'max_depth': -1, 'learning_rate': 0.05, 'num_...        4.0         1.0   \n",
       "5  {'max_depth': -1, 'learning_rate': 0.05, 'num_...        4.0         1.0   \n",
       "3  {'max_depth': -1, 'learning_rate': 0.05, 'num_...        4.0         2.0   \n",
       "6  {'max_depth': -1, 'learning_rate': 0.05, 'num_...        2.0         1.0   \n",
       "9  {'max_depth': 7, 'learning_rate': 0.05, 'num_l...        4.0         1.0   \n",
       "8  {'max_depth': -1, 'learning_rate': 0.05, 'num_...        2.0         2.0   \n",
       "0  {'max_depth': 7, 'learning_rate': 0.05, 'num_l...        4.0         2.0   \n",
       "1  {'max_depth': 7, 'learning_rate': 0.05, 'num_l...        2.0         2.0   \n",
       "\n",
       "   train_score_0  train_score_1  train_score_2  train_score_3  train_score_4  \\\n",
       "2       0.884810       0.878929       0.870778       0.883375       0.880324   \n",
       "4       0.891202       0.895344       0.873181       0.876793       0.884790   \n",
       "7       0.885867       0.880709       0.900700       0.870250       0.892515   \n",
       "5       0.892885       0.882636       0.872181       0.884327       0.873883   \n",
       "3       0.903517       0.900325       0.869330       0.892090       0.878368   \n",
       "6       0.873431       0.859316       0.880181       0.851305       0.846737   \n",
       "9       0.864019       0.859661       0.842670       0.856077       0.847123   \n",
       "8       0.861142       0.847452       0.862335       0.843271       0.839920   \n",
       "0       0.880612       0.855431       0.845372       0.845025       0.853547   \n",
       "1       0.851097       0.873188       0.866143       0.843619       0.862705   \n",
       "\n",
       "   valid_score_0  valid_score_1  valid_score_2  valid_score_3  valid_score_4  \n",
       "2       0.773982       0.775994       0.775934       0.773367       0.775080  \n",
       "4       0.774554       0.776776       0.776170       0.774418       0.776569  \n",
       "7       0.775475       0.776003       0.776658       0.774745       0.776636  \n",
       "5       0.775252       0.776981       0.776253       0.774913       0.776378  \n",
       "3       0.775331       0.776650       0.776021       0.774950       0.777520  \n",
       "6       0.775149       0.777162       0.777237       0.774826       0.776359  \n",
       "9       0.775476       0.777402       0.776402       0.774785       0.776862  \n",
       "8       0.776083       0.777526       0.776533       0.774035       0.777561  \n",
       "0       0.775740       0.776956       0.777227       0.774894       0.777196  \n",
       "1       0.775473       0.777569       0.776632       0.774995       0.777378  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_results = pd.DataFrame(cv_results)\n",
    "df_results.sort_values('mean_valid_score', axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see best parameters, that sample which has the best score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'learning_rate': 0.05,\n",
       " 'max_bin': 255,\n",
       " 'max_depth': 7,\n",
       " 'min_child_samples': 160,\n",
       " 'n_estimators': 10000,\n",
       " 'num_leaves': 32,\n",
       " 'reg_alpha': 2.0,\n",
       " 'reg_lambda': 2.0}"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_index = df_results.mean_valid_score.idxmax()\n",
    "df_results.iloc[best_index]['params']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "fit_time                                                       226.068\n",
       "learning_rate                                                     0.05\n",
       "max_bin                                                            255\n",
       "max_depth                                                            7\n",
       "mean_train_score                                              0.859351\n",
       "mean_valid_score                                              0.776409\n",
       "min_child_samples                                                  160\n",
       "n_estimators                                                     10000\n",
       "num_leaves                                                          32\n",
       "params               {'max_depth': 7, 'learning_rate': 0.05, 'num_l...\n",
       "reg_alpha                                                            2\n",
       "reg_lambda                                                           2\n",
       "train_score_0                                                 0.851097\n",
       "train_score_1                                                 0.873188\n",
       "train_score_2                                                 0.866143\n",
       "train_score_3                                                 0.843619\n",
       "train_score_4                                                 0.862705\n",
       "valid_score_0                                                 0.775473\n",
       "valid_score_1                                                 0.777569\n",
       "valid_score_2                                                 0.776632\n",
       "valid_score_3                                                 0.774995\n",
       "valid_score_4                                                 0.777378\n",
       "Name: 1, dtype: object"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_results.iloc[best_index]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model K-Fold Execution\n",
    "Keep parameters separate to remember what parameters were used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "base_parameters = {\n",
    "    'learning_rate': 0.1,\n",
    "    'n_estimators': 5000,\n",
    "    'num_leaves': 31,\n",
    "    'max_depth': -1,\n",
    "    'min_child_samples': 20,\n",
    "    'reg_alpha': 0.1,\n",
    "    'reg_lambda': 0.1,\n",
    "    'max_bin': 255\n",
    "}\n",
    "\n",
    "previous_params = {\n",
    "    'learning_rate': 0.005,\n",
    "    'n_estimators': 32000,\n",
    "    'num_leaves': 32,\n",
    "    'max_depth': 7,\n",
    "    'min_child_samples': 160,\n",
    "    'reg_alpha': 2.0,\n",
    "    'reg_lambda': 2.0,\n",
    "    'max_bin': 255\n",
    "}\n",
    "\n",
    "best_params = {\n",
    "    'learning_rate': 0.01,\n",
    "    'n_estimators': 32000,\n",
    "    'num_leaves': 16,\n",
    "    'max_depth': 5,\n",
    "    'min_child_samples': 80,\n",
    "    'reg_alpha': 2.0,\n",
    "    'reg_lambda': 2.0,\n",
    "    'max_bin': 127\n",
    "}\n",
    "\n",
    "tuned_params = {\n",
    "    'learning_rate': 0.005,\n",
    "     'max_bin': 127,\n",
    "     'max_depth': 4,\n",
    "     'min_child_samples': 60,\n",
    "     'n_estimators': 10000,\n",
    "     'num_leaves': 7,\n",
    "     'reg_alpha': 10.0,\n",
    "     'reg_lambda': 10.0\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To evaluate how model works cross validation is used. K-Fold technique applied, it works the next way, train data data is split into k folds, then one of folds is chosen to be validation set others to be train set. Models are trained and evaluated with each fold, e.g. model1 trained on Fold1 + Fold2 + Fold3 + Fold4 and evaluated on Fold5. The AUC scores are collected for each model and summarized."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.714818\tvalid's auc: 0.709828\n",
      "[200]\ttrain's auc: 0.723555\tvalid's auc: 0.717431\n",
      "[300]\ttrain's auc: 0.73282\tvalid's auc: 0.725476\n",
      "[400]\ttrain's auc: 0.740211\tvalid's auc: 0.731988\n",
      "[500]\ttrain's auc: 0.746389\tvalid's auc: 0.737422\n",
      "[600]\ttrain's auc: 0.752085\tvalid's auc: 0.742595\n",
      "[700]\ttrain's auc: 0.75639\tvalid's auc: 0.74646\n",
      "[800]\ttrain's auc: 0.7599\tvalid's auc: 0.749569\n",
      "[900]\ttrain's auc: 0.762788\tvalid's auc: 0.752042\n",
      "[1000]\ttrain's auc: 0.765243\tvalid's auc: 0.754161\n",
      "[1100]\ttrain's auc: 0.767398\tvalid's auc: 0.755984\n",
      "[1200]\ttrain's auc: 0.769263\tvalid's auc: 0.75755\n",
      "[1300]\ttrain's auc: 0.770922\tvalid's auc: 0.758975\n",
      "[1400]\ttrain's auc: 0.772411\tvalid's auc: 0.760161\n",
      "[1500]\ttrain's auc: 0.773804\tvalid's auc: 0.761281\n",
      "[1600]\ttrain's auc: 0.775103\tvalid's auc: 0.762338\n",
      "[1700]\ttrain's auc: 0.776241\tvalid's auc: 0.763148\n",
      "[1800]\ttrain's auc: 0.77729\tvalid's auc: 0.763883\n",
      "[1900]\ttrain's auc: 0.778322\tvalid's auc: 0.764616\n",
      "[2000]\ttrain's auc: 0.77924\tvalid's auc: 0.765281\n",
      "[2100]\ttrain's auc: 0.780117\tvalid's auc: 0.765891\n",
      "[2200]\ttrain's auc: 0.780955\tvalid's auc: 0.766478\n",
      "[2300]\ttrain's auc: 0.781767\tvalid's auc: 0.767047\n",
      "[2400]\ttrain's auc: 0.782559\tvalid's auc: 0.767545\n",
      "[2500]\ttrain's auc: 0.783315\tvalid's auc: 0.768014\n",
      "[2600]\ttrain's auc: 0.784024\tvalid's auc: 0.768408\n",
      "[2700]\ttrain's auc: 0.784721\tvalid's auc: 0.76879\n",
      "[2800]\ttrain's auc: 0.785399\tvalid's auc: 0.769187\n",
      "[2900]\ttrain's auc: 0.786048\tvalid's auc: 0.769516\n",
      "[3000]\ttrain's auc: 0.786675\tvalid's auc: 0.769843\n",
      "[3100]\ttrain's auc: 0.787292\tvalid's auc: 0.770155\n",
      "[3200]\ttrain's auc: 0.787917\tvalid's auc: 0.770484\n",
      "[3300]\ttrain's auc: 0.788531\tvalid's auc: 0.77078\n",
      "[3400]\ttrain's auc: 0.789132\tvalid's auc: 0.771087\n",
      "[3500]\ttrain's auc: 0.789699\tvalid's auc: 0.77135\n",
      "[3600]\ttrain's auc: 0.790262\tvalid's auc: 0.771628\n",
      "[3700]\ttrain's auc: 0.790811\tvalid's auc: 0.771857\n",
      "[3800]\ttrain's auc: 0.791364\tvalid's auc: 0.772091\n",
      "[3900]\ttrain's auc: 0.791898\tvalid's auc: 0.772299\n",
      "[4000]\ttrain's auc: 0.792434\tvalid's auc: 0.772525\n",
      "[4100]\ttrain's auc: 0.792949\tvalid's auc: 0.772721\n",
      "[4200]\ttrain's auc: 0.793444\tvalid's auc: 0.772902\n",
      "[4300]\ttrain's auc: 0.793948\tvalid's auc: 0.773087\n",
      "[4400]\ttrain's auc: 0.794443\tvalid's auc: 0.77328\n",
      "[4500]\ttrain's auc: 0.794953\tvalid's auc: 0.773455\n",
      "[4600]\ttrain's auc: 0.795436\tvalid's auc: 0.77364\n",
      "[4700]\ttrain's auc: 0.795915\tvalid's auc: 0.773796\n",
      "[4800]\ttrain's auc: 0.796393\tvalid's auc: 0.773915\n",
      "[4900]\ttrain's auc: 0.796889\tvalid's auc: 0.774081\n",
      "[5000]\ttrain's auc: 0.797365\tvalid's auc: 0.774225\n",
      "[5100]\ttrain's auc: 0.797828\tvalid's auc: 0.774372\n",
      "[5200]\ttrain's auc: 0.798297\tvalid's auc: 0.774505\n",
      "[5300]\ttrain's auc: 0.798714\tvalid's auc: 0.774616\n",
      "[5400]\ttrain's auc: 0.799152\tvalid's auc: 0.774757\n",
      "[5500]\ttrain's auc: 0.799607\tvalid's auc: 0.774914\n",
      "[5600]\ttrain's auc: 0.800064\tvalid's auc: 0.775024\n",
      "[5700]\ttrain's auc: 0.800497\tvalid's auc: 0.775131\n",
      "[5800]\ttrain's auc: 0.800925\tvalid's auc: 0.775204\n",
      "[5900]\ttrain's auc: 0.801358\tvalid's auc: 0.775292\n",
      "[6000]\ttrain's auc: 0.801784\tvalid's auc: 0.775396\n",
      "[6100]\ttrain's auc: 0.802187\tvalid's auc: 0.775483\n",
      "[6200]\ttrain's auc: 0.802609\tvalid's auc: 0.775576\n",
      "[6300]\ttrain's auc: 0.803035\tvalid's auc: 0.77566\n",
      "[6400]\ttrain's auc: 0.803446\tvalid's auc: 0.775739\n",
      "[6500]\ttrain's auc: 0.803845\tvalid's auc: 0.775809\n",
      "[6600]\ttrain's auc: 0.80425\tvalid's auc: 0.7759\n",
      "[6700]\ttrain's auc: 0.804659\tvalid's auc: 0.775959\n",
      "[6800]\ttrain's auc: 0.805046\tvalid's auc: 0.776047\n",
      "[6900]\ttrain's auc: 0.805454\tvalid's auc: 0.77614\n",
      "[7000]\ttrain's auc: 0.805837\tvalid's auc: 0.776232\n",
      "[7100]\ttrain's auc: 0.80622\tvalid's auc: 0.776296\n",
      "[7200]\ttrain's auc: 0.806604\tvalid's auc: 0.776375\n",
      "[7300]\ttrain's auc: 0.806999\tvalid's auc: 0.776451\n",
      "[7400]\ttrain's auc: 0.80737\tvalid's auc: 0.776493\n",
      "[7500]\ttrain's auc: 0.807748\tvalid's auc: 0.776565\n",
      "[7600]\ttrain's auc: 0.808129\tvalid's auc: 0.776634\n",
      "[7700]\ttrain's auc: 0.808513\tvalid's auc: 0.776697\n",
      "[7800]\ttrain's auc: 0.808894\tvalid's auc: 0.776766\n",
      "[7900]\ttrain's auc: 0.809275\tvalid's auc: 0.776805\n",
      "[8000]\ttrain's auc: 0.809651\tvalid's auc: 0.776861\n",
      "[8100]\ttrain's auc: 0.810011\tvalid's auc: 0.776944\n",
      "[8200]\ttrain's auc: 0.810392\tvalid's auc: 0.776978\n",
      "[8300]\ttrain's auc: 0.810755\tvalid's auc: 0.777018\n",
      "[8400]\ttrain's auc: 0.811115\tvalid's auc: 0.77703\n",
      "[8500]\ttrain's auc: 0.811461\tvalid's auc: 0.777079\n",
      "[8600]\ttrain's auc: 0.811827\tvalid's auc: 0.777122\n",
      "[8700]\ttrain's auc: 0.812186\tvalid's auc: 0.777163\n",
      "[8800]\ttrain's auc: 0.812538\tvalid's auc: 0.777219\n",
      "[8900]\ttrain's auc: 0.812887\tvalid's auc: 0.77725\n",
      "[9000]\ttrain's auc: 0.813243\tvalid's auc: 0.777294\n",
      "[9100]\ttrain's auc: 0.813587\tvalid's auc: 0.777343\n",
      "[9200]\ttrain's auc: 0.81394\tvalid's auc: 0.777396\n",
      "[9300]\ttrain's auc: 0.814283\tvalid's auc: 0.777423\n",
      "[9400]\ttrain's auc: 0.814625\tvalid's auc: 0.777445\n",
      "[9500]\ttrain's auc: 0.814967\tvalid's auc: 0.7775\n",
      "[9600]\ttrain's auc: 0.815298\tvalid's auc: 0.777522\n",
      "[9700]\ttrain's auc: 0.815662\tvalid's auc: 0.777559\n",
      "[9800]\ttrain's auc: 0.816006\tvalid's auc: 0.777564\n",
      "[9900]\ttrain's auc: 0.816354\tvalid's auc: 0.777602\n",
      "[10000]\ttrain's auc: 0.81667\tvalid's auc: 0.777629\n",
      "Did not meet early stopping. Best iteration is:\n",
      "[10000]\ttrain's auc: 0.81667\tvalid's auc: 0.777629\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.713338\tvalid's auc: 0.712691\n",
      "[200]\ttrain's auc: 0.722321\tvalid's auc: 0.72096\n",
      "[300]\ttrain's auc: 0.731629\tvalid's auc: 0.729271\n",
      "[400]\ttrain's auc: 0.739066\tvalid's auc: 0.736036\n",
      "[500]\ttrain's auc: 0.745307\tvalid's auc: 0.741558\n",
      "[600]\ttrain's auc: 0.750821\tvalid's auc: 0.746399\n",
      "[700]\ttrain's auc: 0.755442\tvalid's auc: 0.750283\n",
      "[800]\ttrain's auc: 0.758996\tvalid's auc: 0.753297\n",
      "[900]\ttrain's auc: 0.762036\tvalid's auc: 0.755815\n",
      "[1000]\ttrain's auc: 0.764489\tvalid's auc: 0.757858\n",
      "[1100]\ttrain's auc: 0.766687\tvalid's auc: 0.759592\n",
      "[1200]\ttrain's auc: 0.768506\tvalid's auc: 0.76102\n",
      "[1300]\ttrain's auc: 0.770184\tvalid's auc: 0.762352\n",
      "[1400]\ttrain's auc: 0.771645\tvalid's auc: 0.763516\n",
      "[1500]\ttrain's auc: 0.773037\tvalid's auc: 0.764545\n",
      "[1600]\ttrain's auc: 0.774332\tvalid's auc: 0.765496\n",
      "[1700]\ttrain's auc: 0.775487\tvalid's auc: 0.766284\n",
      "[1800]\ttrain's auc: 0.776542\tvalid's auc: 0.76707\n",
      "[1900]\ttrain's auc: 0.777553\tvalid's auc: 0.767775\n",
      "[2000]\ttrain's auc: 0.778479\tvalid's auc: 0.768398\n",
      "[2100]\ttrain's auc: 0.779404\tvalid's auc: 0.768984\n",
      "[2200]\ttrain's auc: 0.780288\tvalid's auc: 0.769556\n",
      "[2300]\ttrain's auc: 0.781117\tvalid's auc: 0.770096\n",
      "[2400]\ttrain's auc: 0.781909\tvalid's auc: 0.770514\n",
      "[2500]\ttrain's auc: 0.782651\tvalid's auc: 0.770872\n",
      "[2600]\ttrain's auc: 0.783376\tvalid's auc: 0.77126\n",
      "[2700]\ttrain's auc: 0.784087\tvalid's auc: 0.771638\n",
      "[2800]\ttrain's auc: 0.784748\tvalid's auc: 0.771966\n",
      "[2900]\ttrain's auc: 0.785412\tvalid's auc: 0.772289\n",
      "[3000]\ttrain's auc: 0.786052\tvalid's auc: 0.772571\n",
      "[3100]\ttrain's auc: 0.786676\tvalid's auc: 0.772861\n",
      "[3200]\ttrain's auc: 0.787298\tvalid's auc: 0.773113\n",
      "[3300]\ttrain's auc: 0.78792\tvalid's auc: 0.773369\n",
      "[3400]\ttrain's auc: 0.788514\tvalid's auc: 0.773585\n",
      "[3500]\ttrain's auc: 0.789093\tvalid's auc: 0.773801\n",
      "[3600]\ttrain's auc: 0.789656\tvalid's auc: 0.774033\n",
      "[3700]\ttrain's auc: 0.790212\tvalid's auc: 0.774239\n",
      "[3800]\ttrain's auc: 0.790776\tvalid's auc: 0.774438\n",
      "[3900]\ttrain's auc: 0.791301\tvalid's auc: 0.77464\n",
      "[4000]\ttrain's auc: 0.791822\tvalid's auc: 0.774763\n",
      "[4100]\ttrain's auc: 0.792347\tvalid's auc: 0.774954\n",
      "[4200]\ttrain's auc: 0.792865\tvalid's auc: 0.77514\n",
      "[4300]\ttrain's auc: 0.793384\tvalid's auc: 0.775303\n",
      "[4400]\ttrain's auc: 0.793886\tvalid's auc: 0.775424\n",
      "[4500]\ttrain's auc: 0.794375\tvalid's auc: 0.775563\n",
      "[4600]\ttrain's auc: 0.794874\tvalid's auc: 0.775692\n",
      "[4700]\ttrain's auc: 0.795353\tvalid's auc: 0.775814\n",
      "[4800]\ttrain's auc: 0.795838\tvalid's auc: 0.775954\n",
      "[4900]\ttrain's auc: 0.796328\tvalid's auc: 0.776094\n",
      "[5000]\ttrain's auc: 0.796811\tvalid's auc: 0.776207\n",
      "[5100]\ttrain's auc: 0.797269\tvalid's auc: 0.776295\n",
      "[5200]\ttrain's auc: 0.797722\tvalid's auc: 0.776404\n",
      "[5300]\ttrain's auc: 0.798171\tvalid's auc: 0.77655\n",
      "[5400]\ttrain's auc: 0.798601\tvalid's auc: 0.776644\n",
      "[5500]\ttrain's auc: 0.799054\tvalid's auc: 0.776709\n",
      "[5600]\ttrain's auc: 0.799492\tvalid's auc: 0.776772\n",
      "[5700]\ttrain's auc: 0.79996\tvalid's auc: 0.776851\n",
      "[5800]\ttrain's auc: 0.800404\tvalid's auc: 0.776939\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[5900]\ttrain's auc: 0.800858\tvalid's auc: 0.777029\n",
      "[6000]\ttrain's auc: 0.801296\tvalid's auc: 0.777101\n",
      "[6100]\ttrain's auc: 0.801725\tvalid's auc: 0.777173\n",
      "[6200]\ttrain's auc: 0.802137\tvalid's auc: 0.77722\n",
      "[6300]\ttrain's auc: 0.802553\tvalid's auc: 0.777307\n",
      "[6400]\ttrain's auc: 0.802963\tvalid's auc: 0.777385\n",
      "[6500]\ttrain's auc: 0.803372\tvalid's auc: 0.77745\n",
      "[6600]\ttrain's auc: 0.803798\tvalid's auc: 0.777483\n",
      "[6700]\ttrain's auc: 0.804216\tvalid's auc: 0.77756\n",
      "[6800]\ttrain's auc: 0.804627\tvalid's auc: 0.77762\n",
      "[6900]\ttrain's auc: 0.805035\tvalid's auc: 0.777661\n",
      "[7000]\ttrain's auc: 0.805427\tvalid's auc: 0.777687\n",
      "[7100]\ttrain's auc: 0.805835\tvalid's auc: 0.777735\n",
      "[7200]\ttrain's auc: 0.806212\tvalid's auc: 0.777784\n",
      "[7300]\ttrain's auc: 0.806594\tvalid's auc: 0.777823\n",
      "[7400]\ttrain's auc: 0.806983\tvalid's auc: 0.777838\n",
      "[7500]\ttrain's auc: 0.807368\tvalid's auc: 0.777891\n",
      "[7600]\ttrain's auc: 0.807758\tvalid's auc: 0.777923\n",
      "[7700]\ttrain's auc: 0.808139\tvalid's auc: 0.777959\n",
      "[7800]\ttrain's auc: 0.808558\tvalid's auc: 0.778048\n",
      "[7900]\ttrain's auc: 0.80894\tvalid's auc: 0.778093\n",
      "[8000]\ttrain's auc: 0.809335\tvalid's auc: 0.77813\n",
      "[8100]\ttrain's auc: 0.809712\tvalid's auc: 0.778181\n",
      "[8200]\ttrain's auc: 0.81008\tvalid's auc: 0.778204\n",
      "[8300]\ttrain's auc: 0.810449\tvalid's auc: 0.778246\n",
      "[8400]\ttrain's auc: 0.810791\tvalid's auc: 0.778283\n",
      "[8500]\ttrain's auc: 0.811152\tvalid's auc: 0.778323\n",
      "[8600]\ttrain's auc: 0.811517\tvalid's auc: 0.778341\n",
      "[8700]\ttrain's auc: 0.811877\tvalid's auc: 0.778379\n",
      "Early stopping, best iteration is:\n",
      "[8681]\ttrain's auc: 0.811814\tvalid's auc: 0.778383\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.713353\tvalid's auc: 0.708871\n",
      "[200]\ttrain's auc: 0.722229\tvalid's auc: 0.71843\n",
      "[300]\ttrain's auc: 0.731397\tvalid's auc: 0.72764\n",
      "[400]\ttrain's auc: 0.738628\tvalid's auc: 0.734562\n",
      "[500]\ttrain's auc: 0.745035\tvalid's auc: 0.740726\n",
      "[600]\ttrain's auc: 0.750547\tvalid's auc: 0.746038\n",
      "[700]\ttrain's auc: 0.754798\tvalid's auc: 0.750176\n",
      "[800]\ttrain's auc: 0.758352\tvalid's auc: 0.753421\n",
      "[900]\ttrain's auc: 0.761335\tvalid's auc: 0.75607\n",
      "[1000]\ttrain's auc: 0.763769\tvalid's auc: 0.758251\n",
      "[1100]\ttrain's auc: 0.765913\tvalid's auc: 0.760012\n",
      "[1200]\ttrain's auc: 0.767868\tvalid's auc: 0.76166\n",
      "[1300]\ttrain's auc: 0.769643\tvalid's auc: 0.763119\n",
      "[1400]\ttrain's auc: 0.771187\tvalid's auc: 0.764356\n",
      "[1500]\ttrain's auc: 0.772557\tvalid's auc: 0.765385\n",
      "[1600]\ttrain's auc: 0.77385\tvalid's auc: 0.766336\n",
      "[1700]\ttrain's auc: 0.775\tvalid's auc: 0.767112\n",
      "[1800]\ttrain's auc: 0.776109\tvalid's auc: 0.767836\n",
      "[1900]\ttrain's auc: 0.777119\tvalid's auc: 0.768489\n",
      "[2000]\ttrain's auc: 0.778075\tvalid's auc: 0.769021\n",
      "[2100]\ttrain's auc: 0.77899\tvalid's auc: 0.769538\n",
      "[2200]\ttrain's auc: 0.779869\tvalid's auc: 0.770047\n",
      "[2300]\ttrain's auc: 0.780718\tvalid's auc: 0.770535\n",
      "[2400]\ttrain's auc: 0.781513\tvalid's auc: 0.77096\n",
      "[2500]\ttrain's auc: 0.782278\tvalid's auc: 0.771346\n",
      "[2600]\ttrain's auc: 0.783003\tvalid's auc: 0.771718\n",
      "[2700]\ttrain's auc: 0.783708\tvalid's auc: 0.772056\n",
      "[2800]\ttrain's auc: 0.784395\tvalid's auc: 0.772352\n",
      "[2900]\ttrain's auc: 0.785065\tvalid's auc: 0.772661\n",
      "[3000]\ttrain's auc: 0.785738\tvalid's auc: 0.772928\n",
      "[3100]\ttrain's auc: 0.786363\tvalid's auc: 0.773215\n",
      "[3200]\ttrain's auc: 0.787004\tvalid's auc: 0.77351\n",
      "[3300]\ttrain's auc: 0.787607\tvalid's auc: 0.773716\n",
      "[3400]\ttrain's auc: 0.788193\tvalid's auc: 0.773924\n",
      "[3500]\ttrain's auc: 0.788764\tvalid's auc: 0.774143\n",
      "[3600]\ttrain's auc: 0.789336\tvalid's auc: 0.77435\n",
      "[3700]\ttrain's auc: 0.789865\tvalid's auc: 0.774537\n",
      "[3800]\ttrain's auc: 0.790425\tvalid's auc: 0.774711\n",
      "[3900]\ttrain's auc: 0.790986\tvalid's auc: 0.774902\n",
      "[4000]\ttrain's auc: 0.791512\tvalid's auc: 0.775053\n",
      "[4100]\ttrain's auc: 0.792053\tvalid's auc: 0.775216\n",
      "[4200]\ttrain's auc: 0.792566\tvalid's auc: 0.775387\n",
      "[4300]\ttrain's auc: 0.793099\tvalid's auc: 0.775512\n",
      "[4400]\ttrain's auc: 0.793606\tvalid's auc: 0.775673\n",
      "[4500]\ttrain's auc: 0.794101\tvalid's auc: 0.775821\n",
      "[4600]\ttrain's auc: 0.794605\tvalid's auc: 0.775948\n",
      "[4700]\ttrain's auc: 0.795088\tvalid's auc: 0.776055\n",
      "[4800]\ttrain's auc: 0.795564\tvalid's auc: 0.77618\n",
      "[4900]\ttrain's auc: 0.796028\tvalid's auc: 0.776282\n",
      "[5000]\ttrain's auc: 0.796516\tvalid's auc: 0.776398\n",
      "[5100]\ttrain's auc: 0.796992\tvalid's auc: 0.776497\n",
      "[5200]\ttrain's auc: 0.797441\tvalid's auc: 0.776601\n",
      "[5300]\ttrain's auc: 0.797899\tvalid's auc: 0.776707\n",
      "[5400]\ttrain's auc: 0.798355\tvalid's auc: 0.776803\n",
      "[5500]\ttrain's auc: 0.798824\tvalid's auc: 0.77692\n",
      "[5600]\ttrain's auc: 0.799291\tvalid's auc: 0.777013\n",
      "[5700]\ttrain's auc: 0.799732\tvalid's auc: 0.777124\n",
      "[5800]\ttrain's auc: 0.800168\tvalid's auc: 0.777199\n",
      "[5900]\ttrain's auc: 0.800601\tvalid's auc: 0.777299\n",
      "[6000]\ttrain's auc: 0.801032\tvalid's auc: 0.777389\n",
      "[6100]\ttrain's auc: 0.801467\tvalid's auc: 0.777458\n",
      "[6200]\ttrain's auc: 0.801896\tvalid's auc: 0.777529\n",
      "[6300]\ttrain's auc: 0.802326\tvalid's auc: 0.777612\n",
      "[6400]\ttrain's auc: 0.802734\tvalid's auc: 0.777674\n",
      "[6500]\ttrain's auc: 0.803161\tvalid's auc: 0.777753\n",
      "[6600]\ttrain's auc: 0.803575\tvalid's auc: 0.777836\n",
      "[6700]\ttrain's auc: 0.803984\tvalid's auc: 0.777876\n",
      "[6800]\ttrain's auc: 0.804378\tvalid's auc: 0.777953\n",
      "[6900]\ttrain's auc: 0.804787\tvalid's auc: 0.777993\n",
      "[7000]\ttrain's auc: 0.805194\tvalid's auc: 0.778024\n",
      "[7100]\ttrain's auc: 0.805581\tvalid's auc: 0.778074\n",
      "[7200]\ttrain's auc: 0.805967\tvalid's auc: 0.778138\n",
      "[7300]\ttrain's auc: 0.806358\tvalid's auc: 0.778185\n",
      "[7400]\ttrain's auc: 0.806753\tvalid's auc: 0.778221\n",
      "[7500]\ttrain's auc: 0.807159\tvalid's auc: 0.778283\n",
      "[7600]\ttrain's auc: 0.807544\tvalid's auc: 0.778327\n",
      "[7700]\ttrain's auc: 0.807922\tvalid's auc: 0.77839\n",
      "[7800]\ttrain's auc: 0.808293\tvalid's auc: 0.778448\n",
      "[7900]\ttrain's auc: 0.808668\tvalid's auc: 0.778479\n",
      "[8000]\ttrain's auc: 0.809047\tvalid's auc: 0.778524\n",
      "[8100]\ttrain's auc: 0.809431\tvalid's auc: 0.778573\n",
      "[8200]\ttrain's auc: 0.809798\tvalid's auc: 0.778642\n",
      "[8300]\ttrain's auc: 0.810159\tvalid's auc: 0.778678\n",
      "[8400]\ttrain's auc: 0.810518\tvalid's auc: 0.778702\n",
      "[8500]\ttrain's auc: 0.810871\tvalid's auc: 0.778721\n",
      "[8600]\ttrain's auc: 0.811242\tvalid's auc: 0.778747\n",
      "[8700]\ttrain's auc: 0.811611\tvalid's auc: 0.778791\n",
      "[8800]\ttrain's auc: 0.811965\tvalid's auc: 0.778816\n",
      "[8900]\ttrain's auc: 0.812311\tvalid's auc: 0.778857\n",
      "[9000]\ttrain's auc: 0.812672\tvalid's auc: 0.778884\n",
      "[9100]\ttrain's auc: 0.813032\tvalid's auc: 0.778921\n",
      "[9200]\ttrain's auc: 0.813387\tvalid's auc: 0.778986\n",
      "[9300]\ttrain's auc: 0.813745\tvalid's auc: 0.779004\n",
      "[9400]\ttrain's auc: 0.814112\tvalid's auc: 0.779046\n",
      "[9500]\ttrain's auc: 0.814467\tvalid's auc: 0.779074\n",
      "[9600]\ttrain's auc: 0.814806\tvalid's auc: 0.779091\n",
      "[9700]\ttrain's auc: 0.815164\tvalid's auc: 0.779145\n",
      "[9800]\ttrain's auc: 0.815511\tvalid's auc: 0.779181\n",
      "[9900]\ttrain's auc: 0.815861\tvalid's auc: 0.779209\n",
      "[10000]\ttrain's auc: 0.816196\tvalid's auc: 0.779232\n",
      "Did not meet early stopping. Best iteration is:\n",
      "[10000]\ttrain's auc: 0.816196\tvalid's auc: 0.779232\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.714076\tvalid's auc: 0.71285\n",
      "[200]\ttrain's auc: 0.722476\tvalid's auc: 0.72013\n",
      "[300]\ttrain's auc: 0.731133\tvalid's auc: 0.728306\n",
      "[400]\ttrain's auc: 0.73895\tvalid's auc: 0.735613\n",
      "[500]\ttrain's auc: 0.745424\tvalid's auc: 0.74144\n",
      "[600]\ttrain's auc: 0.750892\tvalid's auc: 0.746243\n",
      "[700]\ttrain's auc: 0.755358\tvalid's auc: 0.750169\n",
      "[800]\ttrain's auc: 0.758905\tvalid's auc: 0.753184\n",
      "[900]\ttrain's auc: 0.7618\tvalid's auc: 0.755658\n",
      "[1000]\ttrain's auc: 0.764242\tvalid's auc: 0.757685\n",
      "[1100]\ttrain's auc: 0.76643\tvalid's auc: 0.759535\n",
      "[1200]\ttrain's auc: 0.768299\tvalid's auc: 0.761047\n",
      "[1300]\ttrain's auc: 0.769995\tvalid's auc: 0.762419\n",
      "[1400]\ttrain's auc: 0.771522\tvalid's auc: 0.763608\n",
      "[1500]\ttrain's auc: 0.7729\tvalid's auc: 0.764606\n",
      "[1600]\ttrain's auc: 0.774126\tvalid's auc: 0.765445\n",
      "[1700]\ttrain's auc: 0.775325\tvalid's auc: 0.766196\n",
      "[1800]\ttrain's auc: 0.776433\tvalid's auc: 0.766889\n",
      "[1900]\ttrain's auc: 0.777442\tvalid's auc: 0.767478\n",
      "[2000]\ttrain's auc: 0.778418\tvalid's auc: 0.768083\n",
      "[2100]\ttrain's auc: 0.779346\tvalid's auc: 0.768662\n",
      "[2200]\ttrain's auc: 0.780231\tvalid's auc: 0.769145\n",
      "[2300]\ttrain's auc: 0.78107\tvalid's auc: 0.769605\n",
      "[2400]\ttrain's auc: 0.78188\tvalid's auc: 0.770023\n",
      "[2500]\ttrain's auc: 0.78264\tvalid's auc: 0.770428\n",
      "[2600]\ttrain's auc: 0.78337\tvalid's auc: 0.770725\n",
      "[2700]\ttrain's auc: 0.784096\tvalid's auc: 0.771068\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2800]\ttrain's auc: 0.784782\tvalid's auc: 0.771362\n",
      "[2900]\ttrain's auc: 0.785455\tvalid's auc: 0.771651\n",
      "[3000]\ttrain's auc: 0.786096\tvalid's auc: 0.771897\n",
      "[3100]\ttrain's auc: 0.786722\tvalid's auc: 0.772173\n",
      "[3200]\ttrain's auc: 0.787334\tvalid's auc: 0.772447\n",
      "[3300]\ttrain's auc: 0.787939\tvalid's auc: 0.772646\n",
      "[3400]\ttrain's auc: 0.788548\tvalid's auc: 0.772889\n",
      "[3500]\ttrain's auc: 0.789134\tvalid's auc: 0.773106\n",
      "[3600]\ttrain's auc: 0.7897\tvalid's auc: 0.773259\n",
      "[3700]\ttrain's auc: 0.790281\tvalid's auc: 0.773485\n",
      "[3800]\ttrain's auc: 0.790857\tvalid's auc: 0.773676\n",
      "[3900]\ttrain's auc: 0.791407\tvalid's auc: 0.773824\n",
      "[4000]\ttrain's auc: 0.791949\tvalid's auc: 0.774002\n",
      "[4100]\ttrain's auc: 0.79248\tvalid's auc: 0.77415\n",
      "[4200]\ttrain's auc: 0.793011\tvalid's auc: 0.774319\n",
      "[4300]\ttrain's auc: 0.793535\tvalid's auc: 0.774481\n",
      "[4400]\ttrain's auc: 0.794063\tvalid's auc: 0.774619\n",
      "[4500]\ttrain's auc: 0.794547\tvalid's auc: 0.774737\n",
      "[4600]\ttrain's auc: 0.795055\tvalid's auc: 0.774845\n",
      "[4700]\ttrain's auc: 0.795559\tvalid's auc: 0.774966\n",
      "[4800]\ttrain's auc: 0.796031\tvalid's auc: 0.775076\n",
      "[4900]\ttrain's auc: 0.796531\tvalid's auc: 0.775177\n",
      "[5000]\ttrain's auc: 0.796994\tvalid's auc: 0.775288\n",
      "[5100]\ttrain's auc: 0.797475\tvalid's auc: 0.775374\n",
      "[5200]\ttrain's auc: 0.797941\tvalid's auc: 0.775461\n",
      "[5300]\ttrain's auc: 0.798397\tvalid's auc: 0.775542\n",
      "[5400]\ttrain's auc: 0.798837\tvalid's auc: 0.775631\n",
      "[5500]\ttrain's auc: 0.799284\tvalid's auc: 0.775751\n",
      "[5600]\ttrain's auc: 0.799728\tvalid's auc: 0.775835\n",
      "[5700]\ttrain's auc: 0.800174\tvalid's auc: 0.775896\n",
      "[5800]\ttrain's auc: 0.800627\tvalid's auc: 0.775993\n",
      "[5900]\ttrain's auc: 0.801054\tvalid's auc: 0.776062\n",
      "[6000]\ttrain's auc: 0.801491\tvalid's auc: 0.776139\n",
      "[6100]\ttrain's auc: 0.801921\tvalid's auc: 0.776183\n",
      "[6200]\ttrain's auc: 0.802332\tvalid's auc: 0.776243\n",
      "[6300]\ttrain's auc: 0.802755\tvalid's auc: 0.776284\n",
      "[6400]\ttrain's auc: 0.803179\tvalid's auc: 0.776293\n",
      "[6500]\ttrain's auc: 0.8036\tvalid's auc: 0.776346\n",
      "[6600]\ttrain's auc: 0.804013\tvalid's auc: 0.776382\n",
      "[6700]\ttrain's auc: 0.804412\tvalid's auc: 0.776425\n",
      "[6800]\ttrain's auc: 0.804808\tvalid's auc: 0.77647\n",
      "[6900]\ttrain's auc: 0.805203\tvalid's auc: 0.776511\n",
      "[7000]\ttrain's auc: 0.805624\tvalid's auc: 0.776551\n",
      "[7100]\ttrain's auc: 0.806028\tvalid's auc: 0.776613\n",
      "[7200]\ttrain's auc: 0.806434\tvalid's auc: 0.776659\n",
      "[7300]\ttrain's auc: 0.806834\tvalid's auc: 0.776708\n",
      "[7400]\ttrain's auc: 0.807223\tvalid's auc: 0.776747\n",
      "[7500]\ttrain's auc: 0.807601\tvalid's auc: 0.776787\n",
      "[7600]\ttrain's auc: 0.807993\tvalid's auc: 0.776834\n",
      "[7700]\ttrain's auc: 0.808365\tvalid's auc: 0.776882\n",
      "[7800]\ttrain's auc: 0.808751\tvalid's auc: 0.7769\n",
      "[7900]\ttrain's auc: 0.80912\tvalid's auc: 0.776921\n",
      "[8000]\ttrain's auc: 0.809475\tvalid's auc: 0.776928\n",
      "[8100]\ttrain's auc: 0.809823\tvalid's auc: 0.776953\n",
      "[8200]\ttrain's auc: 0.810211\tvalid's auc: 0.776966\n",
      "[8300]\ttrain's auc: 0.810577\tvalid's auc: 0.777006\n",
      "[8400]\ttrain's auc: 0.810936\tvalid's auc: 0.777038\n",
      "[8500]\ttrain's auc: 0.811299\tvalid's auc: 0.777074\n",
      "[8600]\ttrain's auc: 0.811658\tvalid's auc: 0.777103\n",
      "[8700]\ttrain's auc: 0.812034\tvalid's auc: 0.777159\n",
      "[8800]\ttrain's auc: 0.812409\tvalid's auc: 0.777185\n",
      "[8900]\ttrain's auc: 0.812789\tvalid's auc: 0.777249\n",
      "[9000]\ttrain's auc: 0.813139\tvalid's auc: 0.777272\n",
      "[9100]\ttrain's auc: 0.813464\tvalid's auc: 0.777312\n",
      "[9200]\ttrain's auc: 0.813818\tvalid's auc: 0.777342\n",
      "[9300]\ttrain's auc: 0.81418\tvalid's auc: 0.777362\n",
      "[9400]\ttrain's auc: 0.814519\tvalid's auc: 0.777366\n",
      "[9500]\ttrain's auc: 0.814875\tvalid's auc: 0.777387\n",
      "[9600]\ttrain's auc: 0.815221\tvalid's auc: 0.777397\n",
      "[9700]\ttrain's auc: 0.815559\tvalid's auc: 0.777421\n",
      "[9800]\ttrain's auc: 0.815895\tvalid's auc: 0.777442\n",
      "[9900]\ttrain's auc: 0.816243\tvalid's auc: 0.777435\n",
      "Early stopping, best iteration is:\n",
      "[9849]\ttrain's auc: 0.816061\tvalid's auc: 0.777448\n",
      "Training until validation scores don't improve for 100 rounds.\n",
      "[100]\ttrain's auc: 0.713124\tvalid's auc: 0.712505\n",
      "[200]\ttrain's auc: 0.721477\tvalid's auc: 0.72013\n",
      "[300]\ttrain's auc: 0.730824\tvalid's auc: 0.728856\n",
      "[400]\ttrain's auc: 0.737977\tvalid's auc: 0.735668\n",
      "[500]\ttrain's auc: 0.744354\tvalid's auc: 0.741779\n",
      "[600]\ttrain's auc: 0.749902\tvalid's auc: 0.747025\n",
      "[700]\ttrain's auc: 0.754378\tvalid's auc: 0.751229\n",
      "[800]\ttrain's auc: 0.758019\tvalid's auc: 0.754493\n",
      "[900]\ttrain's auc: 0.761005\tvalid's auc: 0.757083\n",
      "[1000]\ttrain's auc: 0.763482\tvalid's auc: 0.75921\n",
      "[1100]\ttrain's auc: 0.76561\tvalid's auc: 0.760905\n",
      "[1200]\ttrain's auc: 0.767619\tvalid's auc: 0.762569\n",
      "[1300]\ttrain's auc: 0.769395\tvalid's auc: 0.763988\n",
      "[1400]\ttrain's auc: 0.770919\tvalid's auc: 0.765157\n",
      "[1500]\ttrain's auc: 0.77236\tvalid's auc: 0.766218\n",
      "[1600]\ttrain's auc: 0.77366\tvalid's auc: 0.767163\n",
      "[1700]\ttrain's auc: 0.774848\tvalid's auc: 0.767974\n",
      "[1800]\ttrain's auc: 0.775955\tvalid's auc: 0.768693\n",
      "[1900]\ttrain's auc: 0.776973\tvalid's auc: 0.769318\n",
      "[2000]\ttrain's auc: 0.777926\tvalid's auc: 0.769893\n",
      "[2100]\ttrain's auc: 0.778832\tvalid's auc: 0.770437\n",
      "[2200]\ttrain's auc: 0.779713\tvalid's auc: 0.770929\n",
      "[2300]\ttrain's auc: 0.78055\tvalid's auc: 0.771379\n",
      "[2400]\ttrain's auc: 0.781338\tvalid's auc: 0.771803\n",
      "[2500]\ttrain's auc: 0.782122\tvalid's auc: 0.772213\n",
      "[2600]\ttrain's auc: 0.782857\tvalid's auc: 0.772579\n",
      "[2700]\ttrain's auc: 0.783545\tvalid's auc: 0.772886\n",
      "[2800]\ttrain's auc: 0.784224\tvalid's auc: 0.773197\n",
      "[2900]\ttrain's auc: 0.784872\tvalid's auc: 0.773505\n",
      "[3000]\ttrain's auc: 0.785514\tvalid's auc: 0.77379\n",
      "[3100]\ttrain's auc: 0.786146\tvalid's auc: 0.774048\n",
      "[3200]\ttrain's auc: 0.786766\tvalid's auc: 0.77433\n",
      "[3300]\ttrain's auc: 0.787393\tvalid's auc: 0.774606\n",
      "[3400]\ttrain's auc: 0.787981\tvalid's auc: 0.774843\n",
      "[3500]\ttrain's auc: 0.788583\tvalid's auc: 0.775062\n",
      "[3600]\ttrain's auc: 0.789155\tvalid's auc: 0.775289\n",
      "[3700]\ttrain's auc: 0.789716\tvalid's auc: 0.775454\n",
      "[3800]\ttrain's auc: 0.790286\tvalid's auc: 0.775631\n",
      "[3900]\ttrain's auc: 0.790838\tvalid's auc: 0.77579\n",
      "[4000]\ttrain's auc: 0.791391\tvalid's auc: 0.775954\n",
      "[4100]\ttrain's auc: 0.791957\tvalid's auc: 0.776124\n",
      "[4200]\ttrain's auc: 0.792497\tvalid's auc: 0.776282\n",
      "[4300]\ttrain's auc: 0.79301\tvalid's auc: 0.776413\n",
      "[4400]\ttrain's auc: 0.793521\tvalid's auc: 0.776555\n",
      "[4500]\ttrain's auc: 0.794042\tvalid's auc: 0.77671\n",
      "[4600]\ttrain's auc: 0.794553\tvalid's auc: 0.776827\n",
      "[4700]\ttrain's auc: 0.795044\tvalid's auc: 0.776942\n",
      "[4800]\ttrain's auc: 0.795533\tvalid's auc: 0.77706\n",
      "[4900]\ttrain's auc: 0.796023\tvalid's auc: 0.777189\n",
      "[5000]\ttrain's auc: 0.796475\tvalid's auc: 0.777274\n",
      "[5100]\ttrain's auc: 0.796938\tvalid's auc: 0.77737\n",
      "[5200]\ttrain's auc: 0.797423\tvalid's auc: 0.777506\n",
      "[5300]\ttrain's auc: 0.797909\tvalid's auc: 0.777617\n",
      "[5400]\ttrain's auc: 0.798341\tvalid's auc: 0.777665\n",
      "[5500]\ttrain's auc: 0.79879\tvalid's auc: 0.77776\n",
      "[5600]\ttrain's auc: 0.799239\tvalid's auc: 0.777835\n",
      "[5700]\ttrain's auc: 0.799707\tvalid's auc: 0.777916\n",
      "[5800]\ttrain's auc: 0.800146\tvalid's auc: 0.778003\n",
      "[5900]\ttrain's auc: 0.800587\tvalid's auc: 0.778078\n",
      "[6000]\ttrain's auc: 0.801021\tvalid's auc: 0.778139\n",
      "[6100]\ttrain's auc: 0.801458\tvalid's auc: 0.77823\n",
      "[6200]\ttrain's auc: 0.801871\tvalid's auc: 0.778292\n",
      "[6300]\ttrain's auc: 0.802305\tvalid's auc: 0.778353\n",
      "[6400]\ttrain's auc: 0.802743\tvalid's auc: 0.778406\n",
      "[6500]\ttrain's auc: 0.803147\tvalid's auc: 0.778462\n",
      "[6600]\ttrain's auc: 0.803572\tvalid's auc: 0.778503\n",
      "[6700]\ttrain's auc: 0.803983\tvalid's auc: 0.778556\n",
      "[6800]\ttrain's auc: 0.804386\tvalid's auc: 0.778618\n",
      "[6900]\ttrain's auc: 0.804813\tvalid's auc: 0.778682\n",
      "[7000]\ttrain's auc: 0.805224\tvalid's auc: 0.778713\n",
      "[7100]\ttrain's auc: 0.80561\tvalid's auc: 0.778756\n",
      "[7200]\ttrain's auc: 0.806031\tvalid's auc: 0.778804\n",
      "[7300]\ttrain's auc: 0.806409\tvalid's auc: 0.778858\n",
      "[7400]\ttrain's auc: 0.806801\tvalid's auc: 0.778891\n",
      "[7500]\ttrain's auc: 0.80719\tvalid's auc: 0.778931\n",
      "[7600]\ttrain's auc: 0.807564\tvalid's auc: 0.778934\n",
      "[7700]\ttrain's auc: 0.807951\tvalid's auc: 0.778999\n",
      "[7800]\ttrain's auc: 0.808342\tvalid's auc: 0.779041\n",
      "[7900]\ttrain's auc: 0.80873\tvalid's auc: 0.779069\n",
      "[8000]\ttrain's auc: 0.809113\tvalid's auc: 0.779088\n",
      "[8100]\ttrain's auc: 0.809478\tvalid's auc: 0.779111\n",
      "[8200]\ttrain's auc: 0.809854\tvalid's auc: 0.779139\n",
      "[8300]\ttrain's auc: 0.810209\tvalid's auc: 0.779173\n",
      "[8400]\ttrain's auc: 0.810592\tvalid's auc: 0.77922\n",
      "[8500]\ttrain's auc: 0.810973\tvalid's auc: 0.779256\n",
      "[8600]\ttrain's auc: 0.811321\tvalid's auc: 0.779257\n",
      "[8700]\ttrain's auc: 0.811676\tvalid's auc: 0.779275\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[8800]\ttrain's auc: 0.81204\tvalid's auc: 0.779307\n",
      "[8900]\ttrain's auc: 0.812403\tvalid's auc: 0.779335\n",
      "[9000]\ttrain's auc: 0.812756\tvalid's auc: 0.779366\n",
      "[9100]\ttrain's auc: 0.813108\tvalid's auc: 0.779376\n",
      "[9200]\ttrain's auc: 0.813458\tvalid's auc: 0.779388\n",
      "Early stopping, best iteration is:\n",
      "[9157]\ttrain's auc: 0.813302\tvalid's auc: 0.779402\n"
     ]
    }
   ],
   "source": [
    "features = np.array(train)\n",
    "test_features = np.array(test)\n",
    "\n",
    "k_fold = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)\n",
    "\n",
    "valid_scores = []\n",
    "train_scores = []\n",
    "test_predictions = np.zeros(test_features.shape[0])\n",
    "\n",
    "params = tuned_params\n",
    "\n",
    "for train_indices, valid_indices in k_fold.split(features, targets):\n",
    "    model = lightgbm.LGBMClassifier(n_estimators=params['n_estimators'], objective='binary', \n",
    "                                class_weight='balanced', learning_rate=params['learning_rate'], \n",
    "                                num_leaves=params['num_leaves'], max_depth=params['max_depth'],\n",
    "                                reg_alpha=params['reg_alpha'], reg_lambda=params['reg_lambda'],\n",
    "                                min_child_sample=params['min_child_samples'],\n",
    "                                subsample=0.8, n_jobs=6, random_state=4242, max_bin=params['max_bin'])\n",
    "    \n",
    "    # DataFrame version\n",
    "    train_features, train_labels = train.iloc[train_indices], targets.iloc[train_indices]\n",
    "    valid_features, valid_labels = train.iloc[valid_indices], targets.iloc[valid_indices]\n",
    "    test_features = test\n",
    "    \n",
    "    # The ‘balanced’ mode uses the values of y to automatically adjust weights inversely proportional\n",
    "    # to class frequencies in the input data as n_samples / (n_classes * np.bincount(y))\n",
    "    model.fit(train_features, train_labels, eval_metric='auc', categorical_feature=cat_features,\n",
    "                  eval_set=[(valid_features, valid_labels), (train_features, train_labels)],\n",
    "                  eval_names=['valid', 'train'], early_stopping_rounds=100, verbose=100)\n",
    "\n",
    "    best_iteration = model.best_iteration_\n",
    "    \n",
    "    test_predictions += model.predict_proba(test_features, num_iteration = best_iteration)[:, 1] / k_fold.n_splits\n",
    "    \n",
    "    valid_score = model.best_score_['valid']['auc']\n",
    "    train_score = model.best_score_['train']['auc']\n",
    "        \n",
    "    valid_scores.append(valid_score)\n",
    "    train_scores.append(train_score)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7f7af3a46588>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VPW9+P/XO5N9IQkhrGGJimFfFUVB44JFrVpbq9hNu9Fal9bb3ntpe3+W2vZ+vd1se8ttS61drIqU1kpblGoloi0ooMgq+xbWAGEJWWfm/fvjcxKGkGQSyMkkk/fz8ZjHnOVzZt7nw3De+Xw+ZxFVxRhjjGlJQqwDMMYY0/lZsjDGGBOVJQtjjDFRWbIwxhgTlSULY4wxUVmyMMYYE5UlCxNXRORrIvJEO3zObBH5fazj8IuIvCgi97R3WRO/xK6zMH4TkZ1Af6C/qh6OWP4OMA4oVNWdUT6jGPi9qhb4F+kZ3zcbuEhVP9YR39cWIqLAUFXdGutYTPdhLQvTUXYAd9fPiMhoIL09v0BEEtvz885VrOOI9feb+GTJwnSUp4BPRMzfA/wusoCIpIjI90Vkt4gcFJGfi0iaiGQALwL9RaTCe/X3uooWiMjvReQEcG/j7iMRmSIi/xKRYyKyR0TubSo4ESkUkddE5KSIvAz0ilhXLCKljcrvFJHrvekW4xCRISKiInKPt2+HReTrEZ+VJiK/FZFyEdkoIv/R+Psiyi71Jt/16uGu+vhE5D9F5ADwaxHJFZG/ikiZ97l/FZGCiM8pEZHPeNP3isgbXt2Xi8gOEbnxHMsWishSrx5fEZE559OdZzoPSxamoywHeojIcBEJADOAxgeRx4CLcV1TFwEDgEdU9RRwI7BPVTO91z5vm9uABUAO8HTkh4nIYFyS+V8g3/vc1c3E9wywCpckvoVLZm3RbBwRpgBFwHXAIyIy3Fv+DWAIcAEwDWi260tVr/Imx3r18Jw33xfoCQwGZuL+b//amx8EVAE/bSH+y4BNuP3/LvArEZFzKPsM8BaQB8wGPt7Cd5ouxJKF6Uj1rYtpwEZgb/0K72AzE3hYVY+q6kngv3FJpSXLVPXPqhpW1apG6z4CvKKqz6pqnaoeUdWzkoWIDAIuBf4/Va1R1aXAX9q4by3FUe+bqlqlqu8C7wJjveV3Av+tquWqWgr8pI3fDRAGvuHFX+Xt6x9VtdKry+8AV7ew/S5V/aWqhoDfAv2APm0pG1GPj6hqraq+ASw8h30xnZD1bZqO9BSwFCikURcU7i//dGBVxB+0AgSifOaeFtYNBLa1Iq7+QLnXgqm3y9u+tVqKo96BiOlKIDPi+yO3b81nNVamqtX1MyKSDjwOTAdyvcVZIhLwDvLNxqaqld6/QWYT5Voq2ws4qqqVjfalLfVoOilrWZgOo6q7cAPdNwF/arT6MK6rZKSq5nivbFWtP2A1d9peS6fz7QEubEVo+4Fcb2yk3qCI6VNEDMZ73Wj5bYijNd8feZbXuRxcG3//l3FdXpepag+gvvuqua6l9rAf6OklqnqWKOKEJQvT0T4NXNvor3hUNQz8EnhcRHoDiMgAEXmfV+QgkCci2W34rqeB60XkThFJFJE8ERnXuJCXxFYC3xSRZBGZAtwSUWQzkCoiN4tIEvBfQEob4ohmPvBVb1B6APBAlPIHceMbLcnCJd9jItITNy7iq4h6nO3V42TOrEfThVmyMB1KVbep6spmVv8nsBVY7p1V9Arur2NU9T3gWWC7d2ZT/1Z8125cK+bLwFHc4PbYZop/BDdwexR3YG3oJlPV48AXgCdw4yyngCbPVjpHj3qftwO3zwuAmhbKzwZ+69XDnc2U+RGQhmuxLQdeardoW/ZRYDJwBPg28Bwt74vpIuyiPGM6GRG5D5ihqi0NSHcJIvIc8J6q+t6yMf6yloUxMSYi/UTkShFJEJEiXEvo+VjHdS5E5FIRudDbl+m4U4r/HOu4zPmzs6GMib1k4Be4s8SOAfOA/4tpROeuL+7khTxc19p9qvpObEMy7cG6oYwxxkRl3VDGGGOiiptuqJycHL3oootiHUandurUKTIyMqIX7Kasflpm9RNdV6yjVatWHVbVxtcNnSVukkWfPn1YubK5MzINQElJCcXFxbEOo9Oy+mmZ1U90XbGORGRXa8pZN5QxxpioLFkYY4yJypKFMcaYqOJmzKIpdXV1lJaWUl1dHb1wHEhNTaWgoICkpKRYh2KMiTNxnSxKS0vJyspiyJAhNP8cl/igqhw5coTS0lIKCwtjHY4xJs7EdTdUdXU1eXl5cZ8oAESEvLy8btOKMsZ0rLhOFkC3SBT1utO+GmM6VtwnC2OMMefPkkUnk5npHgy3b98+7rjjjibLFBcX2wWIxpgOZcmik+rfvz8LFiyIdRjGGANYsvDdrFmzmDNnTsP87Nmz+fa3v811113HhAkTGD16NC+88MJZ2+3cuZNRo0YBUFVVxYwZMxg+fDi33347VVVVHRa/McZAnJ86G+mbf1nPhn0n2vUzR/TvwTduGdlimbvuuosvfelL3H///QDMnz+fxYsX89BDD9GjRw8OHz7M5Zdfzq233trsAPXPfvYz0tPT2bhxI2vWrGHChAntuh/GGBNNt0kWsTJ+/HgOHTrEvn37KCsrIzc3l759+/Lwww+zdOlSEhIS2Lt3LwcPHqRv375NfsbSpUt56KGHABgzZgxjxozpyF0wxpjukyyitQD89OEPf5gFCxZw4MAB7rrrLp5++mnKyspYtWoVSUlJDBkyxK6PMMZ0ajZm0QHuuusu5s2bx4IFC/jwhz/M8ePH6d27N0lJSSxZsoRdu1q+Q/BVV13FM888A8C6detYs2ZNR4RtjDENuk3LIpZGjhzJyZMnGTBgAP369eOjH/0ot9xyC6NHj+aSSy5h2LBhLW5/33338clPfpLhw4czfPhwJk6c2EGRG2OMY8mig6xdu7ZhulevXixbtqzJchUVFQAMGTKEdevWAZCWlsa8efP8D9IYY5ph3VDGGGOi8jVZiMh0EdkkIltFZFYT6weJyBIReUdE1ojITd7yaSKySkTWeu/X+hmnMcaYlvnWDSUiAWAOMA0oBVaIyEJV3RBR7L+A+ar6MxEZASwChgCHgVtUdZ+IjAIWAwP8itUYY0zL/GxZTAK2qup2Va0F5gG3NSqjQA9vOhvYB6Cq76jqPm/5eiBNRFJ8jNUYY0wL/BzgHgDsiZgvBS5rVGY28HcReRDIAK5v4nM+BLytqjV+BGmMMSa6WJ8NdTfwG1X9gYhMBp4SkVGqGgYQkZHA/wA3NLWxiMwEZgLk5+dTUlJyxvrs7GxOnjzpY/idT3V19Vn1UK+ioqLZdcbqJxqrn+jiuo5U1ZcXMBlYHDH/VeCrjcqsBwZGzG8HenvTBcBm4MrWfN/FF1+sjW3YsOGsZR2tvLxc58yZ0+btbrzxRi0vL2/zdi3t85IlS9r8ed2J1U/LrH6i64p1BKzUVhxj/RyzWAEMFZFCEUkGZgALG5XZDVwHICLDgVSgTERygL8Bs1T1nz7G6Ltjx47xf//3f2ctDwaDLW63aNEicnJy/ArLGGPaxLdkoapB4AHcmUwbcWc9rReRR0XkVq/Yl4HPisi7wLPAvV6mewC4CHhERFZ7r95+xeqnWbNmsW3bNsaNG8ell17K1KlTufXWWxkxYgQAH/jAB5g4cSIjR45k7ty5DdsNGTKEw4cPs3PnToYPH85nP/tZRo4cyQ033GC3KDfGdDhfxyxUdRHudNjIZY9ETG8Armxiu28D327XYF6cBQfWRi/XFn1Hw42PtVjkscceY926daxevZqSkhJuvvlm1q1bR2FhIQBPPvkkPXv2pKqqiksvvZQPfehD5OXlnfEZW7Zs4dlnn+WXv/wld955J3/84x/52Mc+1r77YowxLYj1AHe3M2nSpIZEAfCTn/yE559/HoA9e/awZcuWs5JFYWEh48aNA2DixIns3Lmzw+I1xhjoTskiSgugo2RkZDRMl5SU8Morr7Bs2TLS09MpLi5u8lblKSmnLzEJBALWDWWMOT+hOjhe6l6t1H2SRYxkZWU1e/ru8ePHyc3NJT09nffee4/ly5d3cHTGmLhUU+Elgz3udSzi/dhuOLkfd01061my8FleXh5XXnklo0aNIi0tjT59+jSsmz59Oj//+c8ZPnw4RUVFXH755TGM1BjTZVQdcwf9yARwfPfp6aqjZ5ZPSIQeAyBnEFxQDDkDIXuge//mNa36SksWHaD+wUWNpaSk8OKLLza5rn5colevXg23Kgf4yle+0u7xGWM6EVWoPOK1CryuomO7vdcu9159/MxtEtNcIsgZCP3He9ODXELILoCsvpAQOK+wLFkYY0xHq6lwyaB8l0sA5TsjXrug7tSZ5ZPSIWewSwADLz+dDOpf6Xkg4mvIliyMMaa91Z5yLYDyXREtgl3NdxMlpUPuEPcqvBpyB59uFWQPhPSevieDaOI+WagqEuNK7ijuekZjjO/CITixt1GLYCcTdq2FFeVwquzM8ompp1sBkd1EOYMgtxAyesU8GUQT18kiNTWVI0eOkJeXF/cJQ1U5cuQIqampsQ7FmPhQffysZNDwOrYbwhG37ElIhOwCQoEeMPRG12WUO8RLCIMhs3enTwbRxHWyKCgooLS0lLKysuiF40BqaioFBQWxDsOYriFU53UV7fTGDRqNH1SVn1k+radLAP3GwYgPnO42yh3izjQKJPJuSQnFxcUdux8dJK6TRVJS0hlXSxtjupmq8jMHjst3wNEdbv54KWjodNmEJHc2UW4h9J9wZjLIHQyp2bHYg04jrpOFMSbOhcNQccAd/I/u8JLBdjd9dDtUHzuzfHqeSwYFl8KYO89MCFn9zvv00nhmycIY07mFgu6Cs8gkUJ8YyndCMOIWOZLgzh7qWQijPugSQ89Cb/xgMKT2aO5bTBSWLIwxsRcKuusOjm5zieDINm96u+s+CtedLpuU7pJA3kVw0fUuEfQsdMuyB0Jicsx2I55ZsjDGdIzIFsKR7V4LYbtLCmclhAzoeQH0GQnDb3XTeRe698w+Xf7Moq7IkoUxpv2Ew3Byn2sZHNl6+v3oNtdlFHm6aX1C6D0Cht/ipnte6JKCJYROx9dkISLTgR8DAeAJVX2s0fpBwG+BHK/MLO+BSYjIV4FPAyHgIVVd7GesxphWUoXKoy4BHNka8drmXsGIW+gnprmDf30Lob510PPCuLj2oDvxLVmISACYA0wDSoEVIrLQezpevf/CPW71ZyIyAvdUvSHe9AxgJNAfeEVELlaNPM/NGOOruip38D+8GY5sY9jGN2DLt1xiiDzLKCHRGze40N2qIu9CN56Qd5F3hpFvT282HcjPlsUkYKuqbgcQkXnAbUBkslCg/vSEbGCfN30bME9Va4AdIrLV+7xlPsZrTPej6p5tcHgLHNni3utfx/cQ+cyDnJQ86D8SRt4OvYaeTgg5gyCQFLt9MB3Cz2QxANgTMV8KXNaozGzg7yLyIJABXB+xbeSTgEq9ZWcQkZnATID8/HxKSkraI+64VVFRYXXUgniun4RQDWlV+0mvLCW9cq/3KiWtai+JodOnnoYSUqlMH0Bl+hAqh1xJZfoAqtIGUJnenxNVQTIzM13BGtyfdvv2cOZ/8+4tnn9DsR7gvhv4jar+QEQmA0+JyKjWbqyqc4G5AEVFRRqvl9m3l5I4vhVBe+jy9dOGVgLZA6HPUMi73rUSeg2FvKEEevQnS4SsJj6+y9dPB4jnOvIzWewFBkbMF3jLIn0amA6gqstEJBXo1cptjemewiF3ZtGhDVD2HpRt9sYVtkJtxelySRnQ6yIYdBnkfcxN97rYjS0kp8csfNM1+ZksVgBDRaQQd6CfAXykUZndwHXAb0RkOJAKlAELgWdE5Ie4Ae6hwFs+xmpM5xOsdUmh7D0o2+TeD29yLYXIq5azB7qWwcCPntFKoEd/O9vItBvfkoWqBkXkAWAx7rTYJ1V1vYg8CqxU1YXAl4FfisjDuDbyveoeyrBeRObjBsODwP12JpSJW3VVrmVw6D0o2+jeD292iSLyZ589CPKL3BlH+cOgzwjoVQQpmTEL3XQfvo5ZeNdMLGq07JGI6Q3Alc1s+x3gO37GZ0yHCta4VkHZe3Bo4+n38h2gYVcmIdG1CvqOgpEfcNP5F1tSMDEX6wFuY+JPqM6NH0QmhLL33DUL9S0FCZy+WG30Ha6l0HuEW2anoZpOyJKFMedKFU7sg4Pr4eA6N+B8cIPrQqq/z5EkuCuW84fBiNu8pDDcXZ+QmBLb+I1pA0sWxrRGsNa1Dg6ugwNr3evgujOfptajwLUULr7BtRLyh7mzj5LsUbem67NkYUxjlUfPTAgH1rqzkepbC4lpbnB5+K3Qd7RLEL2HQ1pubOM2xkeWLEz3FQ67W2QfdIlh9PrXYNV97q6p9TL7uoQwdJqXGEa7cQV7oprpZixZmO4hWOMGmg+sgf3vutfBDVB3yq2XACnpBXDhVOgzyp2N1Gc0ZObHNm5jOglLFib+1Fa6weZ975xODIc2nu5GSunhWgkTPu61FkZB/jBW/nN53N6qwZjzZcnCdG01J2Hfatj3tns/sMadolp/H6T0POg3Fq54wL33HeMev2m3zTamTSxZmK6jttINOO97x732vu1OU61PDNmDoN8YGHWH60bqNw6yC+yWF8a0A0sWpnMKh919kPaugtKVsHelG2Oov6gtozcMmACjPuTe+4+HjF6xjdmYOGbJwnQOJ/a7xLB3pff+DtSedOtSsmHAeJjysEsK/cfbTfKM6WCWLEzHq6tyXUilK7zEsApOeHegT0h0A85j74IBE2HAJe5qZxtjMCamLFkY/506DLuXw+5l7n3/u6fPTModAoMud0mh4BJ3dlJSWkzDNcaczZKFaV+q7myk3ctgz3KXHI5sdesCya61MPl+lyAKLrVxBmO6CEsW5vwEa93pqvWtht3LofKwW5eWCwMvh/Efd8mh3zi7T5IxXZQlC9M21Sdgz5unE8Pelaef2pZb6G6LMehyGDTZPYvBxhqMiQu+JgsRmQ78GPekvCdU9bFG6x8HrvFm04HeqprjrfsucDOQALwMfNF7ip7pSKGgu+Bt2xLY9qoblNaQex5DvzFwyadg4GUuQWT1jXW0xhif+JYsRCQAzAGmAaXAChFZ6D0dDwBVfTii/IPAeG/6CtwT9MZ4q98ArgZK/IrXRCjf6RLDtldh+1KoOQ6IO2V1ypeg8Co3IG1PbjOm2/CzZTEJ2Kqq2wFEZB5wG+652k25G/iGN61AKpAMCJAEHPQx1u6trhp2vg5b/g5bX3F3YgXIHggjb4MLr3XPfU7vGds4jTEx42eyGADsiZgvBS5rqqCIDAYKgVcBVHWZiCwB9uOSxU9VdWMT280EZgLk5+dTUlLSnvHHnYqKioY6Sqo9Qc+jK+l1+C16Hn2HQLiaUEIyx3LGcPSi6zjacxxVaQPchW9lQNmamMbeESLrx5ytq9ZPWJWwQkhx72Hv3Vse9jq36/u41SsbUgiGlWD49LaqoJzeTqlf5uarqqt589lXUIUwoN53KBFlIz4/FNaGzwaaLKsNyzVin858RcbhvtvN1G8LZ+5n+PRHtVpnGeCeASxQdfdyEJGLgOFAgbf+ZRGZqqqvR26kqnOBuQBFRUVqdwxt2ZuLnuWy5B2w6UV39pKGIasfTPgIFN1EYMhU8pJSyYt1oDFSUlJid53FHeDqQkptKExdMExdOEwwpLzxr2UMG3EJQW++NuTeg6EwdWGlLhgmGA5TF1L3HtSGbetC3nKvbDAUPr2sUdlgOExtUBu+p84rGwy7uMJhJaxKSJVgSAmFXdlQGEJhVy5y+bkcGM+dADXn/SkJAgkiJIiAQEAEbxLFzQcCQmKCICIN6xPqy0VsL+Atk4bPxZtvyz0Q/EwWe4GBEfMF3rKmzADuj5i/HViuqhUAIvIiMBl4vYltTXNU3a26N7wAG17gsrL33PI+o2DqV6DoRjcOYbfN6FTCYaU6GKKqNkR1MOze69yrqi5EdV3Ye49YXhumOhhyB/dQmNqIg2xt0Ht509XBMDV1Ia+cW1YX0jPKNWvpa+2yj0kBITEhgcSAkBRIaJhPCgiJgYSIZW59enIiifXbJHgHxgQhKUEIeMvqD54JIg2fk5ggBBLc8vr5+vUJCe4gm5Bw+qAK7r9DYiCB5PqYEhO875HTB9yEMw/QCSIkJMCqFSuZNOnShs8MNDp417/Xx5MUcJ8bqD+wewfxjiRfal05P5PFCmCoiBTiksQM4CONC4nIMCAXWBaxeDfwWRH5f7hkejXwIx9jjS/lO2H1M7DuT3BkC0gCDL6SLdlTGHrzQ5A7ONYRdnnhsFJZF+JUTZCKmiAV1cHT0zVuurI2RGXt6YN8VW2jd2+6cRKoDbZwsG5BUkBIDngHt0CCm/YOmsmBBJIT3Ss7LYnUrBSSEr3lgQSSEoXkQMCVCUhD2aRAgjuoJQjbtmxi1MgRblmCeAfR+gO+eMsjD/jSZNn6v4bj0YGsBIb2yYp1GL7wLVmoalBEHgAW406dfVJV14vIo8BKVV3oFZ0BzGt0WuwC4FpgLa7V9ZKq/sWvWONCXTW891d4+3ew4zVAoHAqXH4fDL8FMnuzt6SEod04UYTCyqnapg/sFTUh3tlVx/olW5s9+J+eDnGqNkhrT+ROTUogLSlAWlKA1ORAw3RmSiK9MlMa5tOSA6R4ZVOTTperX5aWfHp5alKg4XNTvVcgwd8DcEnldorHDfD1O0zn5euYhaouAhY1WvZIo/nZTWwXAj7nZ2xxY/+78M7vYc18qD4GOYOg+Gsw7iOQMzD69l2MqlJZG+LoqVrKKmo4XlXHiao6ys+YD3Ki2i0/UR3kRFUdJ6uDVNWFon/Bxk0kJgiZqYlkJCeSlZpIRkoi2enJFOSmk5ESICMlkawUtzwjxSuTHDGdkujKJSeSlhQgweeDuDEdobMMcJu2qCqHtQtcK+LAGgikuNbDhI/DkKu63FXTqsqp2hCHTlRz6GSNe52opqyihrITbv5wRQ1HT9VyrLKu2T71QILQIzWRrNQkstOS6JGWSN/sVHqkJpGVmkh6ciKZKYkuEZxxwA+QlZLE6lVvcsM1V5GSmBC33STGnCtLFl1FOAw7l7pWxIaFEKpxd2i98Xsw+o5OeQ1EOKwcPlXDgePVHDxRw6GT1Rw+WUtZhXs/XOGSwKGTNVTWnv1Xf3IggfysFPKzUijITWdsQQ45GUnkpifTMyOZXpnJZKclk52WSE56Mj3Tk8/rr/jtyUJqUuB8dtmYuGXJorMLh9xA9evfh7L3IDUbJnwCxn8M+o+LWViqSllFDfuPVbP/eDX7j1dx4Lib3nesiv3Hqzl0spq60Nkd+znpSeRnptArM4XRBTnkZ6bQu0dKw3vvrFR6Z6WQk55kf+Eb00lYsuisQkFY+weXJI5shfzhcPtcGHFrhz3vobouxJ6jlew8UsmuI6fYfbSSPUcr2X20ktLyKmoanbWTnJhAv+xU+vZI5bLCnvTJTm2Y79MjlfwslyCSE7tWN5kxxpJF5xOshTXz4PUfuFNg+4yGO5+CYe/3ZSxCVTl4oobNB0+y+eBJdhw+xY7Dp9h5+BT7T1SfccZPVmoiA3PTuah3JtcU9aYgN43+Oe7VLzuVnhnJ1hIwJk5ZsugsgjVuPOKNH8Hx3e7ZDzOedRfOtdMBuLJOWbbtSENi2HzwJJsOnOREdbChTHZaEoW9MrjsgjyG5GUwpFc6Q/IyGJyXTk56crvEYYzpeixZxFo4BKufhiX/D07uc3dzff8P4aLrzytJVNQEWb/3OOv3nWDdvuOs3n2M7Ycr4R/LAddKGNY3i1vG9qeobxYX98liaO9M8jJT2mvPjDFxxJJFLO17B/7yJdi/2j1i9ANz4IJr2pwkVJXth0+xcudRVuws5+1d5ew4cqqhCyk/K4WxBTmMz63llinjGN6vB72zUqzLyBjTapYsYiFYC0u/C6//EDLy4YNPuNNf23DwLi2vpGRTGf/adpjl249y9FQtAD0zkpk4OJfbxg1gTEE2I/v3oHcP9yjTkpISiot6+7JLxpj4Zsmiox1YC8/fBwfXwti7YfpjkJYTdbO6UJi3dhzlHxsP8fqWMrYcqgCgX3Yq1xT1ZlJhLpcM6ckFvTKsxWCMaXeWLDpKKAj/fBxK/gfSct3g9bCbWtykNhhmyaZD/G3NfpZsOsTJ6iApiQlMKuzJnZcM5NrhvS05GGM6hCWLjlC2CZ7/vHuW9cgPwk3fh4zmnxqxft9x/rCylIXv7uPoqVp6ZiTzvpF9mTaiD1OH9iI92f7ZjDEdy446fgqHYNkcePXbkJwBd/waRn2wyaLlp2pZtG4/z63Yw5rS4yQHEpg2og8fmjiAq4bmkxiwC9mMMbFjycIvx0thwadhz3Iouhlu+RFknj24vO9YFXOXbmfeit1U14Up6pPFN24Zwe3jB9h1DcaYTsOShR92L4fnPg51VXD7L2DMXWed6bTj8Cl+VrKV59/ZiyrcPn4A91wxhJH9e9gYhDGm07Fk0d5W/Rb+9mX3LIl7/wr5RWes3nTgJP/76hYWrd1PYiCBuycNYuZVF1CQmx6jgI0xJjpfk4WITAd+jHtS3hOq+lij9Y8D13iz6UBvVc3x1g0CnsA9x1uBm1R1p5/xnpdwGBZ/Dd78GVx4LdzxpDvryXOsspYf/2MLv1u2i9TEBD571QV8ekohvbNSYxi0Mca0jm/JQkQCwBxgGlAKrBCRhaq6ob6Mqj4cUf5BYHzER/wO+I6qviwimcC5PZi4I4Tq3NlO6xbA5V+Aad+CgKvaulCYp5bt4sf/2MLJ6jpmTBrEv99QRG6GjUcYY7oOP1sWk4CtqrodQETmAbcBG5opfzfwDa/sCCBRVV8GUNUKH+M8P8FamP8J2PwiXD8bpjTkP0o2HeLRv25ge9kppg7txddvHs6wvj1iFqoxxpwr0dY+db6tHyxyBzBdVT/jzX8cuExVH2ii7GBgOVCgqiER+QDwGaAWKAReAWZ5z+aO3G4mMBMgPz9/4vz5833Zl2ZpiBEbfkDvsn+yeejn2TfgRgBO1Cq/XV/DqoMh+qR/Gx3QAAAWb0lEQVQLdw9LZmx+IOYD1xUVFWRmZsY0hs7M6qdlVj/RdcU6uuaaa1ap6iXRynWWAe4ZwIKIZJAITMV1S+0GngPuBX4VuZGqzgXmAhQVFWlxcXEHhQuowl8fhrJ/wg3f5uIrHuRiYMmmQzz6hzWcqFL+/X1FfGZqISmJneNRnSUlJXRoHXUxVj8ts/qJLp7ryM9ksRc3OF2vwFvWlBnA/RHzpcDqiC6sPwOX0yhZxNQ/HoVVv4Yp/wZXPEhdKMx3X3qPX76+g6I+WTz16UkM72ddTsaY+OBnslgBDBWRQlySmAF8pHEhERkG5ALLGm2bIyL5qloGXAus9DHWtnnrl/DGD2HiJ+G6RzhwvJoHnnmblbvK+cTkwXztpuGkJnWO1oQxxrQH35KFqgZF5AFgMe7U2SdVdb2IPAqsVNWFXtEZwDyNGDzxxi2+AvxDXEf/KuCXfsXaJnvfdqfIDr0Bbv4B6/ef4J4nV1BVG+R/7x7PLWP7xzpCY4xpd61KFiJyObBeVU968z2A4ar6ZkvbqeoiYFGjZY80mp/dzLYvA2NaE1+HqSqHP9wDGb3h9l+wZt9JPv6rt8hIDvDs/VcytE9WrCM0xhhftPbudD8DIk9frfCWdR+q8Of74cQ++PBvWHkIPvrEm2SlJvLc5yZbojDGxLXWJgtp1E0UpvOcSdUx3vw5bPobTPsW/6ot5KNPvEl+ZgrPfW4yA3varTqMMfGttcliu4g8JCJJ3uuLwHY/A+tUju6AV2bDxdPZesHH+NxTqxjUM50/fH4yA3LSYh2dMcb4rrXJ4vPAFbizmkqBy/Auhot7qrDo3yEhkaPXPMa9v1lJSmKAX3/yUvIyU2IdnTHGdIhWdSWp6iHcWUvdz4YXYOvLhG74b+5beICykzXM/9xku0usMaZbae3ZUL/G3fn1DKr6qXaPqDOpPgEvzYK+o/nO4Sm8uaOUx+8ay9iBObGOzBhjOlRrB6n/GjGdCtwO7Gv/cDqZJf8NJw/wzuT/5cmFpdx7xRBuH18Q66iMMabDtbYb6o+R8yLyLPCGLxF1Fvvfhbd+Qe34e7n/tQQuzM9g1o3DYh2VMcbERGsHuBsbCpz9QOl48vI3IC2Xx+ru5MCJar734bF2Cw9jTLfV2jGLk5wes1DgIPAffgUVczvfgO1L2Dnhqzz5r3I+O7WQCYNyo29njDFxqrXdUFki0hPXoqh/Dqg/D8KINVV49TtoZl8+s3Eshb3S+PINRdG3M8aYONbalsVngC/ibjO+Gne78GW4u8HGl+0lsPtfvDH0P9m6Nswznx1l3U/GmG6vtWMWXwQuBXap6jW4hxId8y2qWFGFkscIZfXnwU2juW5Yb664sFesozLGmJhrbbKoVtVqABFJUdX3gPjrm9nxGuxZzks5d3OiLoGv3Tw81hEZY0yn0NrrLEpFJAf4M/CyiJQDu/wLK0aWfp9gRl/+c/tYPjxxIBfmd61n6RpjjF9aO8B9uzc5W0SWANnAS75FFQu7/gU7X+cvve8neCKZL00bGuuIjDGm02jzdRaq+pqqLlTV2mhlRWS6iGwSka0iMquJ9Y+LyGrvtVlEjjVa30NESkXkp22Ns81e/yHBtHxm7b6UT11ZSL9su5usMcbU8+2ZFCISAOYA03B3ql0hIgtVdUN9GVV9OKL8g7iB80jfApb6FWODgxtg68ssyvs0SZXpzLzqAt+/0hhjupJzvYK7NSYBW1V1u9cKmQfc1kL5u4Fn62dEZCLQB/i7jzE6y35KODGNR/ZO4t4rhpCTnuz7VxpjTFfi59PuBgB7Iubrn4NxFhEZDBQCr3rzCcAPgI8B1zf3BSIyE++5Gvn5+ZSUlLQ5yKTaY0x+9zn+nnQtNYlZjAzso6Rkf5s/pyuoqKg4pzrqLqx+Wmb1E10811FneTTqDGCBqoa8+S8Ai1S1VESa3UhV5wJzAYqKirS4uLjt31zyP6BBvlcxjU9edSE3Xh+/NwssKSnhnOqom7D6aZnVT3TxXEd+Jou9wMCI+QJvWVNmAPdHzE8GporIF4BMIFlEKlT1rEHy8xKshZW/YlPWZewpH8gnryxs1483xph44WeyWAEMFZFCXJKYAXykcSERGQbk4m4fAoCqfjRi/b3AJe2eKAA2LoSKg3wvdC93TCwgP8sek2qMMU3xbYBbVYPAA8BiYCMwX1XXi8ijInJrRNEZwDxV7fgbE775c46mDmRJcDQzp9oZUMYY0xxfxyxUdRGwqNGyRxrNz47yGb8BftPOocHeVVC6gl9wL9NHDWBIr4x2/wpjjIkXnWWAu+O9OZfaQDpPn5rCs1dfGOtojDGmU/PzOovOq/o4uv55XtCrGHfRIEYXZMc6ImOM6dS6Z7LYsBAJ1fB01RV83loVxhgTVfdMFmvnsz+hH5X5Y7nyorxYR2OMMZ1e90sWJ/ahO15nfu1k7r5sMC1d9GeMMcbpfsli3R8RlL/pFG4fPyDW0RhjTJfQ7c6GCq+Zz0YuZOiI8XbDQGOMaaXu1bI4so2EA2v4U91kZkwaGL28McYYoLsli/f+BsDqzKlMuahXjIMxxpiuo1t1QwU3/o0tOpjxo8fYwLYxxrRB92lZVJQRKH2TxaGJ3Dquf6yjMcaYLqX7JIvNLyEo2/OuZkxBTqyjMcaYLqXbdENVrFnIMe3F+EuvinUoxhjT5XSPlkXtKVJ3v8bLoYncNMa6oIwxpq26R7LYtoTEcA07el1Nnx6psY7GGGO6nG7RDVWx5gVCms7g8dfHOhRjjOmSfG1ZiMh0EdkkIltF5KzHoorI4yKy2nttFpFj3vJxIrJMRNaLyBoRueucgwgFSdz6d14Nj+eG0XYhnjHGnAvfWhYiEgDmANOAUmCFiCxU1Q31ZVT14YjyDwLjvdlK4BOqukVE+gOrRGSxqh5rcyB73iS17hgbsqZwe8/089gjY4zpvvxsWUwCtqrqdlWtBeYBt7VQ/m7gWQBV3ayqW7zpfcAhIP9cgqhe8zw1mkiP0dPPZXNjjDH4O2YxANgTMV8KXNZUQREZDBQCrzaxbhKQDGxrYt1MYCZAfn4+JSUlZxbQEBPf/QP/Co+jR1352eu7mYqKim5fBy2x+mmZ1U908VxHnWWAewawQFVDkQtFpB/wFHCPqoYbb6Sqc4G5AEVFRVpcXHxmge2vwWvHWNnjer5+63U+hd51lJSUcFYdmQZWPy2z+okunuvIz26ovUDkiHKBt6wpM/C6oOqJSA/gb8DXVXX5uQRwbMU8KjSVgkkfOJfNjTHGePxMFiuAoSJSKCLJuISwsHEhERkG5ALLIpYlA88Dv1PVBef07XXVpGz+C6/qRG65xJ6zbYwx58O3ZKGqQeABYDGwEZivqutF5FERuTWi6AxgnqpqxLI7gauAeyNOrR3Xlu8PbvgLaaGTbO1/Gz0z7CFHxhhzPnwds1DVRcCiRsseaTQ/u4ntfg/8/ny+++jypwlpT8ZdfWv0wsYYY1oUn7f7OHWYnvuX8o/AFK66uE+sozHGmC4vLpPFyZXzSCREzci7SAzE5S4aY0yH6iynzrar42//ib3hgVxzVXGsQzHGmLgQf392V5+g4Pgq1mdewQX5mbGOxhhj4kLcJYuqt34HQOiCa2IciTHGxI/4ShaqBP/5Ew5rDy685H2xjsYYY+JGXCUL3b+arJqDPBm+hQmDc2MdjjHGxI24ShYytxiAQZPej4jENhhjjIkjcZUs6g0fd0WsQzDGmLgSN8lCcHcL+W1wGiP794hxNMYYE1/iJlkQqgVgZbjILsQzxph2FjdH1XCwDoCCoWNjHIkxxsSfuEkWCWHXsrjx6ikxjsQYY+JP3CSLgNaxV/PIzs6OdSjGGBN34iZZpIVOUqmp9EhNinUoxhgTd+ImWQAkU0dWalzeG9EYY2LK12QhItNFZJOIbBWRWU2sfzziSXibReRYxLp7RGSL97qnNd/3fHiKnQlljDE+8O3PcBEJAHOAaUApsEJEFqrqhvoyqvpwRPkHgfHedE/gG8AlgAKrvG3LW/rOzLS0dt8PY4wx/rYsJgFbVXW7qtYC84DbWih/N/CsN/0+4GVVPeoliJeB6dG+sH9e1nmGbIwxpil+dvAPAPZEzJcClzVVUEQGA4XAqy1sO6CJ7WYCMwEm9kugtrqKkpKS8w48XlVUVFj9tMDqp2VWP9HFcx11ltHgGcACVQ21ZSNVnQvMBbikf0DzevVhanGxD+HFh5KSEoqtfppl9dMyq5/o4rmO/OyG2gsMjJgv8JY1ZQanu6Daum0DCXSW3GeMMfHFz2SxAhgqIoUikoxLCAsbFxKRYUAusCxi8WLgBhHJFZFc4AZvWYskYNdYGGOMH3z7U1xVgyLyAO4gHwCeVNX1IvIosFJV6xPHDGCeqmrEtkdF5Fu4hAPwqKoejfadkmjJwhhj/OBrv42qLgIWNVr2SKP52c1s+yTwZFu+TwLJbYzQGGNMa8TVFWzBtLxYh2CMMXEprpJFckp6rEMwxpi4FFfJokeadUMZY4wf4ipZkBBfu2OMMZ1FXB1dRSTWIRhjTFyKs2QRV7tjjDGdRlwdXS1ZGGOMP+Lm6LpP8whm9o91GMYYE5fiJlmEUnLo1eesG9MaY4xpB3GTLHJShN49UmMdhjHGxKW4SRbGGGP8Y8nCGGNMVJYsjDHGRGXJwhhjTFSWLIwxxkRlycIYY0xUviYLEZkuIptEZKuIzGqmzJ0iskFE1ovIMxHLv+st2ygiPxG78ZMxxsSMb0/KE5EAMAeYBpQCK0RkoapuiCgzFPgqcKWqlotIb2/5FcCVwBiv6BvA1UCJX/EaY4xpnp8ti0nAVlXdrqq1wDzgtkZlPgvMUdVyAFU95C1XIBVIBlKAJOCgj7EaY4xpgZ/P4B4A7ImYLwUua1TmYgAR+ScQAGar6kuqukxElgD7AQF+qqobG3+BiMwEZgLk5+dTUlLS7jsRTyoqKqyOWmD10zKrn+jiuY78TBat/f6hQDFQACwVkdFAL2C4twzgZRGZqqqvR26sqnOBuQBFRUVaXFzcQWF3TSUlJVgdNc/qp2VWP9HFcx352Q21FxgYMV/gLYtUCixU1TpV3QFsxiWP24HlqlqhqhXAi8BkH2M1xhjTAj+TxQpgqIgUikgyMANY2KjMn3GtCkSkF65bajuwG7haRBJFJAk3uH1WN5QxxpiO4VuyUNUg8ACwGHegn6+q60XkURG51Su2GDgiIhuAJcC/q+oRYAGwDVgLvAu8q6p/8StWY4wxLfN1zEJVFwGLGi17JGJagX/zXpFlQsDn/IzNGGNM69kV3MYYY6KyZGGMMSYqSxbGGGOismRhjDEmKksWxhhjorJkYYwxJipLFsYYY6KyZGGMMSYqSxbGGGOismRhjDEmKksWxhhjorJkYYwxJipLFsYYY6KyZGGMMSYqcXcJ7/pE5CSwKdZxdHK9gMOxDqITs/ppmdVPdF2xjgaran60QrF+Bnd72qSql8Q6iM5MRFZaHTXP6qdlVj/RxXMdWTeUMcaYqCxZGGOMiSqeksXcWAfQBVgdtczqp2VWP9HFbR3FzQC3McYY/8RTy8IYY4xPLFkYY4yJKi6ShYhMF5FNIrJVRGbFOp6OIiIDRWSJiGwQkfUi8kVveU8ReVlEtnjvud5yEZGfePW0RkQmRHzWPV75LSJyT6z2yQ8iEhCRd0Tkr958oYi86dXDcyKS7C1P8ea3euuHRHzGV73lm0TkfbHZE3+ISI6ILBCR90Rko4hMtt/QaSLysPf/a52IPCsiqd3yN6SqXfoFBIBtwAVAMvAuMCLWcXXQvvcDJnjTWcBmYATwXWCWt3wW8D/e9E3Ai4AAlwNvest7Atu991xvOjfW+9eO9fRvwDPAX735+cAMb/rnwH3e9BeAn3vTM4DnvOkR3u8qBSj0fm+BWO9XO9bPb4HPeNPJQI79hhrqZgCwA0iL+O3c2x1/Q/HQspgEbFXV7apaC8wDbotxTB1CVfer6tve9ElgI+7HfRvuAID3/gFv+jbgd+osB3JEpB/wPuBlVT2qquXAy8D0DtwV34hIAXAz8IQ3L8C1wAKvSOP6qa+3BcB1XvnbgHmqWqOqO4CtuN9dlyci2cBVwK8AVLVWVY9hv6FIiUCaiCQC6cB+uuFvKB6SxQBgT8R8qbesW/Gau+OBN4E+qrrfW3UA6ONNN1dX8VyHPwL+Awh783nAMVUNevOR+9pQD9764175eK6fQqAM+LXXVfeEiGRgvyEAVHUv8H1gNy5JHAdW0Q1/Q/GQLLo9EckE/gh8SVVPRK5T1wbuludHi8j7gUOquirWsXRiicAE4GeqOh44het2atDNf0O5uFZBIdAfyCB+WkxtEg/JYi8wMGK+wFvWLYhIEi5RPK2qf/IWH/S6BvDeD3nLm6ureK3DK4FbRWQnrnvyWuDHuK6T+vuiRe5rQz1467OBI8Rv/YD7C7dUVd/05hfgkof9hpzrgR2qWqaqdcCfcL+rbvcbiodksQIY6p2dkIwbVFoY45g6hNcX+itgo6r+MGLVQqD+bJR7gBciln/CO6PlcuC419WwGLhBRHK9v6Ru8JZ1aar6VVUtUNUhuN/Fq6r6UWAJcIdXrHH91NfbHV559ZbP8M50KQSGAm910G74SlUPAHtEpMhbdB2wAfsN1dsNXC4i6d7/t/r66X6/oViPsLfHC3eGxmbcGQZfj3U8HbjfU3DdA2uA1d7rJlwf6T+ALcArQE+vvABzvHpaC1wS8Vmfwg26bQU+Get986Guijl9NtQFuP+oW4E/ACne8lRvfqu3/oKI7b/u1dsm4MZY70871804YKX3O/oz7mwm+w2d3q9vAu8B64CncGc0dbvfkN3uwxhjTFTx0A1ljDHGZ5YsjDHGRGXJwhhjTFSWLIwxxkRlycIYY0xUliyM8YhIhfc+REQ+0s6f/bVG8/9qz883xm+WLIw52xCgTcki4mre5pyRLFT1ijbGZExMWbIw5myPAVNFZLX3LIOAiHxPRFZ4z3D4HICIFIvI6yKyEHdVLyLyZxFZ5T3/YKa37DHcXUtXi8jT3rL6Vox4n71ORNaKyF0Rn10ip58z8bR3BTEi8pi4Z5isEZHvd3jtmG4p2l9DxnRHs4CvqOr7AbyD/nFVvVREUoB/isjfvbITgFHqbjsN8ClVPSoiacAKEfmjqs4SkQdUdVwT3/VB3BXUY4Fe3jZLvXXjgZHAPuCfwJUishG4HRimqioiOe2+98Y0wVoWxkR3A+5+SKtxt4DPw93bB+CtiEQB8JCIvAssx904bigtmwI8q6ohVT0IvAZcGvHZpaoaxt3KZQjultfVwK9E5INA5XnvnTGtYMnCmOgEeFBVx3mvQlWtb1mcaigkUoy7S+lkVR0LvIO7V9C5qomYDgGJ6p6RMAl3d9j3Ay+dx+cb02qWLIw520ncY2rrLQbu824Hj4hc7D0gqLFsoFxVK0VkGO6xo/Xq6rdv5HXgLm9cJB/31Lpm70bqPbskW1UXAQ/juq+M8Z2NWRhztjVAyOtO+g3uGRhDgLe9QeYyTj9GM9JLwOe9cYVNuK6oenOBNSLytrrbpNd7HpiMez6zAv+hqge8ZNOULOAFEUnFtXj+7dx20Zi2sbvOGmOMicq6oYwxxkRlycIYY0xUliyMMcZEZcnCGGNMVJYsjDHGRGXJwhhjTFSWLIwxxkT1/wMBbw6PYfFyQgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f7af3a68940>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lightgbm.plot_metric(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "#lightgbm.plot_tree(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7f7b845dbc88>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgYAAAEWCAYAAAAdAV+mAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXu8lWP6/9+fyiEdkEQHqRzSeZMZ8h3NbkyFnMIgDDkMfmQwyjhP+TIlGmn4DqMohEQmh6aJasc4Fx0UiWnrRE0RtUs6XL8/nnvtnr322nuvvXftw+p6v17r1fPc9/3c93Wt6LnWfbg+MjMcx3Ecx3EAalS2AY7jOI7jVB08MHAcx3EcJx8PDBzHcRzHyccDA8dxHMdx8vHAwHEcx3GcfDwwcBzHcRwnHw8MHMdx0kTSI5LuqGw7HGdnIs9j4DjOzkZSLnAAsDVWfLiZrShHn9nA02bWrHzWVU8kjQaWmdntlW2Lk1n4jIHjOBXFqWZWN/Ypc1CwI5BUqzLHLw+Sala2DU7m4oGB4ziViqRjJb0jaa2kOWEmIFF3iaRPJa2T9B9JV4byOsA/gSaS1odPE0mjJd0dez5b0rLYfa6kP0qaC+RJqhWee1HSfyUtlvT7YmzN7z/Rt6SbJK2S9LWkMySdLOlzSd9KujX27EBJL0gaF/z5SFKnWH0bSTnhe5gv6bSkcf8maZKkPOAy4ALgpuD7K6HdzZK+DP0vkNQ71kdfSf+WdL+k74KvJ8XqG0h6QtKKUP+PWN0pkmYH296R1DHtv2Cn2uGBgeM4lYakpsBrwN1AA6A/8KKk/UOTVcApQH3gEuABSUeZWR5wErCiDDMQfYBewD7ANuAVYA7QFDgBuF5SzzT7OhDYMzx7J/AYcCHQGTgeuENSy1j704HxwddngH9I2k3SbsGOKUAj4FpgrKTWsWfPB+4B6gFPAmOBocH3U0ObL8O4ewODgKclNY71cQywEGgIDAVGSVKoewrYC2gXbHgAQNKRwOPAlcB+wKPAy5L2SPM7cqoZHhg4jlNR/CP84lwb+zV6ITDJzCaZ2TYzex2YCZwMYGavmdmXFjGD6MV5fDntGGFmS81sI/AzYH8zu8vMfjKz/xC93M9Ls6/NwD1mthl4juiF+6CZrTOz+cACoFOs/SwzeyG0/wtRUHFs+NQFhgQ7pgGvEgUxCSaa2dvhe/oxlTFmNt7MVoQ244BFwM9jTb4ys8fMbCswBmgMHBCCh5OAq8zsOzPbHL5vgCuAR83sfTPbamZjgE3BZicDqbZrbI7jVDvOMLM3ksoOBn4j6dRY2W7AdIAw1f0n4HCiHzJ7AfPKacfSpPGbSFobK6sJvJVmX2vCSxZgY/hzZax+I9ELv9DYZrYtLHM0SdSZ2bZY26+IZiJS2Z0SSRcBfwBahKK6RMFKgm9i428IkwV1iWYwvjWz71J0ezBwsaRrY2W7x+x2MgwPDBzHqUyWAk+Z2e+SK8JU9YvARUS/ljeHmYbE1HeqI1V5RMFDggNTtIk/txRYbGaHlcX4MnBQ4kJSDaAZkFgCOUhSjVhw0Bz4PPZssr8F7iUdTDTbcQLwrpltlTSb7d9XcSwFGkjax8zWpqi7x8zuSaMfJwPwpQTHcSqTp4FTJfWUVFPSnmFTXzOiX6V7AP8FtoTZgx6xZ1cC+0naO1Y2Gzg5bKQ7ELi+hPE/ANaFDYm1gw3tJf1sh3lYkM6SzgwnIq4nmpJ/D3gf2EC0mXC3sAHzVKLliaJYCbSK3dchChb+C9HGTaB9OkaZ2ddEmzn/T9K+wYauofox4CpJxyiijqRekuql6bNTzfDAwHGcSsPMlhJtyLuV6IW2FBgA1DCzdcDvgeeB74g2370ce/Yz4FngP2HfQhOiDXRzgFyi/QjjShh/K9HmxixgMbAaGEm0eW9nMBE4l8if3wJnhvX8n4gCgZOCDf8HXBR8LIpRQNvEng0zWwAMA94lCho6AG+XwrbfEu2Z+Ixo0+f1AGY2E/gd8FCw+wugbyn6daoZnuDIcRynApA0EDjUzC6sbFscpzh8xsBxHMdxnHw8MHAcx3EcJx9fSnAcx3EcJx+fMXAcx3EcJx/PY+BUO/bZZx879NBDK9uMHUJeXh516tSpbDN2GJnkTyb5ApnlTyb5AhXnz6xZs1ab2f4ltfPAwKl2HHDAAcycObOyzdgh5OTkkJ2dXdlm7DAyyZ9M8gUyy59M8gUqzh9JX6XTzpcSHMdxHMfJxwMDx3Ecx3Hy8cDAcRzHcZx8PDBwHMdxHCcfDwwcx3Ecx8nHAwPHcRzHcfLxwMBxHMdxnHw8MHAcx3EcJx8PDBzHcRynArn00ktp1KgR7du3zy8bP3487dq1o0aNGgUSuG3evJmLL76YDh060KZNGwYPHlxsPwADBgzgiCOOoGPHjvTu3Zu1a9eWyj4PDBzHcRynAunbty+TJ08uUNa+fXsmTJhA165dC5SPHz+eTZs2MW/ePGbNmsWjjz5Kbm5ukf0AdO/enU8++YS5c+dy+OGHFwgm0mGXDAwkNZM0UdIiSV9KelDS7pKyJX0vabakzyTdn/TciZI+CHWzJY2T1DxWX0vSfyUNSXouR9LM2P3RknLCdbakV8P1E6HfxCdX0sqkvmZLei52/3AoWyBpY+zZsyWNlnR2aLe7pOGSvgh+T5TULNaPSRoWu+8vaWAR399tsXG2xq6vk/SuJIV2NSV9LOk4SQMlLQ/tPpF0WmgTL0989kn379JxHKe60bVrVxo0aFCgrE2bNrRu3bpQW0nk5eWxZcsWNm7cyO677079+vWL7AegR48e1KoVKR4ce+yxLFu2rFT27XJaCeGlNQH4m5mdLqkm8HfgHuA14C0zO0VSbeBjSS+Z2duS2gN/BU4zs09DX6cBLYAlofvuwOfAbyTdYgU1rRtJOsnM/lmUbWZ2SczOGkAO8GSsrA1QEzheUh0zyzOza0JdC+BVM8uKtT8l1v2fgXpAazPbKukSYIKkY4Kdm4AzJQ02s9XFfYdmdk/4vpC0PmnMLsBlwEjgWmCmmb0jqQfwgJndH/x4S1Kj8NgDZnY/abJx81Za3Pxaus2rNDd22ELfDPEFMsufTPIFMsuf6uxL7pBepWp/9tlnM3HiRBo3bsyGDRt44IEHUgYDRfH4449z7rnnlmrMXS4wAH4F/GhmTwCEl+QNwGJgeqKRmW2UNBtoGor+CPw5ERSENi8n9d0HeBD4f0AX4J1Y3X3AbUCRgUEStwL/NbORSf0/BbQBTgeeSacjSXsBlwAtzWxrsP0JSZcSfR9TgS1EAdINwc6ycgPwb0nvAv2Anyc3MLNPJW0BGqbbqaQrgCsAGjbcnzs7bCmHiVWHA2pH/8hlCpnkTyb5ApnlT3X2JScnB4BvvvmGvLw8cnJyWL9+fX752rVrmTVrFuvXrwdg3rx5rF69mmeffZZ169Zx3XXXUbduXZo0aVKon2Sefvpp1q5dS9OmTVPWF8WuGBi0A2bFC8zsB0lLgHwtX0n7AocBb8aeK/JXraQ9gV8DVwL7EL3E44HBu0BvSd2AdcUZKOnnwOXAUUlV5xLNShxB9Gs8rcCAyK8lZvZDUvlMIr+mhvuHgbmShqbZbyHM7GtJw4n8/b2ZfZvcRtIxwDbgv6HoBkkXhuvvzKxbin7/ThS40Lp1a7v2gtPLamKVIicnh3MyTCUuU/zJJF8gs/zJBF9yc3OpU6cO2dnZBdQV99lnHzp37szRRx8NRHsMLr74Yn79618D8Morr1CrVq389vF+4owePZr58+czdepU9tprr1LZtkvuMSiB4yXNAZYD/zKzb5IbSNovrIV/Lql/KD4FmG5mG4EXgTPCMkWcu4HbixtcUl3gaeCy+EtV0tHAajNbQvQiP1JS+vNJaRAChyeB35ezq4eBmmY2Oqn8hjALcz9wbmyp5QEzywqfQkGB4zjOrkrz5s2ZNm0aAHl5ebz33nscccQRxT4zefJkhg4dyssvv1zqoAB2zcBgAdA5XiCpPtAc+IJoj0Enol/Sl0lKrJ/PJ/yCN7M1YV3970DdUN8H+LWkXKIZif2IpunzMbNpQG3g2GLs+ysw0cymJpX3AY4I/X8J1AfOSs9lvgSaS6qXVN45+BVnONEegTpp9l0IM9sGWIqqRABwvJm9Vdb+HcdxqjN9+vShS5cuLFy4kGbNmvHaa6/x0ksv0axZM95991169epFz549AbjmmmtYv3497dq142c/+xmXXHIJHTt2TNnPqFGjAOjXrx/r1q2je/fuZGVlcdVVV5XKvl1xKWEqMETSRWb2ZPhVPwwYDWxINDKzxeF0wR+JXspDgZckvRfbZ7AX5AcWxwMHmdmmUHZJeO71pPHvBh4B/pNsWDhB0ImkwCFsRDwH6GBmK0JZN+AO4LGSHDazPEljgL9Iuirsq7go2D8tqe23kp4nCg4eL6lvx3Ecp3Q8++yzBe4TSwm9e/cu1LZu3bqMHz8+rX4SfPHFF+Wyb5ebMQjT172JTg4sIjpF8CPRZr9kHgG6SmphZvOA64AnJS2U9DbRJsBnQn/TEkFBYCJwqqQ9ksafxPa19WTuAfYHPogf3yMKOpYngoLAm0BbSY3TdP2W4Ofnwe/fAL2TTk4kGEYpNgbuAG5IOq7YogLHdhzHcWLsijMGmNlS4NQUVTnhk2i3ke2nEjCz14iONCbzOTAmaYxviV7yANlJdZ1j1/ljmlnhQ6zbKTCLEE4XHBi7zwXaJ7XpG7veRLRh8dpUnZtZ3dj1SsJsSEnEnyuu3MwGFtFuIJCyznEcx6l4drkZA8dxHMdximaXnDFw0kfSbUTLDnHGhyRHjuM4TobhMwZOsZjZPbGjhImPBwU7mKVLl9KtWzfatm1Lu3btePDBBwH49ttv6d69O4cddhjdu3fnu+++A2Ds2LF07NiRDh06cNxxxzFnzpwC/W3dupUjjzySU045pdBYjuM4xeGBgeNUAWrVqsWwYcNYsGAB7733Hg8//DALFixgyJAhnHDCCSxatIgTTjiBIUMiGY6WLVsyY8YM5s2bxx133MEVV1xRoL8HH3yQNm3aVIYrjuNUczwwqCCSxIZmS7o5iAzNktQ11m6KpN9Iej+0W6JImKnYHfuSLpU0T9LcIFJ0eiiXpNsVCSd9Lmm6pHax59Yn9dNX0kPhOi5wtEBSn6S2/bVdUOrDcAQyIRq1MGbzC8V8L1cFu2dL+rektqX/dqs/jRs35qijokSX9erVo02bNixfvpyJEydy8cUXA3DxxRfzj3/8A4DjjjuOfffdFygskrJs2TJee+01Lr/88gr2wnGcTMD3GFQcG+NiQwkkXQ08JqkzcDawzczGA+NDfV/gaDPrV1THilQSbwOOMrPvQ/bExImIa4DjgE5mtiGIGb0sqZ2Z/ZiG3Qnho8OAWZJeMLPNkq4iSs/885BSuj7Rsc0EF5jZzJQ9FuQZM3sk+HEa8BfgxOIeyDQRpeykstzcXD7++GOOOeYYVq5cSePG0YnUAw88kJUrVxbqY9SoUZx00kn599dffz1Dhw5l3bpiM287juOkxAODSsbM3g+CQwOB84letqWlEZH+wvrQ5/rENVGCpl+a2YZQN0XSO8AFwKhS2LlI0gZgX2AVUd6H7IT+QvhzTDFdFNVvXL+hDqkzJu4yrF+/nrPOOovhw4fnS6smkIQiRet8pk+fzqhRo/j3v/8NwKuvvkqjRo3o3LlzqURTHMdxEnhgUHHUDsmKEgw2s3Hh+hZgKTDczMqSsmoOsBJYLGkqMMHMXgm/4uuYWXKWxYR4UtpIOgpYZGarQr/1UvQbZ6ykjeH6dTMbUEzf1wB/AHYnKY10rE3GqismXuBbtmzhlltu4ZhjjqFBgwbk5ORQv359XnzxRfbbbz/WrFlDvXr18tt/+eWX3HnnnQwZMoR58+YBUSa0KVOmMGHCBH766Sc2bNhA9+7due228ghmpk9cJa66k0m+QGb5k0m+QBX0x8z8UwEfYH0xdWcAK4g0EpLr+gIPpdG/iCSObyHSfBhIpKfwbYq21wF/SWVXGO+v4XogkZjUfGAzcGIor0+kgliULTlEyx+l/Y7OB8aU1O7www+3TGH69OlmZrZt2zb77W9/a9ddd12B+v79+9vgwYPNzGzw4ME2YMAAMzP76quv7JBDDrG333672L579eq1cwwvZsxMIZN8McssfzLJF7OK8weYaWn8W+ybDysZSXWIdBh+BTSSdHJZ+gl/7x+Y2WDgPOAsi6bp8yS1SmoeF0/aKGn3WF0DYHXs/gEza0ck2DRK0p6h3/Up+i0vzxEFSbscb7/9Nk899RTTpk0jKyuLrKwsJk2axM0338zrr7/OYYcdxhtvvMHNN98MwF133cWaNWu4+uqrycrKypdodRzHKS++lFD53Ak8b2afhY2Iz0maZultDARAUhPgQDP7KBRlAV+F6/uAEZJ+Y2YbJf0a+AVwZaifAVwIPC6pNpFY003JY5jZy5IuAy4GHgUGAw9LOteizYd1gTPN7MnSOC/pMDNbFG57AYuKa5+p/OIXv0jMmhRi6tRkoU0YOXIkI0eOLLbP7OzsQhrtjuM4JeGBQcWRvMdgMvAU0U7+TgBm9rGkfxFtGBxUir53A+4PAcKPRCJNCZ3NvxJtGJwnaSvwDXC6RToQEC0rPCrp90TLEU+a2ZtFjHMX8Iykx4C/EUlOfyhpM9FSw7BY2/geg9Vm9usi+uwXgpXNwHdEgYfjOI5TSXhgUEGYWc0iqg5Pavf7pPvRRJLQxfX9FUVs2gvrSoMoItAws+VAyvR4liR8ZGazgLjQ09DwSX4uuzh7k9pel25bx3EcZ+fjewwcx3Ecx8nHZwyqGZLeB/ZIKv6tmc2rDHvSxcWYHMdxqgceGFQzzOyYyrahLIQAwIMAx3GcKo4vJThOJXHppZfSu3dv2rdvn182e/Zsjj322PwjiB988AEA33//PaeeeiqdOnWiXbt2PPHEEwX6+uGHH2jWrBn9+hWZOdtxHCctPDBwnEqib9++3HvvvQXKbrrpJv70pz8xe/Zs7rrrLm66KTo5+vDDD9O2bVvmzJlDTk4ON954Iz/99FP+c3fccQddu3bFcRynvHhgUEG4umKR30tXSR9J2iLp7NJ/s9WXrl27ptRD+OGHSD7i+++/p0mTJvnl69atw8xYv349DRo0oFataCVw1qxZrFy5kh49elSsA47jZCS+x6DicHXF1CwhSsPcP422QPVXV8wd0qvIuuHDh9OzZ0/69+/Ptm3beOeddwDo168fp512Gk2aNGHdunWMGzeOGjVqsG3bNm688Uaefvpp3njjjYpywXGcDMYDg0rGXF0xF0DStuLaZZKIUlwsJS8vj7y8vPyyESNGcNlll/HLX/6S6dOnc+aZZzJs2DBmzJhBw4YNeeaZZ1ixYgWXX345I0eOZMqUKbRu3ZovvviCzz77jOXLl1eqGEuVE4MpB5nkC2SWP5nkC1RBf9IRVPBP+T/AVmB27HNurK4BkAfck+K5vpQgogTUBP5F9Ov7CeDUUF5WEaWHwvVAoH+4Pgp4K9ZvSSJKC2O+3pfG9zMaODud7zKTRJSeffZZa9euXf59/fr1bdu2bWYWCSvVq1fPzMxOPvlke/PNN/PbdevWzd5//307//zz7aCDDrKDDz7Y9ttvP6tXr5798Y9/rFgnYmSSuE0m+WKWWf5kki9mVU9EyWcMKo6USwmBrsD3QPsi6ovFzLZKOhH4GXAC8EBYmvhLmSyFeNL+GyRdQpSh8dRS9JHuUoITo0mTJsyYMYPs7GymTZvGYYcdBkDz5s2ZOnUqxx9/PCtXrmThwoW0atWKsWPH5j87evRoZs6cyZAhQyrLfMdxMgAPDCqZJHXFJySdbGaTSttPiAY/AD6Q9DrwhJkNlJQnqZWZ/SfWvDOReBIEdUUzS2xxT6WueL+k04jUFQ+xaE/B+hT9OqWgT58+TJkyJf+o4aBBg3jssce47rrr2LJlC3vuuSd///vfgejUQd++fenQoQNmxr333kvDhg0r2QPHcTIRDwwqn11aXXFX5tlnnyUnJ6eQAuKsWbMKtW3SpAlTpkwptr++ffvSt2/fHWih4zi7Ih4YVByurpgCST8DXgo2nippkJm1S9XWcRzH2fl4YFBBmKsrpsTMPgSapdvecRzH2bl4giPHcRzHcfLxGYNqhqsrOo7jODsTDwyqGebqio7jOM5OxJcSHGcncemll9KoUaMC6okDBw6kadOmZGVlkZWVxXvvvQfA2LFj88uysrKoUaMGs2dHe1Wzs7Np3bp1ft2qVasqxR/HcXYNPDCoIFxEqcjv5Q+h77mSpko6uPTfbtWkb9++TJ48uVD5DTfcwOzZs/MllgEuuOCC/LKnnnqKli1bkpW1PR/W2LFj8+sbNWpUYT44jrPr4UsJFYeLKKXm4+DfBkn/j+iUw7nFPVDVRZQSIkldu3YlNze31M8/++yznHfeeTvYKsdxnPTwGYNKxszeBxIiSn8GigwAiqGQiJKZLQ51fwT6WUxECUiIKJXGzkVAQkQJIhGl/2cxESUzK4uI0vSEbcB77AJHFx966CE6duzIpZdeyrp16wrVjxs3jj59CkzOcMkll5CVlcX//u//JrQlHMdxdgo+Y1BxJCc4Gmxm48L1LcBSYLiZfVGGvucAK4HFkqYCE8zslfArvk6KtMUzgVIlEZJ0FLDIzFaFfuuVkA45nuDodTMbkMYwlwH/LGL8aqOuGFdJ++abbwqoJ3bs2JFRo0Yhiccff5wRI0ZQr169/PYLFizAzFi9enX+M9dccw37778/GzZs4E9/+hMbNmygZ8+eFehR+lQ5lbhykEm+QGb5k0m+QNXzxwODisNFlIpB0oXA0cAvUxpk9nfg7wCtW7e2ay84vRSmVB65ubnUqVOnUNpjgFatWtGtW7cCdRMnTuTyyy9P2R5g1apVzJw5s8j6yiZViufqSib5ApnlTyb5AlXPH19KqGSSRJQaSTq5LP0EVc0PzGwwcB5wVpjmz5PUKql5Z2B+uN4oafdYXSoRpXbAWUQiSnuGften6LdMBP2G24DTzGzTjuizqvL111/nX7/00ku0bNky/37btm08//zzBfYXbNmyhdWro7+OzZs38+qrrxY45eA4jrOj8cCg8skXUQKuJvq1v2dpOpDUJEz1J0glolQ7tE2IKD0T6hMiSsRElKYnj2FmLxMtQVwcihIiSvXDs3UTpxJKafuRRKJMp5lZRp3D69OnD126dGHhwoU0a9aMUaNGcdNNN9GhQwc6duzI9OnTueaaa/Lbv/nmmxx00EG0arU93tq0aRM9e/akY8eOZGVl0bRpU373u99VhjuO4+wi+FJCxeEiSqm5L/QzXhLAEjM7rRS+V1meffbZQmWXXXZZgfv4umJ2dnZ+XoMEderUSam26DiOs7PwwKCCcBGl1BQTMDiO4ziVgC8lOI7jOI6Tj88YVDNcRMlxHMfZmXhgUM1wESXHcRxnZ+JLCY6zg0glmpRg2LBhSMo/ejh27Nj87IfHHXccc+bMAWDp0qV069aNtm3b0q5dOx588MEK9cFxHMdnDBxnB9G3b1/69evHRRcVPLW5dOlSpkyZQvPmzfPLWrZsyYwZM5gzZw4bN27kiiuu4P3336dWrVoMGzaMo446inXr1tG5c2e6d+9O27ZtK9odx3F2UXzGoBog6QxJJumIcN8i3N8da9NQ0mZJD0m6LaZsGFd1/H0xY1wUVBnnSfpYUv9QPlrS4vD8HEknxJ5JqaKYpMq4SNIESW2TnjtapVSQrOp07dqVBg0aFCq/4YYbGDp0KOE4JgDHHXcc++4byU4ce+yxLFu2DIDGjRtz1FFRSop69erRpk0bli9fXgHWO47jRPiMQfWgD/Dv8OefQtlioBdwe7j/DSGbYXw9X9L6YlIxE9qcBFwP9DCzFZL2AOI/eweY2QuSuhGlJT4sVldU6uMHzOz+0P+5wDRJHczsv4kGif0S6ShIxqlq6ooJNcVUTJw4kaZNm9KpU6ci24waNYqTTjqpcL+5uXz88cccc0y13FbiOE41xQODKk6QUP4F0A14he2BwQbgU0lHhxfzucDzQJMyDHML0N/MVgCEtMSPpWj3LtC0tJ2b2ThJvYDzgTItmldlEaWiRJN+/PFHbr75Zu677778+7fffpu99947v/0777zDo48+yogRIwr0s3HjRq677jouv/xyPvroowr0pnxUNTGY8pBJvkBm+ZNJvkDV88cDg6rP6cBkM/tc0pogjrQm1D0HnCdpJbAVWEHZAoP2QDrp9U4E/pFUlq6K4kfAEWWwDSgootS81aE2bF7V+U8394Ls7dcx0aR58+axZs0a+vWLJkJWr17NtddeywcffMCBBx7I3Llzefjhh5k+fTqHH749z9XmzZs55ZRTuOqqq/jDH/5Q0e6Ui6omBlMeMskXyCx/MskXqHr+VJ1/XZ2i6MP2X9nPhfuHwv1k4H+JJJfHFX50h3GfpD8DzYAuSXXpqiiq5CbpUXu3miwsZvq+qtChQwdWrdou/9CiRQtmzpxJw4YNWbJkCWeeeSa33HJLgaDAzLjsssto06ZNtQsKHMfJDHzzYRVGUgOiVMcjJeUCA4hEjgRgZj8R/dK/EXihHEPNJ1JcLIoBZnY4kYbD42Uc40jg0zI+Wy1IJZpUFHfddRdr1qxh+PDhZGVlcfTRRwPw9ttv89RTTzFt2jSysrLIyspi0qRJFeWC4ziOzxhUcc4GnjKzKxMFkmYAB8XaDANmmNm38V3vpWQw0axALzP7JsgwX2RmI5PaPQRcKqmnmf0r3c4lnQX0IApgMpZUoklxcnNz869HjhzJyJEjC00h/uIXvyCSt3Acx6kcPDCo2vQB7k0qe5FosyAAZjafcBqhrJjZJEkHAG8oii6MFDMDZpY4InkTkAgMilJRvEHShUAd4BPgV/ETCY7jOE7VxAODKoyZdUtRNgIYUUT70SQpMZpZ3TTHegJ4IkV536T7F4mCkyJVFIMq48BixspOuh9NCQqSjuM4TsXgewwcx3Ecx8nHZwx2IVzh0HEcxykJDwx2IVzh0HEcxykJX0pwnB1EadQVP/vsM7p06UKPHj24//77C7R94IEHaNeuHe3bt6dPnz78+OMprWmPAAAgAElEQVSPFWK/4zgOeGDgODuMvn37Mnny5ELlqdQVGzRowIgRIzjnnHMKtF2+fDkjRoxg5syZfPLJJ2zdupXnnntup9vuOI6TwAODEpDUTNLEoBL4paQHwzl/JP1c0ptBYfBjSSMl7RXqTpI0U9KCUDcslI+WdHbSGOvDny0kbQwKgwskPSKpRqzd9ZJ+lLR3uO8ZUyRcH1M6fFJStqRXY8+eIWmupE+DguIZsbrRQQ1xj3DfMCRUSvV9dIiN+W1MefENSZ9J6hBrO0DSo0X5lVSe+FyUatzqQGnUFRs1asTPfvYzatUqvJq3ZcsWNm7cyJYtW9iwYQNNmpQly7XjOE7Z8D0GxRDO9E8A/mZmp0uqSZSv/x5J9wPjgfPM7N3Q/mygnqRWRMmAepnZZ+G5K9Ic9kszy5JUC5gGnBFsgCivwYfAmcATIcnQv8LYOURCSDPDfXbMj07A/UB3M1ssqSXwuqT/mNnc0GwrcCnwt+KMM7N5QFbodzTwqpkl5JZPBP5PUlcizYargKOBvYvw66NEeZrfDZB56opxmjZtSv/+/WnevDm1a9emR48e9OjRY0eZ6jiOUyIeGBTPr4Afwxl/zGyrpBuIJI8NGJMICkJ94gV5H3CPmX2WeI4SXrjJmNkWSe8Ah4Y+DwHqAlcDt5Ei50Ax9Af+bGaLQ9+LJQ0mSrH829BmOFFSolSqiunaPFnSpUSSzb2AgWb2XWKGI4VfacsGZqq64k8//cSXX36Z38e6desYM2YMTz/9NHXr1mXgwIHcdtttdO/evYK9KhtVTSWuPGSSL5BZ/mSSL1D1/PHAoHjakaQ6aGY/SFpC9GIbU8Rz7YlSFZeZsCRxAnBnKDqPSETpLaC1pAPMbGWa3bUjmjGIMxO4Jna/BPg3UaDwSlntBq4HPgAWmdlTyZUp/DpE0uxYk2vN7K3k5+Lqiq1bt7ZrLzi9HCbuPEqjrggwevRoDjnkkPy0yOPHj+fII4/kjDOilZ4VK1bw3nvvVSnlteKoaipx5SGTfIHM8ieTfIGq548HBhVPqkT48bLEi9KAiWb2z1DeB+htZtskvUiUj+AhdiyDgYlAmefpzWyFpGnAq0lVhfyS1IIyLCVUF4pTVyyK5s2b895777FhwwZq167N1KlT8wWWHMdxKgLffFg8C0hSHZRUH2gOfJlcF6M4tcI1wL6x/hoAq2P1X5pZlpkdGVILEzb0HUa0LyCXaPagT3n8CPcFNBbMbBEwm0jBsTxsC584hfzKNEqjrvjNN9/QrFkzxo8fz913302zZs344YcfOOaYYzj77LM56qij6NChA9u2beOKK9LdnuI4jlN+fMageKYCQyRdZGZPhk2Ew4jy+t8PfCDpNTN7H0DSmcDbwH3ABEn/NrPPw8mCK8zsESAHuF7SmCCb3BeYXoIdfYjW6wcnCsJpgIPN7Ks0/LgfGC9pmpnlhl/qtxKpNyZzD+WYMdiVKY264oEHHsiyZctSTiEOGjSIQYMG7QQLHcdxSsZnDIrBIv3b3sBvJC0CPgd+BG4N6/vnAfeHY4KfAj2BdWGn//XAs6H8E6BV6PNVon0Cs8LU+v8AfyzBlPOAl5LKXgrl6fgxO4zxiqTPiPYQ3BTKk9vOpxSbAncAhyQdV/x9BY7tOI7jJOEzBiVgZkuBU4uoexc4voi6Vym8zp6oGwQU+kloZrlEGxeTy1ulKPtD0n120n0O0exE4n4C2489JvfVN+n+zFTtSnqumP5ySe1XLlA7nbEcx3GcisFnDBzHcRzHyafUMwaS9gUOiiXGcTKUsOkx+cjhJjM7pjLscRzHcXY+ac0YSMqRVD/soP8IeEzSX3auaU5lY2bzwkmC+MeDghiphJPuuOMOOnbsSFZWFj169GDFihXAduGkPfbYo5BwUosWLejQoQNZWVl+PNFxnEol3aWEvc3sB6JUvE+Gl8Ovd55ZjlM9SCWcNGDAAObOncvs2bM55ZRTuOuuu4Dtwkn9+/dP2df06dOZPXs2M2fO3Ol2O47jFEW6gUEtSY2Jzren3FDn7DwkbQ079udLmiPpxri4UmgzPAgh1Qj37SR9Lql2rM1rkvpIOkDSq6GvBZImFTN2XOhojqR3JLUOdflCTZL6SvpvaPdZSB2NpNtiJw62xq5/L2mgpP5J4+VKKjoDUBUjlXBS/fr186/z8vLyxZMSwkm77bZbhdroOI5TGtLdY3AXkVjP22b2YRAJWrTzzHKS2JjIDiipEfAMUB/4UyirQXSscinwS2C6mc2XNIFIV+F2RWqKu5nZs5IeBV43swfD8x1LGP/L2PhXEuVAuDhFu3Fm1k/SfsBCSS+Y2T1EuRGQtD6e5VDSwDJ9GVVERKk48aTbbruNJ598kr333pvp00tKUwGS6NGjB5K48sorPamR4ziVRlqBgZmNJ1ISTNz/BzhrZxnlFI2ZrQqCQh9KGhhyLWQTZTEcR5QMKfEmugv4WNILwBC2H7tsDEyJ9VmajaT1ge9KsHGNpC/COEtL0XeRVEURpYToSVw4KUH37t3p3r07Y8eOpX///lxyySX5dbm5udSuXZucnJx88ZShQ4ey//77891339G/f382btyYtiJjVaKqicGUh0zyBTLLn0zyBaqeP2kFBpIOJ1IHPMDM2odfmKeZ2d071TonJWb2n5CFsRGwkigYeJZI5+DPknYzs81mtiFM1b8J/CWkPAZ4GBgnqR/wBpGE84pihkzoHNQD9gKK3YAoqTmwJ5BOwHGDpAtj901SNYqLKDVvdagNm1f5KThyL8iO/owJJyXTqlUrTj75ZMaM2a63lZOTQ926dcnOzk6Z+XDOnDls3ry5SomqpEtVE4MpD5nkC2SWP5nkC1Q9f9L91/UxIoneRyH6hSnpGcADg0pG0u7AycAfzGydpPeJMjC+CmBmr0haC/xf4hkz+1dYDjoROIloVqG9mf23iGHiSwnnEr2gT0zR7lxJXYEjgH5m9mMaLjxgZvlb9IMWRLHU3q0mC4uZxq9sFi1axGGHHQbAxIkTOeKII4ptn5eXx7Zt26hXrx55eXlMmTKFO++8s9hnHMdxdhbpBgZ7mdkHiU1Ugcqfy91FCS/1rcAq4BRgH2Be+PvZC9hIwU2ihUSNzOxbor0Kz4QNhF2BF9MY/mXgiSLqEnsMjgamSHrZzL5J27FqSJ8+fcjJyWH16tU0a9aMQYMGMWnSJBYuXEiNGjU4+OCDeeSRR4BoyeHoo4/mhx9+oEaNGgwfPpxHH32UlStX0rt3bwC2bNnC+eefz4knpoq7HMdxdj7pBgarJR1CkAeWdDbw9U6zyikSSfsDjwAPmZlJ6gNcbmbPhvo6wGJJe5nZhiL6+BXwXlhqqAccAixJ04RfEClLFomZzZT0FHAdcEua/VZLUgknXXbZZSnbJoST4uTk5NCqVSvmzJmzU+xzHMcpLekGBtcQTR8fIWk5sBi4YKdZ5SRTO6zx70Y0U/MU8BdJexFN6V+VaGhmeZL+TbTRcFwR/XUGHpK0hejI6kgz+7CY8RN7DAT8BFyehs33Ah9J+rOZrUujveM4jlMFKDEwCEfhjjazX4dfozX8H/qKxcxqFlG1AWiQXJgsgmRmLZLu7yOShk5n7FyKEDqKCzWZ2WgiOepE3QrgwKT2dZPuB6bos0VymeM4jlNxlJjgyMy2ATeF6zwPChzHcRwnc0l3KeGNcOxtHJCXKAwb2JwMwAWTHMdxHEg/MDg3/HlNrMyAVjvWHKeyMLN5QFaJDR3HcZyMJi2tBDNrmeLjQYGzS5FKSXH8+PG0a9eOGjVqFBA/ev311+ncuTMdOnSgc+fOTJs2Lb/up59+4oorruDwww/noosu4sUX0zkl6jiOUzGkm/nwolTlZvbkjjXHcaouffv2pV+/flx00fb/Hdq3b8+ECRO48sorC7Rt2LAhr7zyCk2aNOGTTz6hZ8+eLF++HIB77rmHRo0a8fnnnzNt2jQ6dixJqsJxHKfiSFdd8Wexz/HAQOC0nWTTLoekMySZpCPCfYtwf3esTUNJmyU9VJxiYQnjzJb0XFLZ6KDKuEdsnNwkO66NtX9IUt9wnROSGRFr/0m4zg4KjpfE7PtJ0rxwPV5FqD+W/ZvcuaRSUmzTpg2tW7cu1PbII4+kSZMou3O7du3YuHEjmzZtAuDxxx/nllui9A41atSgYcNqIybpOM4uQLoiStfG7yXtAzxXRHOn9PQB/h3+/FMoWwz0Am4P978hEkqiOMXCopDUBqgJHC+pjpnlxaq3ApcS6WEkswq4TtKjZvZTaR0zsycImRJDwNHNzFaH+yGkUH8sqc/KUFcsTkmxJF588UWOOuoo9thjD9auXQvAHXfcQU5ODvXr12fcuHEccMABO8pUx3GcclFWJZo8oOWONGRXRVJdomyC3YBX2B4YbAA+lXS0mc0k2gD6PEWIDKVBH6JTB22A04nSIScYTiRm9FiK5/4LvE0ks5yqvjwUpf5YiMpWVyxOSRFg7dq1zJo1i/Xr1xcoX7x4MbfffjtDhw4lJyeH77//nmXLlrH33nvzl7/8haeffprf/va33HrrrRXkyc6lqqnElYdM8gUyy59M8gWqnj/p7jF4hZAOmWj5oS0xGWanXJwOTDazzyWtkdQZWBPqngPOk7SS6Ff9CsoeGJwLdCcSOLqWgoHBEqIZi98SBSfJ3Av8U9LjZRw7JcWoP6Zqm6+u2Lp1a7v2gtN3pClpU5SS4j777EPnzp05+uj8lRWWLVvGFVdcwfPPP8///M//AGBm7LXXXtxxxx3UqFGDVatWMWjQoCqlrFYeqppKXHnIJF8gs/zJJF+g6vmT7ozB/bHrLcBXZrasqMZOqegDPBiunwv3D4X7ycD/EkkrF5XeuETCPoDVZrYkpLR+XFKDpDwUg4lkmwvN0QeZ5/eB85OrUgyXqqxIUqk/ZgJr166lV69eDBkyJD8oAJDEqaeeSk5ODr/61a/46KOPaNu2bSVa6jiOU5B0Nx+ebGYzwudtM1sm6d6datkugKQGwK+AkWH9fQBwDpEmAWFNfxZwI/BCOYbqQ6RzkUskgFQfOCveIPxanx3GT8WfgT8mbAusAfaN3TcAVpfBvkLqj1WRPn360KVLFxYuXEizZs0YNWoUL730Es2aNePdd9+lV69e9OzZE4CHHnqIL774grvuuousrCyysrJYtWoVAPfeey8DBw6kY8eOTJkyhWHDhlWmW47jOAVId8agO9FLIc5JKcqc0nE28JSZ5Z91kzQDOCjWZhgww8y+TZK9TougdXEO0CHoFyCpG3AHhfcM3EOKGQMAM/tM0gKifQAJwaUc4EJJb5iZEe1DmF5qI6sJqZQUgXzJ5Di33347t99+e4rWcPDBB/Pmm28C0RRi8+bNd5yRjuM45aTYGQNJ/0/SPKC1pLmxz2JgbsWYmNH0AV5KKnuRmFSxmc03szHlGON4YHkiKAi8CbSV1Dje0MzmAx8V09c9QLPY/d+BdcAcSXOAuhRcdnIcx3GqGSXNGDwD/JNo/fnmWPk610koP2bWLUXZCGBEEe1HE1MwDGV1U7WN1c8Ajk0q28p25cO+SXVnxq5zgfax+znEgsmw1NGviHFzCMqLsbIWRbRNWe44juNUPMUGBmb2PfA90S9bJDUC9gTqSqprZkt2vomO4ziO41QUaW0+lHSqpEVESXdmALlEMwlOFSIpI2Lic1tl2+U4juNUH9LdfHg30XT0G2Z2ZNi8duHOM8spC/GMiI7jOI5TFtI9rrjZzNYANSTVMLPpwNElPeQ4mUJplBUBBg8ezKGHHkrr1q3517/+VWw/juM4VYl0A4O1IXXvW8BYSQ8SpUXOWCpC2EjSheGUx3xJcySNDDoUSNpd0nBJX0haJGmipGaxZ5uFskWSvpT0oKTdQ122pO8lfSxpoaQ3JZ0Se7Z1EECaLelTSX8vxsZEX4m2f0pR/pmk+2PP9JX0UOz+IkmfBAGlj0O2w4SA0+LYd/VO6f6WKo6+ffsyefLkAmUJZcWuXbsWKF+wYAHPPfcc8+fPZ/LkyVx99dVs3bq1yH4cx3GqEukuJZwObASuBy4A9ibKc5/J7FRhI0knAjcAJ5nZckk1ifIAHACsJUooVA9obWZbJV0CTJB0TOhiAvA3Mzs9PPv3MP6AUP+WmZ0SxsoC/iFpo5lNJTr18ICZTQz1HUr4Lt4ys1Mk1QFmhxTZ8fLaRJoHL5nZ20l+nkT0300PM1uhSMUxLuM9wMxKlbypokWUcof0omvXruTm5hYob9OmTcr2EydO5LzzzmOPPfagZcuWHHrooXzwwQd06dIlZT+O4zhVibRmDIIS30FAdjhTPxIotdJedUHbhY0uA86LVeULG4X7hLBRWbgN6G9myyE6Qmhmj5vZQkl7AZcAN4SjhQmVwk1EmRJ/BfwYyhLHD28ALg3PFsDMZhMFcomjhY2BZbH6eekYHP47mAUcmlS+kShrYtMUj90S/FwR2m4ysx0txlSlWL58OQcdtD1HVbNmzVi+fHklWuQ4jpM+6Yoo/Y5I2a4BcAjRC+AR4ISdZ1qlUhHCRu0oOpnQocASM/shqXxmeA6iF3Q+ZvaDpCUkvbRjfMT22YQHgGlh6n4K8ISZrS3JYEn7EW1C/V9g/1j5vsBhRImTkmmfbGsS90lKzMDMN7MLihi70tQVS6usuHz5cj799NP8dl9//TXz58+nYcOGhfqpaqpq5SWT/MkkXyCz/MkkX6Dq+ZPuUsI1wM+B9yHKqx9yGmQqO13YKE6Yyn+KaOngVuDTHdFv8jCJCzN7QtK/gBOJgqArJXUys01FPHu8pI+J9AyGmNl8SdmhfA5RUDDczL4pg11pLSVUBXXFdJUV3333XYD8doMHD6ZHjx506dKlUD9VTVWtvGSSP5nkC2SWP5nkC1Q9f9LdfLgpZLkDQFItSqmiV11QxQkbzQeOCn3OC3sS/gnUJhI6ai6pXtIzncNzC8J13O76QHPgiyLGO5JYwGFmK8LSxelEipnFbZN/y8yONLPOZvZIUnknolmMy8JehlR+dk5RnrGcdtppPPfcc2zatInFixezaNEifv7zn1e2WY7jOGmRbmAwQ9KtQG1J3YHxwCslPFNdSQgbHWxmLczsIKJNh8nCRn8sZ1rowcD98ZMGREFBYi1/DPCXsLEQSRcBewHTgKnAXqGM0GYYMNrMNiQPJKkjkWjSw+H+REm7hesDgf2AMi+Cm9liYAipRbUGEy0XHBjG213S5WUdq7IojbJiu3btOOecc2jbti0nnngiDz/8MDVr1kzZz2uvVdwmSsdxnHRIdynhZqKNePOAK4FJRBsQM5E+QLKkdCFhI8JphLJiZpMk7Q/8M7zY1wKfAIlD77cQCRJ9Lmkb8BnQO6gYIqk38H+S7iAK8CYRLUMkSEz/7wWsAn4fTiQA9AAelPRjuB9QxmWAOI8A/SW1SOHnAcAbkkQ00/R4rEl8jwHAz+OzU1WF0igrAtx2223cdlvhpJPJ/VSldUXHcRwoITCQ1NzMlpjZNiKJ3ozeTQ4VI2wUazeGaGYgVd0m4NrwSVW/lEgCOVVdDtGR0qLG/QPwhzRtzCFJDClVeTiZkDiVMJrYdxJOTzyRoo++6djgOI7jVBwlLSX8I3Eh6cWdbIvjOI7jOJVMSUsJil232pmGZCqKRIx+k1Q8PiREqjJI6knhJZTFZpZ6rtxxHMfJSEoKDKyIaydNqouwkZn9i+37GxzHcZxdlJKWEjpJ+kHSOqBjuP5B0jpJycl3HCfjSCV69O2339K9e3cOO+wwunfvznfffQfA2LFj6dixIx06dOC4445jzpw5+c+0aNGCDh06kJWVlZ/vwHEcpypSbGBgZjXNrL6Z1TOzWuE6cV+/oox0nMoilejRkCFDOOGEE1i0aBEnnHACQ4YMAaBly5bMmDGDefPmcccdd3DFFVcUeG769OnMnj27kBKj4zhOVSLdPAY7HUkHSnouKAXOkjRJ0uGKFA2vjbV7KKj3PRwU+RZI2hhT6Du7iP5HJ+qCsuDMWN3RknLC9V6SxgYlwE8k/VvSwbH+v5G0PHafUDQsoMYYylpI+qQIWzbEExgpUlI0SQ3DfVyhcbakm4uzXVLPWNv1ilQVZ0t6sojvI66OOFvSG8nfU6ztekkdYm2/1XZVxDdCX6+W8H0vVKQg+WE8EZKk3PBdJ/pOefqjsujatSsNGjQoUDZx4kQuvvhiAC6++GL+8Y9oj+5xxx3HvvvuC8Cxxx7LsmXLcBzHqW6km8dgpxLOt78EjDGz80JZJyKlwVXAdZIejZ9vN7NrQrsWwKslqRmmoJGkk8zsn0nl1wErzaxD6L818E2if0kDgfVmdn/Sc6nUGIvjC6J0xE9LqkGUbTGeZGhjMT4Vsj2+RyAEOf3NrKSfpvkKjCURhJYS38Foou/8hXCfnUYXF5jZTEUqkfcB3WN13cxsdTp2QMWpK+YO6ZWyfOXKlTRu3BiAAw88kJUrVxZqM2rUKE466aT8e0n06NEDSVx55ZWFZhMcx3GqClUiMAC6AZvj6XbNbE546f8XeJtIknhH5lG4j0jhMDkwaAx8FbNjYUkdabsaYzeijJDpBAbPEakzPg1kE/l4UnEPxCjK9urAu2wXc0obVYKIUlHiSVu2bCmQmGjr1q0F7j/++GP++te/MmLEiPzyoUOHsv/++/Pdd9/Rv39/Nm7cSKdOnaqceEp5ySR/MskXyCx/MskXqHr+VJXAoCQFvnuJMgQ+Xkyb0vIu0FtSN2BdrPxxYEqYBp9KNIuxqIS+Cqkxmllx/gB8DpymSJmwD1GAEA8MakuaHbsfbGYJ0aaibC8tx8fGqKgjlCcSy48RmC5pa7geY2YPJD8UF1Fq3upQGzZv5/+nm3tBdvRnknhS06ZNad26NY0bN+brr7+mSZMm+XVz587loYce4vXXX+fwww9P2e+cOXPYvHmziyhVcTLJF8gsfzLJF6h6/lSVwKBYzOw/kt4Hzt/BXd8N3E4sx7+ZzZbUiiht8K+BDyV1MbPiFA9TqTGWFBgATADOA44hSjUdp7ilhJS2l4FUSwmpjqWWdFS1qPp4+diwH6MuYUkiRqmWEmrvVpOFRUzzVwSnnXYaY8aM4eabb2bMmDGcfnqk9LhkyRLOPPNMnnrqqQJBQV5eHtu2baNevXrk5eUxZcoU7rzzzsoy33Ecp1iqSmAwn0i8qDj+TKRmOGNHDWpm0yTdDRybVL6e6KU9QZFOwckUIYWs7WqMHSQZUBMwSelMl48jCiDGmNm2aKtF+WzfAawB9k3cBP9KemkXeCaQ/NwFRL7eB/wVOLPcllYAffr0IScnh9WrV9OsWTMGDRrEzTffzDnnnMOoUaM4+OCDef755wG46667WLNmDVdffTUAtWrVYubMmaxcuTJfU2HLli2cf/75nHjiiZXmk+M4TnFUlcBgGvBnSVeEKeOEImB+vn8z+0zSAiJ9gA934Nh3EwkA/SeM+z/AAjP7LvzCbUsKrYAYCTXG/F/8kmYAxwNLihvYzL5SlBnxjR1h+w4iB7he0piw2bMvML2EZxYBTSS1MbNPJR0MdALiSyGYmSkSffpS0hFm9tkOtHunUJR40tSpUwuVjRw5kpEjC2uLtWrVqkBOA8dxnKpMlQgMwgujNzBc0h+BH4Fc4PqkpvcAH+/gsSdJ+m+s6BDgb+GkRA3gNSJ1xaIoSo0xUd5aUvzc2g1J4z9aRL/Jewwmm9nNJdhebszsVUmdgVlh3f9L4KoSntkk6ULgCUl7ApuBy83s+xRtN0oaRrQB8bJQHN9jMNfMLtpR/jiO4zilo0oEBgBmtgI4J0VV+1ibOSTlXjCz3HibYvrvG7vOTqrrHLt+Ekh59j/UD0y6L0qNMcFuKboZX0TfLWLXNYtok51037mkNkX0k0MRMyFmNggYVMyzfVOUvU0RyxopbB4Wu25Rkq2O4zhOxVFlEhw5juM4jlP5VJkZgx2FpIeB/0kqftDMnqgMeyobuWqi4ziOUwoyLjBIZER0Ilw10XEcxykNvpTgOCl48MEHad++Pe3atWP48OEADBgwgCOOOIKOHTvSu3dv1q5dm99+7ty5dOnShXbt2tGhQwd+/PHHyjLdcRynXHhg4DhJfPLJJzz22GN88MEHzJkzh1dffZUvvviC7t2788knnzB37lwOP/xwBg8eDES5CS688EIeeeQR5s+fT05ODrvtlmrPqeM4TtWn2i4lSDqDSHipTchx0AJYDNxjZreHNg2Br4FHw5+/CY93AOaF68eTThHEx7gIuIkog98WYKyZ3R9EhH4JfA8I+IOZTQ3P5BDpLWwM3XxhZmcH8aXfEWk/1Anj325mC2LP9QceBvYgShBUm+3CSmeEExjJNl5KdATSiAK928xsYrKQUkxsqn0QPZoO/M7MRob6LKKjoANSCEQlxkr4/UOw7T3gVjNbFupziVI0J44evmlmv0/6vvYEnjWzQZJeAloSZUPcn+jvD+BqM3snlQ2w80WU7uu8gWOOOYa99toLgF/+8pdMmDCBm266Kb/NscceywsvvADAlClT6NixI506dQJgv/3222m2OY7j7Gyq84xBXM0wwWIgniv3N0RZFTGze8wsK6QZ3pi4LiYoOIkoj0KPoLR4LNGLLcGA0Nf1REmG4lwQ6z+e0fGBUHYYUdbDaZL2jz9oZseEfu8ExsX6yU1hYzMiMaVfmFnHYOPcVP6k4BMKHg/tA6SThWeAmXUCWhMFEtNCIqgE3WI2/z7puSyidMgXS2ppZr1D2eVE6ZkTzxUZFFQE7du356233mLNmjVs2LCBSZMmsXTp0gJtHn/88Xz1xM8//xxJ9OzZk6OOOoqhQ4dWhtmO4zg7hGo5Y1CMmuEG4FNJR4dfyucCzwNNyjDMLUS/uFdAlMSH1OqO7wJNS9u5mY2T1ItI/+HBktoXQSOiX+jrQ5/rE9dp8BVQX1JC2vpEYFK6A5uZAQ+ExFQnARPTfJyqjNAAACAASURBVHTP8GdeumNBxaorrly5ktNPP50uXbpQu3ZtWrRowddff52vfvb000+zdu1amjZtSk5ODgsXLuSNN97gkUceYY899uDGG2+kZs2adO5cKMVEIaqaqlp5ySR/MskXyCx/MskXqHr+VMvAgBRqhkT5+iESMTpP0kqiKe0VlC0wKEnxMUEqtcCxkhJLCa+bWVG6CR8BR5TBtgRzgJXAYklTgQlm9kopnn+BaFbl42DLpjLYkPAhERgUpZR4n6TbgUOBEWa2qjSDxNUVW7dubddecHoZTE2f7Oxs7rvvPgBuvfVWmjVrRnZ2NqNHj2b+/PlMnTo1f6nhm2++YcOGDfliSh9++CHbtm1LSy2tqqmqlZdM8ieTfIHM8ieTfIGq5091XUroQxQAwHY1wwSTge5EqoXj2HncJ+lz4BkK5wmILyUUJ6aUvmpSCsxsK1FgcjaRjPMDYS8DpKeS+DxRYNAHSC0KUDLJPsSXEuLyyYmlhAOBEyQdV8bxKoRVq6K4ZcmSJUyYMIHzzz+fyZMnM3ToUF5++eX8oACgZ8+ezJs3jw0bNrBlyxZmzJhB27ZtK8t0x3GcclHtZgyKUjMk2rSHmf0kaRZwI5EA0mllHGo+0JlI4CkVA8zsBUnXAo+HtqXlSGBmGe0D8qf0PwA+kPQ68AQwkMKKh4VUEs3sG0mbiQKp64CyvKyPBAorChVt7/qwMfIXQKXuJSiOs846izVr1rDbbrvx8MMPs88++9Dv/7d35uFSVVfefn8yCIiKNmorSFAxKAJelKB0on0FQQSCRolKY4Con4m2Rk00Jh+dBExiNGBoCXQbQwSiURRUHOLQiqAmNijgZVZAwSg4gaAIqBdY/cfeVRyKqjsPdcv1Pk8995w9nL3XOVBn1R7W76qr+Pzzz+nTpw8QFiDecccdHHTQQfzwhz/ka1/7GpLo378/AwbUnyy04zhOdWhwjgG51QyPTJS5DXjezD6qjJRxBr8hjAoMiC/QpsCw1Cr+BBOASySdFYMJVQhJ5wN9CQ5MlZB0BPDPZrYwJhUR1g5A0EG4WNKz0XkYTnaVxJ8Dh5rZzsrcqygydTVhB8ZTlajXGDiFIL2ct7z44ot7pa1evTpn+YsvvpiLL764NrvkOI5TJzRExyCXmuFPUydmtoy4G6GqROXCw4Bn40vQCCMDmeVM0q8I2xpTjkFyjcEGMzszHl8XVQj3I+wK6GVm1VFHbAKMjQ7CZ4StkCklxDsJc/+L4sjKfBL3KNH/yv5qHxOlk1sQtiueYUGeOUUupcTUGoOmhBGGhyrZruM4jlMHNDjHoAw1w6zbDs1sCjAlI61lBduaTBiaz0wfkXH+IFGaOZeyYVRlHFVGW8UZ51PI6HeWOm8RplWy5X0BXJUjbw5ZlBUzlSOz5I8oJ799Fetl7Y/jOI5T9zTUxYeO4ziO49QCDW7EoKaRNJLdERFTTDezX9dHf3IhaR4hImKS75jZkmzlq9mWK1Q6juN8SfnSOwbRAcgrJyAbZnZKHbb1pVaoHDduHJMmTUISXbp0YfLkyfTp04ctW7YAYStjjx49mDlzJps2beKSSy7hjTfeoFmzZtx111107ty5ni1wHMepOj6V4DgJ1q1bx/jx45k/fz5Lly5l586dTJs2jRdffJGSkhJKSkro2bMn5513HgA333wzRUVFLF68mD//+c9cc8019WyB4zhO9XDHoB6R9M+Spkl6Q9ICSU9I+qoki/ERUuUmSBohaaKkEknLJW2PxyWSBue4/hRJa2KZhZJ6JvIaS/pQ0i2JtF9LujVx/hVJb0pqJWmOpH8osadR0kxJn8bj9hl9KokiVEhaK+nBRL3BsW/fTZT9QtKSeJzuU32wY8cOtm/fzo4dO9i2bRtHHLE7cOYnn3zCc889x7nnngvA8uXL6dUrrP887rjjWLt2Le+//3699NtxHKcm+NJPJdQX8QX7MCFs8EUx7UQgpV1wjaQ/JLcCpob4E0qJRRVoKhWIqS9BZbJrTO9DiJb4bUk/jbEOfgWUSJpiZisIGg4/M7PN0R/YTFh78DdJrQgxDJK8UUafTpbUKaUmGe1J7/qIyoxnmNmGHPXT1Ja64tpbBtCmTRuuv/562rVrR/Pmzenbty99+/ZNl5k5cya9e/fmgAMOAODEE0/koYce4rTTTuPll1/mrbfe4p133uGwww6r8f45juPUBe4Y1B9nAKVmllZmNLNF8aX/IfB3QlCibMJNVeEFgk5BiiGEF/8VQE/gJTPbLuk6YKKkscD+ZvaXRJ1phFDTfwPOI8QiOKGC7d9GUIIcWpXO14WI0pw5c9iyZQtTp07lnnvuoWXLlowaNYqRI0emox1OnDiR/v37pwVPvv71rzNhwgQ6dOjA0UcfTYcOHXj11VfT6xHKI9/EU6pLIdlTSLZAYdlTSLZA/tnjjkH9UZ5I063Ak5L2CqpURb4JLAGQ1Aw4E/ge0IrgJLwE6cBOlwJTCWGLk8wC/iipEcFBuBz4WSL/GEklifOrzSwVQvAB4EpJSeekwiRFlNod3cFuW1Lz/3TXDi1m+vTpdOvWLT1VsH79eubOnUtxcTEbNmxg9erV3HjjjTRr1ixdLxX+2Mw46qijuOCCC9IjCuWRb+Ip1aWQ7CkkW6Cw7CkkWyD/7HHHIE8xszfjFsV/q+alUhEHPwQujWkDgdlxhOBB4GeSro2iTBB0J5qb2esZ19pJGC24KOavzQijXNZUwk5gDCH64pPVMah5k0a8fkvtaBG0a9eOuXPnsm3bNpo3b86sWbPo3r07ADNmzGDgwIF7OAWbN2+mRYsWNG3alEmTJnH66adX2ClwHMfJR3zxYf2REmkqi5uBG6meCuMNUemwj5ktjWlDgDPjvP4C4J/YM4LirvjJxjRClMkHqtCXu4HT2VPXIq845ZRTGDx4MCeddBJdunRh165dXH755QBMmzaNIUOG7FF+xYoVdO7cmY4dO/Lkk09y++2310e3HcdxagwfMag/ngNulnR5HCZHUlfgwFQBM3tN0nLCNMArNdGopAOA04AjzezzmPZdgrPwTAUu8SJBYKrSMs1mVippHPATcqtW1jujR49m9OjRe6VnmwPs2bMnK1eurINeOY7j1A0+YlBPxF0A3yL8cn9D0jLCC/e9jKK/BtrWYNPfAp5LOQWRR4BvSsqMrLgXFhibY/fAMRnbFX+QpcyfcIfUcRwnb/Ev6HrEzNYDF2TJ6pwos4gMB87M1ibLlHH9EVnSphIWFibTPgIOSZzPIUPUqAxxqJaJPjXPUaZ94vhz4IiyyjiO4zj1h48YOI7jOI6TxkcMCgAXPXIcx3FqCncMCoAvu+iR4ziOU3P4VILjJBg3bhwnnHACnTt3ZsiQIXz22WeMGDGCo446iqKiIoqKiigpKdmjziuvvELjxo2ZMWNGPfXacRyn5vARA8eJpJQVly9fTvPmzbnggguYNm0aAGPGjGHw4L21qnbu3MmNN964h56C4zhOQ8ZHDKqIpHOjCuJx8bx9PP9VokxrSaVRHXFkYhvfznK29KXqD5O0NKoOvirp+pieVE1cJKl3os4cSa8nrj8jpo+StC6mrZL0kKROGfW6S5oXy/wjqi+mrtM+Rx8vif1bHPt6TvJ6iXLtJS2Nx8XxXl2WyC+KaddX9lnUJGUpK2bj97//Peeffz6HHnpoHfXQcRyndvERg6ozhBAeeAjwi5i2BhgA/Ec8/zYhwiFm9mtCTAIkfVqeMqKks4Frgb5mtj7GGBiWKJJSTTyDoCFwbCJvqJnNz3LZcWY2Nl7/QuA5SV3M7MNUATM7JeaPALqb2VVl9LEtQRjpJDP7WFJLEtsey2EpYavmpHg+BFhUkYq1qa6YTVnx3nvvZeTIkdx000307t2bW265hX333Zd169bx8MMPM3v2bF55pUbiTzmO49Q77hhUgfgC/AZBIfExdjsG24AVkrrHF/OFhNDBZf/szM5PgetjrIPU/v9sSov/C7Sp7MXN7H5JAwhaDFWN43sosAX4NF7z09RxBXgLOEBSSma6H/BErsJ1oa742GOPZVVW/OY3v8nw4cMpLS3ltttu4/vf/z7Dhw9n1KhRXHjhhbzwwgu89957LFu2jNatW1eqzXxTVasuhWRPIdkChWVPIdkC+WePOwZV4xzgKTNbKWmjpJOBjTFvGnCRpPcJwkHrqZpjUJ76Yop+wMyMtL9I2h6PnzGzG3LUXQgcV4W+pVgEvA+skTQLeMjMHqtE/RmEUZVXY18+z1Uwqa7YsWNHu3roOVXudC5yKSuef/756TJNmzZl7NixFBcX89Zbb/Hb3/4WgA0bNrBw4UJOPPHEdP2KkG+qatWlkOwpJFugsOwpJFsg/+xxx6BqDGH3r+xp8XxCPH8K+CXhhXl/LfZhjKSbCeGSe2bk5ZpKyKQ64kyY2U5J/YCvAb2BcZJONrNRgGWrknH+AOEeHUfQXviX6vSnuuRSVnz33Xc5/PDDMTNmzpxJ584h6OSaNWvSdUeMGMHAgQMr5RQ4juPkI+4YVBJJBxOUCLtIMqAR4YU3EcDMvpC0APgR0AkYVMWmUuqLucSGUmsMrgbuonylxmx0AyriQOQkaj68DLws6RlgMjCKMIJyUKLowcCGjLrvSSoF+gDXUM+OQVJZsXHjxnTr1o3LL7+cs88+mw8//BAzo6ioiDvuuKM+u+k4jlOruGNQeQYDd5vZ91IJkp5nTynh24Dnzewjqco/yn9DGBUYEF+gTYFhZjYpo9wE4BJJZ5nZ0xW9uKTzgb4EB6ZKSDoC+GczWxiTighrByBoLVws6dnoPAwHZme5zM+BQ+PoQ1W7UmNkU1Z87rnyhSCnTJlSSz1yHMepW9wxqDxDgFsz0h4kLBYEwMyWEXcjVBUzeyIuzHtW4Y1phJGBzHKpLZI/BlKOQXKNwQYzOzMeXyfpYmA/wq6AXskdCVWgCTA2OgifAR8C3495dxKmCBbFkZX5JO5Rov8vVaN9x3Ecp4Zxx6CSmNkZWdLGA+NzlJ8CTMlIa1nBtiYThuYz00dknD9IcE7KUkEcRRjiz9VWccb5FDL6naXOW4RplWx5XwBZtzpmU29M9NFxHMepRzzAkeM4juM4aXzEoJ6RNJKwZS/J9BgQKW+QNA/YNyP5O2a2pD764ziO49QO7hjUM8mIiPlMKiJioTNu3DgmTZqEJLp06cLkyZO59NJLmT9/Pk2aNKFHjx784Q9/oEmTJkDYf3zttddSWlpK69atef755+vZAsdxnOrhUwmOE0mJKM2fP5+lS5eyc+dOpk2bxtChQ3nttddYsmQJ27dvZ9KksDFk8+bNXHnllTz66KMsW7aM6dOn17MFjuM41ccdgyqQEEFaFkWMfiRpn4wy/xlFi/aJ5ydIWimpeaLMXyUNkXSYpMfjtZZLKis0cKYY0cdRYOl1SS9IGlhO35NiSkslDYrpUyQNzij7aaLN7QnRppckdUz04fEs7aRFlMoQWsrZZn2RTUSpf//+SEISPXr04J133gHg3nvv5bzzzqNdu3YALqTkOE5B4FMJVWN7SgRJ0qHAvcABRM2E6Ax8C3gb+Fdgtpktk/QQQXToPySdCzQxs/sk/YEQuvj2WL9rJfryopkNjPWKgJmStpvZrDLqjDOzsZKOB16MNpTHGwmbvwf8f0JsgjKpptBSVmpDRGntLQNo06ZNVhGlFKWlpdx9993cfnsIerly5UpKS0spLi5my5YtXHPNNQwbNixXE47jOA0CHzGoJmb2AUHc5yrtjtBTTIhj8N+EuAcpbgK+HV/gtwD/HtMPB95JXHNxFftSEtvIqYiYUX4FsAOonPJPcII2VbDsXkJLZram7Cr1w6ZNm3jkkUdYs2YN69evZ+vWrdxzzz3p/CuvvJLTTz+d0047DQijCwsWLOCvf/0rTz/9NL/85S9ZuXJlfXXfcRynRvARgxrAzN6U1IjwEnyf4AzcBzwC3CypiZmVmtk2SdcDLwC/M7NV8RITgfslXQU8C0xOqSpWgYVALtGkPZB0CrCLEJioPI6RVALsD7QAKroYsTyhpTGS/iN71T36WqvqinPmzGHOnDk0a9aMZctCbKrjjz+e6dOn07ZtW6ZOncqqVau46aab0ipoX3zxBR07dkxLLh977LHce++9lRJDyTdVtepSSPYUki1QWPYUki2Qh/aYmX8q+QE+zZK2GTgMaAqsA/aP6Q8BAzPK/oMQBjiZdjBBAvluwov0kBxttweWxuNi4PGM/G7AijL6Pir2rwR4ETgtpk8GBmeU3ZLZZjy/kKAumbUPMX0O0D0eC+hBiHy4GhgV06dkaXOve5v5+epXv2q1wdy5c61Tp062detW27Vrlw0bNszGjx9vf/zjH61nz562bdu2PcovX77cevXqZaWlpbZ161Y74YQTbMmSJZVqc/bs2TVoQf1TSPYUki1mhWVPIdliVnf2APOtAu84HzGoASQdTZBY/gAYCLQClsSZhRbAdiC5QG9X/KQxs48IaxXujYv5TidGM6wk3YAV5ZQZZ2ZjM9L2ED2KYlEbyM6jZInImIv4DzKb0FJekUtEab/99uMrX/kKPXsGEcvzzjuPn//85xx//PH069ePrl27ss8++3DZZZellRcdx3EaKu4YVBNJhwB3ABPMzCQNAS4zs/ti/n6EYfQWZrYtxzV6AXMtTDXsDxxDGFWobF+6Aj8DLquCKXOAayVNtRDOeATZRY8AvgG8UcE+lSW0lHdkE1HasSP3tMUNN9zADTdUaObGcRynQeCOQdVoHufbmxAW790N/E5SC6Afu4WEMLOtkv4GfBO4P8f1TgYmSNpBWBA6ycxeqWBfTpP0KmFk4gPgB1b2joSsmNnjkk4GFkjaSXjxfz9RJLXGQMAX7Ol89Jb0TuI8GcmxLKElx3EcJ89wx6AKmFmjHFnbCGsFMsufl3HePuN8DDCmgm2vBTrH4znAgRWpl6g/qoy80cDoLOlrgeZ7Vdjdh2x5xYnjXEJLI7KkVUhgynEcx6kdfLui4ziO4zhpfMQgT5HUhTBFkeRzq6BmQUMRZ3Icx3HyC3cM8hQLqoVF1ajfIMSZHMdxnPzCpxIcJ8G4ceM44YQT6Ny5M0OGDOGzzz5j6NChdOzYkc6dO3PJJZdQWloKhBggP/jBD+jQoQNdu3Zl4cKF5VzdcRwn/3HHwHEilVVXfPLJJ1m1ahWrVq3izjvv5IorrqhnCxzHcapPrToGks6VZJKOi+ft4/mvEmVaSyqVNEHSyKjgV5JQMCyR9IMy2hgWFfuWRJXB62P6FElrEoqAvRN15kQ1wtT1Z8T0pPLgKkkPSeqUUa+7pHmxzD8kfZi4TvscfVwr6cWMtBJFlcRE2h6KjDHth5LuSpwPlVSmgpCkyyW9Fj8vS/pGTP+FpN9klC2StCLRzyUJe8ZX4l4uju1NkNQq5qWVIBPlR+V4RiWSXirLrrqgMuqKjzzyCMOGDUMSp556Kps3b+bdd9+tZwscx3GqR22vMRgC/C3+/UVMWwMMAFLx8b9NEBzaY15c0qcW1fxyIels4Fqgr5mtl7QvkJS3u8HMZkg6A7gTODaRN9TM5me5bDoqoKQLgeckdTGztJ5AagGgpBGEsL8VES3aX9KRZva2gqphpi17KTLGrPHAfElfJ9ynXwG9M+snrjMQ+B7wDTPbIOkkguJiD4J+w1OE0MQpLorpKc4ws2wRD8u9l5KaAr8haET8a847keW6FSwL5I+64rp16zjyyCPT+W3btmXdunUcfvjhNdo3x3GcuqTWHAMFed1vAGcAj7HbMdgGrJDUPb6YLwQeAI6oQjM/Ba63KDhkZp8Df8xS7n+BNpW9uJndL2kAQcPg9ir0L8kDBFvHsltk6TuJ/GLCi//+mD879mGHpCuB/yKEFb7LzN4so50bCS/bDbH+QklTgX83s59J2iTpFDObF8tfAJxVCTty3ksz+0LSj4HVkk4EPq7EdctEdSCitGXLFqZOnco999xDy5YtGTVqFCNHjqRPnz4AjB07lqOPPpqdO3cyZ84cNm7cyKuvvpqOjLhp0yYWLFjAp59+WuF28048pZoUkj2FZAsUlj2FZAvknz21OWJwDkFoZ6WkjTGq3saYNw24SNL7BI2B9VTNMegMLKhAuX7AzIy0v0jaHo+fMbNccW0XAsdVoW+ZPEjQCBhLiII4lD0dg6yKjABm9lIc7j8T2Gu0IYMT2PuezAeGx+P7CKME8ySdCnxku1UeAWYrRD4EmGpm4zKule1epjGznZIWEe7ZvFzlEiTVFZeZ2dAc172TMFJBu6M72G1Lavaf7tqhxUyfPp1u3bpx7rnnArB+/Xrmzp1LcXExo0ePpnHjxjzwwAPss0+Y6enatSutW7dOqylu3bqVQYMGVWrEYM6cOZVSY8x3CsmeQrIFCsueQrIF8s+e2nQMhrD7V/a0eD4hnj8F/JKgIpgrTHBNMEbSzUBboGdGXq6phExUQ33ZCGySdBFB5CitmxCH4PsDPzSzLZLmEX7FPx7zWwLdCeGFDwHeoercD7wk6UfsPY0AuacSyrqXmaTumeXIT6ZXeiqheZNGvH7LgMpUqRDt2rVj7ty5bNu2jebNmzNr1iy6d+/OpEmTePrpp5k1a1baKQAYNGgQEyZM4KKLLmLevHkceOCBPo3gOE6Dp1YcAwVlvl5AF0kGNCK8DCZCesh5AfAjoBMwqIpNLSPoDDyXIz81L341cFcsW1m6EX5x1wT3E+7BiIz0syhbkXE0cA/BkRrH3oGLkixn73tyMrvXcbwtaQ1hDcD5lP+ST1GheympEdCF4PzsodgYOZiwziTvqKy6Yv/+/XniiSfo0KEDLVq0YPLkCgtOOo7j5C21NWIwGLjbzL6XSpD0PHBkosxtwPNm9lF8GVaF3xB+yQ4ws/fiL+9hZjYpo9wE4BJJZ5nZ0xW9uKTzgb4EB6YmeBg4HHiaPadOcioyEpQWBxCCHX0BXCqpj5k9k6ON3wK3SupnZhslFREckWTExPsIDsabZlbZ0Yec91JSE8Li0bfNbHFMe1dSLzN7LjqM/aj+eo1aozLqipKYOHFiXXTLcRynzqgtx2AIcGtG2oMkVsOb2TLir9iqYmZPSDoMeFbBuzDCr9nMcqktkj8mvJRhzzUGG8zszHh8naSLgf2ApUCv5I6EavZ3C/G+pJwhla/IeDVwnZl9FstfAfxZUlGUR85s41FJbQjTBQZsAS42s+Q+uumE3Q5XZ+lmco3BYjNL7vIo615+DuwLPEtYX5JiGDBR0u/i+WgzS0o2J9cYAPTIZpfjOI5TN8gs1zSw4+QnHTt2tNdff72+u1Ej5Nuio+pSSPYUki1QWPYUki1Qd/ZIWmBm3csr55EPHcdxHMdJ0yBElNRAlALjboJ9M5K/EwWRarqtBnFPHMdxnIZFg3AMGopSYEUlkWuorQZxTxzHcZyGhU8lOI7jOI6Txh0Dx3Ecx3HSuGPgOI7jOE4adwwcx3Ecx0njcQycBoekLUBhBDKA1kA2bYqGSiHZU0i2QGHZU0i2QN3Z8xUzO6S8Qg1iV4LjZPB6RYJ0NAQkzS8UW6Cw7CkkW6Cw7CkkWyD/7PGpBMdxHMdx0rhj4DiO4zhOGncMnIbInfXdgRqkkGyBwrKnkGyBwrKnkGyBPLPHFx86juM4jpPGRwwcx3Ecx0njjoHjOI7jOGncMXAaDJL6SXpd0mpJP6nv/uRC0pGSZktaLmmZpGti+sGSnpG0Kv49KKZL0vho12JJJyWuNTyWXyVpeD3a1EjSq5Iej+dHSZoX+3y/pKYxfd94vjrmt09c46cx/XVJZ9WPJSCplaQZkl6TtEJSz4b6bCRdF/+NLZV0n6RmDenZSLpL0geSlibSauxZSDpZ0pJYZ7wk1bEtY+K/s8WSHpbUKpGX9Z7n+p7L9VxrBTPzj3/y/gM0At4AjgaaAouATvXdrxx9PRw4KR7vD6wEOgG/BX4S038C3BqP+wNPAgJOBebF9IOBN+Pfg+LxQfVk0w+Be4HH4/kDwEXx+A7ginh8JXBHPL4IuD8ed4rPbF/gqPgsG9WTLVOBy+JxU6BVQ3w2QBtgDdA88UxGNKRnA5wOnAQsTaTV2LMAXo5lFeueXce29AUax+NbE7ZkveeU8T2X67nWxsdHDJyGQg9gtZm9aWZfANOAc+q5T1kxs3fNbGE83gKsIHyJn0N4KRH/nhuPzwH+bIG5QCtJhwNnAc+Y2Udmtgl4BuhXh6YAIKktMACYFM8F9AJmxCKZtqRsnAH0juXPAaaZ2edmtgZYTXimdYqkAwlf4H8CMLMvzGwzDfTZEILUNZfUGGgBvEsDejZm9gLwUUZyjTyLmHeAmc218Db9c+JadWKLmf2Pme2Ip3OBtglbst3zrN9z5fyfq3HcMXAaCm2AtxPn78S0vCYO13YD5gGHmdm7Mes94LB4nMu2fLH5P4EfA7vi+T8BmxNfeMl+pfsc8z+O5fPFlqOAD4HJcWpkkqT9aIDPxszWAWOBfxAcgo+BBTTcZ5Oipp5Fm3icmV5fXEIYtYDK21LW/7kaxx0Dx6klJLUEHgSuNbNPknnxF0ze7xWWNBD4wMwW1HdfaojGhOHe/zazbsBWwnB1mgb0bA4i/PI8CjgC2I/6GbWoNRrKsygPSSOBHcBf6rsvFcEdA6ehsA44MnHeNqblJZKaEJyCv5jZQzH5/Ti8Sfz7QUzPZVs+2Px1YJCktYRhzV7A7YRh3JTWSrJf6T7H/AOBjeSHLRB+ab1jZvPi+QyCo9AQn82ZwBoz+9DMSoGHCM+roT6bFDX1LNaxe+g+mV6nSBoBDASGRkcHKm/LRnI/1xrHHQOnofAKcGxcmduUsHjq0XruU1bifOCfgBVm9rtE1qNAasX0cOCRRPqwuOr6VODjOJT6NNBX0kHx12HfmFZnmNlPzaytmbUn3PPnzGwoMBsYnMOWlI2DY3mL6RfFlfFHAccSFobVKWb2HvC2ORIhJgAABNFJREFUpI4xqTewnAb4bAhTCKdKahH/zaVsaZDPJkGNPIuY94mkU+P9GZa4Vp0gqR9hGm6QmW1LZOW651m/5+JzyvVca57aWtXoH//U9IewKnklYdXuyPruTxn9/AZh+HMxUBI//QnzhLOAVcCzwMGxvICJ0a4lQPfEtS4hLExaDXy3nu0qZveuhKMJX2SrgenAvjG9WTxfHfOPTtQfGW18nVpcHV4BO4qA+fH5zCSsZG+QzwYYDbwGLAXuJqxybzDPBriPsD6ilDCac2lNPguge7w3bwATiNF+69CW1YQ1A6nvgTvKu+fk+J7L9Vxr4+MhkR3HcRzHSeNTCY7jOI7jpHHHwHEcx3GcNO4YOI7jOI6Txh0Dx3Ecx3HSuGPgOI7jOE4adwwcx8kbJO2UVJL4tK/CNVpJurLme5e+/iDVsbqnpHMldarLNp0vL75d0XGcvEHSp2bWsprXaE+It9C5kvUamdnO6rRdG8Rod5MINs0or7zjVBcfMXAcJ6+R1Cjq2r8Sde2/F9NbSpolaaGkJZJSapu3AMfEEYcxkoolPZ643oQYphZJayXdKmkh8G1Jx0h6StICSS9KOi5Lf0ZImhCPp0j6b0lzJb0Z27pL0gpJUxJ1PpU0TtKy2OdDYnpRrLtY0sMxch+S5kj6T0nzgRuBQcCYaNMxkv5fvB+LJD0oqUWiP+MlvRT7MzjRhxvjfVok6ZaYVq69zpePxuUXcRzHqTOaSyqJx2vM7FuECHIfm9nXJO0L/F3S/xAiyn3LzD6R1BqYK+lRgihSZzMrApBUXE6bG83spFh2FvB9M1sl6RTgvwj6EGVxENCT8PJ+lKBXcBnwiqQiMyshCBzNN7PrJP0c+AVwFUEK+Goze17STTH92njdpmbWPfbrWBIjBpI2m9kf4/Gv4j36fax3OCH65nGxPzMknU0QXDrFzLZJOjiWvbMK9joFjjsGjuPkE9tTL/QEfYGuiV+/BxJiy78D3CzpdIIkdBt2S/RWhvshrYb5L8D0EFofCCGGy+MxMzNJS4D3zWxJvN4yoD0hFO6uVDvAPcBDkg4EWpnZ8zF9KiHU7R79ykHn6BC0Alqyp07DTDPbBSyXlLofZwKTLcbrN7OPqmGvU+C4Y+A4Tr4jwq/qPUSK4nTAIcDJZlaqoADZLEv9Hew5bZpZZmv8uw9B8z7TMSmPz+PfXYnj1Hmu79iKLO7aWkbeFOBcM1sU70Nxlv5AuHe5qKq9ToHjawwcx8l3ngauUJCyRtJXJe1HGDn4IDoFZwBfieW3APsn6r8FdIpKdq0IKoR7YWafAGskfTu2I0kn1pAN+7BbGe/fgL+Z2cfAJkmnxfTvAM9nq8zeNu0PvBvvydAKtP8M8N3EWoSDa9lepwHjjoHjOPnOJIKc8EJJS4E/EH6J/wXoHofwhxFUBjGzjYR1CEsljTGzt4EHCCp7DwCvltHWUOBSSYuAZYR5+ZpgK9Aj9r8XcFNMH05YVLiYoPp4U47604AbJL0q6RjgZ8A84O9Eu8vCzJ4irDeYH9dwXB+zastepwHj2xUdx3FqGdXANkzHqSt8xMBxHMdxnDQ+YuA4juM4ThofMXAcx3EcJ407Bo7jOI7jpHHHwHEcx3GcNO4YOI7jOI6Txh0Dx3Ecx3HS/B87jAr7QKcC8AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f7b845a8358>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lightgbm.plot_importance(model, max_num_features=15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Submission"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'kaggle competitions submit -c home-credit-default-risk -f /home/inc/work/edu/ndml/capstone/submissions/submission-lightgbm.csv -m \"lightgbm\"'"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "submission = pd.DataFrame({'SK_ID_CURR': test_ids, 'TARGET': test_predictions})\n",
    "save_prediction(submission, 'lightgbm')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}