"cells": [
{
"metadata": {
"init_cell": true,
"trusted": true
},
"cell_type": "code",
"source": "%%HTML\n<style type="text/css">\ntable.dataframe td, table.dataframe th {\n border: 1px black solid !important;\n color: black !important;\n}\n</style>\n",
"execution_count": 37,
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": "<IPython.core.display.HTML object>",
"text/html": "<style type="text/css">\ntable.dataframe td, table.dataframe th {\n border: 1px black solid !important;\n color: black !important;\n}\n</style>\n"
},
"metadata": {}
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "# Predicting heart disease using machine learning\n\nThis notebook will introduce some foundation machine learning and data science concepts by exploring the problem of heart disease classification.\n\nIt is intended to be an end-to-end example of what a data science and machine learning proof of concept might look like.\n\n## What is classification?\n\nClassification involves deciding whether a sample is part of one class or another (single-class classification). If there are multiple class options, it's referred to as multi-class classification.\n\n## What we'll end up with\n\nSince we already have a dataset, we'll approach the problem with the following machine learning modelling framework.\n\n\n\n\n\n6 Step Machine Learning Modelling Framework\n\nMore specifically, we'll look at the following topics.\n\n* Exploratory data analysis (EDA) - the process of going through a dataset and finding out more about it.\n* Model training- create model(s) to learn to predict a target variable based on other variables.\n* Model evaluation - evaluating a models predictions using problem-specific evaluation metrics.\n* Model comparison - comparing several different models to find the best one.\n* Model fine-tuning - once we've found a good model, how can we improve it?\n* Feature importance - since we're predicting the presence of heart disease, are there some things which are more important for prediction?\n* Cross-validation - if we do build a good model, can we be sure it will work on unseen data?\n* Reporting what we've found - if we had to present our work, what would we show someone?\n\nTo work through these topics, we'll use pandas, Matplotlib and NumPy for data anaylsis, as well as, Scikit-Learn for machine learning and modelling tasks.\n\n\n
\nTools which can be used for each step of the machine learning modelling process.\n\nWe'll work through each step and by the end of the notebook, we'll have a handful of models, all which can predict whether or not a person has heart disease based on a number of different parameters at a considerable accuracy.\n\nYou'll also be able to describe which parameters are more indicative than others, for example, sex may be more important than age.\n"
},
{
"metadata": {},
"cell_type": "markdown",
"source": "## 1. Problem Definition\n\nIn our case, the problem we will be exploring is binary classification (a sample can only be one of two things).\n\nThis is because we're going to be using a number of differnet features (pieces of information) about a person to predict whether they have heart disease or not.\n\nIn a statement,\n> Given clinical parameters about a patient, can we predict whether or not they have heart disease?\n\n## 2. Data\n\nWhat you'll want to do here is dive into the data your problem definition is based on. This may involve, sourcing, defining different parameters, talking to experts about it and finding out what you should expect.\n\nThe original data came from the cleveland database from the UCI Machine Learning Repository. \n\nTher is also a version of it avalable on Kaggle.\n\nThe original database contains 76 attributes, but here only 14 attributes will be used. Attributes (also called features) are the variables what we'll use to predict our target variable.\n\nAttributes and features are also referred to as independent variables and a target variable can be referred to as a dependent variable.\n\n> We use the independent variables to predict our dependent variable.\n\nOr in our case, the independent variables are a patients different medical attributes and the dependent variable is whether or not they have heart disease.\n\n## 3. Evaluation\n\nThe evaluation metric is something you might define at the start of a project.\n\nSince machine learning is very experimental, you might say something like,\n\n>If we can reach 95% accuracy at predicting whether or not a patient has heart disease during the proof of concept, we'll pursure this project.\n\nThe reason this is helpful is it provides a rough goal for a machine learning engineer or data scientist to work towards.\n\nHowever, due to the nature of experimentation, the evaluation metric may change over time.\n\n## 4. Features\n\nFeatures are different parts of the data. During this step, you'll want to start finding out what you can about the data.\n\nOne of the most common ways to do this, is to create a data dictionary.\n\n### Heart Disease Data Dictionary\n\nA data dictionary describes the data you're dealing with. Not all datasets come with them so this is where you may have to do your research or ask a subject matter expert (someone who knows about the data) for more.\n\nThe following are the features we'll use to predict our target variable (heart disease or no heart disease).\n\n1. age - age in years\n2. sex - 1 = male; 0 = female\n3. cp - chest pain type\n * 0: Typical angina: chest pain related decrease blood supply to the heart\n * 1: Atypical angina: chest pain not related to heart\n * 2: Non-anginal pain: typically esophgael spasms (non heart related)\n * 3: Asymptomatic: chest pain not showing signs of disease\n4. trestbps - resting blood pressure (in mm Hg on admission to the hospital)\n5. chol - serum cholestoral in mg/dl\n * serum = LDL + HDL + .2 * triglycerides\n * above 200 is cause for concern\n6. fbs(fasting blood sugar > 120 mg/dl) (1 = true; 0 = false)\n * '>126' mg/dL signals diabetes\n7. restecg - resting electrocardiographic results\n * 0: Nothing to note\n * 1: ST-T Wave abnormality\n * can range from mild symptoms to severe problems\n * signals non-normal heart beat\n * 2: Possible or definite left venticular hypertrophy\n * Enlarged heart's main pumping chamber\n8. thalach - maximum heart rate achieved\n9. exang - exercise induced angina (1 = yes; 0 = no)\n10. oldpeak - ST depression induced by exercise relative to rest\n * looks at stress of heart during excercise\n * unhealthy heart will stress more\n11. slope - the slope of the peak exercise ST segment\n * 0: Unsloping: better heart rate with excercise (uncommon)\n * 1: Flatsloping: minimal change (typical healthy heart)\n * 2: Downsloping: Signs of unhealthy heart\n12. ca - number of major vessels (0-3) colored by flourosopy\n * colored vessel means the doctor can see the blood passing through\n * the more blood movement the better (no clots)\n13. thal - thalium stress result\n * 1,3 = normal \n * 6 = fixed defect: Used to be defect but ok now \n * 7 = reversable defect: no proper blood movement when excercising\n14. target - have disease or not (1=yes, 0=no)(= the predicted attribute\n\nNote: No personal identifiable information (PPI) can be found in the dataset.\n\nIt's a good idea to save these to a Python dictionary or in an external file, so we can look at them later without coming back here."
},
{
"metadata": {},
"cell_type": "markdown",
"source": "## Preparing the tools\n\nAt the start of any project, it's custom to see the required libraries imported in a big chunk like you can see below.\n\nHowever, in practice, your projects may import libraries as you go. After you've spent a couple of hours working on your problem, you'll probably want to do some tidying up. This is where you may want to consolidate every library you've used at the top of your notebook (like the cell below).\n\nThe libraries you use will differ from project to project. But there are a few which will you'll likely take advantage of during almost every structured data project.\n\n* pandas for data analysis.\n* NumPy for numerical operations.\n* Matplotlib/seaborn for plotting or data visualization.\n* Scikit-Learn for machine learning modelling and evaluation."
},
{
"metadata": {
"init_cell": true,
"trusted": true
},
"cell_type": "code",
"source": "# Import all the tools we need\n\n# Regular EDA (exploratory data analysis) and plotting libraries\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n# we want our plots to appear inside the notebook\n%matplotlib inline\n\n# Models from Scikit-Learn\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.neighbors import KNeighborsClassifier\nfrom sklearn.ensemble import RandomForestClassifier\n\n# Model Evaluations\nfrom sklearn.model_selection import train_test_split, cross_val_score\nfrom sklearn.model_selection import RandomizedSearchCV, GridSearchCV\nfrom sklearn.metrics import confusion_matrix, classification_report\nfrom sklearn.metrics import precision_score, recall_score, f1_score\nfrom sklearn.metrics import plot_roc_curve",
"execution_count": 38,
"outputs": []
},
{
"metadata": {},
"cell_type": "markdown",
"source": "## Load Data\n\nThere are many different kinds of ways to store data. The typical way of storing tabular data, data similar to what you'd see in an Excel file is in .csv format. .csv stands for comma seperated values.\n\nPandas has a built-in function to read .csv files called read_csv() which takes the file pathname of your .csv file. You'll likely use this a lot."
},
{
"metadata": {
"init_cell": true,
"trusted": true
},
"cell_type": "code",
"source": "df = pd.read_csv("heart-disease.csv")\ndf.shape",
"execution_count": 39,
"outputs": [
{
"output_type": "execute_result",
"execution_count": 39,
"data": {
"text/plain": "(303, 14)"
},
"metadata": {}
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "## Data Exploration (Exploratory data analysis or EDA)\n\nOnce you've imported a dataset, the next step is to explore. There's no set way of doing this. But what you should be trying to do is become more and more familiar with the dataset.\n\nCompare different columns to each other, compare them to the target variable. Refer back to your data dictionary and remind yourself of what different columns mean.\n\nYour goal is to become a subject matter expert on the dataset you're working with. So if someone asks you a question about it, you can give them an explanation and when you start building models, you can sound check them to make sure they're not performing too well(overfitting) or they might be performing poorly (underfitting).\n\nSince EDA has no real set methodolgy, the following is a short check list you might want to walk through.\n\n1. Ehat question(s) are you trying to solve (or prove wrong)?\n2. What kind of data do you have and how do you treat different types?\n3. What's missing from the data and how do you deal with it?\n4. Where are the outliers and why should you care about them?\n5. How can you add, change or remove features to get more out of your data?\n\nOne of the quickest and easiest ways to check your data is with the
head()
function. Calling it on any dataframe will print the top 5 rows, tail()
calls the bottom 5. You can also pass a number to them like head(10)
to show the top 10 rows."
},
{
"metadata": {
"init_cell": true,
"trusted": true
},
"cell_type": "code",
"source": "df.head()",
"execution_count": 40,
"outputs": [
{
"output_type": "execute_result",
"execution_count": 40,
"data": {
"text/plain": " age sex cp trestbps chol fbs restecg thalach exang oldpeak slope \\n0 63 1 3 145 233 1 0 150 0 2.3 0 \n1 37 1 2 130 250 0 1 187 0 3.5 0 \n2 41 0 1 130 204 0 0 172 0 1.4 2 \n3 56 1 1 120 236 0 1 178 0 0.8 2 \n4 57 0 0 120 354 0 1 163 1 0.6 2 \n\n ca thal target \n0 0 1 1 \n1 0 2 1 \n2 0 2 1 \n3 0 2 1 \n4 0 2 1 ",
"text/html": "
value_counts()
allows you to show how many times each of the values of a categorical column appear."
},
{
"metadata": {
"init_cell": true,
"trusted": true
},
"cell_type": "code",
"source": "# Let's see how many positive (1) and negative (0) samples we have in our dataframe\ndf.target.value_counts()",
"execution_count": 42,
"outputs": [
{
"output_type": "execute_result",
"execution_count": 42,
"data": {
"text/plain": "1 165\n0 138\nName: target, dtype: int64"
},
"metadata": {}
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "Since these two values are close to even, our target column can be considered balanced. An unbalanced target column, meaning some classes have far more samples, can be harder to model than a balanced set, all of your target classes have the same number of samples.\n\nIf you'd prefer these values in percentages, value_counts()
takes a parameter, normalize which can be set to true."
},
{
"metadata": {
"init_cell": true,
"trusted": true
},
"cell_type": "code",
"source": "# Normalized value counts\ndf.target.value_counts(normalize=True)",
"execution_count": 43,
"outputs": [
{
"output_type": "execute_result",
"execution_count": 43,
"data": {
"text/plain": "1 0.544554\n0 0.455446\nName: target, dtype: float64"
},
"metadata": {}
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "We can plot the target column value counts by calling the plot()
function and telling it what kind of plot we'd like, in this case, bar is good"
},
{
"metadata": {
"init_cell": true,
"trusted": true
},
"cell_type": "code",
"source": "# Plot the value_counts with a bar graph\ndf.target.value_counts().plot(kind="bar", color=["salmon", "lightblue"]);",
"execution_count": 44,
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": "<Figure size 432x288 with 1 Axes>",
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD1CAYAAACrz7WZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAOMElEQVR4nO3dbYxmZ13H8e/PLq0CMS3stJZ9cBfdgoVgaIZSJRqkCq0StjGQbIO6wSYTtSAIhrbyovqiCfgASFSSla5dkqalqdVuCKJ1bW2MtmXKQ+l2Kd200A67stMU8IGksPD3xZzqeHPPzsx97nuGvfb7eXPf539d55z/i9nfnlxzzpxUFZKktvzAejcgSRo/w12SGmS4S1KDDHdJapDhLkkNMtwlqUEb1rsBgI0bN9a2bdvWuw1JOqncf//9T1bV1LCx74tw37ZtG7Ozs+vdhiSdVJJ8eakxl2UkqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDfq+eIjpZPHtP3jXerfQlGdd+yfr3YLULK/cJalBhrskNWjZcE+yN8mxJA8O1N+W5OEkB5P84aL6NUkOd2Ovm0TTkqQTW8ma+w3AnwEffaaQ5OeAncDLqurpJGd39fOBXcBLgBcA/5jkvKr6zrgblyQtbdkr96q6G3hqoPybwHur6uluzrGuvhO4uaqerqrHgMPAhWPsV5K0AqOuuZ8H/EySe5P8c5JXdPVNwBOL5s11NUnSGhr1VsgNwFnARcArgFuSvBDIkLk17ABJZoAZgK1bt47YhiRpmFGv3OeA22rBfcB3gY1dfcuieZuBI8MOUFV7qmq6qqanpoa+SESSNKJRw/1vgdcAJDkPOB14EtgP7EpyRpLtwA7gvnE0KklauWWXZZLcBLwa2JhkDrgW2Avs7W6P/Bawu6oKOJjkFuAh4DhwpXfKSNLaWzbcq+ryJYZ+ZYn51wHX9WlKktSPT6hKUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhq0bLgn2ZvkWPfWpcGx301SSTZ220nyoSSHkzyQ5IJJNC1JOrGVXLnfAFwyWEyyBfgF4PFF5UtZeG/qDmAG+HD/FiVJq7VsuFfV3cBTQ4Y+ALwbqEW1ncBHa8E9wJlJzh1Lp5KkFRtpzT3JG4CvVNXnBoY2AU8s2p7rapKkNbTsC7IHJXk28B7gtcOGh9RqSI0kMyws3bB169bVtiFJOoFRrtx/DNgOfC7Jl4DNwKeT/AgLV+pbFs3dDBwZdpCq2lNV01U1PTU1NUIbkqSlrPrKvao+D5z9zHYX8NNV9WSS/cBbk9wMvBL4RlUdHVezkoa77WH/mY3TL7/o5P9V4UpuhbwJ+DfgRUnmklxxgumfAB4FDgN/CfzWWLqUJK3KslfuVXX5MuPbFn0v4Mr+bUmS+vAJVUlqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSg1byJqa9SY4leXBR7Y+SfCHJA0n+JsmZi8auSXI4ycNJXjepxiVJS1vJlfsNwCUDtTuAl1bVy4AvAtcAJDkf2AW8pNvnL5KcNrZuJUkrsmy4V9XdwFMDtX+oquPd5j3A5u77TuDmqnq6qh5j4V2qF46xX0nSCoxjzf3Xgb/rvm8Cnlg0NtfVJElrqFe4J3kPcBy48ZnSkGm1xL4zSWaTzM7Pz/dpQ5I0YORwT7IbeD3w5qp6JsDngC2Lpm0Gjgzbv6r2VNV0VU1PTU2N2oYkaYiRwj3JJcBVwBuq6puLhvYDu5KckWQ7sAO4r3+bkqTV2LDchCQ3Aa8GNiaZA65l4e6YM4A7kgDcU1W/UVUHk9wCPMTCcs2VVfWdSTUvSRpu2XCvqsuHlK8/wfzrgOv6NCVJ6scnVCWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDVo23JPsTXIsyYOLas9LckeSR7rPs7p6knwoyeEkDyS5YJLNS5KGW8mV+w3AJQO1q4EDVbUDONBtA1zKwntTdwAzwIfH06YkaTWWDfequht4aqC8E9jXfd8HXLao/tFacA9wZpJzx9WsJGllRl1zP6eqjgJ0n2d39U3AE4vmzXU1SdIaGvcvVDOkVkMnJjNJZpPMzs/Pj7kNSTq1jRruX31muaX7PNbV54Ati+ZtBo4MO0BV7amq6aqanpqaGrENSdIwo4b7fmB39303cPui+q91d81cBHzjmeUbSdLa2bDchCQ3Aa8GNiaZA64F3gvckuQK4HHgTd30TwC/CBwGvgm8ZQI9S5KWsWy4V9XlSwxdPGRuAVf2bUqS1I9PqEpSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGtQr3JP8TpKDSR5MclOSH0yyPcm9SR5J8rEkp4+rWUnSyowc7kk2Ab8NTFfVS4HTgF3A+4APVNUO4GvAFeNoVJK0cn2XZTYAP5RkA/Bs4CjwGuDWbnwfcFnPc0iSVmnkcK+qrwB/zMILso8C3wDuB75eVce7aXPApr5NSpJWp8+yzFnATmA78ALgOcClQ6bWEvvPJJlNMjs/Pz9qG5KkIfosy/w88FhVzVfVt4HbgJ8GzuyWaQA2A0eG7VxVe6pquqqmp6amerQhSRrUJ9wfBy5K8uwkAS4GHgLuBN7YzdkN3N6vRUnSavVZc7+XhV+cfhr4fHesPcBVwDuTHAaeD1w/hj4lSauwYfkpS6uqa4FrB8qPAhf2Oa4kqR+fUJWkBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNahXuCc5M8mtSb6Q5FCSn0ryvCR3JHmk+zxrXM1Kklam75X7nwKfrKoXAz8JHAKuBg5U1Q7gQLctSVpDI4d7kh8GfpbuHalV9a2q+jqwE9jXTdsHXNa3SUnS6vS5cn8hMA/8VZLPJPlIkucA51TVUYDu8+wx9ClJWoU+4b4BuAD4cFW9HPhvVrEEk2QmyWyS2fn5+R5tSJIG9Qn3OWCuqu7ttm9lIey/muRcgO7z2LCdq2pPVU1X1fTU1FSPNiRJg0YO96r6d+CJJC/qShcDDwH7gd1dbTdwe68OJUmrtqHn/m8DbkxyOvAo8BYW/sO4JckVwOPAm3qeQ5K0Sr3Cvao+C0wPGbq4z3ElSf34hKokNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUG9wz3JaUk+k+Tj3fb2JPcmeSTJx7q3NEmS1tA4rtzfDhxatP0+4ANVtQP4GnDFGM4hSVqFXuGeZDPwS8BHuu0ArwFu7absAy7rcw5J0ur1vXL/IPBu4Lvd9vOBr1fV8W57DtjU8xySpFUaOdyTvB44VlX3Ly4PmVpL7D+TZDbJ7Pz8/KhtSJKG6HPl/irgDUm+BNzMwnLMB4Ezk2zo5mwGjgzbuar2VNV0VU1PTU31aEOSNGjkcK+qa6pqc1VtA3YB/1RVbwbuBN7YTdsN3N67S0nSqkziPvergHcmOczCGvz1EziHJOkENiw/ZXlVdRdwV/f9UeDCcRxXkjQan1CVpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBo0c7km2JLkzyaEkB5O8vas/L8kdSR7pPs8aX7uSpJXoc+V+HHhXVf0EcBFwZZLzgauBA1W1AzjQbUuS1tDI4V5VR6vq0933/wQOAZuAncC+bto+4LK+TUqSVmcsa+5JtgEvB+4Fzqmqo7DwHwBw9hL7zCSZTTI7Pz8/jjYkSZ3e4Z7kucBfA++oqv9Y6X5Vtaeqpqtqempqqm8bkqRFeoV7kmexEOw3VtVtXfmrSc7txs8FjvVrUZK0Wn3ulglwPXCoqt6/aGg/sLv7vhu4ffT2JEmj2NBj31cBvwp8Pslnu9rvAe8FbklyBfA48KZ+LUqSVmvkcK+qfwGyxPDFox5XktSfT6hKUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkho0sXBPckmSh5McTnL1pM4jSfpeEwn3JKcBfw5cCpwPXJ7k/EmcS5L0vSZ15X4hcLiqHq2qbwE3AzsndC5J0oA+L8g+kU3AE4u254BXLp6QZAaY6Tb/K8nDE+rlVLQReHK9m1jW779/vTvQ2js5fjZPHj+61MCkwn3Yi7Pr/21U7QH2TOj8p7Qks1U1vd59SIP82Vw7k1qWmQO2LNreDByZ0LkkSQMmFe6fAnYk2Z7kdGAXsH9C55IkDZjIskxVHU/yVuDvgdOAvVV1cBLn0lAud+n7lT+bayRVtfwsSdJJxSdUJalBhrskNchwl6QGTeo+d0kiyYtZeDp9EwvPuhwB9lfVoXVt7BTglXvDkrxlvXvQqSvJVSz86ZEA97Fwi3SAm/xjgpPn3TINS/J4VW1d7z50akryReAlVfXtgfrpwMGq2rE+nZ0aXJY5ySV5YKkh4Jy17EUa8F3gBcCXB+rndmOaIMP95HcO8DrgawP1AP+69u1I/+sdwIEkj/B/f0hwK/DjwFvXratThOF+8vs48Nyq+uzgQJK71r4daUFVfTLJeSz8CfBNLFxwzAGfqqrvrGtzpwDX3CWpQd4tI0kNMtwlqUGGuyQ1yHCXpAYZ7pLUoP8B4304fv57Ts0AAAAASUVORK5CYII=\n"
},
"metadata": {
"needs_background": "light"
}
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "df.info()
shows a quick insight to the number of missing values you have and what type of data your working with.\n\nin our case, there are no missing values and all our columns are numerical in nature."
},
{
"metadata": {
"init_cell": true,
"trusted": true
},
"cell_type": "code",
"source": "df.info()",
"execution_count": 45,
"outputs": [
{
"output_type": "stream",
"text": "<class 'pandas.core.frame.DataFrame'>\nRangeIndex: 303 entries, 0 to 302\nData columns (total 14 columns):\n # Column Non-Null Count Dtype \n--- ------ -------------- ----- \n 0 age 303 non-null int64 \n 1 sex 303 non-null int64 \n 2 cp 303 non-null int64 \n 3 trestbps 303 non-null int64 \n 4 chol 303 non-null int64 \n 5 fbs 303 non-null int64 \n 6 restecg 303 non-null int64 \n 7 thalach 303 non-null int64 \n 8 exang 303 non-null int64 \n 9 oldpeak 303 non-null float64\n 10 slope 303 non-null int64 \n 11 ca 303 non-null int64 \n 12 thal 303 non-null int64 \n 13 target 303 non-null int64 \ndtypes: float64(1), int64(13)\nmemory usage: 33.3 KB\n",
"name": "stdout"
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "Another way to get some quick insides on your dataframe is to use df.describe()
.\ndescribe()
shows a range of different metrics about your numerical columns such as mean, max and standard deviation."
},
{
"metadata": {
"init_cell": true,
"trusted": true
},
"cell_type": "code",
"source": "df.describe()",
"execution_count": 46,
"outputs": [
{
"output_type": "execute_result",
"execution_count": 46,
"data": {
"text/plain": " age sex cp trestbps chol fbs \\ncount 303.000000 303.000000 303.000000 303.000000 303.000000 303.000000 \nmean 54.366337 0.683168 0.966997 131.623762 246.264026 0.148515 \nstd 9.082101 0.466011 1.032052 17.538143 51.830751 0.356198 \nmin 29.000000 0.000000 0.000000 94.000000 126.000000 0.000000 \n25% 47.500000 0.000000 0.000000 120.000000 211.000000 0.000000 \n50% 55.000000 1.000000 1.000000 130.000000 240.000000 0.000000 \n75% 61.000000 1.000000 2.000000 140.000000 274.500000 0.000000 \nmax 77.000000 1.000000 3.000000 200.000000 564.000000 1.000000 \n\n restecg thalach exang oldpeak slope ca \\ncount 303.000000 303.000000 303.000000 303.000000 303.000000 303.000000 \nmean 0.528053 149.646865 0.326733 1.039604 1.399340 0.729373 \nstd 0.525860 22.905161 0.469794 1.161075 0.616226 1.022606 \nmin 0.000000 71.000000 0.000000 0.000000 0.000000 0.000000 \n25% 0.000000 133.500000 0.000000 0.000000 1.000000 0.000000 \n50% 1.000000 153.000000 0.000000 0.800000 1.000000 0.000000 \n75% 1.000000 166.000000 1.000000 1.600000 2.000000 1.000000 \nmax 2.000000 202.000000 1.000000 6.200000 2.000000 4.000000 \n\n thal target \ncount 303.000000 303.000000 \nmean 2.313531 0.544554 \nstd 0.612277 0.498835 \nmin 0.000000 0.000000 \n25% 2.000000 0.000000 \n50% 2.000000 1.000000 \n75% 3.000000 1.000000 \nmax 3.000000 1.000000 ",
"text/html": "pd.crosstab(column_1, column_2)
.\n\nThis is helpful if you want to start gaining an intuition about hoyour independent variables interact with your dependent variables.\n\nLet's compare our target column with the sex column.\n\nRemember from our data dictionary, for the target column, 1 = heart disease present, 0 = no heart disease. and for sex 1 = male, 0 = female"
},
{
"metadata": {
"init_cell": true,
"trusted": true
},
"cell_type": "code",
"source": "df.sex.value_counts()",
"execution_count": 48,
"outputs": [
{
"output_type": "execute_result",
"execution_count": 48,
"data": {
"text/plain": "1 207\n0 96\nName: sex, dtype: int64"
},
"metadata": {}
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "There are 207 males and 96 females in our study."
},
{
"metadata": {
"init_cell": true,
"trusted": true
},
"cell_type": "code",
"source": "# Compare target column with sex column\npd.crosstab(df.target, df.sex)",
"execution_count": 49,
"outputs": [
{
"output_type": "execute_result",
"execution_count": 49,
"data": {
"text/plain": "sex 0 1\ntarget \n0 24 114\n1 72 93",
"text/html": "plot()
function and passing it a few parameters such as, kind
(type of plot you want), figsize=(length, width)
(how big you want it to be) and color=[color_1, color_2]
(the different colors you'd like to use).\n\nDifferent metrics are represented best with different kinds of plots. In our case, a bar graph is great. We'll see examples of more later. And with a bit of practice, you'll gainan intuition of which plot to use with different variables."
},
{
"metadata": {
"init_cell": true,
"trusted": true
},
"cell_type": "code",
"source": "# Create a plot of crosstab\npd.crosstab(df.target, df.sex).plot(kind="bar",\n figsize=(10, 6),\n color=["salmon", "lightblue"]);",
"execution_count": 50,
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": "<Figure size 720x432 with 1 Axes>",
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAlYAAAFvCAYAAACIOIXnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAATzUlEQVR4nO3df6zd9X3f8debXFteSjIbuCDDhdpRgDYmS5sYFlKNZCXIgZDAMiKBGkYCEZoGBdaQYcaklH+qVurWLUrHZDUsntQCaUZnRhJ3mRuPdSNQu2XBMQUjWOCCBxcvoSkbw3Y/++OeZBbYA+75XJ9zrx8PCd17vj/f5o+rp77f7/3eaq0FAIDhHTXqAQAAFgthBQDQibACAOhEWAEAdCKsAAA6mRj1AEly3HHHtVWrVo16DACA17V9+/YXWmuTB1s3FmG1atWqbNu2bdRjAAC8rqr6/qHWuRUIANCJsAIA6ERYAQB0MhbPWAEAR5a9e/dmeno6L7/88qhHOaRly5ZlamoqS5YsecP7CCsA4LCbnp7O2972tqxatSpVNepxXqO1lj179mR6ejqrV69+w/u5FQgAHHYvv/xyjj322LGMqiSpqhx77LFv+oqasAIARmJco+rH5jKfsAIA6ERYAQB0IqwAADoRVgDAgvDSSy/lox/9aN7znvfkjDPOyF133ZXt27fngx/8YN73vvdl3bp12b17d/bt25czzzwzW7duTZLcfPPNueWWWw7LjF63AAAsCJs3b86JJ56Yr3/960mSF198Meeff342bdqUycnJ3HXXXbnlllty++235ytf+UouueSSfPGLX8zmzZvzwAMPHJYZhRUAsCC8+93vzo033pibbropF154YVasWJEdO3bkvPPOS5Ls378/K1euTJKsWbMml19+eT72sY/l/vvvz9KlSw/LjMIKAFgQTjvttGzfvj3f+MY3cvPNN+e8887LmjVrcv/99x90+4cffjjLly/Pc889d9hmFFYsOHc/unvUI4ydT5y+ctQjAMy7Z599Nsccc0w+9alP5eijj86GDRsyMzOT+++/P2effXb27t2bxx57LGvWrMndd9+dPXv25L777suFF16YBx98MMuXL5/3GYUVALAgPPzww/n85z+fo446KkuWLMltt92WiYmJXHfddXnxxRezb9++3HDDDTnhhBOyfv36bNmyJSeffHKuvfbaXH/99dm4ceO8zyisAIAFYd26dVm3bt1rlt93332vWfbYY4/95PvrrrtuXuc6kNctAAB0IqwAADoRVgAAnQgrAIBOhBUAQCfCCgCgE69bAABGbu+tn+t6vCVf+Kevu83mzZtz/fXXZ//+/fnsZz+b9evXD31eV6wAgCPO/v37c8011+Sb3/xmdu7cmTvuuCM7d+4c+rjCCgA44jz44IN55zvfmXe84x1ZunRpLr300mzatGno4worAOCI88wzz+Tkk0/+yeepqak888wzQx9XWAEAR5zW2muWVdXQxxVWAMARZ2pqKk8//fRPPk9PT+fEE08c+rjCCgA44px55pnZtWtXnnzyybzyyiu588478/GPf3zo43rdAgAwcm/k9Qg9TUxM5Etf+lLWrVuX/fv358orr8yaNWuGP26H2QAAFpwLLrggF1xwQddjuhUIANCJsAIA6ERYAQB0IqwAADoRVgAAnQgrAIBOvG4BABi5ux/d3fV4nzh95etuc+WVV+bee+/N8ccfnx07dnQ5rytWAMAR6dOf/nQ2b97c9ZjCCgA4Ip1zzjk55phjuh5TWAEAdCKsAAA6EVYAAJ0IKwCATrxuAQAYuTfyeoTeLrvssmzdujUvvPBCpqamcuutt+aqq64a6pjCCgA4It1xxx3dj+lWIABAJ8IKAKCT1w2rqrq9qp6vqh0HLDumqr5VVbsGX1cMlldVfbGqHq+q71bVe+dzeABg4WqtjXqE/6+5zPdGrlh9JclHXrVsfZItrbVTk2wZfE6S85OcOvjv6iS3vemJAIBFb9myZdmzZ8/YxlVrLXv27MmyZcve1H6v+/B6a+2+qlr1qsUXJfnQ4PuNSbYmuWmw/N+02f9L36mq5VW1srXW9y8rAgAL2tTUVKanpzMzMzPqUQ5p2bJlmZqaelP7zPW3Ak/4cSy11nZX1fGD5SclefqA7aYHy14TVlV1dWavauWUU06Z4xgAwEK0ZMmSrF69etRjdNf74fU6yLKDXuNrrW1ora1tra2dnJzsPAYAwOE317B6rqpWJsng6/OD5dNJTj5gu6kkz859PACAhWOuYXVPkisG31+RZNMBy//e4LcD35/kRc9XAQBHitd9xqqq7sjsg+rHVdV0ki8k+fUkX62qq5I8leSTg82/keSCJI8n+V9JPjMPMwMAjKU38luBlx1i1bkH2bYluWbYoQAAFiJvXgcA6ERYAQB0IqwAADoRVgAAnQgrAIBOhBUAQCfCCgCgE2EFANCJsAIA6ERYAQB0IqwAADoRVgAAnQgrAIBOJkY9AACMg7sf3T3qEcbOJ05fOeoRFhxXrAAAOhFWAACdCCsAgE6EFQBAJ8IKAKATYQUA0ImwAgDoRFgBAHQirAAAOhFWAACdCCsAgE6EFQBAJ8IKAKATYQUA0ImwAgDoRFgBAHQirAAAOhFWAACdCCsAgE6EFQBAJ8IKAKATYQUA0ImwAgDoRFgBAHQirAAAOhFWAACdCCsAgE6EFQBAJ8IKAKATYQUA0ImwAgDoZKiwqqp/WFXfq6odVXVHVS2rqtVV9UBV7aqqu6pqaa9hAQDG2ZzDqqpOSnJdkrWttTOSvCXJpUl+I8lvtdZOTfKDJFf1GBQAYNwNeytwIslfq6qJJG9NsjvJLyb52mD9xiQXD3kOAIAFYc5h1Vp7JslvJnkqs0H1YpLtSX7YWts32Gw6yUkH27+qrq6qbVW1bWZmZq5jAACMjWFuBa5IclGS1UlOTPJTSc4/yKbtYPu31ja01ta21tZOTk7OdQwAgLExzK3ADyd5srU201rbm+TuJB9IsnxwazBJppI8O+SMAAALwjBh9VSS91fVW6uqkpybZGeSbye5ZLDNFUk2DTciAMDCMMwzVg9k9iH1P03y8OBYG5LclORXqurxJMcm+XKHOQEAxt7E629yaK21LyT5wqsWP5HkrGGOCwCwEHnzOgBAJ8IKAKATYQUA0ImwAgDoRFgBAHQirAAAOhFWAACdCCsAgE6EFQBAJ8IKAKATYQUA0ImwAgDoRFgBAHQirAAAOhFWAACdCCsAgE4mRj0AAIfX3ls/N+oRxtOlN456AhYBV6wAADoRVgAAnQgrAIBOhBUAQCfCCgCgE2EFANCJsAIA6ERYAQB0IqwAADoRVgAAnQgrAIBOhBUAQCfCCgCgE2EFANCJsAIA6ERYAQB0IqwAADoRVgAAnQgrAIBOhBUAQCfCCgCgE2EFANCJsAIA6ERYAQB0IqwAADoRVgAAnQgrAIBOhBUAQCfCCgCgk6HCqqqWV9XXqurPq+qRqjq7qo6pqm9V1a7B1xW9hgUAGGfDXrH6F0k2t9Z+Jsl7kjySZH2SLa21U5NsGXwGAFj05hxWVfX2JOck+XKStNZeaa39MMlFSTYONtuY5OJhhwQAWAiGuWL1jiQzSf51Vf1ZVf1OVf1UkhNaa7uTZPD1+IPtXFVXV9W2qto2MzMzxBgAAONhmLCaSPLeJLe11n4+yUt5E7f9WmsbWmtrW2trJycnhxgDAGA8DBNW00mmW2sPDD5/LbOh9VxVrUySwdfnhxsRAGBhmHNYtdb+R5Knq+r0waJzk+xMck+SKwbLrkiyaagJAQAWiIkh9//lJL9bVUuTPJHkM5mNta9W1VVJnkryySHPAQCwIAwVVq21h5KsPciqc4c5LgDAQuTN6wAAnQgrAIBOhBUAQCfCCgCgE2EFANCJsAIA6ERYAQB0IqwAADoRVgAAnQgrAIBOhBUAQCfCCgCgE2EFANCJsAIA6ERYAQB0IqwAADoRVgAAnQgrAIBOhBUAQCfCCgCgE2EFANCJsAIA6ERYAQB0IqwAADoRVgAAnQgrAIBOhBUAQCfCCgCgE2EFANCJsAIA6ERYAQB0IqwAADoRVgAAnQgrAIBOhBUAQCfCCgCgE2EFANCJsAIA6ERYAQB0IqwAADoRVgAAnQgrAIBOhBUAQCfCCgCgE2EFANDJ0GFVVW+pqj+rqnsHn1dX1QNVtauq7qqqpcOPCQAw/npcsbo+ySMHfP6NJL/VWjs1yQ+SXNXhHAAAY2+osKqqqSQfTfI7g8+V5BeTfG2wycYkFw9zDgCAhWLYK1b/PMk/SvJXg8/HJvlha23f4PN0kpMOtmNVXV1V26pq28zMzJBjAACM3pzDqqouTPJ8a237gYsPsmk72P6ttQ2ttbWttbWTk5NzHQMAYGxMDLHvLyT5eFVdkGRZkrdn9grW8qqaGFy1mkry7PBjAgCMvzlfsWqt3dxam2qtrUpyaZI/aq39UpJvJ7lksNkVSTYNPSUAwAIwH++xuinJr1TV45l95urL83AOAICxM8ytwJ9orW1NsnXw/RNJzupxXACAhcSb1wEAOhFWAACdCCsAgE6EFQBAJ8IKAKATYQUA0ImwAgDoRFgBAHQirAAAOhFWAACdCCsAgE6EFQBAJ8IKAKATYQUA0ImwAgDoRFgBAHQirAAAOhFWAACdCCsAgE6EFQBAJ8IKAKATYQUA0ImwAgDoRFgBAHQirAAAOhFWAACdCCsAgE6EFQBAJ8IKAKATYQUA0ImwAgDoRFgBAHQirAAAOhFWAACdCCsAgE6EFQBAJ8IKAKATYQUA0MnEqAfg0Pbe+rlRjzCeLr1x1BMAwEG5YgUA0ImwAgDoRFgBAHQirAAAOhFWAACdCCsAgE6EFQBAJ3MOq6o6uaq+XVWPVNX3qur6wfJjqupbVbVr8HVFv3EBAMbXMFes9iX5XGvtZ5O8P8k1VfWuJOuTbGmtnZpky+AzAMCiN+ewaq3tbq396eD7HyV5JMlJSS5KsnGw2cYkFw87JADAQtDlGauqWpXk55M8kOSE1truZDa+khx/iH2urqptVbVtZmamxxgAACM1dFhV1dFJ/m2SG1prf/FG92utbWitrW2trZ2cnBx2DACAkRsqrKpqSWaj6ndba3cPFj9XVSsH61cmeX64EQEAFoZhfiuwknw5ySOttX92wKp7klwx+P6KJJvmPh4AwMIxMcS+v5Dk8iQPV9VDg2X/OMmvJ/lqVV2V5KkknxxuRACAhWHOYdVa++MkdYjV5871uAAAC5U3rwMAdCKsAAA6EVYAAJ0IKwCAToQVAEAnwgoAoBNhBQDQibACAOhEWAEAdCKsAAA6EVYAAJ0IKwCAToQVAEAnwgoAoBNhBQDQibACAOhEWAEAdCKsAAA6EVYAAJ0IKwCAToQVAEAnwgoAoBNhBQDQibACAOhEWAEAdCKsAAA6EVYAAJ0IKwCAToQVAEAnwgoAoBNhBQDQibACAOhEWAEAdCKsAAA6EVYAAJ0IKwCAToQVAEAnwgoAoBNhBQDQibACAOhEWAEAdCKsAAA6EVYAAJ0IKwCAToQVAEAn8xJWVfWRqnq0qh6vqvXzcQ4AgHHTPayq6i1JfjvJ+UneleSyqnpX7/MAAIyb+bhidVaSx1trT7TWXklyZ5KL5uE8AABjpVprfQ9YdUmSj7TWPjv4fHmSv9lau/ZV212d5OrBx9OTPNp1EBaz45K8MOohgEXHzxbeqJ9urU0ebMXEPJysDrLsNfXWWtuQZMM8nJ9Frqq2tdbWjnoOYHHxs4Ue5uNW4HSSkw/4PJXk2Xk4DwDAWJmPsPqTJKdW1eqqWprk0iT3zMN5AADGSvdbga21fVV1bZI/TPKWJLe31r7X+zwc0dxCBuaDny0MrfvD6wAARypvXgcA6ERYAQB0IqwAADoRVgAAnczHC0Khm6r6mcz+SaSTMvui2WeT3NNae2SkgwHAQbhixdiqqpsy+7cmK8mDmX1HWiW5o6rWj3I2YPGqqs+MegYWLq9bYGxV1WNJ1rTW9r5q+dIk32utnTqayYDFrKqeaq2dMuo5WJjcCmSc/VWSE5N8/1XLVw7WAcxJVX33UKuSnHA4Z2FxEVaMsxuSbKmqXUmeHiw7Jck7k1w7sqmAxeCEJOuS/OBVyyvJfz3847BYCCvGVmttc1WdluSszD68Xpn9I99/0lrbP9LhgIXu3iRHt9YeevWKqtp6+MdhsfCMFQBAJ34rEACgE2EFANCJsALGWlUtr6p/cBjO86Gq+sB8nwdY3IQVMO6WJ3nDYVWz5vKz7UNJhBUwFA+vA2Otqu7M7J81ejTJt5P8jSQrkixJ8k9aa5uqalWSbw7Wn53k4iQfTnJTZv8M0q4k/6e1dm1VTSb5V5l9dUcy+1qPZ5J8J8n+JDNJfrm19p8Px78PWFyEFTDWBtF0b2vtjKqaSPLW1tpfVNVxmY2hU5P8dJInknygtfadqjoxs+8iem+SHyX5oyT/bRBWv5fkX7bW/riqTknyh621n62qX03yl6213zzc/0Zg8fAeK2AhqSS/VlXnZPbt+yfl/70l+/utte8Mvj8ryX9qrf3PJKmq309y2mDdh5O8q6p+fMy3V9XbDsfwwOInrICF5JeSTCZ5X2ttb1X99yTLButeOmC7evWOBzgqydmttf994MIDQgtgzjy8Doy7HyX58RWlv57k+UFU/e3M3gI8mAeTfLCqVgxuH/7dA9b9hxzwJ5Gq6ucOch6AORFWwFhrre1J8l+qakeSn0uytqq2Zfbq1Z8fYp9nkvxakgeS/MckO5O8OFh93eAY362qnUn+/mD5v0/yd6rqoar6W/P2DwIWNQ+vA4tSVR3dWvvLwRWrP0hye2vtD0Y9F7C4uWIFLFa/WlUPJdmR5Mkk/27E8wBHAFesAAA6ccUKAKATYQUA0ImwAgDoRFgBAHQirAAAOvm/egLRRdRturYAAAAASUVORK5CYII=\n"
},
"metadata": {
"needs_background": "light"
}
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "Nice! But our plot is looking pretty bare. Let's add some attributes.\n\nWe'll create the plot again with crosstab()
and plot()
, then add some helpfull labels to it with plt.title(), plt.xlabel()
and more.\n\nTo add the attributes, you call them on plt
within the same cell as where you create the graph."
},
{
"metadata": {
"init_cell": true,
"trusted": true
},
"cell_type": "code",
"source": "# Create a plot of crosstab\npd.crosstab(df.target, df.sex).plot(kind="bar",\n figsize=(10, 6),\n color=["salmon", "lightblue"])\n\nplt.title("Heart Disease Frequency for Sex")\nplt.xlabel("0 = No disease, 1 = disease")\nplt.ylabel("Amount")\nplt.legend(["Female", "Male"])\nplt.xticks(rotation=0)",
"execution_count": 51,
"outputs": [
{
"output_type": "execute_result",
"execution_count": 51,
"data": {
"text/plain": "(array([0, 1]), <a list of 2 Text xticklabel objects>)"
},
"metadata": {}
},
{
"output_type": "display_data",
"data": {
"text/plain": "<Figure size 720x432 with 1 Axes>",
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAmQAAAGDCAYAAACFuAwbAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3de5xdVX338c8XAgYFkUvkgUQIWkQRImiCFrwgUBAvoFUUWi0KFNt6wRuK9kHQPrZarRbRoqgVRItaEMFqK2iJVAVJKAhI5CJQiCBGLuFS0AR/zx9nJxyHSXKSzMyayXzer9d5zdlrr733b+8zZL6stc85qSokSZLUznqtC5AkSZrsDGSSJEmNGcgkSZIaM5BJkiQ1ZiCTJElqzEAmSZLUmIFM0lpLsm2S+5Ks37oWDS7Jnkmu6167l7WuR5rMDGTSOJLkpiT7Dml7XZIfjOIxK8kfrGT965I81P3Rvi/JjUm+kOTJy/pU1c1VtXFVPTRadY6U7ho/0Hc+9yXZpnVdjXwA+GT32n1jbXeWZEaSs5L8OsniJFcmed3alymt+wxk0iSVZMpqdL+oqjYGNgX2BR4ALk2y86gUN/pe2oWQZY9bh3ZYzeszUW0H/HRNNlzB9TkduKXb7xbAnwG3r3F10iRiIJMmmCTbdKMQi7rRqrf0rds9yUVJ7k5yW5JPJtmwb30leWOS64DrklzYrfpJN1L06pUdu6oeqqqfV9VfAd8HTuj2O7Pb95Ru+XVJbkhyb1fjn/bVcHiSBUnuSvKdJNv1rTsxyS1J7klyaZLnDjm3+d2625N8rG/ds5P8qDvvnyTZaw2u67JzOCLJzcB/rmrfSbZP8v3uPM/vrveXunV7JVk45BjLR0CTrJfk2CQ/T3JHkq8l2XxILYclubkbcfrrvv2sn+S93bb3dtfqCUk+leQfhhzzm0neOsz5/hx4IvDN7rV/VPe7dW6SO5Ncn+TP+/qfkOTMJF9Kcg/wumEu4xzg1Kq6v6qWVtVlVfXvffsY9lom2TzJwiQv7ZY37o7/ZwO8dNK6oap8+PAxTh7ATcC+Q9peB/yge74ecCnwPmBDen9QbwD279Y/E3g2MAWYCSwA3tq3rwLOBzYHNupr+4OV1LT8+EPaDwdu757P7PYzBXgMcA+wY7dua+Bp3fOXAdcDT+36/l/gR337fA29kZUpwDuAXwJTu3UXAa/tnm8MPLt7Ph24A3hRd33+qFueNug1HnIOX+zOYaNV7bur6WPAo4DnAfcCX+rW7QUsXNGxgbcCFwMzuu0/A5wxpJbPdnU8HfgN8NRu/THAlcCOQLr1WwC7A7cC63X9tgT+F9hqkGtBL2T/EzAV2BVYBOzTrTsBWNK9huvR/f4M2d93gR8ChwDbDlm3qmu5X/d6P7477zNb//fow8dYPpoX4MOHj4cf3R/I+4C7+x7/y8OB7FnAzUO2eQ/whRXs763A2X3LBew9pM+aBrIXAku658sCxLJAdjfwiqF/tIF/B47oW16vO7/tVnDsu4Cnd88vBN4PbDmkz7uB04e0fQc4bMBr/I0h5/DEQfYNbAssBR7Tt+5fGDyQLVgWdrrlrbvAM6Wvlhl96y8BDumeXwMctILzWwD8Uff8TcC3V/H7tqyeJwAPAZv0rf87eiNe0AtkF67i93cz4EP0pkEfAi4H5gz6OgEn0QuatwJbtPrv0IePFg+nLKXx52VV9bhlD+Cv+tZtB2zTTfncneRu4L3AVgBJnpzk35L8sptW+lt6oyT9bhmhOqcDdw5trKr7gVcDfwHcluRbSZ7SV/+JfbXfSW+EZ3pX/zu66czF3fpN++o/Angy8LMk85K8pG+fBw+5Js+hF3BWpP8aD313Yf/1Wdm+twHu6s53mf9ZyTGH2g44u2+/C+iFmK36+vyy7/n/0hsZhF54+vkK9nsavZFGup+nD1jPNsCdVXVvX9v/0L02nZX+7lTVXVV1bFU9jd55XA58I0kY7HU6BdiZ3v9g3DFg3dI6wUAmTSy3ADf2B7aq2qSqXtStPxn4GbBDVT2WXljLkH3UCNXycuC/hltRVd+pqj+i98f2Z/SmoJbV/4Yh9W9UVT/q7hd7N/AqYLMujC5eVn9VXVdVh9Kb0vowcGaSx3T7PH3IPh9TVR9aw/Pqvz4r2/dtwGZdDcts2/f8fuDRyxbS+0iQaUP2fcCQfU+tql8MUOMtwJNWsO5LwEFJnk5vanjQd0/eCmyeZJO+tm2B/noG/t2pql8DH6UX9DZnFa9Td30+Q2/K+C+zknf+SusiA5k0sVwC3JPk3Uk26m7u3jnJnG79JvTu37qvG5X6ywH2eTu9e9FWqTve9klOojcl9/5h+myV5MAuqPyG3vTgso/D+DTwniRP6/pumuTgvtqX0rtvaUqS9wGP7dvva5JMq6rf0ZtqpNvvl4CXJtm/q29qd0P9jEHOaRVWuO+q+h9gPvD+JBsmeQ7w0r5trwWmJnlxkg3o3S/3qL71nwY+mO5NDUmmJTlowLo+B/xNkh3SMyvJFgBVtRCYR29k7KyqemCQHVbVLcCPgL/rznMWvVHJLw9YE0k+3P0+TumC3V8C13ejXat6nd7b/TycXpD7YvxcO00iBjJpAqne53y9lN4N1zcCv6b3x3nTrss7gT+hd3P5Z4GvDrDbE4DTummkV62gzx8muY9e2JtLLyjNqaorh+m7Hr0b8m+lNyX5fLpp16o6m97o1le6KdWrgAO67b5D7x6za+lNlT3I70+RvRD4aVfHifTup3qwCxIH0fuDvqjb5hhG4N+3Afb9J/Tu67sTOJ7e6M6ybRd35/05eqNM9wP977o8ETgXOC/JvfRu8H/WgKV9DPgacB691+Tz9G7+X+Y0YBcGn65c5lB696/dCpwNHF9V56/G9o/utrub3ptNtgMOhJVfyyTPBN4O/Fn3O/5heqNxx65m/dKElaqRmr2QpMktyQn03iDxmlX1HeU6nkdvRGpmN6IoaZxzhEyS1iHd9OjRwOcMY9LEYSCTpHVEkqfSmy7cGvjHxuVIWg1OWUqSJDXmCJkkSVJjBjJJkqTGprQuYG1sueWWNXPmzNZlSJIkrdKll17666qaNty6CR3IZs6cyfz581uXIUmStEpJVvj1ak5ZSpIkNWYgkyRJasxAJkmS1NiEvodMkiSNvSVLlrBw4UIefPDB1qWMS1OnTmXGjBlssMEGA29jIJMkSatl4cKFbLLJJsycOZMkrcsZV6qKO+64g4ULF7L99tsPvJ1TlpIkabU8+OCDbLHFFoaxYSRhiy22WO3RQwOZJElabYaxFVuTa2MgkyRJE87666/Prrvuuvxx0003jdqxTj31VN70pjeN2v7Be8gkSdJaWvL+d4zo/jY4/h9W2WejjTbi8ssvH9HjtuQImSRJWic89NBDHHPMMcyZM4dZs2bxmc98BoC5c+fy/Oc/n1e96lU8+clP5thjj+XLX/4yu+++O7vssgs///nPAfjmN7/Js571LHbbbTf23Xdfbr/99kccY9GiRbziFa9gzpw5zJkzhx/+8IcjUruBTJIkTTgPPPDA8unKl7/85QB8/vOfZ9NNN2XevHnMmzePz372s9x4440A/OQnP+HEE0/kyiuv5PTTT+faa6/lkksu4cgjj+Skk04C4DnPeQ4XX3wxl112GYcccgh///d//4jjHn300bztbW9j3rx5nHXWWRx55JEjcj5OWUqSpAlnuCnL8847jyuuuIIzzzwTgMWLF3Pdddex4YYbMmfOHLbeemsAnvSkJ7HffvsBsMsuu3DBBRcAvY/zePWrX81tt93Gb3/722E/tuK73/0uV1999fLle+65h3vvvZdNNtlkrc7HQCZJktYJVcVJJ53E/vvv/3vtc+fO5VGPetTy5fXWW2/58nrrrcfSpUsBePOb38zb3/52DjzwQObOncsJJ5zwiGP87ne/46KLLmKjjTYa0doNZJo0vn7Nba1LGHf+eMetW5cgSSNm//335+STT2bvvfdmgw024Nprr2X69OkDb7948eLl/U877bRh++y333588pOf5JhjjgHg8ssvZ9ddd13r2r2HTJIkrROOPPJIdtppJ57xjGew884784Y3vGH56NcgTjjhBA4++GCe+9znsuWWWw7b5xOf+ATz589n1qxZ7LTTTnz6058ekdpTVSOyoxZmz55d8+fPb12GJghHyB7JETJJa2LBggU89alPbV3GuDbcNUpyaVXNHq6/I2SSJEmNGcgkSZIaM5BJkiQ1ZiCTJElqzEAmSZLUmIFMkiSpMQOZJEmacJLw2te+dvny0qVLmTZtGi95yUtWut3cuXNX2acFP6lfkiStlZH+nMdBPiPxMY95DFdddRUPPPAAG220Eeeff/5qfSr/eOMImSRJmpAOOOAAvvWtbwFwxhlncOihhy5fd8kll7DHHnuw2267sccee3DNNdc8Yvv777+fww8/nDlz5rDbbrtxzjnnjFntQxnIJEnShHTIIYfwla98hQcffJArrriCZz3rWcvXPeUpT+HCCy/ksssu4wMf+ADvfe97H7H9Bz/4Qfbee2/mzZvHBRdcwDHHHMP9998/lqewnFOWkiRpQpo1axY33XQTZ5xxBi960Yt+b93ixYs57LDDuO6660jCkiVLHrH9eeedx7nnnstHP/pRAB588EFuvvnmJl8LZSCTJEkT1oEHHsg73/lO5s6dyx133LG8/bjjjuMFL3gBZ599NjfddBN77bXXI7atKs466yx23HHHMax4eE5ZSpKkCevwww/nfe97H7vsssvvtS9evHj5Tf6nnnrqsNvuv//+nHTSSVQVAJdddtmo1royBjJJkjRhzZgxg6OPPvoR7e9617t4z3vew5577slDDz007LbHHXccS5YsYdasWey8884cd9xxo13uCmVZKpyIZs+eXfPnz29dhiaIkX5b9rpgkLeWS9JQCxYsaHKf1UQy3DVKcmlVzR6uvyNkkiRJjRnIJEmSGjOQSZIkNWYgkyRJq20i34M+2tbk2hjIJEnSapk6dSp33HGHoWwYVcUdd9zB1KlTV2s7PxhWkiStlhkzZrBw4UIWLVrUupRxaerUqcyYMWO1tjGQSZKk1bLBBhuw/fbbty5jneKUpSRJUmMGMkmSpMYMZJIkSY0ZyCRJkhozkEmSJDVmIJMkSWrMQCZJktSYgUySJKkxA5kkSVJjBjJJkqTGRi2QJfnnJL9KclVf2+ZJzk9yXfdzs649ST6R5PokVyR5xmjVJUmSNN6M5gjZqcALh7QdC3yvqnYAvtctAxwA7NA9jgJOHsW6JEmSxpVRC2RVdSFw55Dmg4DTuuenAS/ra/9i9VwMPC7J1qNVmyRJ0ngy1veQbVVVtwF0Px/ftU8Hbunrt7Bre4QkRyWZn2T+okWLRrVYSZKksTBeburPMG01XMeqOqWqZlfV7GnTpo1yWZIkSaNvrAPZ7cumIrufv+raFwJP6Os3A7h1jGuTJElqYqwD2bnAYd3zw4Bz+tr/rHu35bOBxcumNiVJktZ1U0Zrx0nOAPYCtkyyEDge+BDwtSRHADcDB3fdvw28CLge+F/g9aNVlyRJ0ngzaoGsqg5dwap9hulbwBtHqxZJkqTxbLzc1C9JkjRpGcgkSZIaM5BJkiQ1ZiCTJElqzEAmSZLUmIFMkiSpMQOZJElSYwYySZKkxgxkkiRJjRnIJEmSGjOQSZIkNWYgkyRJasxAJkmS1NiU1gVIkjSRff2a21qXMO788Y5bty5hwnGETJIkqTEDmSRJUmMGMkmSpMYMZJIkSY0ZyCRJkhozkEmSJDVmIJMkSWrMQCZJktSYgUySJKkxA5kkSVJjBjJJkqTGDGSSJEmNGcgkSZIaM5BJkiQ1ZiCTJElqzEAmSZLUmIFMkiSpMQOZJElSYwYySZKkxgxkkiRJjRnIJEmSGjOQSZIkNWYgkyRJasxAJkmS1JiBTJIkqTEDmSRJUmMGMkmSpMYMZJIkSY0ZyCRJkhozkEmSJDVmIJMkSWqsSSBL8rYkP01yVZIzkkxNsn2SHye5LslXk2zYojZJkqSxNuaBLMl04C3A7KraGVgfOAT4MPDxqtoBuAs4YqxrkyRJaqHVlOUUYKMkU4BHA7cBewNndutPA17WqDZJkqQxNeaBrKp+AXwUuJleEFsMXArcXVVLu24LgeljXZskSVILLaYsNwMOArYHtgEeAxwwTNdawfZHJZmfZP6iRYtGr1BJkqQx0mLKcl/gxqpaVFVLgK8DewCP66YwAWYAtw63cVWdUlWzq2r2tGnTxqZiSZKkUdQikN0MPDvJo5ME2Ae4GrgAeGXX5zDgnAa1SZIkjbkW95D9mN7N+/8NXNnVcArwbuDtSa4HtgA+P9a1SZIktTBl1V1GXlUdDxw/pPkGYPcG5UiSJDXlJ/VLkiQ1ZiCTJElqzEAmSZLUmIFMkiSpMQOZJElSYwYySZKkxgxkkiRJjRnIJEmSGjOQSZIkNWYgkyRJasxAJkmS1JiBTJIkqTEDmSRJUmMGMkmSpMYMZJIkSY0ZyCRJkhozkEmSJDU2pXUBkqSJYcn739G6hPHpkHe2rkDrAEfIJEmSGjOQSZIkNWYgkyRJasxAJkmS1JiBTJIkqTEDmSRJUmMGMkmSpMYMZJIkSY0ZyCRJkhozkEmSJDVmIJMkSWrMQCZJktSYgUySJKkxA5kkSVJjBjJJkqTGVhnIknxvkDZJkiStmSkrWpFkKvBoYMskmwHpVj0W2GYMapMkSZoUVhjIgDcAb6UXvi7l4UB2D/CpUa5LkiRp0lhhIKuqE4ETk7y5qk4aw5okSZImlZWNkAFQVScl2QOY2d+/qr44inVJkiRNGqsMZElOB54EXA481DUXYCCTJEkaAasMZMBsYKeqqtEuRpIkaTIa5HPIrgL+z2gXIkmSNFkNMkK2JXB1kkuA3yxrrKoDR60qSZKkSWSQQHbCaBchSZI0mQ3yLsvvj0UhkiRJk9Ug77K8l967KgE2BDYA7q+qx45mYZIkSZPFICNkm/QvJ3kZsPuoVSRJkjTJDPIuy99TVd8A9l6bgyZ5XJIzk/wsyYIkf5hk8yTnJ7mu+7nZ2hxDkiRpohhkyvKP+xbXo/e5ZGv7mWQnAv9RVa9MsiG9LzF/L/C9qvpQkmOBY4F3r+VxJEmSxr1B3mX50r7nS4GbgIPW9IBJHgs8D3gdQFX9FvhtkoOAvbpupwFzMZBJkqRJYJB7yF4/wsd8IrAI+EKSpwOXAkcDW1XVbd0xb0vy+OE2TnIUcBTAtttuO8KlSZIkjb1V3kOWZEaSs5P8KsntSc5KMmMtjjkFeAZwclXtBtxPb3pyIFV1SlXNrqrZ06ZNW4syJEmSxodBbur/AnAusA0wHfhm17amFgILq+rH3fKZ9ALa7Um2Buh+/motjiFJkjRhDBLIplXVF6pqafc4FVjjoamq+iVwS5Idu6Z9gKvphb7DurbDgHPW9BiSJEkTySA39f86yWuAM7rlQ4E71vK4bwa+3L3D8gbg9fTC4deSHAHcDBy8lseQJEmaEAYJZIcDnwQ+Tu/jLn7Uta2xqrqc3sdnDLXP2uxXkiRpIhrkXZY3AweOQS2SJEmT0iAfDLs9vSnGmf39q8qQJkmSNAIGmbL8BvB5eu+u/N3oliNJkjT5DBLIHqyqT4x6JZIkSZPUIIHsxCTHA+cBv1nWWFX/PWpVSZIkTSKDBLJdgNcCe/PwlGV1y5IkSVpLgwSylwNP7L4EXJIkSSNskE/q/wnwuNEuRJIkabIaZIRsK+BnSebx8D1kVVUHjV5ZkiRJk8cggez4vucBnkPv65MkSZI0AlY5ZVlV3wcWAy8GTqX39UafHt2yJEmSJo8VjpAleTJwCA9/mfhXgVTVC8aoNkmSpElhZVOWPwP+C3hpVV0PkORtY1KVJEnSJLKyKctXAL8ELkjy2ST70LuHTJIkSSNohYGsqs6uqlcDTwHmAm8DtkpycpL9xqg+SZKkdd4gN/XfX1VfrqqXADOAy4FjR70ySZKkSWKQD4ZdrqrurKrPVJVfmyRJkjRCViuQSZIkaeQZyCRJkhozkEmSJDVmIJMkSWrMQCZJktSYgUySJKkxA5kkSVJjBjJJkqTGDGSSJEmNGcgkSZIaM5BJkiQ1ZiCTJElqzEAmSZLUmIFMkiSpMQOZJElSYwYySZKkxgxkkiRJjRnIJEmSGjOQSZIkNWYgkyRJasxAJkmS1JiBTJIkqTEDmSRJUmMGMkmSpMYMZJIkSY0ZyCRJkhozkEmSJDVmIJMkSWqsWSBLsn6Sy5L8W7e8fZIfJ7kuyVeTbNiqNkmSpLHUcoTsaGBB3/KHgY9X1Q7AXcARTaqSJEkaY00CWZIZwIuBz3XLAfYGzuy6nAa8rEVtkiRJY63VCNk/Au8CftctbwHcXVVLu+WFwPQWhUmSJI21MQ9kSV4C/KqqLu1vHqZrrWD7o5LMTzJ/0aJFo1KjJEnSWGoxQrYncGCSm4Cv0Juq/EfgcUmmdH1mALcOt3FVnVJVs6tq9rRp08aiXkmSpFE15oGsqt5TVTOqaiZwCPCfVfWnwAXAK7tuhwHnjHVtkiRJLYynzyF7N/D2JNfTu6fs843rkSRJGhNTVt1l9FTVXGBu9/wGYPeW9UiSJLUwnkbIJEmSJiUDmSRJUmMGMkmSpMYMZJIkSY0ZyCRJkhozkEmSJDVmIJMkSWrMQCZJktSYgUySJKkxA5kkSVJjBjJJkqTGDGSSJEmNGcgkSZIaM5BJkiQ1ZiCTJElqzEAmSZLUmIFMkiSpMQOZJElSYwYySZKkxgxkkiRJjRnIJEmSGjOQSZIkNWYgkyRJasxAJkmS1JiBTJIkqTEDmSRJUmMGMkmSpMYMZJIkSY0ZyCRJkhozkEmSJDVmIJMkSWrMQCZJktSYgUySJKkxA5kkSVJjBjJJkqTGDGSSJEmNGcgkSZIaM5BJkiQ1NqV1ARp5S97/jtYljE+HvLN1BZIkDcsRMkmSpMYMZJIkSY0ZyCRJkhozkEmSJDVmIJMkSWrMQCZJktSYgUySJKmxMQ9kSZ6Q5IIkC5L8NMnRXfvmSc5Pcl33c7Oxrk2SJKmFFiNkS4F3VNVTgWcDb0yyE3As8L2q2gH4XrcsSZK0zhvzQFZVt1XVf3fP7wUWANOBg4DTum6nAS8b69okSZJaaHoPWZKZwG7Aj4Gtquo26IU24PEr2OaoJPOTzF+0aNFYlSpJkjRqmgWyJBsDZwFvrap7Bt2uqk6pqtlVNXvatGmjV6AkSdIYaRLIkmxAL4x9uaq+3jXfnmTrbv3WwK9a1CZJkjTWWrzLMsDngQVV9bG+VecCh3XPDwPOGevaJEmSWpjS4Jh7Aq8Frkxyedf2XuBDwNeSHAHcDBzcoDZJkqQxN+aBrKp+AGQFq/cZy1okSZLGAz+pX5IkqTEDmSRJUmMGMkmSpMYMZJIkSY0ZyCRJkhozkEmSJDVmIJMkSWrMQCZJktSYgUySJKkxA5kkSVJjBjJJkqTGDGSSJEmNGcgkSZIaM5BJkiQ1ZiCTJElqzEAmSZLUmIFMkiSpMQOZJElSYwYySZKkxgxkkiRJjRnIJEmSGjOQSZIkNWYgkyRJasxAJkmS1JiBTJIkqTEDmSRJUmMGMkmSpMYMZJIkSY0ZyCRJkhozkEmSJDVmIJMkSWrMQCZJktSYgUySJKkxA5kkSVJjBjJJkqTGDGSSJEmNGcgkSZIaM5BJkiQ1ZiCTJElqzEAmSZLUmIFMkiSpMQOZJElSYwYySZKkxgxkkiRJjRnIJEmSGhtXgSzJC5Nck+T6JMe2rkeSJGksjJtAlmR94FPAAcBOwKFJdmpblSRJ0ugbN4EM2B24vqpuqKrfAl8BDmpckyRJ0qgbT4FsOnBL3/LCrk2SJGmdNqV1AX0yTFs9olNyFHBUt3hfkmtGtSqtO0742JbAr1uXIWkd478tGtx2K1oxngLZQuAJfcszgFuHdqqqU4BTxqoorTuSzK+q2a3rkLRu8d8WjYTxNGU5D9ghyfZJNgQOAc5tXJMkSdKoGzcjZFW1NMmbgO8A6wP/XFU/bVyWJEnSqBs3gQygqr4NfLt1HVpnOdUtaTT4b4vWWqoecd+8JEmSxtB4uodMkiRpUjKQaZ3nV3JJGg1J/jnJr5Jc1boWTXwGMq3T/EouSaPoVOCFrYvQusFApnWdX8klaVRU1YXAna3r0LrBQKZ1nV/JJUka9wxkWtcN9JVckiS1ZCDTum6gr+SSJKklA5nWdX4llyRp3DOQaZ1WVUuBZV/JtQD4ml/JJWkkJDkDuAjYMcnCJEe0rkkTl5/UL0mS1JgjZJIkSY0ZyCRJkhozkEmSJDVmIJMkSWrMQCZJktSYgUya5JK8MMk1Sa5PcuwI7G9mkkry5r62TyZ53Vrs877u5zZJzlzbGkdDkv9IcneSfxul/Z+a5JXd888l2Wk0jiOpDQOZNIklWR/4FHAAsBNw6Aj9of8VcHT3YbwjpqpurapXjuQ+R9BHgNeOxYGq6siqunosjiVpbBjIpMltd+D6qrqhqn4LfAU4aAT2uwj4HnDY0BVJdk1ycZIrkpydZLNh+myf5KIk85L8TV/7zCRXdc+fluSSJJd3+9qha39NX/tnutBJkpOTzE/y0yTv79vnh5Jc3e3jo13btCRndcefl2TPVZ1wVX0PuHe1r9QKpOeTXW3fAh7ft25uktlJ1u9Gzq5KcmWSt3Xrn9SN2F2a5L+SPKVrf2mSHye5LMl3k2zVtT+/u16Xd+s26dqP6c7/iv5rJmnkGcikyW06cEvf8sKu7fd0f5gvH+bxiZXs+0PAO5YFoj5fBN5dVbOAK4Hjh9n2RODkqpoD/HIF+/8L4MSq2hWYDSxM8lTg1cCeXftDwJ92/f+6qmYDs4DnJ5mVZHPg5cDTunr+X9/xP94d/xXA51ZyngNbzev4cmBHYBfgz4E9humzKzC9qnauql2AL3TtpwBvrqpnAu8E/qlr/wHw7KrajV74flfX/k7gjd01ey7wQJL9gB3ohfZdgWcmed7aXQFJKzKldQGSmsowbY/4+o6q+gi9KbmBVdWNSS4B/mT5wZJNgcdV1fe7ptOAfx1m8z3pBSGA04EPD9PnIuCvk8wAvl5V1yXZB3gmMC8JwEUSrnsAAAL9SURBVEb0pk8BXpXkKHr/7m1Nb4r2auBB4HPdKNSy+7/2BXbq9gHw2CSbVNVajYCt5nV8HnBGVT0E3JrkP4fpcwPwxCQnAd8CzkuyMb3w9q999T+q+zkD+GqSrYENgRu79h8CH0vyZXrXcmEXyPYDLuv6bEwvoF04YP2SVoOBTJrcFgJP6FueAdw6tFOSY3h4pKnfhVX1lpXs/2+BM1mzP+Ir/V63qvqXJD8GXgx8J8mR9ALmaVX1nv6+SbanNwo0p6ruSnIqMLWqlibZHdiH3hfPvwnYm97swR9W1QNrUPcKrcF1XNU1uCvJ04H9gTcCrwLeCtzdjXYNdRLwsao6N8lewAndfj7UBdIXARcn2Zfetfy7qvrMQCcnaa04ZSlNbvOAHbp7tjakF0rOHdqpqj5SVbsO81hZGKOqfkZvFOol3fJi4K4kz+26vBb4/jCb/rCrBYYPMCR5InBDVX2iq3kWvfvWXpnk8V2fzZNsBzwWuB9Y3N03dUC3fmNg06r6Nr0gsyzEnEcvnC071q7dz92TfHFl57wyq3kdLwQO6e4T2xp4wTDXYEtgvao6CzgOeEZV3QPcmOTgrk+60AawKfCL7vlhfft5UlVdWVUfBuYDTwG+AxzeXSOSTF92XSWNPEfIpEmsGyF6E70/vusD/1xVPx3hw3yQh6e9oBcEPp3k0fSm3F4/zDZHA/+S5GjgrBXs99XAa5IsoXef2Qeq6s4k/5fe1N16wBJ690ZdnOQy4KfdMX/Y7WMT4JwkU+mNCL2ta38L8KkkV9D7d/JCevesbQsMO2qW5L/oBZmNkywEjqiq76zswqzC2fRG664ErmX44Dod+EJ3rgDLRgb/FDi5uxYb0Ltf7Cf0RsT+NckvgIuB7bv+b03yAnr33F0N/HtV/aa7J++iburzPuA1PDwFLGkEpWqlI+KSpE6SjwCnV9UVrWuRtG4xkEmSJDXmPWSSJEmNGcgkSZIaM5BJkiQ1ZiCTJElqzEAmSZLUmIFMkiSpMQOZJElSY/8f7keDI+OZPRsAAAAASUVORK5CYII=\n"
},
"metadata": {
"needs_background": "light"
}
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "### Age vs. Max Heart Rate for Heart Disease\n\nLet's try combining a couple of independent variables, such as, age and talach (maximum heart rate) and then comparing them to our target variable heart disease\n\nBecause there are so many different values for age and talach, we'll use a scatter plot."
},
{
"metadata": {
"init_cell": true,
"trusted": true
},
"cell_type": "code",
"source": "# Create another figure\nplt.figure(figsize=(10, 6))\n\n# Scatter with possitive examples\nplt.scatter(df.age[df.target == 1], df.thalach[df.target == 1], c="salmon")\n\n# Scatter with negative examples\nplt.scatter(df.age[df.target == 0], df.thalach[df.target == 0], c="lightblue")\n\n# Add some helpful info\nplt.title("Heart disease in function of Age and Max Heart Rate")\nplt.xlabel("Age")\nplt.ylabel("Max Heart Rate")\nplt.legend(["disease", "No Disease"])",
"execution_count": 52,
"outputs": [
{
"output_type": "execute_result",
"execution_count": 52,
"data": {
"text/plain": "<matplotlib.legend.Legend at 0x1f4bf44be80>"
},
"metadata": {}
},
{
"output_type": "display_data",
"data": {
"text/plain": "<Figure size 720x432 with 1 Axes>",
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAmQAAAGDCAYAAACFuAwbAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzde5xcVZnv/8/TF0kHQhsIBEjTBhUMCeQCIQjKzSCZcPWgeD0SHBFnZAQdPIiOPwUP6Mwc54gM5wyHA4oXRqIIRxiHYSCOoAjGxJBgSJiAQNMQIOHSCaSBTvf6/bF3JdXdVd1rd9eqVbvq+3698kr3ql17r33rWvWstZ9lzjlEREREJJ6m2BUQERERaXRqkImIiIhEpgaZiIiISGRqkImIiIhEpgaZiIiISGRqkImIiIhEpgaZyDiZ2XQzc2bWkv5+h5ktiV2v0ZjZl83sujG+t83MbjezHjP7aaXrNsq215rZ8VXeppnZ98zsJTNbXs1th2ZmT5jZibHrIdLo1CCTmlbqw8LMzjGz3wTcpjOzt4/1/c65xc6571eyTiE4577hnDt3jG//ADAV2NM5d1YFqzWImd1gZpcXlznnZjnnfhVqm2W8G3gv0OGcW1BuITM7Pr1+Lq5e1cJJj78zs9OHlF+Zlp8TYJs1d8+n2+83s1fMbIuZrTazUzOsf9h1LDKUGmQiqUKES7y8BfhP59z22BWpkrcATzjnXh1luSXAi+n/9eI/Kdqf9D45C3gsWo0qJOM9f79zbjfgzcD/Bm4yszeHqZk0IjXIJPfMbD8z+5mZbTKzx83sgqLXFpjZ/Wb2spltNLOrzexNRa87MzvfzDYAG8zs3vSl1em34Q+V2F6zmX3LzDab2Z+AU4a8/iszOzf9+e1mdk/atbfZzJYWLTfDzO4ysxfN7BEz+2DRa6eY2ar02/hTZnZp0WsTzOxHZvZCul+/N7Op6WvtZnZ9uq9Pm9nlZtZc5rhdamY/Sn8udLsuMbOutK5/U+Z9lwFfBT6UHqNPFq9ryPoK3bi/MrP/bmb3mdlWM/t3M5tStPy7zey36f48lUYkzgM+Blycbuf2dNkdERQz2yWN1jyT/rvSzHZJXzvezLrN7CIzez49Jp8otU/p8vuZ2W3p+XjUzD6Vln8SuA44Kq3HZWXeP5Ekcng+cKCZzR/y+tlm9mR63v6/IfvRZGaXmNlj6es/MbM9ymxnspn9S3q9v5T+3FH0+mjH+uNF9Sh5joe4HXiXmU1Of/8zYA3wbNE632Zmv0zXudnMbrS0sZK+9qKZHVZ0nDfbOLqdrcr3fDHn3ADwQ2BX4MCi9f7UzJ615F6/18xmpeXlruOy+yCNSQ0yyTUzayL5wFgNTAMWAp8zs0XpIv3A54EpwFHp658Zspr3AUcCM51zx6Zlc5xzuznnljLcp4BTgXnAfJIP4XL+O/DvwGSgA/jHtN67AncB/wzsDXwE+N+FP+LAq8DZJN/GTwH+0szel762BGgH9gf2BP4C6E1f+z6wHXh7Wr+TgCzdku8G3kFynL5qZgcPXcA59zXgG8DS9Bhd77nujwKfINnfNwFfADCzTuAOkmOzFzAXeNA5dy1wI/D36XZOK7HOvwHemb5nDrAA+ErR6/uQHKtpwCeB/1XUsBjqx0A3sB/JOf2GmS1M9+8vSCMk6f6X8n7gFeCnwJ0k5490H2eSRFU+BuxbVKeCC0iuw+PS7b8E/K8y22kCvkcSteskOfdXD1mm3LGeCfwT8PF0O3uSXJcjeQ24Dfhw+vvZwA+GLGPAN9N1HkxybV4K4Jx7DPgicGPaaP0ecMNYu50j3fPF228mObZ9wJNFL91B0kDbG/gDybVLqevYYx+kETnn9E//avYf8ATJh9zLRf+2Ab9JXz8S6Bryni8B3yuzvs8Btxb97oD3DFnGAW8foU6/BP6i6PeT0ve0pL//Cjg3/fkHwLUkY4+K1/Eh4NdDyv4P8LUy27wS+Hb6858DvwVmD1lmKvA60FZU9hHgP8qs81LgR+nP09N96Ch6fTnw4dHeW+b3wvqKj8lXil7/DPBvRefr1jLbuQG4vMQ1cWL682PAyUWvLSLpWgQ4nqSx0lL0+vPAO0tsZ3+SD/JJRWXfJGk4AJxTuOZGuC7uBq4sOu6bgNb0968CPy5adiLwRtF+rAMWFr2+L8kHfstI20yXnQu8VPT7SMf6q8BNRa/tWlyPcsefpKF+P0lD8jmgDfgNcE6Z970PWDWk7DbgIZLo2i4j7M8T1N49fw7JF52X0/PSC3xwhOXfnK6zvdR1nHUf9K8x/ilCJnnwPufcmwv/GPxt9y3Afmn3xMtm9jLwZZLGCWZ2UNql86yZbSGJ7EwZsv6nMtZnvyHvebLcgsDFJNGD5ZY8HfjnRfU+cki9P0YS0cHMjjSz/0i7M3pIIjSFev+QJAJzU9pN9/dm1pqusxXYWLTO/0Pyjd3Xs0U/bwN2y/Desa57f8Y+Hmk/Bh//J9Oyghfc4HFu5fZpP+BF59zWIeuaVmLZYcxsf+AE0qgI8HNgAju7swddM865bcALRat4C3Br0XlbR9JAnFpiWxPN7P+k3Y5bgHuBN9vgrulyx3poPV4dUo+SnHO/IYlefgX4F+dcb/HrZra3md1kSTf5FuBHDL/P/i9wCPCPzrnXR9lkrd3zAA+kdZlM0rg8pvCCJcMY/jbtct5C0qikxHa99kEakxpkkndPAY8X//F2zk1yzp2cvv5PwHrgQOfc7iR/9GzIOlzGbW4kaUQUdJZb0Dn3rHPuU865/YBPk3RLvj2t9z1D6r2bc+4v07f+M8kf/f2dc+3ANYV6O+f6nHOXOedmAkeTdJ+ena7zdWBK0Tp3d87NIrxXSaI+BftkeO9TwNvKvDbauXmG5MOtoDMty+oZYA8zmzRkXU97vv/jJH9PbzezZ4E/kTTICt2WGynqGjSzNpLuwoKngMVDrocJzrlS27+IpFv5yPSaLnS5Db2uSxl07aZdiHuWX3yQH6XbHtpdCUk00ZFEbXcH/mtxfcxsN5Io7/XApVZmfJynGPf8zjc69wpJA/HjZjYvLf4ocAZwIkkUcXpaXtju0O2Ntg/SgNQgk7xbDmwxsy9akhur2cwOMbMj0tcnAVuAV8xsBvCXZde003PAW0d4/SfABWbWkY5HuqTcgmZ2lu0ccP0SyR/mfuBfgIMsGWDdmv47omjM1iSSiM1rZraA5A9+YZ0nmNmhaURkC0kXSr9zbiPJeLV/MLPdLRko/jYzO85jn8frQeBYM+s0s3aS7hdfNwInmtkHzazFzPY0s7npa6Odix8DXzGzvSwZuP5VkoZDJs65p0i6gb9pyUMTs0nGnN048jt3OBu4jKT7sPDv/cApZrYncDNwmpkdbckA88sY3Ei4BrjCzN4CkO7PGWW2NYmky+zltGFTbkxbKTcDp1ryEMWbgK/j/zlwFUnqj3tLvDaJtJvRzKYB/23I698BVrokzcovSPZ3rGLc84M4514gedDjq0XbfJ0k2jiRJCo30vpH2wdpQGqQSa455/qB00g+AB8HNpP8oWxPF/kCSWNmK0mXyYgDdlOXAt9PuxI+WOL1/0vSZbiaZPDuLSOs6wjgd2b2CknE60Ln3ONp19hJJAOlnyHpYvo7YJf0fZ8Bvm5mW0n+6P+kaJ37kHywbiHp2rqHnY2Qs0kGcT9M0gC8mWQ8UlDOubtIju0aYCVJg9P3vV3AySTRlxdJGndz0pevB2am5+L/lXj75cCKdLsPkZyPseZ7+ghJZOMZ4FaS8Xx3jfYmM3tn+r7/lUZEC/9uAx4FPuKcWwt8FriJJEq1lWQ8W6Hr7jsk18e/p+f8AZJxRqVcSTKGa3O63L/57mBaj/NJIrAbSa6Rbs/3vuicW+acKxVdugw4DOghaXDtuCfShuWfkXS7A/w1cJiZfcy33kPqEeOeL+VK4OS08f4Dki7up0nuvQeGLDvoOvbYB2lAVvreEhGRUNIuvJdJutUej10fEYlPETIRkSows9PSAfm7At8iieg9EbdWIlIr1CATEamOM0i6Q58hyVf14TLdfyLSgNRlKSIiIhKZImQiIiIikalBJiIiIhJZlpnua86UKVPc9OnTY1dDREREZFQrV67c7Jzbq9RruW6QTZ8+nRUrVsSuhoiIiMiozKzsVHvqshQRERGJTA0yERERkcjUIBMRERGJLNdjyERERMRPX18f3d3dvPbaa7GrUvcmTJhAR0cHra2t3u9Rg0xERKQBdHd3M2nSJKZPn46Zxa5O3XLO8cILL9Dd3c0BBxzg/T51WYqIiDSA1157jT333FONscDMjD333DNzJFINMhERkQahxlh1jOU4q8tSREREorj00kvZbbfd2LJlC8ceeywnnnhi7CpFowaZiIiIRPX1r389dhWiU5flCPofWknflZfTd9lF9F15Of0PrYxdJRERkaoI9Rl4xRVX8I53vIMTTzyRRx55BIBzzjmHm2++GYBLLrmEmTNnMnv2bL7whS8AsGnTJt7//vdzxBFHcMQRR3DfffcBsHz5co4++mjmzZvH0UcfvWN9a9euZcGCBcydO5fZs2ezYcMGAH70ox/tKP/0pz9Nf39/RfapEhQhK6P/oZUM3P5T6OtLCnpeSn4Hmg89PGLNREREwgr1Gbhy5UpuuukmVq1axfbt2znssMM4/PCd63vxxRe59dZbWb9+PWbGyy+/DMCFF17I5z//ed797nfT1dXFokWLWLduHTNmzODee++lpaWFu+++my9/+cv87Gc/45prruHCCy/kYx/7GG+88Qb9/f2sW7eOpUuXct9999Ha2spnPvMZbrzxRs4+++yxH6gKUoOsjIFld+y8EAv6+hhYdocaZCIiUtdCfQb++te/5r/8l//CxIkTATj99NMHvb777rszYcIEzj33XE455RROPfVUAO6++24efvjhHctt2bKFrVu30tPTw5IlS9iwYQNmRl9a56OOOoorrriC7u5uzjzzTA488ECWLVvGypUrOeKIIwDo7e1l7733HvO+VJoaZOX0vJStXEREpF4E/Awc6QnElpYWli9fzrJly7jpppu4+uqr+eUvf8nAwAD3338/bW1tg5b/7Gc/ywknnMCtt97KE088wfHHHw/ARz/6UY488kh+8YtfsGjRIq677jqccyxZsoRvfvOb496HEDSGrJz2ydnKRURE6kWgz8Bjjz2WW2+9ld7eXrZu3crtt98+6PVXXnmFnp4eTj75ZK688koefPBBAE466SSuvvrqHcsVynt6epg2bRoAN9xww47X//SnP/HWt76VCy64gNNPP501a9awcOFCbr75Zp5//nkg6R598sknx7U/lRSsQWZm+5vZf5jZOjNba2YXpuV7mNldZrYh/X9yWm5mdpWZPWpma8zssFB189G0cDEMnfKgtTUpFxERqWOhPgMPO+wwPvShDzF37lze//73c8wxxwx6fevWrZx66qnMnj2b4447jm9/+9sAXHXVVaxYsYLZs2czc+ZMrrnmGgAuvvhivvSlL/Gud71r0AD9pUuXcsghhzB37lzWr1/P2WefzcyZM7n88ss56aSTmD17Nu9973vZuHHjuPanksw5F2bFZvsC+zrn/mBmk4CVwPuAc4AXnXN/a2aXAJOdc180s5OBzwInA0cC33HOHTnSNubPn+9WrFgRpP6QDmpcdkcSom2fTNPCxRo/JiIiubRu3ToOPvhg7+X1GTg+pY63ma10zs0vtXywMWTOuY3AxvTnrWa2DpgGnAEcny72feBXwBfT8h+4pIX4gJm92cz2TdcTRfOhh+viExGRhqTPwOqqyhgyM5sOzAN+B0wtNLLS/wuPOEwDnip6W3daNnRd55nZCjNbsWnTppDVFhEREamK4A0yM9sN+BnwOefclpEWLVE2rD/VOXetc26+c27+XnvtValqioiIiEQTtEFmZq0kjbEbnXO3pMXPpePLCuPMnk/Lu4H9i97eATwTsn4iIiIitSDkU5YGXA+sc879z6KXbgOWpD8vAX5eVH52+rTlO4GemOPHRERERKolZGLYdwEfBx4yswfTsi8Dfwv8xMw+CXQBZ6Wv/SvJE5aPAtuATwSsm4iIiEjNCBYhc879xjlnzrnZzrm56b9/dc694Jxb6Jw7MP3/xXR555w73zn3Nufcoc65cPksREREpOrMjIsuumjH79/61re49NJLvd9/ww03sNdeezFv3jwOPPBAFi1axG9/+9sdr3/1q1/l7rvvrmSVq0aZ+kVERKQqdtllF2655RY2b9485nV86EMfYtWqVWzYsIFLLrmEM888k3Xr1gHw9a9/nRNPPLFS1a0qNchERERkmK6ebdzx2HPc8shG7njsObp6to17nS0tLZx33nk7MvAXe/LJJ1m4cCGzZ89m4cKFdHV1jbq+E044gfPOO49rr70WgHPOOYebb74ZgEsuuYSZM2cye/ZsvvCFLwCwadMm3v/+93PEEUdwxBFHcN999wGwfPlyjj76aObNm8fRRx/NI488AsDatWtZsGABc+fOZfbs2WzYsAGAH/3oRzvKP/3pTw+aJWCs1CATEckgxIeUSK3p6tnGqud66N0+AEDv9gFWPddTkev9/PPP58Ybb6Snp2dQ+V/91V9x9tlns2bNGj72sY9xwQUXeK3vsMMOY/369YPKXnzxRW699VbWrl3LmjVr+MpXvgLAhRdeyOc//3l+//vf87Of/Yxzzz0XgBkzZnDvvfeyatUqvv71r/PlL38ZgGuuuYYLL7yQBx98kBUrVtDR0cG6detYunQp9913Hw8++CDNzc3ceOON4z0sQQf1i4jUlcKHVH+aIbHwIQXQ2T4xYs1EKmvt5q07rvOCfpeUj/da33333Tn77LO56qqraGtr21F+//33c8stSYasj3/841x88cVe6ys1BeTuu+/OhAkTOPfccznllFM49dRTAbj77rt5+OGHdyy3ZcsWtm7dSk9PD0uWLGHDhg2YGX19fQAcddRRXHHFFXR3d3PmmWdy4IEHsmzZMlauXMkRRxwBQG9vL3vvvfewOmSlBpnIGGmet8YT8kNKpJYUImO+5Vl97nOf47DDDuMTnyifUCHJnjW6VatWDZszsqWlheXLl7Ns2TJuuukmrr76an75y18yMDDA/fffP6ghCPDZz36WE044gVtvvZUnnniC448/HoCPfvSjHHnkkfziF79g0aJFXHfddTjnWLJkCd/85jez7fQo1GUpMgb9D61k4PafJo0xgJ6XGLj9p/Q/tDJuxSSo0B9SIrWiraV086BceVZ77LEHH/zgB7n++ut3lB199NHcdNNNANx44428+93vHnU999xzD9deey2f+tSnBpW/8sor9PT0cPLJJ3PllVfy4INJ9q2TTjqJq6++esdyhfKenh6mTUtma7zhhht2vP6nP/2Jt771rVxwwQWcfvrprFmzhoULF3LzzTfz/PNJXvsXX3yRJ598cgxHYTA1yETGYGDZHZCGtHfo60vKpW6F/pASqRWzpkyieUiAqtmS8kq56KKLBj1tedVVV/G9732P2bNn88Mf/pDvfOc7Jd+3dOlS5s6dy0EHHcQ3vvENfvaznw2LkG3dupVTTz2V2bNnc9xxx+14iOCqq65ixYoVzJ49m5kzZ3LNNdcAcPHFF/OlL32Jd73rXYMG6C9dupRDDjmEuXPnsn79es4++2xmzpzJ5ZdfzkknncTs2bN573vfy8aN489jb6X6XvNi/vz5bsUKpSuT6uu77KKyr7V+7R+qWBOppqFjyCD5kJo3tV1dllLz1q1bN6zhMpKunm2s3byV3u0DtLU0MWvKJF3nGZQ63ma20jk3v9TyGkMmMhbtk3d2Vw4tl7pV+DDSh5Q0gs72ibq2q0gNsgaiQeiV07RwcTKGrLjbsrWVpoWL41VKqkIfUiISghpkDWLHIPRCAyIdhA6oUTYGhWOmBq6IiFSCGmQNYqRB6GpEjE3zoYfr2IlIrjjnvNNJyNiNZXy+Hg1qFKXGO41ULiIidWXChAm88MILY2osiD/nHC+88AITJkzI9D5FyBqFBqFLA9K4SZGdOjo66O7uZtOmTbGrUvcmTJhAR0dHpveoQdYgNAhdGo3GTYoM1traygEHHBC7GlKGuiwbRPOhh9N02lk7I2Ltk2k67Sx9MEndUvJeEckTRcgaiAahS0PRuEkRyRFFyESkPpUbH6lxkyJSgxQhE6khGoReORo3KSJ5ogaZSI3QIPTKUvJeEckTNcikahT9GZmS91aexk2KSF6oQSZVoeiPBw1CFxFpWGqQSVUo+uMhR8l7Y0c7Y29fRKTS9JSlVIeiP6NqWrgYWlsHF9bgIPQd0c7CuUujnf0PrWyI7YuIhKAGmVSHUhCMKi/Je2MnXI29fRGRENRlKVWhFAR+sgxCj9ZtFzvaGXv7IiIBqEEmVaEUBJUV9SGJ2GPdYm9fRCQANcikapSCoHJiPiQRO9qZZfsa/C8ieaEGmUgeRey2ix3t9N2+Uq2ISJ6oQSaSR5G77WJHO322r1QrIpInespSJIfykiIjKg3+F5EcUYRMSoo59qarZxtrN2+ld/sAbS1NzJoyic72iVWrZx7GHcXuNswFDf73lodrXqTeqUEmw8Qce9PVs41Vz/XQ75Lfe7cPsOq5HoBhjbIQ9czTuKPY3Ya1LvbDB3mRp2tepJ6py1KGiZl4c+3mrTsaYwX9LikfKkQ9lXS0fuQl0W5suuZFaoMiZDJcxLE3vdsH/Msz1NO7GzTQvqtLKA5FET1orJ1ITQgWITOz75rZ82b2x6KyuWb2gJk9aGYrzGxBWm5mdpWZPWpma8zssFD1Eg8Rpzlqayl9SZYs96xnoRu00KgrdIN29Wwb8zqz0NyLUtM0rZlITQjZZXkD8GdDyv4euMw5Nxf4avo7wGLgwPTfecA/BayXjCLmE3yzpkyi2QaXNVtSPpRvPbN0g4bY9yxdQv0PraTvysvpu+wi+q68vGYbbXmpp4xOT+yK1IZgXZbOuXvNbPrQYmD39Od24Jn05zOAHzjnHPCAmb3ZzPZ1zm0MVT8pL+YTfIVuRJ/uRd96ZukGDbLvnl1CeRlcnZd6ih89sStSG6o9huxzwJ1m9i2S6NzRafk04Kmi5brTMjXIIok59qazfWLZNBdj0dbSVLLxVa57tOI80y/UQiJTn7FutVBPqay8jLXTWEypZ9V+yvIvgc875/YHPg9cn5ZbiWVdiTLM7Lx0/NmKTZs2Baqm5IHv2Kws3aAhxnt5dwlFHlztve8aBC4RaCym1LtqR8iWABemP/8UuC79uRvYv2i5DnZ2Zw7inLsWuBZg/vz5JRtt0hh8IzWd7RMZ6H6CdX0t9E7YjbbXXuHg1u10tu875nUC9P3iZlj5O3ADYE1w+JG0nvKBYev07hKypmRdQ1l1vjd573uOEq7mJaKSl3rGpMis1LtqN8ieAY4DfgW8B9iQlt8G/JWZ3QQcCfRo/JiMKsPYrGm3/5RpQxKE9g+UyEnluc6+X9wMK+7fWeAGYMX99EHZRtmoHxqlGmMjlVea577nJeFqXsa65aWe0SkyK3UuZNqLHwP3A+8ws24z+yTwKeAfzGw18A2SJyoB/hX4E/Ao8H+Bz4Sql9QRz8f1MyW+9E0BsPJ3pZcrV+4jdvoBz+3nJeFqXhKe5qWe0cW+P0QCC/mU5UfKvDTsr3b6dOX5oeoi9ck7UpPhm7X3OgNEs5oWLqZr1WrWH3I0vRN3p23bFmb88bd0zpsz5nVm3b5v5CsXg8DzElHJSz0jy0tkVmSslKlfcst7bFaGMU8xx3s93Xkwa1r3oT9dR++u7axZsIjmfSfTOea1+qu79Ad5GeuWl3pGVnfXp8gQapBJrvlEarJ+s/aK/hx+5OAxZMXlY7R289YdjbGCfmti7eatFU0DMpJcRL485SWikpd61oJ6uj5FhlKDTOpeiG/Wrad8gD7wesrSV6Z5PGVUeYmo5KWeIhKWJcO38mn+/PluxYoVsashUhF3PPZc2QS2i982dVh5zFQJStMgIpKdma10zs0v9Vq1E8OKSBkHv/YCzdsHP23XvL2Pg197YdiyMZNkKkGniEjlqcuygWSJaigCUn3T7roF1z6V9XOO2fmU5epfM63nOZg1c9CyMZNkKkFnXE+sfXhYkuPpQ64P8dfVs81r7lyR0NQgaxBZkk8qUWUkPS/R0fMSHV3rvZbNVF5JStMQzRNrH2Y1u9HflkzF1ds2idXb+2Dtw2qUjUFXzzZWPddDfzpyp3f7AKue6wFQo0yqTl2WDSJL8kklqowkS+LLmEkylaAzmnV9LfS3DJ4Xtb+llXV9+m49Fms3b93RGCvod0m5SLXpLm4UWaIakSMgvnNExlbpbt2mhYsZ+PlS6O/fWdjcXDL9QcxUCbWQpiFEN1Meuul7J+zmXa6uuNHpyWapJWqQNYosyScjJqrMOkdkLMG6dYc+9VzmKeiYqRJip2kI0c2Ul276tt6t9E7cvWR5MXXF+WlraSr7ZLNItalB1iCyRDWiRkBGmiOySg0ynwhdiIHtA8vugIEhHw4DA2XX6ZskM0TkJ2aCzpG6mYY2Nnz3PS8PKsx4+AHWzD1hULdl8/Y+Zjz8AMx7x46yLMcoTyp9Lc+aMmlQwxWg2ZLyWqqnNAY1yBpElqhG1AhIgDkis/CO0IXo1g2wzrxEfrLw7WbKtO85eVCh49E18MYbw57EHfogSG9fP5gNe39vX/+wsrwIcS0XGqeV7Nqtx3tOqkMNsgaSJaoRLQISYI7ITHwjdBm7db2+MQfoKq6FyE+lowW+3UyZ9j0v80m2T6aja/3wJ3GH1LPttVfobRse5Wl77ZVxbT5m5CfUtdzZPrGiUcNauOckn9RRLrWl3FyQ45gjMhPPCF3TwsXQOvhpt3Ldur6JVLOs01vkyE+IJLKzpkyieUjwp2Q3U4Z9D3LsA/Ct595PbSg5HnHvpzaMedvREwLnJIqZm3pKzVGETKrG59t1iDkiM/GM0GXp1vX9xtx86OEMdD0+eN/nzB9f8t7IkZ8Q0QLvbqYM+57lfMaMEvnW8/n9DxzeZWmWlI/RwLI76N73bcO7S6sV+clRFDMX9ZSaowaZVEWWcRWtp3ygagP4hzn8yMFjyIrLhzDhq8EAACAASURBVPDu1vX8xtz/0EpYvWJng9ANwOoV9HceMObkvdFTVASKFvh0M2Xdd5/zWQvjg3zqmSU9hq/u9qmsWbBoxwMFvbu2s2bBIlh+JweMea3+ol/LnvJST6k96rKUqshLstnWUz4A84/aGRGzJph/1PgidJ6JVEMk720+9HCaTjtr57baJ9N02lnVG8sSMYlsiH3Py3Xc1tqcqdzH+nnHlUxKu37ecWNeZxbRr2VPeamn1B5FyKQ6cjSuotIROu9vzIGS98ZMURE7WjDQ9ThsSfJvsaWHga7Hx3csAl3HeUjnkDXqFiIx7dOdB7P2tI7B6xzXGsOIec9JfqlBJtXRwOMqvMcntU2E3m3DV9BW4kMsJ8czZgqVIEmGAxz3vKRzaGttLv10a4moW4jEtEp2K/VODTKpitiRkixCTN1U6W/MsY9nluhHtGjByt/R3Tlj+CD0cSQZDnHcsz744BtNq3Q6hyxRtxCJaes12a1IgRpkUhWxp9vxFXXqplLRsTLlMY9nXiIV3fsfVHoQOox5EHqQ456hGzTmQwVZom4h5ojUvJNS79Qgk6rJxbiKmFM3ZewOi3U88xKpWD/n2NKD0OccW5WnAr1lOO+xk476Rt1CzBGpeSel3ulKFikWceqmvCQnzUukondi6QHs5cp9hEiOmum85+ThGO/kvZHXKVJLFCETKRZx6qa8dOvWQqTCZxxVlkHovkKM98p03nPyMEeIhwpCrFOklqhBJlIsQ2LYEPLQrbvPxF14fEtvyfJq8B1HFSL1Q6jxXr7nPfbDHFlU+qGCUOsUqRXqshQpEiQxbJ15dtvrmcorzTc5a2f7ROZNbd8RuWtraWLe1PbxfaBnSHQbIomsko6K1C9FyESGCDF1U8z5Dyst+hiyDFGqSkdUmhYuZuDnS6G/f2dhc3NVx3v5RtOeWPsw6/pa6J2wG22vvcLBrduZPmvmuLbdyEIkuhUppgiZSGAhBoLHVG6sWNXGkEWcjgkA50b+vSBiPZ9Y+zCr2Y3etklgRm/bJFazG0+sfTj4tutRIdVL4UtHIdVLV0+ZVDUiY6AGmUhgeZn/0Ffsp91iPo06sOwOGBgSCRwYKHkuQ9Wzq2cbdzz2HLc8spE7HnuuZKNgXV9LyZQf6/pKd4r0P7SSvisvp++yi+i78vLcflkIZaRULyKVoi5LkdBykqrAV+yn3aI+jZpxDlGobD19k/JmmXcyZrLZvIjeTS8NQQ0ykSEqPt4rJ6kKssgyNsv3eGYZ8xTtadRAyXt9j5FvUt62115JuiuHaHvtlWFlsZPN5mF8ZdZULxpvJmOhLkuRItETf9YZ3+OZmzFPB87IVu4hyzXX29c/rKxU+cGt22nePriR1by9j4Nbtw9/c8QIbl7GV2bpptd4MxkrRchEimSJFvh+s28+9HAGuh4fPGH5nPk1FwUIwfd4rutrob+txJin3hamj2P7FY++bFifrdxDlmvON/I1fdZM3D2/Yv2kfemdOIm2bVuZsXUj0487fngF2iez5m3z6Hr7XJwZ5hydjz7I7MdWjXmffMWOzvnK0k2fl6nFIB/RyVBqMYqpBplIMc9oQZZxN/0PrYTVK3bOAOAGYPUK+jsPqP8/fp7HM8uYJ19BxkaFiCZlWOeMVfcMmjAdksjXjFX3wNyDdpT1P7SSjvv+jY4hCWT795g0bN/XnPQhnmzaFSwJATkznjxwHrztIIJfnTkaX+nbTZ+X8WaNPHbQdyxmtanLUqSYZ6qCLE9O1ttTlpmUm3JqSHnbttJPq5Ur9xHkuIdIZeF5jAA6ep5j9vI7aXu1B5yj7dUeZi+/k46e5wYtl2Xfu5p329EY27ltS8pDi53CJIDoaWE8NfLfpVp9alYRMpEi3lPTZPlmn6MoQMV5TtY+Y/W9pSM/q++Fw94xtm33vER35wzWzzmG3om707ZtCzNW/5qOrrF3L2adusirSyjDhPZNCxfTcftPB+9Da2uSvb9YhmuuTBa1suW+fLqE8jQVlK8gU3aF0MB/l2o1ihmsyW5m3zWz583sj0PKP2tmj5jZWjP7+6LyL5nZo+lri0LVS2Qk3lPTZPlm31YmBF6uvJ54HiffyE8W3TMOZ82CRfTu2p48KLBrO2sWLKJ7xti7Y7JMXeQ9YD3DtRTi+rQyiW3LlfvwHdhej1NBBZmyK4Q6jE76qtUoZsgI2Q3A1cAPCgVmdgJwBjDbOfe6me2dls8EPgzMAvYD7jazg5xzpR8pEgnIJ1VBnr7Zxxy463ucvCM/Gayfcwz9NvhPXH9LK+vnHMMBJZav9HHyHbCe9VryvT67Vq1m/SFH74wO/vG3dM6bM2zZzicf5sm3zBzcbekcnU8+DDP2G7Ssb2qSLAPbn+48mLWndQyOpI24d7VvWtc69h1yLVFjjcw8/Q2rtFqNYgZrkDnn7jWz6UOK/xL4W+fc6+kyz6flZwA3peWPm9mjwALg/lD1ExmPTEk/e8s87l6uvIJiD9z1PU4hkqj2Wuk/b6XKfY9TpuPp2SUUYt+f7jyYNa370J+OQytEB5v3nTyssTP7gTvgjdeHP2X5h1/CohN3LFdITVJ4Gra3bRKrt/fB2oeHNcp8u4RqdXD1eMS+53xFTbAcWezk1uVUewzZQcAxZnYF8BrwBefc74FpwANFy3WnZSI1yzs5acTEsLWQViBWElej9DgoK1Hme5wyHc8M5z3LMfKJ5K3dvHVHY2zH+6ypdOqF9snssfkZnp/2Nnon7s6E3q3ssfmZYfXMkprEN5FqnlJE+KqFe85XtATLNSBLcutqqXaHaQswGXgn8N+An5iZUfpvZMkBDGZ2npmtMLMVmzZtCldTkQqJmhg2JwN3QyQIzTRY3fc4ZTieIc6773HKMmj56feeWXKs3dPvPXPwezOkJvFNpFqrg6vHJSf3nNSeakfIuoFbnHMOWG5mA8CUtHz/ouU6gGdKrcA5dy1wLcD8+fPH+yCQyDCVThgYtWsgJ9M2hYgqZJruxvc4WVPppyJLpKgIcd59j1OWfV83YU/6hyzb39LKupY9B0W+skzH5NsllHVKIl9Z7uGKJwjNyT0ntafaDbL/B7wH+JWZHQS8CdgM3Ab8s5n9T5JB/QcCy6tcN5FgY1pidQ3kZuBugKjCwa+9kIx5GpJK4+DtrwBTBy3rfZwypKiAAOfd8zhlGbTsG6U6uHU7q7f3DT+epaZjwq9LKMTg6iz3cIj7PTf3nNScYA0yM/sxcDwwxcy6ga8B3wW+m6bCeANYkkbL1prZT4CHge3A+XrCUmKotzEtuRm4myHy5GvaXbfg2qcOy0M2rec5GDII3fs4BahnJp7RlyyDln2jVNNnzYS1D7Ou128CeB8hBldnuYdD3O+5ueek5oR8yvIjZV76r2WWvwK4IlR9RHzU45iWXAzczRh58kpR0fMSHT0veSeC9TpOGetZaVmiL76DlmdNmcSqjS8Negig2Q0wa0r78O13TIfNW2H7AExqp6kCaQIqnSIiyz0c6n7PxT0nNae25nIQiaxWEwbWvQxJKkMkXA1RzxBCJFKd1rWuZFLeaV3rBi3nm+w1ixAPc2S5h9tc6e7WcuUiIWnqJCmp4gNdc6JWEwaOR5aEp7GSyGaJ/GRJuOqbHDVTPX++FPqLRlQ0N1d1fJBv9MX3Hh5YdkcSSXx87eDyF58ZtJ0Q3XshHubIcg/PWP1r1hzy7uFTdv3xNzDjo2PavshYqUEmw9RjskZftZowcKyyJKmMmdAy07gbz4HtWZKjZjJ0SqFxTDEUSqZ72PN4BuneC/AwR5Z7uGP9Stj2apn5TtUgk+pSg0yGqbeB7VnVYsLAscoSgYid0LLSiXYzJUfFL6I0sOwOGBjSABkYKHuMYkUcM93DnsczSIqKQCkivMeltU+mo2v98HGGSlEhEWhgjAxTjwPbG1aWCEROElr6JlzNch17j4/KcIxCjI/ylWXffY/nPhN3KbnOcuU+YibPDbV9kbFShEyGCZWsUSLIEoHISULL5kMPZ6DrcVj5u+TpRmuCOfOHRZ6yXMfeEaUMxyhmxDHLvvt2Fz+77fWS2ypX7qP50MN5qqnNa8JyyBDF9DzuSlEhtUQNMhmmHge2N6osg+XzktCy/6GVsHrFzlQTbgBWr6C/84BBH6QhkqNmOkYRI45ZkuKCX3dxiMh5V882VrdOpj/9JOptm8Rqg6aebWNP4prxuCtFhdQKNchkmHob2J5VrHE/EHfaprxEC3wjICGSo2Y6RhmiaZW+5rIkxfUVInIeJIlr7OS9ImOkBpmUVE8D27OI+aRhLUzblItoQYYISKbkqJ7RNN9j5BtNC3LNZUyK6yNE5DxIEtfIyXtFxkpfGUSKjBR9CW2kCIAUCZCctbN9IvOmtu+I9rS1NDFvavu4G8I+SVyDXHM5OUaZkrj6Lhs5ea/IWClCJrlW8e7FiON+so7R8d33LMcoDwmBQ411802VkOV4ekXTAlxzIZLiQuUj51mibr7Lhro+Yg5lkMbg3SAzs12dc6+GrIxIFkG6eiI+aZhljI7vvmc5RnlJCBxirFuI4+ktwDUXLCluhWUZ5+e7bMzrQ2Q8Rm2QmdnRwHXAbkCnmc0BPu2c+0zoyomMJERagZhPGmaJFvjue5ZjlKeEwJUe6xbiePoKcc1lTYobU5aom++yWa4Pn8hXqBQmeZjWTKrHJ0L2bWARcBuAc261mR0btFYiPgJ09cR80jDT062++57hGDV0QuAAx9NXiGuuoc9lBt6RrwDnPS/Tmkn1eHVZOueeMrPiov5yy4pUTaDuRd9v1yG+sXpHC3z3PcMxytJlmoexZpkEOJ5ZPN15MGtP6xh8PMss63Ps6zW5c6XvOe/IV8ZUGr4JbLv3fdvweTRrcFozqQ6fu/OptNvSmdmbzOwLwLrA9RIZVcxpT2JOiwP++57lGM2aMolmG1xWqsvUe5qhHAlxPH1lOZ6+y/qeyzwJcs/5Rr4ypNLwPUfd7VNZs2ARvbu2g9mOcX7d7cMT9+ZlWjMZH58I2V8A3wGmAd3AvwMaPybRxexejD2mxHffsxwj3y7TWhhrVulISYjj6StEctR6TO4c5J4LEBn1PUfr5x03aCYFgP6WVtbPO44DxlpPyTWfBtk7nHMfKy4ws3cB94Wpkoi/aIlMI48pKZT57HuWY+TTZRp7fFKo8TQhjqePIMlRqcPkzoHSg/g8UJHlwQvfc9Q7YbfSy5Uoz8u0ZjI+Pl2W/+hZJtI4AiSfjJmUNossyTxDyMtx8hUkOWo9CnDP+Sbv9V0O/M9RW2tz6eVKlGfZvuRX2QiZmR0FHA3sZWZ/XfTS7kDpK0mkQQT5xpoxAhDrMfjok8/X2XiazMlRN740KKVFsxtg1pT2alQ1qthRIt/IqO/5zHof5WJaMxmXkbos30SSe6wFKL5CtgAfCFkpkVoXZPxa1smoIz0GH318Up2Np8lyPKd1raO/RAb+afPmlJxVoJ7kJeGr7/mMfh9JzSnbIHPO3QPcY2Y3OOeerGKdRHKh0t9Ys0QAsjwyHyJFRczxSVmOU4goYsx0JwPL7kgmDX987eDyF59piOhJlvQgPkI9nON7PutunJ+Mi8+g/m1m9j+AWcCEQqFz7j3BaiXSgLJEAAqPzBee0io8Ms/yOwc9oZWX6ZCy8D1OIaIf0RN01ll3bRZBruUGPp5Se3waZDcCS4FTSVJgLAE2hayUSKPyjbr5PjJfCykqQvA5TiGiH9ETdGboro2dvNd3+74RxyDXco66v2OfTwnP59GcPZ1z1wN9zrl7nHN/DrwzcL1EZAS+j8zHTlERVYjoR+SIim9i2tjJe323nyXZa4hrOWZy6Sxin0+pDp8IWeHr4EYzOwV4BugIVyURGU1ba3PpqXGGPDJfr1PoeAkR/ci4zkpHNXy7a7NGkyo9Ls53+1nGQoa4lmMml86iXiPdMphPg+xyM2sHLiLJP7Y78PmgtRKREYV6tL6ehEiTkGWdocbv+XTXZokmhRgX57t937GQEO5azkM6iYaOdDeQURtkzrl/SX/sAU4AMLNdQ1ZKREamR+tHFyL6kWWdMaMaWaJJIcbFGeDKlBfLMn1QI1/LDR3pbiAjNsjMbBqwL7DGOfeGme0NfA44B9gvfPVEGkuWLi49Mj+6ENEP33VmiWpUumszUwLZDOPifOtZqjFWqjzL9EFZxEqaHEojR7obSdnmtZl9DniQpJvyATNbAqwD2oD8XtkiNSrEwF0NBo7HdwqdEOdoWtc6Zi+/k7ZXe8A52l7tYfbyO5nWtW74wp5TEmWpZ4jpg0I8KJAXne0TmTe1fcfxa2tpYt7Udn0hqzMjRcjOI5lY/EUz6wQeBY51zj1QnaqJ1A+fyEKILq7Yg7tDyUPCV9+oRtZz5HMtZUkg27RwMV0lsv93zpsz5nqGGOOY5UGBqKlJAvGNiOflHpbhRmqQveacexHAOddlZv+pxphIdr6Du0MM3I09uDuEvCR89R3zlLVr0+tBgQzdkE93Hsya1n12dG8WBtY37zt5UBb8LPUMMcbRe/sNnOw1L/ewlDZSg6zDzK4q+n3v4t+dcxeEq5ZI/fD9Zh9i4G7swd0hZEmVkGWdsabQyXKOvKNUGdJzrN28ddBYM4B+axr39Vnp6YO8t18DyV5jRanycg9LaSP9pf9vwMqif0N/FxEPvt/sZ02ZRPOQx9DGO3A30zpzElkopEro3bUdzHZEdLrbp459pRH3fZ+Ju3iX+15LWRKe+q4zSz1D8L2WYyd7jTqGLSf3sJQ20uTi369mRUTqle83+xCP9WdaZw1EFnysn3Ns6VQJc44dlirBW8R9f3bb697lvtdSlvQcvikqstQzBN9rOXay1yxRqopPh5STe1hK80kMKyLjkGXgcohUFr7rDJFINYTeiaUjhuXKfcTc9yxjs/aZuAuPb+ktWT6Ub3oO7xQVNZCc1Pdajprs1TNKFSJxcF7uYSktWFY5M/uumT1vZn8s8doXzMyZ2ZT0dzOzq8zsUTNbY2aHhaqXSLXl5ZH15kMPp+m0s3Z+m26fTNNpZ9Xc2JO2117JVO4j5r77poiAMFEq3+0PjZiNVt6wPNOIjDQecKzycg9LaaNGyMzsXc65+0YrK+EG4GrgB0Peuz/wXqCrqHgxcGD670jgn9L/RUZU8ZB/IHlJ4pqHaWQObt3O6u19g7otm7f3cXDr9nGtN9a+Z4mghkg2O2vKJP7wbA/Fa2hi+PZ9I2lZtx87TUOlt+8bpQoVcczDPSyl+UTI/tGzbBDn3L3AiyVe+jZwMYPv4zOAH7jEA8CbzWxfj7pJA1PS08Y0fdZM5vAKbb1bk6SnvVuZwytMnzUzdtXGJEsENVSy2aGNqlKNrCyRvLwkcQ2xfd8oVZbjKY2hbITMzI4Cjgb2MrO/Lnppd6B0euVRmNnpwNPOudVmgwLd04Cnin7vTss2jmU70hhizhXY6LJEFUJEMafPmsn0ca2htvhGUEMkm127eWvJBtnQZesxiWuo7ftEqTQdkgw1Upflm4Dd0mWKr5AtwAeybsjMJgJ/A5xU6uUSZSUj4WZ2HsksAnR2dpZaRBpELQwybkRZkk+GGLjcyEIkm/Vdti6TuEbcfiNPli6ljZT24h4z+w1wqHPusgps623AAUAhOtYB/MHMFpBExPYvWrYDeKZMva4FrgWYP39+ueEL0gBCJFLNkzwkn1QUs/IqnWw2y7J1l8Q1w/ZD3G95GVsq1THiJ5dzrh/YoxIbcs495Jzb2zk33Tk3naQRdphz7lngNuDs9GnLdwI9zjl1V8qIQiRSzYu8JJ9UFDOOLPdGiPvIN4ls7CSuvtuPPdZNGoNPHrJVZnYb8FPg1UKhc+6Wkd5kZj8GjgemmFk38DXn3PVlFv9X4GSSCcy3AZ/wqJc0uEYO+Ucde5MhqtDoUcxYstwbIe4j3/QcsZO4+m4/9lg3aQw+DbI9gBeA9xSVOWDEBplz7iOjvD696GcHnO9RF5FBGjbkH3HsS5bkkxq4HE+We6PS91FvXz/Y8KHBvX39w8pip2nw2n7ssW7SEEZtkDnnFK0SqTURx95kiWo0chSzkbW99gq9bcMb3eNJ3htV7LFu0hB8EsNOAD4JzAImFMqdc38esF4idaXSqR9iT5GSJarRsFHMBjZj1T2sWbBoWPLeGavugbkHjXm9sRJBx77fpDH4DOT4IbAPsAi4h+QJyLHP7SDSYEIksNUUKVLLOnqeY/byO2l7tSdJ3vtqD7OX30lHz3NjXmfMRNC636QafMaQvd05d5aZneGc+76Z/TNwZ+iKidSLUKkfYo+9ESmnaeFiOm7/KR1d63cWtrYmjZoxip1CRfebhObTICvEaF82s0OAZ6GukmSLBKXUD9JoQjw9qftI6p1Pg+xaM5sM/H8k+cJ2A74atFYidUSpH6TWhRibVemIku4jqXejXsnOueuccy855+5xzr01Te56TTUqJ1IPGjmBrdS+mGOzstB9JPXO5ynLqcA3gP2cc4vNbCZw1AhJXkWkiFI/yGhiPT0I8cdm+cpyH8WaVkxkPHy6LG8AvkcyMTjAfwJLATXIRDwp9YOUE3sC9jyNzfK5j3ZMc1RIUZFOcwSoUSY1zafzfYpz7ifAAIBzbjswPN2yiIhkNlKEqhrKjcHK69iskaY5EqllPhGyV81sT5LpkihM/h20ViIR5aW7I2Y3l/jxOUexI1RZpreKfc15bb/nJbo7Z7B+zjH0Ttydtm1bmLH614NTcITatsg4+DTI/prk6cq3mdl9wF7AB4LWSiSSvHR3xO7mktH5nqPYTw/6js2Kfc35br97xuGsOeTdO2YJ6N21nTULFsHEXTkg8LZFxsNnLss/mNlxwDsAAx5xzvWN8jaRXBqpu6OWGmR5GYjdyHzPUagJ2J9Y+zDr+lronbAbba+9wsGt25k+a2bJZX3GZsW+5ny3v37OMfTb4I+2/pZW1s85ZswNstj7DvmJ3MvYlW2QmdmZZV46yMxwzt0SqE4i8ZSaQHik8khid3PJ6HzPUYincJ9Y+zCr2Y3+tjRK1DaJ1dv7YO3DZRtlo4l9zfluv9dKf6yVK6/ktkPJS+RexmekK/S0IT/fXvS7A9Qgq2MNO16ifXLpxldhDrsaEbubS0aX5RxV+incdX0tOxpjBf0trazrbRnzNCuxrznf7RvpgOchrERZpbcdSqjIvaJutaXs1eSc+0ThH/BU8e/OuT+vYh2lyvKSKDKEpoWLoXXwBxmtrUl5DVGSzNq3z8RdMpVXUu+E3TKV+4h9zfluv1RjbKTySm47mACR+x1Rt8I60qhb/0Mrx7xOGR/fGO54rmXJmVoYLxFLiDn4QlCy2dr37LbXM5VXkjmHs+ExIXNj/1Me+5rz3X6IaFbWfa94D0OAyH1exss2krF3qkvdij1eIrZKz8EXipLN1raY91GpxthI5b5iX3M+2w/1kITvvod4IrNp4eLBY8hg/JH7nIyXbSQjDeq/nZ2Rsbea2W3FrzvnTg9ZMYkn9ngJkXoQ8z5qa20uve3W5uDbji12JC9ED0OQyH1Oxss2kpEiZN8q+vkfQldEakeob5giI6m3B0li3keNfg/HjOSFioxWOnIfJOom41K2Qeacu6eaFZHaEfsbpjSeeky8GfM+0j0cT156GPIyXraRaAyZlBR7rIg0lnp9kCTmfaR72E+lI7N5ik7mZbxso1CDTESia/QHSSSOEJFZRSdlrEZtkJnZBOfca0PKpjjnNoerlojUC58IRF66eSD+WDcl86ycUJFZRSdlLHz+2v3ezN5Z+MXM3g/8NlyVRKRe+CYZjplENYvYSZOVzLOyFJmVWuLTZflR4Ltm9itgP2BP4D0hKyWSB7EjJXngG4HImkQ1VpQo9li3UMk8Q1zLebg/QkVmY+57lm0r2lpbRm2QOeceMrMrgB8CW4FjnXPdwWsmUsPq8anAELwnhM4QqYg50XL0iEqAZJ4hruW83B8hBuDH3Pcs29aE5bVn1K8BZnY98DlgNvAJ4HYzOz90xURq2UiREtmpXKRhaLnvcjBylCi0LPUMolzSznEk8wxxLefl/uhsn8i8qe07zl9bSxPzpraPq+EUc9+zbDvmfSSl+XRZ/hE41znngMfT8WT/M2y1RGpb9EhJTvhGIDJFKiJO+RI7pUHTwsV0rVrN+kOOpnfi7rRt28KMP/6WznlzxrzOENdynu6PSg/Aj7nvmbatqZNqjk+X5beH/N4DfDJYjURyIE9PBcbkmwIgU6qAiFO+xE5p8HTnwaxp3Yd+S66z3l3bWbNgEc37TqZzjOsMcS038v0RdcqsLNvW1Ek1xyftxYHAN4GZwIRCuXPurQHrJVLTYkdK8sQ3AuG7XNPCxQz8fCn09+8sbG6u2pQvMVMarN28dUdjrKDfmsb1UEGIa7mR74+8TJmlqZNqj0+X5feArwHfBk4gGUdmISslUutiR0oannMj/16nQnSHhbiWG/n+yMuUWZo6qfb4NMjanHPLzMycc08Cl5rZr0kaaSINa1rXOvYd8scM/TELbmDZHXR3HMT6OcfsHEe1+td0jDP1Qx4YUKrpWeobcpaUBiGifjEjibHTOeRlyixNnVRbfBpkr5lZE7DBzP4KeBrYO2y1RGqbHhmPp7t9KmsWLKK/pRXYOY6K5XdyQOS6hVYuDji0vJGvz0bed8k3n1GGnwMmAhcAhwMfB5aErJRIrdMj4/Gsn3fcjsZYQX9LK+vnHRepRtXjm3ajka/PRt73POl/aCV9V15O32UX0Xfl5ZptAr+nLH+f/vgKyfgxEdEj49H0TtgtU3k98R603cjXZyPve04oilla2QaZmd020hudc6eP9LqZfRc4FXjeOXdIWvY/gNOAN4DHgE84515OX/sSSTqNfuAC59ydGfZDpLr0yHg0ba3NpR/tb22OUJvq8h603cjXZyPve06EmgIs70aKkB0FPAX8GPgd2Z+svAG4GvhBUdld2KJsgAAAEb5JREFUwJecc9vN7O+ALwFfNLOZwIeBWSTzZd5tZgc55/oRqUFZHxnPy9x2edDIKRXAb9B2I6c0CLXv9XYfRaUoZkkjNcj2Ad4LfIRkgvFfAD92zq31WbFz7l4zmz6k7N+Lfn0A+ED68xnATc6510lmA3gUWADc77MtkWrL8sh4Xua2y4tGTqngq5FTGoTY93q8j6JSFLOksg2yNDr1b8C/mdkuJA2zX5nZ151z/1iBbf85sDT9eRpJA62gOy0TqVm+j4yPNL9c6D/mMbcdUsy0AnnRyCkNKr3v9XofxdLIEdyRjDioP22InULSGJsOXAXcMt6NmtnfANuBGwtFJRYr+YS3mZ0HnAfQ2TnWyUJEqic3c9uJSEm6jyqrkSO4IxlpUP/3gUOAO4DLnHN/rMQGzWwJyWD/hemE5ZBExPYvWqwDeKbU+51z1wLXAsyfP78x0nNLruVmbjs0TkaklEaemzOURo7gljPS1fRx4CDgQuC3ZrYl/bfVzLaMZWNm9mfAF4HTnXPbil66Dfiwme1iZgcABwLLx7INkVoza8okmofEgKs5t53vtgvjZAofPIVxMl0924YtK9JIYt7D0jhGGkM2rqa/mf0YOB6YYmbdJFMtfQnYBbjLzAAecM79hXNurZn9BHiYpCvzfD1hKfUiL3PbaZxM5SniWB/0IIlUg7kcT8o7f/58t2LFitjVEKkLtzyysexrZ75j3yrWpD4MfTIPkqjKvKnt+iAXaVBmttI5N7/Ua+oAFxHAf1oe8TNSxFFEZCifycVFpAE0esJVX77dkHoyT6S6+h9amesnN9UgExFA42R8ZEkQqifzRKqnHubHVINMRHZo5ISrPt+uszz4ECriqAcF6kfeIzq1pB7mx1SDTEQanu+36yzdkCEijprCp37UQ0SnptTB/JhqkElDUFRBRuL77TprN2SlI45KTVJ5vn8bKh3NqoeITk2pg/kxNZhB6p4SnsqoPL9dx04QqgcFKsv3b8OOaFbhekijWf0PrRz7xusgolNLmhYuhtbWwYU5mx9TDTKpe0o/IKMq9y16SHln+0Q6J7XtmHzXgM5JbVWLTik1SWX5/m0YKZo1Zp7XnPhpPvRwmk47a+fxa59M02ln5SraqC5LqXuKKshomhYuHjyeB0p+u+7q2UbX1l4Kn+EO6Nray54T31SVRplSk1SW99+GANEs32tO/OV9fkx9rZK6p6iCjMb323XsaGtn+0TmTW3fce22tTQp8/84eP9tCBDNqoeIjlSWImRS9xRVEB8+364Vba0vvn8bQkWzQkR09ABTfqlBJnVPCU+lUmIne1Xai8ry/dtQaDTVes4wXR/5pgaZNIRGTngqfnzSGmSNtlY6WpEl7YUiJX58/zbkYXyS0qLkmxpkItLwfJN0Zom2hohW+HaZKlLSmNSlnm9qkIlIw8uSpNM3ohIiWuHbZapISWOK3aUu46OzJCISIK1BiGiFb2JaRUoaU+zExTI+ipCJiGSYdsV3bFaIaIVvl6kiJY2ps30iL2x7gye2JLnyqp24WMZHDTIRaXhZEsP6js0KlW7Fp8tUqV4aU+zExTI++rokIg0vRGLYmElclUC2McVOXCzjowiZiAhhEsPGTLeiVC/xxEo5orGD+aYImYiIp1bLVi6Np9CtXWgEFbq1u3q2Bd+2ponLN50lERFPZqVbXuXKpfHE7DbUU5b5pi5LERFPbwy4TOXSeGJ2G2qauHxTg0xExJPSSVRevU3xFPsa0djB/NJfERERT+oSqqyY461C0TUiY6UImYiIp1BdQiGiRHmIPIWa4slnovhQ1G0oY6UGmYhIBpXuEgoxEXheJhcPMd7Kd6L4kNRtKGOhLksRkYhCPJWXlwShIdI0jDRRvEgtU4NMRCSiEFGivCQIDTLeKsBE8SLVoAaZiEhEIaJEeUkQGmSKpxITwo9YLlIjauvuFBFpMLOmTGJoWlljfFGiRn7Sr2nhYmhtHVxYYqJ4kVqjQf0iIpEZ4Ib8Ph55edIvxMMHhYH7sZ6yFBkrNchERCJau3krQ0d2DTD+1A95eNIvVNoLn4niRWqNuixFRCLKywD8EBp530WGUoNMRCSivAzAD6GR911kKF31IiIRNfIA/Ebed5GhgjXIzOy7Zva8mf2xqGwPM7vLzDak/09Oy83MrjKzR81sjZkdFqpeIiK1JEjqh5xo5H0XGSrkoP4bgKuBHxSVXQIsc879rZldkv7+RWAxcGD670jgn9L/RUTqXh4G4IfSyPsuUixYhMw5dy/w4pDiM4Dvpz9/H3hfUfkPXOIB4M1mtm+ouomIiIjUkmqnvZjqnNsI4JzbaGZ7p+XTgKeKlutOyzYOXYGZnQecB9DZ2Rm2tiIiElRXz7aaz5cmUg21Mqi/VB5EV6IM59y1zrn5zrn5e+21V+BqiYhIKIXEsIU0F4XEsF092yLXTKT6qt0ge67QFZn+/3xa3g3sX7RcB/BMlesmIiJVNFJiWJFGU+0G2W3AkvTnJcDPi8rPTp+2fCfQU+jaFBGR+qTEsCI7BRtDZmY/Bo4HpphZN/A14G+Bn5jZJ4Eu4Kx08X8FTgYeBbYBnwhVLxERqQ1tLU0lG19KDCuNKFiDzDn3kTIvLSyxrAPOD1UXERGpPftM3IXHt/SWLBdpNJpcXEREvPQ/tJKBZXdAz0vQPpmmhYvHNYn3s9tez1TuS09uSh6pQSYiIqPqf2glA7f/FPr6koKel5LfYcyNshBjyApPbhYeFig8uQmoUSY1TR31IiIyqoFld+xsjBX09SXlYxRicnE9uSl5pQiZiAjq5hpVz0vZyj3MmjJpUDQLxj+5uJ7clLxShExEGp4SlHpon5yt3EOIycVDRN1EqkERMhFpeCN1cylKlmhauHjwGDKA1laaFi4e13orPbl4iKibSDWoQSYiDU/dXKMrDNyv5FOWIRQad+p+lrxRg0xEGp4SlPppPvTwmmuAlVLpqJtINeivjYg0vFlTJtFsg8vUzSUi1aQImYg0PHVziUhsapCJiKBuLhGJS12WIiIiIpGpQSYiIiISmRpkIiIiIpFpDJmISIPTtFEi8alBJiLSwArTRhUy2xemjQLUKBOpInVZiog0sJGmjRKR6lGDTESkgWnaKJHaoC5LEZEcqfR4L00bJVIbdMeJiOREYbxXoQFVGO/V1bNtzOvUtFEitUENMhGRnAgx3quzfSLzprbviIi1tTQxb2q7BvSLVJm6LEVEciLUeC9NGyUSnyJkIiI5UW5cl8Z7ieSf7mIRkZzQeC+R+qUuSxGRnCh0Kyqrvkj9UYNMRCRHNN5LpD6py1JEREQkMjXIRERERCJTg0xEREQkMjXIRERERCJTg0xEREQkMjXIRERERCJTg0xEREQkMjXIRERERCJTg0xEREQkMjXIRERERCKL0iAzs8+b2Voz+6OZ/djMJpjZAWb2OzPbYGZLzexNMeomIiIiUm1Vb5CZ2TTgAmC+c+4QoBn4MPB3wLedcwcCLwGfrHbdRERERGKI1WXZArSZWQswEdgIvAe4OX39+8D7ItVNREREpKqq3iBzzj0NfAvoImmI9QArgZedc9vTxbqBaaXeb2bnmdkKM1uxadOmalRZREREJKgYXZaTgTOAA4D9gF2BxSUWdaXe75y71jk33zk3f6+99gpXUREREZEqidFleSLwuHNuk3OuD7gFOBp4c9qFCdABPBOhbiIiIiJVF6NB1gW808wmmpkBC4GHgf8APpAuswT4eYS6iYiIiFRdjDFkvyMZvP8H4KG0DtcCXwT+2sweBfYErq923URERERiaBl9kcpzzn0N+NqQ4j8BCyJUR0RERCQqZeoXERERiUwNMhEREZHI1CATERERiUwNMhEREZHI1CATERERiUwNMhEREZHI1CATERERiUwNMhEREZHI1CATERERiUwNMhEREZHI1CATERERiUwNMhEREZHIokwuLiIiUq+6eraxdvNWercP0NbSxKwpk+hsnxi7WlLj1CATERGpkK6ebax6rod+l/zeu32AVc/1AKhRJiNSl6WIiEiFrN28dUdjrKDfJeUiI1GDTEREpEJ6tw9kKhcpUINMRESkQtpaSn+slisXKdAVIiIiUiGzpkyi2QaXNVtSLjISDeoXERGpkMLAfT1lKVmpQSYiIlJBne0T1QCTzNRlKSIiIhKZGmQiIiIikalBJiIiIhKZGmQiIiIikalBJiIiIhKZGmQiIiIikalBJiIiIhKZGmQiIiIikalBJiIiIhKZGmQiIiIikalBJiIiIhKZGmQiIiIikZlzLnYdxszMNgFPxq5HkSnA5tiVkBHpHNU+naPap3NU+3SOatNbnHN7lXoh1w2yWmNmK5xz82PXQ8rTOap9Oke1T+eo9ukc5Y+6LEVEREQiU4NMREREJDI1yCrr2tgVkFHpHNU+naPap3NU+3SOckZjyEREREQiU4RMREREJDI1yMbIzCaY2XIzW21ma83ssrT8ADP7nZltMLOlZvam2HVtZGbWbGarzOxf0t91fmqMmT1hZg+Z2YNmtiIt28PM7krP011mNjl2PRuZmb3ZzG42s/Vmts7MjtI5qh1m9o70/in822Jmn9M5yhc1yMbudeA9zrk5wFzgz8zsncDfAd92zh0IvAR8MmIdBS4E1hX9rvNTm05wzs0tekz/EmBZep6Wpb9LPN8B/s05NwOYQ3JP6RzVCOfcI+n9Mxc4HNgG3IrOUa6oQTZGLvFK+mtr+s8B7wFuTsu/D7wvQvUEMLMO4BTguvR3Q+cnL84gOT+g8xSVme0OHAtcD+Cce8M59zI6R7VqIfCYc+5JdI5yRQ2ycUi7wx4EngfuAh4DXnbObU8X6QamxaqfcCVwMTCQ/r4nOj+1yAH/bmYrzey8tGyqc24jQPr/3tFqJ28FNgHfS7v/rzOzXdE5qlUfBn6c/qxzlCNqkI2Dc64/DRF3AAuAg0stVt1aCYCZnQo875xbWVxcYlGdn/je5Zw7DFgMnG9mx8aukAzSAhwG/397d/NqVRXGcfz7q6tgGVwKkcLq4sQgi0wQXyZSEQQhvdwGh4KI+hOaNDUcNBOCokkDBxJqRCFRBFnQxAa9gAY1qMiLXSUEoQY16Gmwl92Bg0jkrnPP/n4me++z94EHHs7hx1rrnMWbVbUD+AOnvqZSWxN7ADjeuxb9fway66AN338G7Abmk8y1W1uA873qGrl9wIEkPwPvMExVHsb+TJ2qOt+OFxnWvewCLiS5HaAdL/arcPSWgKWqOt2uTzAENHs0fR4DvqqqC+3aHq0hBrJrlGRTkvl2vgF4hGGh6ylgsT32PPB+nwrHrapeqaotVbXAMIT/aVU9i/2ZKkluTnLLlXPgUeAM8AFDf8A+dVVVy8C5JNvaSw8D32GPptGElelKsEdrin8Me42S3M+wSPJGhmB7rKoOJtnKMCJzK/A18FxV/dmvUiXZD7xcVY/bn+nS+vFeu5wDjlbVoSS3AceAu4BfgGeq6lKnMkcvyQMMP45ZD/wIvED73sMeTYUkNwHngK1Vdbm95udoDTGQSZIkdeaUpSRJUmcGMkmSpM4MZJIkSZ0ZyCRJkjozkEmSJHVmIJM0SkmeTFJJ7uldiyQZyCSN1QT4guGPgyWpKwOZpNFJspFhe60XaYEsyQ1J3khyNsnJJB8mWWz3dib5vG2A/vGV7Wgk6XoxkEkaoyeAj6rqB+BSkgeBp4AF4D7gJWAPQJJ1wOvAYlXtBN4GDvUoWtLsmvvvRyRp5kwYNpuHYSutCbAOOF5VfwPLSU61+9uA7cAnSWDYLu3X1S1X0qwzkEkalba/30PA9iTFELCKlT01r3oLcLaq9qxSiZJGyClLSWOzCBypqruraqGq7gR+An4Dnm5ryTYD+9vz3wObkvw7hZnk3h6FS5pdBjJJYzPh6tGwd4E7gCXgDPAWcBq4XFV/MYS415J8C3wD7F29ciWNQaqqdw2SNBWSbKyq39u05pfAvqpa7l2XpNnnGjJJWnEyyTywHnjVMCZptThCJkmS1JlryCRJkjozkEmSJHVmIJMkSerMQCZJktSZgUySJKkzA5kkSVJn/wBUqRtXR5xLYQAAAABJRU5ErkJggg==\n"
},
"metadata": {
"needs_background": "light"
}
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "What can we infer from this?\n\nIt seems the younger someone is, the higher their max heart rate (dots are higher on the left of the graph) and the older someone is, the more blue dots there are. But this may be because there are more dots all together on the right side of the graph (older participants)\n\nBoth of these are observational of course, but this is what we're trying to do, build an understanding of the data.\n\nLet's check the age distribution"
},
{
"metadata": {
"init_cell": true,
"trusted": true
},
"cell_type": "code",
"source": "# Check the distribution of the age column with a histogram\ndf.age.plot.hist();",
"execution_count": 53,
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": "<Figure size 432x288 with 1 Axes>",
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAD4CAYAAADrRI2NAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAQv0lEQVR4nO3dfYxldX3H8ffHXSkPha7IQjesONBsEGLkwSnF0JoKalAoYCJWa83GULcP1GpsY1djqm1qAkkr0rSxbkG7Wh94UIRKq64raJo04PDQoiwGpStuF9lRoYhaKPjtH/eMDLuzu3d299y7M7/3K5nc8/vNOfd8f1n43DO/ex5SVUiS2vGMcRcgSRotg1+SGmPwS1JjDH5JaozBL0mNWTruAoZxxBFH1MTExLjLkKQF5bbbbvteVS3fvn9BBP/ExARTU1PjLkOSFpQk356r36keSWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqzIK4clfSjibW3jiW/W6+5Jyx7Ff7jkf8ktQYg1+SGmPwS1JjDH5JaozBL0mN6TX4kyxLcm2Se5JsSvKiJIcn2ZDk3u71WX3WIEl6ur6P+C8HPldVzwNOAjYBa4GNVbUK2Ni1JUkj0lvwJzkMeDFwJUBVPV5VDwPnA+u71dYDF/RVgyRpR30e8R8HTAMfTnJHkiuSHAIcVVUPAHSvR/ZYgyRpO31eubsUOBV4c1XdkuRy5jGtk2QNsAbgmGOO6adCaS+N6+pZaW/0ecS/BdhSVbd07WsZfBA8mGQFQPe6ba6Nq2pdVU1W1eTy5Ts8JF6StId6C/6q+i7wnSTHd11nAXcDNwCru77VwPV91SBJ2lHfN2l7M/CxJAcA9wFvZPBhc3WSi4D7gQt7rkGSNEuvwV9VdwKTc/zqrD73K0naOa/claTGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNWZpn2+eZDPwQ+BJ4ImqmkxyOHAVMAFsBl5TVQ/1WYck6SmjOOJ/SVWdXFWTXXstsLGqVgEbu7YkaUTGMdVzPrC+W14PXDCGGiSpWX0HfwFfSHJbkjVd31FV9QBA93rkXBsmWZNkKsnU9PR0z2VKUjt6neMHzqiqrUmOBDYkuWfYDatqHbAOYHJysvoqUJJa0+sRf1Vt7V63AdcBpwEPJlkB0L1u67MGSdLT9Rb8SQ5JcujMMvBy4GvADcDqbrXVwPV91SBJ2lGfUz1HAdclmdnPx6vqc0m+Clyd5CLgfuDCHmuQJG2nt+CvqvuAk+bo/z5wVl/7lSTtmlfuSlJjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmP6fPSipEVoYu2NY9v35kvOGdu+FxOP+CWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1Jjeg/+JEuS3JHks1372CS3JLk3yVVJDui7BknSU0ZxxP8WYNOs9qXAZVW1CngIuGgENUiSOkMFf5Ln78mbJ1kJnANc0bUDnAlc262yHrhgT95bkrRnhj3i//sktyb5gyTL5vH+7wfeDvy0az8beLiqnujaW4Cj59owyZokU0mmpqen57FLSdKuDBX8VfWrwOuB5wBTST6e5GW72ibJucC2qrptdvdcb7+Tfa6rqsmqmly+fPkwZUqShjD0vXqq6t4k7wKmgL8BTummbt5ZVZ+eY5MzgPOSvBI4EDiMwV8Ay5Is7Y76VwJb93YQkqThDTvH/4IklzH4kvZM4Deq6oRu+bK5tqmqd1TVyqqaAF4LfKmqXg/cBLy6W201cP3eDUGSNB/DzvH/LXA7cFJVXVxVtwNU1VbgXfPc558Cb0vyTQZz/lfOc3tJ0l4YdqrnlcBPqupJgCTPAA6sqh9X1Ud3t3FV3Qzc3C3fB5y2R9VKkvbasEf8XwQOmtU+uOuTJC0wwx7xH1hVj840qurRJAf3VJMWKB/QIS0Mwx7x/yjJqTONJC8EftJPSZKkPg17xP9W4JokM6dergB+s5+SJEl9Gir4q+qrSZ4HHM/gIqx7qur/eq1MmodxTjNJC818Hrb+y8BEt80pSaiqj/RSlSSpN0MFf5KPAr8E3Ak82XUXYPBL0gIz7BH/JHBiVc15Xx1J0sIx7Fk9XwN+sc9CJEmjMewR/xHA3UluBR6b6ayq83qpSpLUm2GD/z19FiFJGp1hT+f8cpLnAquq6ovdVbtL+i1NktSHYW/L/CYGj0v8YNd1NPCZvoqSJPVn2C93L2bwYJVHYPBQFuDIvoqSJPVn2OB/rKoen2kkWcpOHpkoSdq/DRv8X07yTuCg7lm71wD/3F9ZkqS+DBv8a4Fp4C7gd4F/Yf5P3pIk7QeGPavnp8A/dD/az3nDMkm7Muy9ev6LOeb0q+q4fV6RJKlX87lXz4wDgQuBw/d9OZKkvg01x19V35/1899V9X7gzJ5rkyT1YNipnlNnNZ/B4C+AQ3upSJLUq2Gnev561vITwGbgNfu8GklS74Y9q+clfRciSRqNYad63rar31fV++bY5kDgK8DPdfu5tqreneRY4JMMvhy+HXjD7KuCJUn9GvYCrkng9xncnO1o4PeAExnM8+9srv8x4MyqOgk4GTg7yenApcBlVbUKeAi4aM/LlyTN13wexHJqVf0QIMl7gGuq6nd2tkH3mMZHu+Yzu59icDbQb3X96xnc6/8D8y1ckrRnhj3iPwaYPR3zODCxu42SLElyJ7AN2AB8C3i4qp7oVtnC4C+IubZdk2QqydT09PSQZUqSdmfYI/6PArcmuY7BUfurgI/sbqOqehI4Ocky4DrghLlW28m264B1AJOTk94JVJL2kWHP6nlvkn8Ffq3remNV3THsTqrq4SQ3A6cDy5Is7Y76VwJb51mzJGkvDDvVA3Aw8EhVXQ5s6c7O2akky7sjfZIcBLwU2ATcBLy6W201cP28q5Yk7bFhT+d8N4Mze44HPszgi9p/YvBUrp1ZAaxPsoTBB8zVVfXZJHcDn0zyl8AdwJV7Ub8kaZ6GneN/FXAKg/PuqaqtSXZ5y4aq+s9um+377wNOm2edkqR9ZNipnse70zMLIMkh/ZUkSerTsMF/dZIPMvhi9k3AF/GhLJK0IA17Vs9fdc/afYTBPP+fVdWGXiuTJPVit8HffTn7+ap6KYOLsCRJC9hup3q6i7B+nOQXRlCPJKlnw57V87/AXUk2AD+a6ayqP+qlKklSb4YN/hu7H0nSArfL4E9yTFXdX1XrR1WQJKlfu5vj/8zMQpJP9VyLJGkEdhf8mbV8XJ+FSJJGY3fBXztZliQtULv7cvekJI8wOPI/qFuma1dVHdZrdZKkfW6XwV9VS0ZViCRpNOZzP35J0iJg8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqTG/Bn+Q5SW5KsinJ15O8pes/PMmGJPd2r8/qqwZJ0o76POJ/AvjjqjoBOB24OMmJwFpgY1WtAjZ2bUnSiPQW/FX1QFXd3i3/ENgEHA2cD8w8ynE9cEFfNUiSdjSSOf4kE8ApwC3AUVX1AAw+HIAjd7LNmiRTSaamp6dHUaYkNaH34E/y88CngLdW1SO7W39GVa2rqsmqmly+fHl/BUpSY3oN/iTPZBD6H6uqT3fdDyZZ0f1+BbCtzxokSU/X51k9Aa4ENlXV+2b96gZgdbe8Gri+rxokSTva3TN398YZwBuAu5Lc2fW9E7gEuDrJRcD9wIU91iBJ2k5vwV9V/8bgoexzOauv/UqSds0rdyWpMQa/JDXG4Jekxhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSY/q8H78k7VMTa28cy343X3LOWPbbF4/4JakxBr8kNcbgl6TGGPyS1BiDX5IaY/BLUmMMfklqjMEvSY0x+CWpMb1duZvkQ8C5wLaqen7XdzhwFTABbAZeU1UP9VXDOI3rCkNJ2p0+j/j/ETh7u761wMaqWgVs7NqSpBHqLfir6ivAD7brPh9Y3y2vBy7oa/+SpLmNeo7/qKp6AKB7PXJnKyZZk2QqydT09PTICpSkxW6//XK3qtZV1WRVTS5fvnzc5UjSojHq4H8wyQqA7nXbiPcvSc0bdfDfAKzullcD1494/5LUvN6CP8kngH8Hjk+yJclFwCXAy5LcC7ysa0uSRqi38/ir6nU7+dVZfe1TkrR7++2Xu5Kkfhj8ktQYg1+SGmPwS1JjDH5JaozBL0mNMfglqTEGvyQ1xuCXpMYY/JLUGINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNcbgl6TG9PYELklaLCbW3jiW/W6+5Jxe3tcjfklqjMEvSY0x+CWpMQa/JDXG4Jekxhj8ktSYsZzOmeRs4HJgCXBFVV3S177GdRqWJO2vRn7En2QJ8HfAK4ATgdclOXHUdUhSq8Yx1XMa8M2quq+qHgc+CZw/hjokqUnjmOo5GvjOrPYW4Fe2XynJGmBN13w0yTf2cR1HAN/bx++5kLQ8/pbHDm2Pf0GNPZfu9Vs8d67OcQR/5uirHTqq1gHreisimaqqyb7ef3/X8vhbHju0Pf6Wxz7bOKZ6tgDPmdVeCWwdQx2S1KRxBP9XgVVJjk1yAPBa4IYx1CFJTRr5VE9VPZHkD4HPMzid80NV9fVR10GP00gLRMvjb3ns0Pb4Wx77z6Rqh+l1SdIi5pW7ktQYg1+SGtNE8Cc5MMmtSf4jydeT/HnXf2ySW5Lcm+Sq7svmRSnJkiR3JPls125p7JuT3JXkziRTXd/hSTZ049+Q5FnjrrMPSZYluTbJPUk2JXlRQ2M/vvs3n/l5JMlbWxn/rjQR/MBjwJlVdRJwMnB2ktOBS4HLqmoV8BBw0Rhr7NtbgE2z2i2NHeAlVXXyrHO41wIbu/Fv7NqL0eXA56rqecBJDP4baGLsVfWN7t/8ZOCFwI+B62hk/LvSRPDXwKNd85ndTwFnAtd2/euBC8ZQXu+SrATOAa7o2qGRse/C+QzGDYt0/EkOA14MXAlQVY9X1cM0MPY5nAV8q6q+TZvjf5omgh9+NtVxJ7AN2AB8C3i4qp7oVtnC4HYSi9H7gbcDP+3az6adscPgQ/4LSW7rbgUCcFRVPQDQvR45tur6cxwwDXy4m+a7IskhtDH27b0W+ES33OL4n6aZ4K+qJ7s/+VYyuFHcCXOtNtqq+pfkXGBbVd02u3uOVRfd2Gc5o6pOZXBH2IuTvHjcBY3IUuBU4ANVdQrwIxqc1ui+vzoPuGbctewvmgn+Gd2fujcDpwPLksxcxLZYbx1xBnBeks0M7oR6JoO/AFoYOwBVtbV73cZgjvc04MEkKwC6123jq7A3W4AtVXVL176WwQdBC2Of7RXA7VX1YNdubfw7aCL4kyxPsqxbPgh4KYMvuW4CXt2tthq4fjwV9qeq3lFVK6tqgsGfu1+qqtfTwNgBkhyS5NCZZeDlwNcY3CZkdbfaohx/VX0X+E6S47uus4C7aWDs23kdT03zQHvj30ETV+4meQGDL3GWMPiwu7qq/iLJcQyOgg8H7gB+u6oeG1+l/Ury68CfVNW5rYy9G+d1XXMp8PGqem+SZwNXA8cA9wMXVtUPxlRmb5KczOBL/QOA+4A30v0/wCIfO0CSgxncBv64qvqfrq+Jf/tdaSL4JUlPaWKqR5L0FINfkhpj8EtSYwx+SWqMwS9JjTH4JakxBr8kNeb/AbFph2EDrMb1AAAAAElFTkSuQmCC\n"
},
"metadata": {
"needs_background": "light"
}
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "We can see it's a normal distribution but slightly swaying to the right, which reflects in the scatter plot above.\n\nLet's keep going."
},
{
"metadata": {},
"cell_type": "markdown",
"source": "### Heart Disease Frequency per Chest Pain Type\n\nLet's try another independent variable. This time, cp(chest pain).\n\nWe'll use the same process as we did before with sex."
},
{
"metadata": {
"init_cell": true,
"trusted": true
},
"cell_type": "code",
"source": "pd.crosstab(df.cp, df.target)",
"execution_count": 54,
"outputs": [
{
"output_type": "execute_result",
"execution_count": 54,
"data": {
"text/plain": "target 0 1\ncp \n0 104 39\n1 9 41\n2 18 69\n3 7 16",
"text/html": "df.corr()
which will create a correlation matrix for us, in other words, a big table of numbers telling us how related each variable is to other."
},
{
"metadata": {
"init_cell": true,
"trusted": true
},
"cell_type": "code",
"source": "# Find the correlation between our independent variables\ncorr_matrix = df.corr()\ncorr_matrix",
"execution_count": 56,
"outputs": [
{
"output_type": "execute_result",
"execution_count": 56,
"data": {
"text/plain": " age sex cp trestbps chol fbs \\nage 1.000000 -0.098447 -0.068653 0.279351 0.213678 0.121308 \nsex -0.098447 1.000000 -0.049353 -0.056769 -0.197912 0.045032 \ncp -0.068653 -0.049353 1.000000 0.047608 -0.076904 0.094444 \ntrestbps 0.279351 -0.056769 0.047608 1.000000 0.123174 0.177531 \nchol 0.213678 -0.197912 -0.076904 0.123174 1.000000 0.013294 \nfbs 0.121308 0.045032 0.094444 0.177531 0.013294 1.000000 \nrestecg -0.116211 -0.058196 0.044421 -0.114103 -0.151040 -0.084189 \nthalach -0.398522 -0.044020 0.295762 -0.046698 -0.009940 -0.008567 \nexang 0.096801 0.141664 -0.394280 0.067616 0.067023 0.025665 \noldpeak 0.210013 0.096093 -0.149230 0.193216 0.053952 0.005747 \nslope -0.168814 -0.030711 0.119717 -0.121475 -0.004038 -0.059894 \nca 0.276326 0.118261 -0.181053 0.101389 0.070511 0.137979 \nthal 0.068001 0.210041 -0.161736 0.062210 0.098803 -0.032019 \ntarget -0.225439 -0.280937 0.433798 -0.144931 -0.085239 -0.028046 \n\n restecg thalach exang oldpeak slope ca \\nage -0.116211 -0.398522 0.096801 0.210013 -0.168814 0.276326 \nsex -0.058196 -0.044020 0.141664 0.096093 -0.030711 0.118261 \ncp 0.044421 0.295762 -0.394280 -0.149230 0.119717 -0.181053 \ntrestbps -0.114103 -0.046698 0.067616 0.193216 -0.121475 0.101389 \nchol -0.151040 -0.009940 0.067023 0.053952 -0.004038 0.070511 \nfbs -0.084189 -0.008567 0.025665 0.005747 -0.059894 0.137979 \nrestecg 1.000000 0.044123 -0.070733 -0.058770 0.093045 -0.072042 \nthalach 0.044123 1.000000 -0.378812 -0.344187 0.386784 -0.213177 \nexang -0.070733 -0.378812 1.000000 0.288223 -0.257748 0.115739 \noldpeak -0.058770 -0.344187 0.288223 1.000000 -0.577537 0.222682 \nslope 0.093045 0.386784 -0.257748 -0.577537 1.000000 -0.080155 \nca -0.072042 -0.213177 0.115739 0.222682 -0.080155 1.000000 \nthal -0.011981 -0.096439 0.206754 0.210244 -0.104764 0.151832 \ntarget 0.137230 0.421741 -0.436757 -0.430696 0.345877 -0.391724 \n\n thal target \nage 0.068001 -0.225439 \nsex 0.210041 -0.280937 \ncp -0.161736 0.433798 \ntrestbps 0.062210 -0.144931 \nchol 0.098803 -0.085239 \nfbs -0.032019 -0.028046 \nrestecg -0.011981 0.137230 \nthalach -0.096439 0.421741 \nexang 0.206754 -0.436757 \noldpeak 0.210244 -0.430696 \nslope -0.104764 0.345877 \nca 0.151832 -0.391724 \nthal 1.000000 -0.344029 \ntarget -0.344029 1.000000 ",
"text/html": "df.column.hist()
)\n * Missing values (df.info()
)\n * Outliers\n \nLet's build some models."
},
{
"metadata": {},
"cell_type": "markdown",
"source": "## 5. Modeling"
},
{
"metadata": {},
"cell_type": "markdown",
"source": "We've explored the data, now we'll try to use machine learning to predict our target variable based on the 13 independent variables.\n\nRemember our problem?\n>Given clinical parameters about a patient, can we predict whether or not they have heart disease?\n\nThat's what we'll be trying to answer.\n\nAnd remember our evaluation metric?\n\n>If we can reach 95% accuracy at predicting whether or not a patient has heart disease during the proof of concept, we'll pursure this project.\n\nThat's what we'll be aiming for.\n\nBut before we build a model, we have to get our dataset ready.\n\nLet's look at it again."
},
{
"metadata": {
"init_cell": true,
"trusted": true
},
"cell_type": "code",
"source": "df.head()",
"execution_count": 58,
"outputs": [
{
"output_type": "execute_result",
"execution_count": 58,
"data": {
"text/plain": " age sex cp trestbps chol fbs restecg thalach exang oldpeak slope \\n0 63 1 3 145 233 1 0 150 0 2.3 0 \n1 37 1 2 130 250 0 1 187 0 3.5 0 \n2 41 0 1 130 204 0 0 172 0 1.4 2 \n3 56 1 1 120 236 0 1 178 0 0.8 2 \n4 57 0 0 120 354 0 1 163 1 0.6 2 \n\n ca thal target \n0 0 1 1 \n1 0 2 1 \n2 0 2 1 \n3 0 2 1 \n4 0 2 1 ",
"text/html": "303 rows Ă— 13 columns
\ntest_size
parameter is used to tell the train_test_split()
function how much of our data we want in the test set.\n\nA rule thumb is to use 80% of your data to train on and the other 20% to test on.\n\nFor our problem, a train and test set are enough. But for other problems, you could also use validation(train/validation/test) set or ross-validation (we'll see this in a second).\n\nBut again, each problem will differ. The post, How(and why) to create a good validation set by Rachel Thomas is a good place to go to learn more.\n\nLet's look at our training data."
},
{
"metadata": {
"init_cell": true,
"trusted": true
},
"cell_type": "code",
"source": "X_train.head()",
"execution_count": 63,
"outputs": [
{
"output_type": "execute_result",
"execution_count": 63,
"data": {
"text/plain": " age sex cp trestbps chol fbs restecg thalach exang oldpeak \\n132 42 1 1 120 295 0 1 162 0 0.0 \n202 58 1 0 150 270 0 0 111 1 0.8 \n196 46 1 2 150 231 0 1 147 0 3.6 \n75 55 0 1 135 250 0 0 161 0 1.4 \n176 60 1 0 117 230 1 1 160 1 1.4 \n\n slope ca thal \n132 2 0 2 \n202 2 0 3 \n196 1 0 2 \n75 1 0 2 \n176 2 2 3 ",
"text/html": "model.fit(X_train, y_train)
and for scoring a model model.score(X_test, y_test).score()
returns the ration of predictions (1.0 = 100% correct).\n\nSince the algorithms we've chosen implement the same methods for fitting them to the data as well as evaluating them, let's put them in a dictionary and create a which fits and scores them.",
"attachments": {}
},
{
"metadata": {
"init_cell": true,
"trusted": true
},
"cell_type": "code",
"source": "# Put models in a dictionary\nmodels = {\n "KNN": KNeighborsClassifier(),\n "Logistic Regression": LogisticRegression(),\n "Random Forest": RandomForestClassifier()\n}\n\n\n# Create function to fit and score models\ndef fit_and_score(models, X_train, X_test, y_train, y_test):\n """\n Fits and evaluates given machine learning models.\n models : a dict of different Scikit-Learn machine learning models\n X_train : training data\n X_test : testing data\n y_train : labels assosciated with training data\n y_test : labels assosciated with test data\n """\n # Random seed for reproducible results\n np.random.seed(42)\n # Make a list to keep model scores\n model_scores = {}\n # Loop through models\n for name, model in models.items():\n # Fit the model to the data\n model.fit(X_train, y_train)\n # Evaluate the model and append its score to model_scores\n model_scores[name] = model.score(X_test, y_test)\n return model_scores",
"execution_count": 66,
"outputs": []
},
{
"metadata": {
"init_cell": true,
"trusted": true
},
"cell_type": "code",
"source": "model_scores = fit_and_score(models=models,\n X_train=X_train,\n X_test=X_test,\n y_train=y_train,\n y_test=y_test)\n\nmodel_scores",
"execution_count": 67,
"outputs": [
{
"output_type": "stream",
"text": "C:\Users\eyeso\Desktop\ml_course\heart-disease-project\env\lib\site-packages\sklearn\linear_model\logistic.py:938: ConvergenceWarning: lbfgs failed to converge (status=1):\nSTOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n\nIncrease the number of iterations (max_iter) or scale the data as shown in:\n https://scikit-learn.org/stable/modules/preprocessing.html\nPlease also refer to the documentation for alternative solver options:\n https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n n_iter_i = check_optimize_result(\n",
"name": "stderr"
},
{
"output_type": "execute_result",
"execution_count": 67,
"data": {
"text/plain": "{'KNN': 0.6885245901639344,\n 'Logistic Regression': 0.8852459016393442,\n 'Random Forest': 0.8360655737704918}"
},
"metadata": {}
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "Beautiful! Since our models are fitting, let's compare them visually.\n\n### Model Comparison\n\nSince we've saved our models scores to a dictionary, we can plot them by first converting them to a DataFrame."
},
{
"metadata": {
"init_cell": true,
"trusted": true
},
"cell_type": "code",
"source": "model_compare = pd.DataFrame(model_scores, index=["accuracy"])\nmodel_compare.T.plot.bar();",
"execution_count": 68,
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": "<Figure size 432x288 with 1 Axes>",
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAFPCAYAAABd3jU9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAZyUlEQVR4nO3de5RdZZ3m8e9DAqYJF7lERwiQaIMjQiIQkQYXDaIsQAFlBGHwQppL28Ko4A1tsHuBF9TG9oaj6Ra8IDK0NIqKIGEQphWUIAgiRmlAKXGGCBhQLib4mz/OqViWldQpKLLr7PP9rFWrzt5n16mHCuupt96z97tTVUiS+t86TQeQJE0OC12SWsJCl6SWsNAlqSUsdElqCQtdklpielPfePPNN685c+Y09e0lqS9df/31v66qWWM911ihz5kzhyVLljT17SWpLyX5+eqec8pFklrCQpeklrDQJaklGptDH8uKFSsYGhrikUceaTpKX5oxYwazZ89m3XXXbTqKpAZMqUIfGhpiww03ZM6cOSRpOk5fqSruvfdehoaGmDt3btNxJDVgSk25PPLII2y22WaW+eOQhM0228y/bqQBNqUKHbDMnwB/dtJgm3KFLkl6fKbUHPpoc07+xqS+3p1nvHRSX++JWLlyJdOnT+kf/5Qy2f8vTCVT6f9L9TdH6GN4+ctfzi677MJzn/tcFi1aBMCll17KzjvvzPz589lnn30A+O1vf8vChQvZcccdmTdvHhdeeCEAG2ywwarX+vKXv8xRRx0FwFFHHcVJJ53E3nvvzTve8Q6+//3vs/vuu7PTTjux++67s3TpUgAee+wx3vrWt6563Y9//ONcccUVvOIVr1j1updffjmHHHLI2vhxSOoTDhHHcPbZZ7Ppppvy8MMP8/znP5+DDz6YY489lquvvpq5c+dy3333AXD66aez8cYbc/PNNwNw//33j/vaP/3pT1m8eDHTpk3jgQce4Oqrr2b69OksXryYd73rXVx44YUsWrSIO+64gxtuuIHp06dz3333sckmm3D88cezbNkyZs2axTnnnMPChQuf1J+DpP5ioY/hYx/7GBdddBEAd911F4sWLWLPPfdcdTrgpptuCsDixYs5//zzV33dJptsMu5rH3rooUybNg2A5cuX87rXvY6f/exnJGHFihWrXvf1r3/9qimZ4e/3mte8hnPPPZeFCxdyzTXX8PnPf36S/osltYGFPsq3v/1tFi9ezDXXXMP666/PXnvtxfz581dNh4xUVWOeWTJy3+jTCGfOnLnq8amnnsree+/NRRddxJ133slee+21xtdduHAhBx54IDNmzODQQw91Dl7Sn3AOfZTly5ezySabsP766/OTn/yEa6+9lkcffZSrrrqKO+64A2DVlMu+++7LJz7xiVVfOzzl8vSnP51bb72VP/zhD6tG+qv7XltuuSUAn/3sZ1ft33ffffnUpz7FypUr/+T7bbHFFmyxxRa85z3vWTUvL0nDLPRR9ttvP1auXMm8efM49dRT2W233Zg1axaLFi3ikEMOYf78+bzqVa8C4JRTTuH+++9nhx12YP78+Vx55ZUAnHHGGbzsZS/jRS96Ec94xjNW+73e/va38853vpM99tiDxx57bNX+Y445hq233pp58+Yxf/58zjvvvFXPHXnkkWy11VZsv/32T9JPQFK/SlU18o0XLFhQo9dDv/XWW3nOc57TSJ5+ccIJJ7DTTjtx9NFHj/l8W3+GnrYodSS5vqoWjPWck7B9ZJdddmHmzJmceeaZTUeRNAVZ6H3k+uuvbzqCpCnMOXRJaokpN0Jf3Sl7Gl9T74dIa9Lm9z9gar0HMqVG6DNmzODee++1mB6H4fXQZ8yY0XQUSQ2ZUiP02bNnMzQ0xLJly5qO0peG71gkaTBNqUJfd911vduOJD1OU2rKRZL0+PVU6En2S7I0yW1JTh7j+a2TXJnkhiQ3JTlg8qNKktZk3EJPMg04C9gf2B44Isno685PAS6oqp2Aw4FPTnZQSdKa9TJC3xW4rapur6rfA+cDB486poCNuo83Bu6evIiSpF708qbolsBdI7aHgBeMOuYfgW8l+R/ATODFk5JOktSzXkboY13lM/pE8SOAz1bVbOAA4AtJ/uy1kxyXZEmSJZ6aKEmTq5dCHwK2GrE9mz+fUjkauACgqq4BZgCbj36hqlpUVQuqasGsWbMeX2JJ0ph6KfTrgG2TzE2yHp03PS8edcwvgH0AkjyHTqE7BJektWjcQq+qlcAJwGXArXTOZrklyWlJDuoe9hbg2CQ/BL4EHFVevy9Ja1VPV4pW1SXAJaP2vXvE4x8De0xuNEnSRHilqCS1hIUuSS1hoUtSS1joktQSFroktYSFLkktYaFLUktY6JLUEha6JLWEhS5JLWGhS1JLWOiS1BIWuiS1hIUuSS1hoUtSS1joktQSFroktYSFLkktYaFLUktY6JLUEha6JLXE9KYDrC1zTv5G0xGeVHee8dKmI0hqmCN0SWoJC12SWsJCl6SWsNAlqSUsdElqCQtdklrCQpeklrDQJaklLHRJagkLXZJawkKXpJaw0CWpJSx0SWoJC12SWsJCl6SWsNAlqSV6KvQk+yVZmuS2JCev5pjDkvw4yS1JzpvcmJKk8Yx7x6Ik04CzgJcAQ8B1SS6uqh+POGZb4J3AHlV1f5KnPVmBJUlj62WEvitwW1XdXlW/B84HDh51zLHAWVV1P0BV3TO5MSVJ4+ml0LcE7hqxPdTdN9J2wHZJvpPk2iT7TVZASVJverlJdMbYV2O8zrbAXsBs4P8k2aGqfvMnL5QcBxwHsPXWW084rCRp9XoZoQ8BW43Yng3cPcYxX62qFVV1B7CUTsH/iapaVFULqmrBrFmzHm9mSdIYein064Btk8xNsh5wOHDxqGO+AuwNkGRzOlMwt09mUEnSmo1b6FW1EjgBuAy4Fbigqm5JclqSg7qHXQbcm+THwJXA26rq3icrtCTpz/Uyh05VXQJcMmrfu0c8LuCk7ockqQFeKSpJLWGhS1JLWOiS1BIWuiS1hIUuSS1hoUtSS1joktQSFroktYSFLkktYaFLUktY6JLUEha6JLWEhS5JLWGhS1JLWOiS1BIWuiS1hIUuSS1hoUtSS1joktQSFroktYSFLkktYaFLUktY6JLUEha6JLWEhS5JLWGhS1JLWOiS1BIWuiS1hIUuSS1hoUtSS1joktQSFroktYSFLkktYaFLUktY6JLUEha6JLWEhS5JLWGhS1JL9FToSfZLsjTJbUlOXsNxr0xSSRZMXkRJUi/GLfQk04CzgP2B7YEjkmw/xnEbAm8EvjfZISVJ4+tlhL4rcFtV3V5VvwfOBw4e47jTgQ8Cj0xiPklSj3op9C2Bu0ZsD3X3rZJkJ2Crqvr6ml4oyXFJliRZsmzZsgmHlSStXi+FnjH21aonk3WAfwbeMt4LVdWiqlpQVQtmzZrVe0pJ0rh6KfQhYKsR27OBu0dsbwjsAHw7yZ3AbsDFvjEqSWtXL4V+HbBtkrlJ1gMOBy4efrKqllfV5lU1p6rmANcCB1XVkiclsSRpTOMWelWtBE4ALgNuBS6oqluSnJbkoCc7oCSpN9N7OaiqLgEuGbXv3as5dq8nHkuSNFFeKSpJLWGhS1JLWOiS1BIWuiS1hIUuSS1hoUtSS1joktQSFroktYSFLkktYaFLUktY6JLUEha6JLWEhS5JLWGhS1JLWOiS1BIWuiS1hIUuSS1hoUtSS1joktQSFroktYSFLkktYaFLUktY6JLUEha6JLWEhS5JLWGhS1JLWOiS1BIWuiS1hIUuSS1hoUtSS1joktQSFroktYSFLkktYaFLUktY6JLUEha6JLWEhS5JLWGhS1JL9FToSfZLsjTJbUlOHuP5k5L8OMlNSa5Iss3kR5Ukrcm4hZ5kGnAWsD+wPXBEku1HHXYDsKCq5gFfBj442UElSWvWywh9V+C2qrq9qn4PnA8cPPKAqrqyqh7qbl4LzJ7cmJKk8fRS6FsCd43YHuruW52jgW+O9USS45IsSbJk2bJlvaeUJI2rl0LPGPtqzAOTVwMLgA+N9XxVLaqqBVW1YNasWb2nlCSNa3oPxwwBW43Yng3cPfqgJC8G/h7466p6dHLiSZJ61csI/Tpg2yRzk6wHHA5cPPKAJDsBnwYOqqp7Jj+mJGk84xZ6Va0ETgAuA24FLqiqW5KcluSg7mEfAjYA/i3JjUkuXs3LSZKeJL1MuVBVlwCXjNr37hGPXzzJuSRJE+SVopLUEha6JLWEhS5JLWGhS1JLWOiS1BIWuiS1hIUuSS1hoUtSS1joktQSFroktYSFLkktYaFLUktY6JLUEha6JLWEhS5JLWGhS1JLWOiS1BIWuiS1hIUuSS1hoUtSS1joktQSFroktYSFLkktYaFLUktY6JLUEha6JLWEhS5JLWGhS1JLWOiS1BIWuiS1hIUuSS1hoUtSS1joktQSFroktYSFLkktYaFLUktY6JLUEha6JLVET4WeZL8kS5PcluTkMZ5/SpL/1X3+e0nmTHZQSdKajVvoSaYBZwH7A9sDRyTZftRhRwP3V9VfAv8MfGCyg0qS1qyXEfquwG1VdXtV/R44Hzh41DEHA5/rPv4ysE+STF5MSdJ4pvdwzJbAXSO2h4AXrO6YqlqZZDmwGfDrkQclOQ44rrv52yRLH0/oPrE5o/77n0zxb6LJ5L9df2v7v982q3uil0Ifa6Rdj+MYqmoRsKiH79n3kiypqgVN59DE+W/X3wb536+XKZchYKsR27OBu1d3TJLpwMbAfZMRUJLUm14K/Tpg2yRzk6wHHA5cPOqYi4HXdR+/EvjfVfVnI3RJ0pNn3CmX7pz4CcBlwDTg7Kq6JclpwJKquhj4DPCFJLfRGZkf/mSG7hMDMbXUUv7b9beB/feLA2lJagevFJWklrDQJaklLHRJfS3Job3sGwTOoWvgJdkD+Ec6F2xMp3NdRVXVM5vMpd4k+UFV7TzevkHQy4VFGkeSd6/h6aqq09daGD0enwFOBK4HHms4i3qUZH/gAGDLJB8b8dRGwMpmUjXLQp8cvxtj3/rAMXSWQLDQp7blVfXNpkNowu4GlgAH0fllPOxBOr+gB45TLpMsyYbAm+isQHkBcGZV3dNsKq1JkjPoXGPx78Cjw/ur6geNhVLPkqxbVSu6jzcBtqqqmxqO1QhH6JMkyabAScCRdFae3Lmq7m82lXo0vNjcyPU/CnhRA1k0cZcnOYhOn90ILEtyVVWd1HCutc5CnwRJPgQcQucKtR2r6rcNR9IEVNXeTWfQE7JxVT2Q5BjgnKr6hyQDOUL3tMXJ8RZgC+AU4O4kD3Q/HkzyQMPZNI4kGyf5cJIl3Y8zk2zcdC71bHqSZwCHAV9vOkyTLPRJUFXrVNVfVNWGVbXRiI8Nq2qjpvNpXGfTeSPtsO7HA8A5jSbSRJxGZ62p/6yq65I8E/hZw5ka4ZuiGnhJbqyq5423T5rqHKFPguGple7nB0dsP5RkIM+H7TMPJ3nh8Eb3QqOHG8yjCUiyXZIrkvyouz0vySlN52qCI/QnQffUxTcAfwtcVFVvaTiS1iDJ8+icmbQxnatE7wOOqqofNhpMPUlyFfA24NNVtVN334+qaodmk619nuUyiZI8FXgz8FrgPOD5VXVvs6k0nqq6EZifZKPutm9k95f1q+r7o+5LP5B/GVvokyDJ5nTOdHkVnTfYdqqq5c2m0niSvLqqzk1y0qj9AFTVhxsJpon6dZJn0b2PcZJXAr9qNlIzLPTJ8XNgGZ0zIx4Cjh45WrAYpqyZ3c8bNppCT9TxdK4B+a9JfgncQecCv4HjHPokSPKPdEcHY6iqOm0txpEGRpJ1gFdW1QVJZgLrVNWDTedqioU+CZLMrqqh1Tx3YFV9bW1nUu+SfBB4D50zWy4F5gNvrqpzGw2mniS5uqr2bDrHVOBpi5PjiiRzRu9MshD4yFpPo4nat/tG6MuAIWA7OmdNqD9cnuStSbZKsunwR9OhmuAc+uQ4kc7/VAdU1c8AkrwT+O/AXzeaTL1Yt/v5AOBLVXXfqDMmNLX9Tffz8SP2FTBwNyix0CdBVV2S5FHgm0leTmcd9OcDe7riYl/4WpKf0JlyeUOSWcAjDWdSj6pqbtMZpgrn0CdR92rDrwDfBQ6rKkuhT3TX0X6gqh5Lsj6wUVX936ZzaXxJ1gX+DhieR/82nYuMVjQWqiEW+iRI8iCdP/ECPAVYQedWZsP3pnSBrimse0PhS6vqwe4l4zsD7/EGF/0hyb/SmTb7XHfXa4DHquqY5lI1w0LXwEtyU1XN6/6F9X7gn4B3VdULxvlSTQFJflhV88fbNwg8y0X6442hXwr8z6r6KrBeg3k0MY91rxQFoLt87kDe7Ns3RSX4ZZJPAy8GPpDkKTjY6SdvA65Mcjudac5tgIXNRmqGUy4aeN03QfcDbq6qn3XvfrNjVX2r4WjqUfeX8LPpFPpPqurRcb6klRyFaOBV1UPAPcDwmugrGdA73vSTJO8bsblnVd1UVT8c1DIHR+gSSf4BWAA8u6q2S7IF8G9VtUfD0bQGSX5QVTuPfjzIHKFL8ArgIOB3AFV1N67AqD7km6IS/L6qKsnwetozx/sCTQlP665lnxGPVxnEZastdAku6J7l8tQkx9JZG+RfGs6k8f0Lf/xLauTjgeUcugQkeQmwL53R3mVVdXnDkaQJs9A10JJMo1PgL246i/RE+aaoBlpVPQY8lGTjprNIT5Rz6FJnqdybk1xO90wXgKp6Y3ORpImz0CX4RvdDfSjJU4HXAnMY0WmD+AvZOXRJfS3Jd4FrgZuBPwzvr6rPrfaLWspC18BLcjOd9exHWg4sobMu+r1rP5V65VWif2Sha+Al+SCd5VbP6+46nM7pi8uBF1bVgU1l0/iSnAj8Fvg6sGodl6q6r7FQDbHQNfCSfGf0ui3D+5LcXFU7NpVN40tyPPBe4Df88S+tqipvEi0NoA2SvKCqvgeQZFdgg+5zK5uLpR6dBPxlVf266SBNs9AlOAY4O8lwiT8IHNNd0+X9zcVSj24BHmo6xFTglIvU1b24KFX1m6azqHdJLgKeC1zJn86hD9xpi47QNfCSPB14H7BFVe2fZHvgr6rqMw1HU2++0v0YeI7QNfCSfBM4B/j7qpqfZDpwg2+G9o8k6wHbdTeXVtWKJvM0xbVcJNi8qi6ge1FKVa1kQO8a34+S7EXnloFnAZ8Efppkz0ZDNcQpFwl+l2Qzuqe8JdmNzjno6g9nAvtW1VKAJNsBXwJ2aTRVAyx0qXPa28XAs5J8B5gFHNpsJE3AusNlDlBVP02ybpOBmuIcugR0582fTecK0YGdg+1HSc6m89fVF7q7jgSmV9XC5lI1w0KXRunevejtVfWSprNofEmeAhwPvJDOL+SrgU9W1aNr/MIWstA1sJK8CPgUsAWd097eB3yeTim8t6r+vcF40oRZ6BpYSW4ATgSuAfanU+anVtVHGw2mnqxmlcxVqmreWowzJVjoGlijl11N8p9V9awmM6l3SbbpPjy++3nkHPpDVXXa2k/VLAtdAyvJ7cBbR+z6p5HbTrn0hzWtltlUpqZ42qIG2VXAgavZLsBC7w8zk7ywqv4DIMnuwMyGMzXCEbqkvpZkF+BsYOPurt8Af1NVP2guVTMsdEmtkGQjOp02sFf5WuiS+lr3PPT/BsxhxDTyIL4p6hy6pH73VTpr71zPiPXQB5EjdA287j0pvzh8Y4skmwBHVNUnm02mXiT5UVXt0HSOqcDlcyU4duRdiqrqfuDYBvNoYr6bxLXrccpFAlgnSar752qSacB6DWdS714IHJXkDjpTLgFqEK8UtdAluAy4IMmn6Jx//nrg0mYjaQL2bzrAVOEcugZeknWAvwX2oTO6+xbwr1XlXYv6SJKnATOGt6vqFw3GaYSFLqmvJTmIzl2LtgDuAbYBbq2q5zYarAFOuWhgJbmgqg5b3ap9gzgH26dOB3YDFlfVTkn2Bo5oOFMjLHQNsjd1P7+s0RR6olZU1b1J1kmyTlVdmeQDTYdqgqctamBV1a+6D99QVT8f+QG8oclsmpDfJNmAzp2Kvpjko8DKhjM1wjl0DbzR66J3993klEt/SDITeJjOAPVIOot0fbGq7m00WAMsdA2sJH9HZyT+LOC2EU9tCHynql7dSDA9Id3rCA6vqi82nWVts9A1sJJsDGwCvB84ecRTD1bVfc2kUq+6qyseD2wJXAxc3t1+G3BjVR3cYLxGWOgaeEmeBQxV1aNJ9gLmAZ8fuRyApp4kXwXup3NP2H3o/HJeD3hTVd3YZLamWOgaeEluBBbQWX71MjqjvWdX1QFN5tKaJbm5qnbsPp4G/BrYuqoebDZZczzLRYI/VNVK4BDgI1V1IvCMhjNpfCuGH3Sv6r1jkMscPA9dAliR5AjgtfzxnqLrNphHvZmf5IHu4wB/0d0eXpxro+aiNcNCl2AhnQW53ltVdySZC5zbcCaNo6qmNZ1hqnEOXZJawhG6BpZruahtHKFrYCV5RlX9Ksk2Yz3fXQJA6hsWuiS1hFMuGnhJHuTPp1yWA0uAt1TV7Ws/lTRxFroEHwbuBs6jc8rb4cB/AZYCZwN7NZZMmgCnXDTwknyvql4wat+1VbVbkh9W1fymskkT4ZWiEvwhyWHDN0hIctiI5xzxqG84QtfAS/JM4KPAX3V3XQOcCPwS2KWq/qOpbNJEWOiS1BJOuWjgJZmd5KIk9yT5f0kuTDK76VzSRFnoEpxDZ8ncLejcLOFr3X1SX3HKRQMvyY1V9bzx9klTnSN0CX6d5NVJpnU/Xg0M3A2G1f8coWvgJdka+ASds1wK+C7wxqr6RaPBpAmy0KUxJHlzVX2k6RzSRFjo0hiS/KKqtm46hzQRzqFLY0vTAaSJstClsfmnq/qOqy1qYK1m2Vzo3nB4LceRnjDn0CWpJZxykaSWsNAlqSUsdElqCQtdklrCQpeklrDQJakl/j84C56RlmJmsQAAAABJRU5ErkJggg==\n"
},
"metadata": {
"needs_background": "light"
}
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "Beautiful! We can't really see it from the graph but looking at the dictionary, the LogisticRegression() model performs best.\n\nSince you've found the best model. Let's take it to the boss and show her what we've found.\n\n> You: I've found it!\n Her: Nice one! What did you find?
\n You: The best algorithm for predicting heart disease is a LogisticRegression!
\n Her: Excellent. I'm supriced the hyperparameter tuning is finished by now
\n You: wonders what hyperparameter tuning is
\n You: Ummm yeah, me too, it went pretty quick.
\n Her: I'm very proud, how about you put together a classification report to show the team, and be sure to include a confusion matrix, and the cross-validated precission, recall and F1 scores. I'd also be curious to see what features are most important. Oh and don't forget to include a ** ROC curve**.
\n You: ask self, "what are those?"
\n You: Of course! I'll have to you by tomorrow.\n \nAlright, there were a few words in there which could sound made up to someone who's not a budding data scientist like yourself. But being the budding data scientist you are, you know data scientists make up words all the time.\n\nLet's briefly go through each before we see them in action.\n\n* Hyperparameter tuning - Each model you use has a series of dials you can turn to dictate how they perform. Changing these values may increase or decrease model performance.\n* Feature importance - If there are a large amount of features we're using to make predictions, do some have more importance than others? For example, for predicting heart disease, which is more important, sex or age?\n* Confusion matrix - Compares the predicted values with the true values in a tabular way, if 100% correct, all values in the matrix will be top left to bottom right (diagnol line).\n* Cross-validation - Splits your dataset into multiple parts and train and tests your model on each part and evaluates performance as an average.\n* Precision - Proportion of true positives over total number of samples. Higher precision leads to less false positives.\n* Recall - Proportion of true positives over total number of true positives and false negatives. Higher recall leads to less false negatives.\n* F1 score - Combines precision and recall into one metric. 1 is best, 0 is worst.\n* Classification report - Sklearn has a build-in function called
classification_report()
which returns some of the main classification metrics such as precission, recal and f1-score.\n* ROC Curve - Receiver Operating Characteristic is a plot of true positive rate versus false positive rate.\n* Area Under Curve(AUC) - the area underneath the ROC curve. A perfect model achieves a score of 1.0.\n"
},
{
"metadata": {},
"cell_type": "markdown",
"source": "## Hyperparameter tuning and cross-validation\n\nTo cook your favourite dish, you know to set the oven to 180 degrees and turn the grill on. But when your roommate cooks their favourite dish, they set use 200 degrees and the fan-forced mode. Same oven, different settings, different outcomes.\n\nThe same can be done for machine learning algorithms. You can use the same algorithms but change the settings (hyperparameters) and get different results.\n\nBut just like turning the oven up too high can burn your food, the same can happen for machine learning algorithms. You change the settings and it works so well, it overfits (does too well) the data.\n\nWe're looking for the goldilocks model. One which does well on our dataset but also does well on unseen examples.\n\nTo test different hyperparameters, you could use a validation set but since we don't have much data, we'll use cross-validation.\n\nThe most common type of cross-validation is k-fold. It involves splitting your data into k-fold's and then testing a model on each. For example, let's say we had 5 folds (k = 5). This what it might look like.\n\n\n
LogisticRegression
or the RandomForestClassifier
did.\n\nBecause of this, we'll discard KNN and focus on the other two.\n\nWe've tuned KNN by hand but let's see how we can LogisticsRegression
and RandomForestClassifier
using RandomizedSearchCV.\n\nInstead of us having to manually try different hyperparameters by hand, RandomizedSearchCV
tries a number of different combinations, evaluates them and saves the best.\n\n### Tuning models with with RandomizedSearchCV\nReading the Scikit-Learn documentation for LogisticRegression, we find there's a number of different hyperparameters we can tune.\n\nThe same for RandomForestClassifier.\n\nLet's create a hyperparameter grid (a dictionary of different hyperparameters) for each and then test them out."
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "# Create a hyperparameter grid for LogisticRegression\nlog_reg_grid = {"C": np.logspace(-4, 4, 20),\n "solver": ["liblinear"]}\n\n# Create a hyperparameter grid for RandomForestClassifier\nrf_grid = { "n_estimators": np.arange(10, 1000, 50),\n "max_depth": [None, 3, 5, 10],\n "min_samples_split": np.arange(2, 20, 2),\n "min_samples_leaf": np.arange(1, 20, 2)}",
"execution_count": 73,
"outputs": []
},
{
"metadata": {},
"cell_type": "markdown",
"source": "Now let's use RandomizedSearchCV
to try and tune our LogesticRegression
model.\n\nWe'll pass it the different hyperparameters from log_reg_grid
as well as set n_iter = 20
. This means, RandomizedSearchCV
will try 20 different combinations of hyperparameters from log_reg_grid
and save the best ones."
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "# Tune LogisticRegression\n\n# Setup random seed\nnp.random.seed(42)\n\n# Setup random hyperparameter search for LogisticRegression\nrs_log_reg = RandomizedSearchCV(LogisticRegression(),\n param_distributions=log_reg_grid,\n cv=5,\n n_iter=20,\n verbose=True)\n\n# Fit random hyperparameter search model\nrs_log_reg.fit(X_train, y_train);",
"execution_count": 74,
"outputs": [
{
"output_type": "stream",
"text": "Fitting 5 folds for each of 20 candidates, totalling 100 fits\n",
"name": "stdout"
},
{
"output_type": "stream",
"text": "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n[Parallel(n_jobs=1)]: Done 100 out of 100 | elapsed: 0.6s finished\n",
"name": "stderr"
}
]
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "rs_log_reg.best_params",
"execution_count": 75,
"outputs": [
{
"output_type": "execute_result",
"execution_count": 75,
"data": {
"text/plain": "{'solver': 'liblinear', 'C': 0.23357214690901212}"
},
"metadata": {}
}
]
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "rs_log_reg.score(X_test, y_test)",
"execution_count": 76,
"outputs": [
{
"output_type": "execute_result",
"execution_count": 76,
"data": {
"text/plain": "0.8852459016393442"
},
"metadata": {}
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "Now we've tuned LogisticRegression
using RandomizedSearchCV
, we'll do the same for RandomForestClassifier
."
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "# Tune RandomForestClassifier\n\n# Setup random seed\nnp.random.seed()\n\n#Setup random hyperparameter search for RandomForestClassifier\nrs_rf = RandomizedSearchCV(RandomForestClassifier(),\n param_distributions=rf_grid,\n cv=5,\n n_iter=20,\n verbose=True)\n\n# Fit random hyperparameter search model\nrs_rf.fit(X_train, y_train);",
"execution_count": 77,
"outputs": [
{
"output_type": "stream",
"text": "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
"name": "stderr"
},
{
"output_type": "stream",
"text": "Fitting 5 folds for each of 20 candidates, totalling 100 fits\n",
"name": "stdout"
},
{
"output_type": "stream",
"text": "[Parallel(n_jobs=1)]: Done 100 out of 100 | elapsed: 2.2min finished\n",
"name": "stderr"
}
]
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "rs_rf.best_params",
"execution_count": 78,
"outputs": [
{
"output_type": "execute_result",
"execution_count": 78,
"data": {
"text/plain": "{'n_estimators': 60,\n 'min_samples_split': 2,\n 'min_samples_leaf': 11,\n 'max_depth': 3}"
},
"metadata": {}
}
]
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "rs_rf.score(X_test, y_test)",
"execution_count": 79,
"outputs": [
{
"output_type": "execute_result",
"execution_count": 79,
"data": {
"text/plain": "0.8852459016393442"
},
"metadata": {}
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "Excellent! Tuning the hyperparameters for each model saw a slight performance boost in both the RandomForestClassifier
and LogisticRegression
.\n\nThis is akin to tuning the settings on your oven and getting it to cook your favourite dish just right.\n\nBut since LogisticRegression
is pulling out in front, we'll try tuning it further with GridSearchCV
."
},
{
"metadata": {},
"cell_type": "markdown",
"source": "### Tuning a model with GridSearchCV\n\nThe difference between RandomizedSearchCV
and GridSearchCV
is where RandomizedSearchCV
searches over a grid of hyperparameters performing n_iter
combinations, GridSearchCV
will test every single possible combination.\n\nIn short:\n* RandomizedSearchCV
- tries n_iter
combinations of hyperparameters and saves the best.\n* GridSearchCV
- tries every single combination of hyperparameters and saves the best.\n\nLet's see it in action."
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "# Different hyperparameters for our LogisticRegression model\nlog_reg_grid = {"C": np.logspace(-4, 4, 30),\n "solver": ["liblinear"]}\n\n# Setup grid hyperparameter search for LogisticRegression\ngs_log_reg = GridSearchCV(LogisticRegression(),\n param_grid=log_reg_grid,\n cv=5,\n verbose=True)\n\n# Fit grid hyperparameter search model\ngs_log_reg.fit(X_train, y_train);",
"execution_count": 80,
"outputs": [
{
"output_type": "stream",
"text": "Fitting 5 folds for each of 30 candidates, totalling 150 fits\n",
"name": "stdout"
},
{
"output_type": "stream",
"text": "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n[Parallel(n_jobs=1)]: Done 150 out of 150 | elapsed: 0.9s finished\n",
"name": "stderr"
}
]
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "# Check the best hyperparameters\ngs_log_reg.best_params_",
"execution_count": 81,
"outputs": [
{
"output_type": "execute_result",
"execution_count": 81,
"data": {
"text/plain": "{'C': 0.20433597178569418, 'solver': 'liblinear'}"
},
"metadata": {}
}
]
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "# Evaluate the grid search LogisticRegression model\ngs_log_reg.score(X_test, y_test)",
"execution_count": 82,
"outputs": [
{
"output_type": "execute_result",
"execution_count": 82,
"data": {
"text/plain": "0.8852459016393442"
},
"metadata": {}
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "In this case, we get the same results as before since our grid only has a maximum of 20 different hyperparameter combinations.\n\nNote: If there are a large amount of hyperparameters combinations in your grid, GridSearchCV
may take a long time to try them all out. This is why it's a good idea to start with RandomizedSearchCV
, try a certain amount of combinations and then use GridSearchCV
to refine them."
},
{
"metadata": {},
"cell_type": "markdown",
"source": "## Evaluating a classification model, beyond accuracy\n\nNow we've got a tuned model, let's get some of the metrics we discussed before.\n\nWe want:\n* ROC curve and AUC score - plot_roc_curve().\n* Confusion matrix - confusion_matrix().\n* Classification report - classification_report().\n* Precision - precision_score().\n* Recall - recall_score().\n* F1-score - f1_score().\n\nLuckily, Scikit-Learn has these all built-in.\n\nTo access them, we'll have to use our model to make predictions on the test set. You can make predictions by calling predict()
on a trained model and passing it the data you'd like to predict on.\n\nWe'll make predictions on the test data."
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "# Make predictions with tuned model\ny_preds = gs_log_reg.predict(X_test)",
"execution_count": 83,
"outputs": []
},
{
"metadata": {},
"cell_type": "markdown",
"source": "Let's see them."
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "y_preds",
"execution_count": 84,
"outputs": [
{
"output_type": "execute_result",
"execution_count": 84,
"data": {
"text/plain": "array([0, 1, 1, 0, 1, 1, 1, 0, 0, 1, 1, 0, 1, 0, 1, 1, 1, 0, 0, 0, 1, 0,\n 0, 1, 1, 1, 1, 1, 0, 1, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1,\n 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0], dtype=int64)"
},
"metadata": {}
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "They look like our original test data labels, except different where the model has predicted wrong."
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "np.array(y_test)",
"execution_count": 85,
"outputs": [
{
"output_type": "execute_result",
"execution_count": 85,
"data": {
"text/plain": "array([0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 0, 1, 0,\n 0, 1, 1, 0, 0, 1, 0, 1, 1, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1,\n 1, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0], dtype=int64)"
},
"metadata": {}
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "Since we've got our prediction values we can find the metrics we want.\n\nLet's start with the ROC curve and AUC scores.\n\n### ROC Curve and UAC Scores\n\nWhat's a ROC curve?\n\nIt's a way of understanding how your model is performing by comparing the true positive rate to the false positive rate.\n\nIn our case...\n> To get an appropiate example in a real-world problem, consider a diagnostic test that seeks to determine whether a person has a certain disease. A false positive in this case occurs when the person tests positive, but does not actually have the disease.\nA false negeative, on the other hand, occurs when the person tests negative, suggesting they are healthy, when they actually do have the disease.\n\nScikit-Learn implements a function plot_roc_curve
which can help us create a ROC curve as well as calculate the area under the curve (AUC) metric.\n\nReading the documentation on the plot_roc_curve function we can see it takes (estimator, X, y)
as inputs. Where estimator
is a fitted machine learning model and X and y are the data you'd like to test on.\n\nIn our case, we'll use the GridSearchCV version of our LogesticRegression
estimator, gs_log-reg
as well as the test data, X_test
and y_test
."
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "# Plot ROC curve and calculate AUC metric\nplot_roc_curve(gs_log_reg, X_test, y_test)",
"execution_count": 86,
"outputs": [
{
"output_type": "execute_result",
"execution_count": 86,
"data": {
"text/plain": "<sklearn.metrics.plot.roc_curve.RocCurveDisplay at 0x1f4beaa6ee0>"
},
"metadata": {}
},
{
"output_type": "display_data",
"data": {
"text/plain": "<Figure size 432x288 with 1 Axes>",
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3de3wV1fnv8c8jikAlWkE5aIQEQSFAQIgRbFVA6Q+shXqHeuUnUqm3n5Qe7bGt1F6OpbbeSqFUraLctFUBRTlVQesFBQQRQrWIqIG8MCKIVe48548Z0k2ys7NDMnuTzPf9euXlnpm1Z54V4n72WmtmLXN3REQkvg7KdgAiIpJdSgQiIjGnRCAiEnNKBCIiMadEICIScwdnO4Daat26tefl5WU7DBGRBmXJkiWfuvtRyY41uESQl5fH4sWLsx2GiEiDYmYfVndMXUMiIjGnRCAiEnNKBCIiMadEICISc0oEIiIxF1kiMLMHzewTM1tRzXEzs3vNbLWZLTezXlHFIiIi1YuyRfAQMCjF8cFAp/BnFDAxwlhERKQakT1H4O4vm1leiiJDgSkezIO90MyOMLO27l4WVUwSP9Pe+IhZy9ZlOwyRelFwTA63fadrvZ83m2MExwIfJ2yXhvuqMLNRZrbYzBaXl5dnJDhpHGYtW0dJ2ZZshyFyQMvmk8WWZF/SVXLcfTIwGaCoqEgr6UitFLTNYeb3+2Y7DJEDVjZbBKXAcQnbucD6LMUiIhJb2UwEs4HLw7uH+gCfa3xARCTzIusaMrPpQD+gtZmVArcBhwC4+yRgLnA2sBr4ChgRVSwiIlK9KO8aGl7DcQeujer6cuDI5p07JWVbKGibk5VrizQUerJYIpfNO3cK2uYwtGfSm9FEJNTg1iOQhkl37ogcuNQiEBGJOSUCEZGYUyIQEYk5JQIRkZjTYHED1lAmVNMtnCIHNrUIGrCGMqGabuEUObCpRdDA6bZMEakrtQhERGJOiUBEJOaUCEREYk5jBAeY2twJpLtxRKQ+qEVwgKnNnUC6G0dE6oNaBAcg3QkkIpmkFoGISMwpEYiIxJwSgYhIzCkRiIjEnBKBiEjMKRGIiMScEoGISMwpEYiIxJwSgYhIzCkRiIjEnBKBiEjMKRGIiMScEoGISMwpEYiIxJwSgYhIzCkRiIjEXKSJwMwGmdm7ZrbazG5Jcrydmc03s6VmttzMzo4yHhERqSqyRGBmTYAJwGCgABhuZgWViv0EeMzdTwKGAX+MKh4REUkuyhZBMbDa3de4+w5gBjC0UhkH9q6+fjiwPsJ4REQkiSgTwbHAxwnbpeG+ROOAS82sFJgLXJ/sRGY2yswWm9ni8vLyKGIVEYmtKBOBJdnnlbaHAw+5ey5wNvCImVWJyd0nu3uRuxcdddRREYQqIhJfUSaCUuC4hO1cqnb9XAU8BuDurwPNgNYRxiQiIpVEmQgWAZ3MLN/MmhIMBs+uVOYj4EwAM+tCkAjU9yMikkGRJQJ33wVcB8wDVhHcHbTSzG43syFhsR8CV5vZ28B04Ep3r9x9JCIiETo4ypO7+1yCQeDEfT9LeF0CfCPKGEREJDU9WSwiEnNKBCIiMadEICISc0oEIiIxp0QgIhJzSgQiIjGnRCAiEnNKBCIiMadEICISc2k9WRzOFdTO3VdHHE+jNe2Nj5i1bF2N5UrKtlDQNqfGciIi9aXGFoGZfRt4B/h7uN3TzJ6MOrDGZtaydZSUbamxXEHbHIb2rLxsg4hIdNJpEdwOnALMB3D3ZWbWMdKoGqmCtjnM/H7fbIchIrKPdMYIdrr75kr7NEOoiEgjkU6LYJWZXQQcZGb5wI3AwmjDEhGRTEmnRXAd0BvYAzwBbCNIBiIi0gik0yL4L3e/Gbh57w4zO48gKYiISAOXTovgJ0n23VrfgYiISHZU2yIws/8CBgHHmtnvEw7lEHQTiYhII5Cqa+gTYAXBmMDKhP1fALdEGZSIiGROtYnA3ZcCS81sqrtvy2BMIiKSQekMFh9rZr8CCoBme3e6+wmRRSUiIhmTzmDxQ8BfAAMGA48BMyKMSUREMiidRNDC3ecBuPv77v4ToH+0YYmISKak0zW03cwMeN/MrgHWAUdHG5aIiGRKOongJuAw4AbgV8DhwH9HGZSIiGROjYnA3d8IX34BXAZgZrlRBiUiIpmTcozAzE42s++aWetwu6uZTUGTzomINBrVJgIz+7/AVOAS4Dkzu5VgTYK3Ad06KiLSSKTqGhoK9HD3rWZ2JLA+3H43M6GJiEgmpOoa2ubuWwHc/TPgn0oCIiKNT6oWQQcz2zvVtAF5Cdu4+3k1ndzMBgH3AE2A+939jiRlLgLGEax69ra7fy/98EVEpK5SJYLzK23/oTYnNrMmwARgIFAKLDKz2e5eklCmE/Bj4BvuvsnM9HyCiEiGpZp07oU6nrsYWO3uawDMbAbBuENJQpmrgQnuvim85id1vKaIiNRSOlNM7K9jgY8TtkvDfYlOAE4ws1fNbGHYlVSFmY0ys8Vmtri8vDyicEVE4imdJ4v3lyXZ50mu3wnoB+QC/zCzbu6+eZ83uU8GJgMUFRVVPke9m/bGR8xatq5ez1lStoWCtjn1ek4RkfqQdovAzA6t5blLgeMStnMJbkGtXGaWu+909w+AdwkSQ1bNWraOkrIt9XrOgrY5DO1ZuUEkIpJ9NbYIzKwYeIBgjqF2ZtYDGOnu19fw1kVAJzPLJ5iobhhQ+Y6gp4DhwEPh08snAGtqV4VoFLTNYeb3+2Y7DBGRyKXTIrgXOAfYCODub5PGNNTuvgu4DpgHrAIec/eVZna7mQ0Ji80DNppZCcFTyz9y9421r4aIiOyvdMYIDnL3D4OZqCvsTufk7j4XmFtp388SXjswJvwREZEsSCcRfBx2D3n4bMD1wHvRhiUiIpmSTtfQaIJv7O2ADUCfcJ+IiDQC6bQIdrn7sMgjERGRrEinRbDIzOaa2RVm1jLyiEREJKNqTATufjzwS6A38I6ZPWVmaiGIiDQSaT1Q5u6vufsNQC9gC8GCNSIi0gjUmAjM7DAzu8TM5gBvAuXAqZFHJiIiGZHOYPEKYA4w3t3/EXE8IiKSYekkgg7uvifySEREJCuqTQRm9jt3/yHwNzOrMuNnOiuUiYjIgS9Vi2Bm+N9arUwmIiINS6oVyt4MX3Zx932SgZldB9R1BTMRETkApHP76H8n2XdVfQciIiLZkWqM4GKCNQTyzeyJhEMtgc3J3yUiIg1NqjGCNwnWIMgFJiTs/wJYGmVQIiKSOanGCD4APgCez1w4IiKSaam6hl5y9zPMbBP7LjpvBGvKHBl5dCIiErlUXUN7l6NsnYlAREQkO6q9ayjhaeLjgCbuvhvoC3wf+FoGYhMRkQxI5/bRpwiWqTwemAJ0AaZFGpWIiGRMOolgj7vvBM4D7nb364Fjow1LREQyJZ1EsMvMLgQuA54O9x0SXUgiIpJJ6T5Z3J9gGuo1ZpYPTI82LBERyZQap6F29xVmdgPQ0cw6A6vd/VfRhyYiIplQYyIws9OAR4B1BM8Q/C8zu8zdX406OBERiV46C9PcBZzt7iUAZtaFIDEURRmYiIhkRjpjBE33JgEAd18FNI0uJBERyaR0WgRvmdmfCFoBAJegSedERBqNdBLBNcANwP8mGCN4GbgvyqBERCRzUiYCM+sOHA886e7jMxOSiIhkUrVjBGb2fwiml7gE+LuZJVupTEREGrhUg8WXAIXufiFwMjC6tic3s0Fm9q6ZrTazW1KUu8DM3Mx0J5KISIalSgTb3f1LAHcvr6FsFWbWhGBls8FAATDczAqSlGtJMAbxRm3OLyIi9SPVGEGHhLWKDTg+ce1idz+vhnMXEzyFvAbAzGYAQ4GSSuV+AYwHxtYmcBERqR+pEsH5lbb/UMtzHwt8nLBdCpySWMDMTgKOc/enzazaRGBmo4BRAO3atatlGCIikkqqNYtfqOO5LdlpKw6aHUTw1PKVNZ3I3ScDkwGKioq8huIiIlILter3r6VSgtXN9soF1idstwS6AQvMbC3QB5itAWMRkcyKMhEsAjqZWb6ZNQWGAbP3HnT3z929tbvnuXsesBAY4u6LI4xJREQqSefJYgDM7FB3355ueXffZWbXAfOAJsCD7r7SzG4HFrv77NRnqF/T3viIWcvWpVW2pGwLBW1zIo5IROTAkM401MXAA8DhQDsz6wGMDJesTMnd5wJzK+37WTVl+6UT8P6atWxd2h/wBW1zGNpTq3GKSDyk0yK4FziH4Clj3P1tM+sfaVQRKWibw8zv9812GCIiB5R0xggOcvcPK+3bHUUwIiKSeem0CD4Ou4c8fFr4euC9aMMSEZFMSadFMBoYA7QDNhDc5lnreYdEROTAlM7i9Z8Q3PopIiKNUDp3Df2ZhCeC93L3UZFEJCIiGZXOGMHzCa+bAeey7xxCIiLSgKXTNTQzcdvMHgH+HllEIiKSUfszxUQ+0L6+AxERkexIZ4xgE/8ZIzgI+AyodrUxERFpWGpavN6AHsDeSXr2uLumgRYRaURSdg2FH/pPuvvu8EdJQESkkUlnjOBNM+sVeSQiIpIV1XYNmdnB7r4L+CZwtZm9D3xJsPKYu7uSg4hII5BqjOBNoBfw3QzFIiIiWZAqERiAu7+foVhERCQLUiWCo8xsTHUH3f33EcQjIiIZlioRNAEOI2wZiIhI45QqEZS5++0Zi0RERLIi1e2jagmIiMRAqkRwZsaiEBGRrKk2Ebj7Z5kMREREsmN/Zh8VEZFGRIlARCTmlAhERGJOiUBEJOaUCEREYk6JQEQk5pQIRERiTolARCTmIk0EZjbIzN41s9VmVmXBezMbY2YlZrbczF4ws/ZRxiMiIlVFlgjMrAkwARgMFADDzaygUrGlQJG7FwJ/BcZHFY+IiCQXZYugGFjt7mvcfQcwAxiaWMDd57v7V+HmQiA3wnhERCSJKBPBscDHCdul4b7qXAU8m+yAmY0ys8Vmtri8vLweQxQRkSgTQbJprD1pQbNLgSLgt8mOu/tkdy9y96KjjjqqHkMUEZFUC9PUVSlwXMJ2LrC+ciEzOwu4FTjD3bdHGI+IiCQRZYtgEdDJzPLNrCkwDJidWMDMTgL+BAxx908ijEVERKoRWSJw913AdcA8YBXwmLuvNLPbzWxIWOy3BOsiP25my8xsdjWnExGRiETZNYS7zwXmVtr3s4TXZ0V5fRERqZmeLBYRiTklAhGRmFMiEBGJOSUCEZGYUyIQEYk5JQIRkZhTIhARiTklAhGRmFMiEBGJOSUCEZGYUyIQEYk5JQIRkZhTIhARiTklAhGRmFMiEBGJOSUCEZGYUyIQEYk5JQIRkZhTIhARiTklAhGRmFMiEBGJuYOzHYDIgWznzp2Ulpaybdu2bIcikpZmzZqRm5vLIYcckvZ7lAhEUigtLaVly5bk5eVhZtkORyQld2fjxo2UlpaSn5+f9vvUNSSSwrZt22jVqpWSgDQIZkarVq1q3YJVIhCpgZKANCT78/eqRCAiEnNKBCIHuA0bNvC9732PDh060Lt3b/r27cuTTz6ZtOz69eu54IILkh7r168fixcvBuDBBx+ke/fuFBYW0q1bN2bNmhVZ/AB5eXl8+umnSY89++yzFBUV0aVLFzp37szYsWNZsGABffv23afcrl27aNOmDWVlZVXOcffddzNlypR9yrZu3Zof//jHKeNYsGAB55xzTspY6mrJkiV0796djh07csMNN+DuVcps2rSJc889l8LCQoqLi1mxYgUQdE0WFxfTo0cPunbtym233VbxnmHDhvGvf/2rzvEBweBCQ/rp3bu374+LJr3mF016bb/eK/FVUlKS1evv2bPH+/Tp4xMnTqzYt3btWr/33nurlN25c2fKc51xxhm+aNEi//jjj71Dhw6+efNmd3f/4osvfM2aNXWONdX127dv7+Xl5VX2v/POO96hQwdftWpVxTkmTJjgu3fv9tzcXP/ggw8qyj777LM+YMCApNft3r37Ptd/5pln/NRTT/UOHTr4nj17qo1j/vz5/u1vfztlLHV18skn+2uvveZ79uzxQYMG+dy5c6uUGTt2rI8bN87d3VetWlVRzz179vgXX3zh7u47duzw4uJif/31193dfcGCBT5y5Mik10z2dwss9mo+V3XXkEiafj5nJSXrt9TrOQuOyeG273St9viLL75I06ZNueaaayr2tW/fnuuvvx6Ahx56iGeeeYZt27bx5Zdf8uCDD3LOOeewYsUKtm7dyogRIygpKaFLly5s3boVgE8++YSWLVty2GGHAXDYYYdVvH7//fe59tprKS8vp0WLFvz5z3+mc+fOzJkzh1/+8pfs2LGDVq1aMXXqVNq0acO4ceNYv349a9eupXXr1jzyyCPcfPPNzJs3DzPj6quvroj1vvvuY86cOezcuZPHH3+czp07M378eG699VY6d+4MwMEHH8wPfvADAC688EJmzpzJzTffDMCMGTMYPnx40t9Rr169OPjg/3ycTZ8+nRtvvJGJEyeycOHCKq2LZFLFsr/KysrYsmVLxfUvv/xynnrqKQYPHrxPuZKSkorWS+fOnVm7di0bNmygTZs2Ff82O3fuZOfOnRVjAKeddhpXXnklu3bt2qfu+0NdQyIHsJUrV9KrV6+UZV5//XUefvhhXnzxxX32T5w4kRYtWrB8+XJuvfVWlixZAkCPHj1o06YN+fn5jBgxgjlz5lS8Z9SoUdx3330sWbKEO++8s+KD8Jvf/CYLFy5k6dKlDBs2jPHjx1e8Z8mSJcyaNYtp06YxefJkPvjgA5YuXcry5cu55JJLKsq1bt2at956i9GjR3PnnXcCsGLFCnr37p20XsOHD2fGjBkAbN++nblz53L++edXKffqq6/uc46tW7fywgsvcM455zB8+HCmT5+e8ve3V6pYEs2fP5+ePXtW+Tn11FOrlF23bh25ubkV27m5uaxbt65KuR49evDEE08A8Oabb/Lhhx9SWloKwO7du+nZsydHH300AwcO5JRTTgHgoIMOomPHjrz99ttp1S8VtQhE0pTqm3umXHvttbzyyis0bdqURYsWATBw4ECOPPLIKmVffvllbrjhBgAKCwspLCwEoEmTJjz33HMsWrSIF154gZtuuoklS5YwduxYXnvtNS688MKKc2zfvh0Inqe4+OKLKSsrY8eOHfvcoz5kyBCaN28OwPPPP88111xT8Q01Ma7zzjsPgN69e1d86KVy8skn8+9//5t3332XVatW0adPH77+9a9XKVdWVkaXLl0qtp9++mn69+9PixYtOP/88/nFL37BXXfdRZMmTZLeUVPbu2z69+/PsmXL0irrScYDkl3vlltu4cYbb6Rnz550796dk046qeJ32KRJE5YtW8bmzZs599xzWbFiBd26dQPg6KOPZv369WklsFQiTQRmNgi4B2gC3O/ud1Q6figwBegNbAQudve1UcYk0pB07dqVv/3tbxXbEyZM4NNPP6WoqKhi39e+9rVq31/dh5yZUVxcTHFxMQMHDmTEiBGMGTOGI444IumH3PXXX8+YMWMYMmQICxYsYNy4cUmv7+7VXvPQQw8Fgg+2Xbt2VdRvyZIl9OjRI+l7hg0bxowZM1i1alXSbiGA5s2b73Pf/PTp03n11VfJy8sDYOPGjcyfP5+zzjqLVq1asWnTJlq3bg3AZ599VvG6plj2mj9/PjfddFOV/S1atOC1117bZ19ubm7FN3sIEuoxxxxT5b05OTn85S9/AYLfYX5+fpUHwo444gj69evHc889V5EItm3bVpGE6yKyriEzawJMAAYDBcBwMyuoVOwqYJO7dwTuAn4TVTwiDdGAAQPYtm0bEydOrNj31VdfpfXe008/nalTpwJBt8fy5cuB4M6it956q6LcsmXLaN++PTk5OeTn5/P4448DwQfS3m6Hzz//nGOPPRaAhx9+uNprfutb32LSpEkVH/SfffZZyhh/9KMf8etf/5r33nsPgD179vD73/++4vjw4cN59NFHefHFFxkyZEjSc3Tp0oXVq1cDsGXLFl555RU++ugj1q5dy9q1a5kwYUJF91C/fv145JFHgKDL5dFHH6V///5pxbLX3hZB5Z/KSQCgbdu2tGzZkoULF+LuTJkyhaFDh1Ypt3nzZnbs2AHA/fffz+mnn05OTg7l5eVs3rwZCLq8nn/++YoxDID33nuPrl3r3lKNcoygGFjt7mvcfQcwA6j8GxgK7P2r+itwpunpHZEKZsZTTz3FSy+9RH5+PsXFxVxxxRX85jc1f2caPXo0//73vyksLGT8+PEUFxcDwaDj2LFj6dy5Mz179mTmzJncc889AEydOpUHHnig4nbFvbeVjhs3jgsvvJDTTjut4ht0MiNHjqRdu3YUFhbSo0cPpk2bljLGwsJC7r77boYPH06XLl3o1q3bPreHFhQU0KJFCwYMGFBty2fw4MG8/PLLADzxxBMMGDCgovUBMHToUGbPns327dv56U9/yurVq+nRowcnnXQSHTt25NJLL00rlv01ceJERo4cSceOHTn++OMrBoonTZrEpEmTAFi1ahVdu3alc+fOPPvssxX/HmVlZfTv35/CwkJOPvlkBg4cWHG764YNG2jevDlt27atc4yWrA+rPpjZBcAgdx8Zbl8GnOLu1yWUWRGWKQ233w/LfFrpXKOAUQDt2rXr/eGHH9Y6np/PWQkcGP280nCsWrVqn/5nOTCde+65jB8/nk6dOmU7lIy56667yMnJ4aqrrqpyLNnfrZktcfeiKoWJdowg2Tf7ylknnTK4+2RgMkBRUdF+ZS4lAJHG64477qCsrCxWieCII47gsssuq5dzRZkISoHjErZzgfXVlCk1s4OBw4HUnYoiIpWceOKJnHjiidkOI6NGjBhRb+eKcoxgEdDJzPLNrCkwDJhdqcxs4Irw9QXAix5VX5XIftKfpDQk+/P3GlkicPddwHXAPGAV8Ji7rzSz281s7/D/A0ArM1sNjAFuiSoekf3RrFkzNm7cqGQgDYKH6xE0a9asVu+LbLA4KkVFRb534iyRqGmFMmloqluhLFuDxSIN3iGHHFKrlZ5EGiLNNSQiEnNKBCIiMadEICIScw1usNjMyoHaP1ocaA0kXyap8VKd40F1joe61Lm9ux+V7ECDSwR1YWaLqxs1b6xU53hQneMhqjqra0hEJOaUCEREYi5uiWBytgPIAtU5HlTneIikzrEaIxARkari1iIQEZFKlAhERGKuUSYCMxtkZu+a2WozqzKjqZkdamYzw+NvmFle5qOsX2nUeYyZlZjZcjN7wczaZyPO+lRTnRPKXWBmbmYN/lbDdOpsZheF/9YrzSz1WpENQBp/2+3MbL6ZLQ3/vs/ORpz1xcweNLNPwhUckx03M7s3/H0sN7Nedb6ouzeqH6AJ8D7QAWgKvA0UVCrzA2BS+HoYMDPbcWegzv2BFuHr0XGoc1iuJfAysBAoynbcGfh37gQsBb4ebh+d7bgzUOfJwOjwdQGwNttx17HOpwO9gBXVHD8beJZghcc+wBt1vWZjbBEUA6vdfY277wBmAEMrlRkKPBy+/itwppklWzazoaixzu4+392/CjcXEqwY15Cl8+8M8AtgPNAY5pFOp85XAxPcfROAu3+S4RjrWzp1diAnfH04VVdCbFDc/WVSr9Q4FJjigYXAEWZWpxXsG2MiOBb4OGG7NNyXtIwHC+h8DrTKSHTRSKfOia4i+EbRkNVYZzM7CTjO3Z/OZGARSuff+QTgBDN71cwWmtmgjEUXjXTqPA641MxKgbnA9ZkJLWtq+/97jRrjegTJvtlXvkc2nTINSdr1MbNLgSLgjEgjil7KOpvZQcBdwJWZCigD0vl3Ppige6gfQavvH2bWzd03RxxbVNKp83DgIXf/nZn1BR4J67wn+vCyot4/vxpji6AUOC5hO5eqTcWKMmZ2MEFzMlVT7ECXTp0xs7OAW4Eh7r49Q7FFpaY6twS6AQvMbC1BX+rsBj5gnO7f9ix33+nuHwDvEiSGhiqdOl8FPAbg7q8DzQgmZ2us0vr/vTYaYyJYBHQys3wza0owGDy7UpnZwBXh6wuAFz0chWmgaqxz2E3yJ4Ik0ND7jaGGOrv75+7e2t3z3D2PYFxkiLs35HVO0/nbforgxgDMrDVBV9GajEZZv9Kp80fAmQBm1oUgEZRnNMrMmg1cHt491Af43N3L6nLCRtc15O67zOw6YB7BHQcPuvtKM7sdWOzus4EHCJqPqwlaAsOyF3HdpVnn3wKHAY+H4+IfufuQrAVdR2nWuVFJs87zgG+ZWQmwG/iRu2/MXtR1k2adfwj82cxuIugiubIhf7Ezs+kEXXutw3GP24BDANx9EsE4yNnAauArYESdr9mAf18iIlIPGmPXkIiI1IISgYhIzCkRiIjEnBKBiEjMKRGIiMScEoEccMxst5ktS/jJS1E2r7pZGmt5zQXhDJdvh9MznLgf57jGzC4PX19pZsckHLvfzArqOc5FZtYzjff8j5m1qOu1pfFSIpAD0VZ375nwszZD173E3XsQTEj429q+2d0nufuUcPNK4JiEYyPdvaReovxPnH8kvTj/B1AikGopEUiDEH7z/4eZvRX+nJqkTFczezNsRSw3s07h/ksT9v/JzJrUcLmXgY7he88M57l/J5wn/tBw/x32n/Ud7gz3jTOzsWZ2AcF8TlPDazYPv8kXmdloMxufEPOVZnbffsb5OgmTjZnZRDNbbME6BD8P991AkJDmm9n8cN+3zOz18Pf4uJkdVsN1pJFTIpADUfOEbqEnw32fAAPdvRdwMXBvkvddA9zj7j0JPohLwykHLga+Ee7fDVxSw/W/A7xjZs2Ah4CL3b07wZP4o83sSOBcoKu7FwK/THyzu/8VWEzwzb2nu29NOPxX4LyE7YuBmfsZ5yCCKSX2utXdi4BC4AwzK3T3ewnmoenv7v3DaSd+ApwV/i4XA2NquI40co1uiglpFLaGH4aJDgH+EPaJ7yaYQ6ey14FbzSwXeMLd/2VmZwK9gUXh1BrNCZJKMlPNbCuwlmAq4xOBD9z9vfD4w8C1wB8I1je438yeAdKe5trdy81sTThHzL/Ca7wanrc2cX6NYMqFxNWpLjKzUaASv38AAAGySURBVAT/X7clWKRleaX39gn3vxpepynB701iTIlAGoqbgA1AD4KWbJWFZtx9mpm9AXwbmGdmIwmm7H3Y3X+cxjUuSZyUzsySrlERzn9TTDDR2TDgOmBALeoyE7gI+CfwpLu7BZ/KacdJsFLXHcAE4DwzywfGAie7+yYze4hg8rXKDPi7uw+vRbzSyKlrSBqKw4GycI75ywi+De/DzDoAa8LukNkEXSQvABeY2dFhmSMt/fWa/wnkmVnHcPsy4KWwT/1wd59LMBCb7M6dLwimwk7mCeC7BPPozwz31SpOd99J0MXTJ+xWygG+BD43szbA4GpiWQh8Y2+dzKyFmSVrXUmMKBFIQ/FH4AozW0jQLfRlkjIXAyvMbBnQmWA5vxKCD8z/Z2bLgb8TdJvUyN23Eczs+LiZvQPsASYRfKg+HZ7vJYLWSmUPAZP2DhZXOu8moARo7+5vhvtqHWc49vA7YKy7v02wVvFK4EGC7qa9JgPPmtl8dy8nuKNpenidhQS/K4kxzT4qIhJzahGIiMScEoGISMwpEYiIxJwSgYhIzCkRiIjEnBKBiEjMKRGIiMTc/wcqA8pb5yvsOQAAAABJRU5ErkJggg==\n"
},
"metadata": {
"needs_background": "light"
}
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "This is great, our model does far better then guessing which would be a line going from the bottom left corner to the top right corner, AUC = 0.5. But a perfect model would achieve an AUC score of 1.0. So there's still room for improvement.\n\nLet's move onto the next evaluation request, a confusion matrix.\n\n### Confusion matrix\n\nA confusion matrix is a visual way to show where your model made the right predictions and where it made the wrong predictions (or in other words, got confused).\n\nScikit-Learn allows us to create a confusion matrix using confusion_matrix() and passing it the true labels and predicted labels."
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "# Confusion matrix\nprint(confusion_matrix(y_test, y_preds))",
"execution_count": 87,
"outputs": [
{
"output_type": "stream",
"text": "[[25 4]\n [ 3 29]]\n",
"name": "stdout"
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "As you can see, Scikit-Learn's build-in confusion matrix is a bit bland. For a presentation you'd proberbly want to make it visual.\n\nLet's create a function which uses Seaborn's heatmap()
for doing so."
},
{
"metadata": {
"run_control": {
"marked": false
},
"trusted": true
},
"cell_type": "code",
"source": "sns.set(font_scale=1.5)\n\ndef plot_conf_mat(y_test, y_preds):\n """\n Plots a nice looking confusion matrix using seaborn's heatmap()\n """\n fig, ax = plt.subplots(figsize=(3, 3))\n ax = sns.heatmap(confusion_matrix(y_test, y_preds),\n annot=True,\n cbar=False)\n plt.xlabel("True label")\n plt.ylabel("Predicted label")\n \nplot_conf_mat(y_test, y_preds)",
"execution_count": 88,
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": "<Figure size 216x216 with 1 Axes>",
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAOQAAADfCAYAAADm6n/jAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAek0lEQVR4nO3deUBU9f7/8eewCAoo4K6AgriyKJmaclWUcCfDC+pFsdTSctfUm95cuvXLJZcw09Tkq5ImoWaJS7lhhTe7+rVIRUtBIDBcWGRREGZ+f/B1ulxABxiYM/B+/OWcc+bwInt55nPOZ85RaTQaDUIIRTAxdAAhxJ+kkEIoiBRSCAWRQgqhIFJIIRTEzNABqsuDL1cbOkKt0WLcFkNHqFWycm6Uu06OkEIoiBRSCAWRQgqhIFJIIRRECimEgkghhVAQKaQQCiKFFEJBpJBCKIgUUggFkUIKoSBSSCEURAophIJIIYVQECmkEAoihRRCQaSQQiiIFFIIBZFCCqEgUkghFEQKKYSCSCGFUBAppBAKIoUUQkHKvVFyp06dUKlUFdqZSqXiypUrVQ4lRF1VbiFffPHFChdSCFE15RZy5cqVNZlDCEElnu2Rnp7O2bNnSU1NZdiwYTRo0ICMjAzatWtXHfmEqFMqVMiwsDBCQ0PJz89HpVLh4eFBbm4uM2fOZOzYsSxdulQ+5gpRBTqfZT106BCrV6/Gz8+P0NBQNBoNAG5ubvj5+bF3717Cw8OrLagQdYHOhQwLC8Pb25s1a9bQs2dP7fKWLVuyYcMG+vfvT2RkZLWEFKKu0LmQN27cYODAgeWuHzBgAMnJyXoJJURdpfMY0srKiuzs7HLXp6am0qBBA72Eqk3OXvudbSd/Ii7lLiqVCk+nZkwf3B3PNs2024z78EsuJ98t9d7nPdqyJsS3JuMaPTe3jkR/d5C1azaz8r0Nho5TYToXsm/fvuzZs4egoCBMTEoeWK9evcru3bvx8fHRdz6jdv7GLaaHfU275nZMH/wsRWo1n/8rjskfHybs9RF4ODVFo9GQkJbFALc2PO/RtsT7W9pZGya4kTI1NWXzltXUq1fP0FEqTedCvvHGGwQGBjJ8+HB69OiBSqUiIiKC3bt3Ex0djbW1NbNnz67OrEbn/UM/0KKRFeEzXqB+veL/1P7d2xOwZh8bvz7PlleHkpqRQ17BI3zcnBj+jKuBExu3efNfp1Pn9oaOUSU6jyGbN2/O/v378fHx4YcffkCj0XDs2DFiYmLw9fUlMjISR0fH6sxqVO7n5fPrrXT8PF20ZQRobFOf7i4t+fnmbQBu/JEBgEszW4PkrC26uHVgwcJprF71kaGjVEmFrkM2a9aMlStXotFoyMjIoKioCHt7e0xNTSsdICUlhYSEBHJycjAxMcHGxgZnZ2datGhR6X0qgZWlOQfnB5Yo42OZuQ8xMy2+XnsjrbiQzv9XyAcFj6hfz7zmgtYCpqambNq8mujTZ/l870GWLJ1n6EiVVuGZOllZWcTExJCSkoKpqSlOTk4899xzWFtXbLzzzTffEBoaSnx8vPaa5mMqlYo2bdowZ84chgwZUtGIimBqYkKbpo1KLf/1Vjo/JabRp4MDANfTMrCyMGftoXN8/XMCeQWPcLC3YcaQ7gzpJrOfdDF33lRc2rUleOxrmJlV/uCgBBUq5MaNG9m2bRv5+fkllltZWbFw4ULGjBmj034OHjzIm2++ydChQ5k5cyZt2rTBysoKjUZDbm4uiYmJfP3118ydO5dHjx7h7+9fkZiKlZf/iLf2ngFgoo8nADf+yCQ3/xHZDwt4d2w/7j8oYM/3l3lzTzSFRWpGdDfuMVF169S5PQvfnMGCN94mNfUPnJxaGzpSlehcyJ07d7Jx40aee+45xo8fj6OjY/EZwoQEdu7cyfLly7G2tmb48OFP3dfWrVv529/+xrJly8pc36VLF4YOHcry5cvZsmVLrSjkg4JCZu84zq+30pk0oCvPtmsJwF97daRIo2Fsny7abYd0cyFw7QHWH/43Q73aYWoiX1sti4mJCZs2r+KHf11g544IQ8fRC50LuWfPHry9vdm+fXuJ5Z06dWLQoEFMmDCBjz/+WKdCpqSk8Pzzzz91O19fX7744gtdIyrW/Qf5zPqf4/x0M40Xe3Rg5pDu2nVBvTuX2t7S3Izhz7iy5cRF4tMyad/SvibjGo3Zc17F3aMTQ/zGYN/YDgBb2+JhQoP69bFvbEdGemapIZGS6fxP761bt/D1LfsitampKcOHDycxMVGnfTk6OvL9998/dbvo6GijP7mTnvOAV7cc4aebafy1V0eWBf5Fpwn49taWAOQVPKruiEbL168fFhYWnP72IAmJ50lIPM93Zw8BMHvuFBISz+Po2MrAKStG5yNk+/bt+fnnnwkODi5zfVJSEm3bttVpX6+99hoLFizg9u3bDBo0CGdnZ6ytrVGpVOTk5GjHkFFRUbz99tu6RlSc3IcFvP7JMa6lpjO+rxvz/Z8rsT4tK5fXPznGYE8Xpvp5lVh3804WAK3tbWosr7F5a9EKbG0blljWtFkTPglbz2d7vmDvngOkpd0xULrK0bmQS5YsYeLEiTRr1ozJkydja1t8mj4vL499+/axb98+Nm7cqNO+RowYgampKevXr+fw4cOljhgajQYHBwfee+89AgICKvDrKMuKg//iWmo6wX8pXUaA5o2syHlYwIEfrzGurxvWlsUzTP7IzOGr87/Ro11LmtjIdMTy/PTTpVLLHp/UuXkziejoszUdqcpUmnI+YJd1Tx2NRqNdZmtri0qlIisrC7VajaWlJba2tpw+fbpCAZKTk4mPjycnJweNRqO9Dunk5FTJX6nYgy9XV+n9VRWflsmotfuxtqzHghd6YVbGiZnhz7hy6tJN5u06Sbvmtozq1ZG8/EfsjYnjUVERO6b549Lc8BMGWozbYugIOnNyas0vV75lxXuhip3LmpVzo9x1Br+njqOjY62c4XMh/hYAOQ8LWPb5d2VuM/wZVwa6t2X9S8+z/dTPhB45j4W5Kc+6tGTW0Ge1kwVE3VHuEdLYGfoIWZsY0xHSGDzpCKmXC1xqtZrs7GyOHz+uj90JUWfpfFInOzubFStW8M0335CXl1futZ24uDi9hROirtH5CPn+++9z4MABXF1d6d27NxqNhhEjRtC7d2/MzMywsLDgww8/rM6sQtR6Oh8ho6Oj8fPz48MPPyQjI4PevXsTEhKCp6cncXFxjBs3jvj4+OrMKkStp/MRMj09HW9vbwDs7Oxo3rw5sbGxAHTu3JnAwEAOHTpUPSmFqCN0LqSVlRVqtVr72snJiV9//VX7ukOHDqSmpuo3nRB1jM6F9PT05OjRoxQVFQHg6urK+fPntSd3EhISjPpeJkIogc6FnDRpEhcuXGDw4MFkZWUREBBAfHw8kyZNYvny5ezatYtevXpVZ1Yhaj2dC9m7d2+2bt2Ks7MzDRs2xNPTk+XLl/PTTz+xd+9eunTpwqJFi6ozqxC1XpVn6hQUFPDw4UMaNmz49I1rkMzU0R+ZqaNflZrLqqt69erJ2FEIPZEnKAuhIAb/tocQ4k/yBGUhFERuZyaEgkghhVAQKaQQCiKFFEJBpJBCKIgUUggF0evEAJBbeAhRFRWaGHDixAny8/P5y1/+gouLC2q1muTkZM6cOYO1tTVBQUHVHliI2kzniQHh4eGcPn2aL7/8Emdn5xLrfv/9d4KDg2VmjxBVpPMY8pNPPuHll18uVUYABwcHxo8fT2RkpF7DCVHX6FzI7OzsJ36rQ61WU1BQoJdQQtRVOheyW7duhIeHk5aWVmrd9evX2bFjBz179tRrOCHqGp2/Dzlv3jxCQkIYNmwY/fv3x9HRkYKCAhISEvj++++xsbFh4cKF1ZlViFpP50K6u7sTGRnJhg0biI6OJi8vDwBra2v8/f2ZPXu20T9cVQhDq9AdA1xdXdmwYQMajYaMjAxUKhV2dnbVlU2IOqfCt/BIT0/n7NmzpKamMmzYMG0527VrVx35hKhTKlTIsLAwQkNDyc/PR6VS4eHhQW5uLjNnzmTs2LEsXbpUrkUKUQU6n2U9dOgQq1evxs/Pj9DQUO0Nkt3c3PDz82Pv3r2Eh4dXW1Ah6gKdCxkWFoa3tzdr1qwpcXmjZcuWbNiwgf79+8vEACGqSOePrDdu3CAwMLDc9QMGDGDFihV6CaUPNkGhho5QazxILfuR7EL/KvSwnezs7HLXp6am0qBBA72EEqKu0rmQffv2Zc+ePdy7d6/UuqtXr7J792769Omj13BC1DU6P0ogLS2NwMBAHj16RI8ePThx4gSDBw+msLCQ6OhorK2tiYyMxNHRsboz68SsXmtDR6g15COrfpk3cSl3XYWe7XH79m3WrVvHyZMntR9f69evT79+/Zg/f75iyghSSH2SQuqX3gr52OPJAEVFRdjb22NqagoUP3hHKc/5kELqjxRSv55USJ3HkL6+vpw8eRIofoaHvb09TZs21ZYxKiqKvn37VjGqEHVbuZc90tPTuXHjz8dmpaSk8Msvv5T52Dm1Ws3x48fl+5BCVFG5H1lzc3MZOnQod+7c0WlHGo2GYcOGsW7dOr0GrCz5yKo/8pFVvyo9hrx8+TK//vorGo2GxYsXM3r0aLy8vEptZ2Jigr29Pb1798bMrMqPnNQLKaT+SCH160mFfGJ73NzccHNzA4ov/A8aNIgOHTroN50QQkvnkzozZsygoKCAuXPnlpgcsGrVKmbNmlVivCmEqBydC3n+/HmCg4OJiYkhIyNDu7xp06ZcuHCBwMBArl69Wi0hhagrdL4OGRISwv3799m5cye2trYl1mVlZRESEkLz5s3Ztm1btQStKBlD6o+MIfVLL9ch4+LiGDNmTKkyAjRq1IjRo0cTGxtbuYRCCKAChTQzMyvxUfW/5eTkoFar9RJKiLpK50L26tWLTz/9lOTk5FLr0tLS+PTTT+W+rEJUkc5jyPj4eIKCglCr1fTr14+2bduiUqlISkrizJkzqFQqIiIiFHOzKxlD6o+MIfVLb5PLExMTWb9+Pd9++632vqyWlpZ4e3szb948xZQRpJD6JIXUr2r7todarcbOzk47wVxJpJD6I4XUr0rP1CnP4297CCH0q9xC+vr6snjxYnx9fbWvn0alUnHixAn9pROijim3kK1atSpx06pWrVrVSCAh6rJKjSGNgYwh9UfGkPqll5k6QojqV+5H1gkTJlRqh7t27ap0GCHqunIL+fvvv5dadu/ePfLz82nUqBFt2rRBrVaTkpJCRkYGtra2iroOKYQxKreQp06dKvH63LlzvPbaa6xcuZIXXngBE5M/P+1GRUXx1ltvMW7cuOpLKkQdoPMY8t133yUwMJAXX3yxRBkBRowYQXBwMKGh8jwNIapC50ImJSXRtm3bcte3aNGC27dv6yOTEHWWzoV0dnbm8OHDFBUVlVqXn5/P/v376dixo17DCVHX6Dx1bsqUKcybN4/g4GBGjRqFo6Mj+fn53Lx5k88++4zU1FS2bNlSnVmFqPUqNDHgwIEDrF27lnv37mkfXa7RaGjdujVLlizBx8enunJWmEwM0B+ZGKBfev22h1qt5vLly6SkpKBSqXB0dKRLly5VDqlvUkj9kULql16/7WFiYkKzZs1Qq9W4uLhgYWGBWq0udeZVlG+AjzfLl83H07ML9+9ns2//YZYuW0Vubp6hoylezLkLbNnxGVeuXUdloqKrWydmvjqBru6dtdv878+XCN2yk8tXf6OhjTUD+/Vm+uTx2Nk2MmBy3VSoRRcuXGDUqFH4+PgwduxYLl26xI8//oiPjw9Hjhyproy1ik//Phw7+hn16pmz+B/vsXvPfqa8Oo4jUbu1wwBRtn9fjOW1N5aQnZPLrCkv8frEcSSn3OLlGQv55co1AH7831gmz15EQtLvvDphDH/7qz/fnP6eCdMWkHW//CeAK4XOR8jY2FgmTpxIy5Yteemll9ixYwdQfMc5MzMz5s+fj5WVFf3796+urLXCqlVLSEpKYYBvIA8fPgQgKSmVjR++x+BBPhz7+rSBEyrXqtAttGjWlD3b1lPf0hKAF4b68kLwFEK37OST0PdYsX4zpiamfPrxWpwcir+h5Nu/D6MmTGPrrr0smPGqIX+Fp9L5CBkaGoqDgwNffvklU6ZM0S738PDgq6++ol27dnKW9SksLCy4e+ce28P2aMsI8O13/wLAw6NzeW+t87LuZ3PtegKDB/bVlhGgib0dz3p58POlK6TcSuO3+Jv4DxmoLSOASxtHfLx78dVR5X9XV+cj5MWLF5k2bRqWlpY8ePCgxDpra2tGjx7Nhg0b9B6wNsnPz2e4//hSy7t2LX5+SlJySk1HMhrWVg2I+mxbiTI+lpl5H1NTU9Lu3AWgvUvbUts4ObTkxJkYbqXdoWXzptUdt9IqdFLnSU9Hzs/Pl/uyVpCTU2t8+nvz/uql/HIpjoMHjxk6kmKZmprSxrH0mfNr1xO4+MsVvHt1p8H/lTU370Gp7TKzisePd9PTFV1InT+ydu3alaioqDLX5eXlERkZiYeHh96C1XZ2drbEX/+RsO3rsbS0YM6cJeTn5xs6llHJy3vA4nfWADB5fBDtnJ2wtmrA8egY/vNqXn5+ATE/XgCgIP+RQbLqSucj5KxZswgJCWH8+PH4+vqiUqmIjY3lt99+Izw8nNTUVN5++22df3BaWlqFgjZv3rxC2yudRqPhb+Nep149c2ZMn8TXx/YSPH4aBw4cNnQ0o/Dg4UNm/P1trl2P55WQMfTw8gRgwthRbNr+KX9/ezWvhIxGXaRmw7ZdPHhQPGY3NVPeHRL/U4UmBsTExLBs2bJS35Vs2rQpS5YsYdCgQTr/YHd39zLnxZYnLi5O523BuCYGWFpa8vPFk5ibm+Piqry7vyttYsD97BymL1zGxdgrBIwYxD/fnKO9ZKRWq3n/w23s3veVdgjl490LL88urN/8Pxzasw3nNg6GjK+fmToZGRnY2dmh0Wi4cuUKSUlJqNVqWrdujbu7e4WfnBwXF8fUqVMpKCjgjTfeeOr7AwICKrR/YyokwLq1bzNr5is0b+nOvXvlP0PFEJRUyHsZmUyd+w+u/hZP0MihLF0ws8zrt3fTM0hKTqFF86a0atGc0C07CNsdyY/Hv8DCovxzITVBLzN1AgICCAoKYvr06SWerFxZnTt3ZseOHQQFBXHnzh2mTZtWpf0Zg44d23H40G7WrN3Mx1t2llhnY2ONWq0mP7/AQOmULzc3T1vGCWMCWDhrSqltjhyPpklje3o+40kTezvt8gs/XaJLx/YGL+PT6HxSJz09naZN9Xt2ysXFhXnz5vHJJ5+Qnp6u130r0fXrN2nUyIYpU8Zjbm6uXe7k1JpRAcP49tsfyMnJNWBCZXt33Sau/hbP+KCRZZYRYFfEF7y3bhOFhX8Oh86c/ZH/jb3M2FEjaipqpel8hPT39yciIoI+ffrg4KC/z+Bjx46lffv2etufkhUVFTF77hJ27fiQ0yf3s3vPfho3tmPa6xPRaDTMnvuWoSMq1o2bSRw6dhIbays6tW/Hoa9PldrGf/BAJo8LYu5b/4/pC5fxfP8+pNy6za6IA3j36s6IQQMMkLxidB5DLlmyhKioKAoKCnBycqJx48alJpSrVCp27txZzh5qlpLHkIGB/iyYPw13t47k5uZx6nQMS5au4rff4g0drUxKGENGfHGYd9ZsfOI2l2KOAnDkRDTbwyNJ+j2FxvZ2DPcbwCsTRpc5qcAQ9HJSZ+DAgTr9sP++OZahKLmQxkYJhaxN9P70K2MghdQfKaR+Veks66NHj7h+/TqFhYW4urpSv359vYYTQvzpiYXcsWMHH330ETk5OUDxXNbg4GCdrhsKISqu3FYdPHiQlStX0rp1a0aOHImJiQnnzp1jx44dFBUVsXjx4prMKUSdUO4YcvTo0ZiYmLBz504sLCyA4vmXc+fO5fTp0/z73/9+4rc/DE3GkPojY0j9qtTTr27cuIG/v7+2jFB8WePll1+moKCA+HhlnqIXwpiVW8gHDx5gY2NTarmDgwMajYb79+9XazAh6qJyC6lWq8uctGtqWvz1lYp8U0MIoRu5d6MQCvLEaxeZmZmkpqaWWJaVlQUUTzb/73UArVq1KrVMCKGbcs+ydurUqdz7hGo0mjLXqVQqrly5ot+ElSRnWfVHzrLqV6Vm6lT0C8FCiKqTuaziqeQIqV+Vug4phKh5UkghFEQKKYSCSCGFUBAppBAKIoUUQkGkkEIoiBRSCAWRQgqhIFJIIRRECimEgkghhVAQKaQQCiKFFEJBpJBCKIgUUggFkUIKoSBSSCEURAophIJIIYVQkFp7kyshjJEcIYVQECmkEAoihRRCQaSQQiiIFFIIBZFCCqEgUkghFEQKKYSCSCGFUBAppBAKIoU0kKioKIYPH46npydDhw7l4MGDho5k9OLi4nBzc+OPP/4wdJRKk0IawNGjR5k/fz7e3t589NFH9OzZk7///e8cO3bM0NGMVnx8PFOnTqWwsNDQUapEJpcbgJ+fH+7u7qxfv167bM6cOVy7do2jR48aMJnxKSwsJCIigrVr12Jubk5mZiZnzpyhRYsWho5WKXKErGHJyckkJSUxaNCgEssHDx5MfHw8ycnJBkpmnC5cuMCaNWuYNGkS8+fPN3ScKpNC1rD4+HgAnJ2dSyxv06YNAAkJCTWeyZi1a9eOEydOMGPGDExNTQ0dp8rMDB2grsnOzgbA2tq6xHIrKysAcnJyajyTMWvSpImhI+iVHCFr2OMhu0qlKnO5iYn8ldRl8rdfw2xsbIDSR8Lc3NwS60XdJIWsYY/HjklJSSWWJyYmllgv6iYpZA1r06YNDg4Opa45fvPNN7Rt25ZWrVoZKJlQAjmpYwDTp09n0aJFNGrUCB8fH06dOsXRo0dLXJcUdZMU0gBGjRpFQUEBYWFhREZG4ujoyKpVqxg2bJihowkDk5k6QiiIjCGFUBAppBAKIoUUQkGkkEIoiBRSCAWRQgqhIHIdUgHefPNNvvjii6duFxAQwMqVK2sgUfkGDhxI69atCQ8Pr5H31dT+lEIKqQBjxoyhd+/e2tcXLlwgIiKCMWPG0L17d+1yJycnQ8QTNUgKqQBeXl54eXlpXxcVFREREUG3bt0YOXKkAZOJmiZjSCEURApphAYOHMhbb73F4sWL8fDwoF+/fqSnpzNw4EBCQkLK3P6/l1+8eJGJEydqj86TJk0iNja2wlk0Gg2fffYZgYGBeHl54eHhwZAhQ9i6dStlzcqMjIzE19cXDw8PgoKC+O6770pto69sxkgKaaQOHz7M1atX+cc//sHo0aOxt7fX+b0xMTGEhISQnZ3N7Nmzef3110lNTWXcuHGcP3++Qjk++OADli9fjqurK4sWLWLevHlYWFiwdu3aUveavXTpEu+++y7Dhg1j3rx53L9/n6lTp3L27NlqyWaUNEJx9u/fr+nQoYNm//79Za4fMGCAplOnTprExMRSy8ePH1/m9o+XFxUVaXx9fTVjx47VFBYWarfJzc3V+Pn5aUaOHPnEbP+5r4KCAs0zzzyjmTt3boltsrOzNe7u7pqpU6eWeF+HDh000dHR2mUZGRmanj17agICAiqcrbzf1djJEdJIOTk5Veqs65UrV0hOTub5558nKyuL9PR00tPTefjwIQMGDCAuLk7nO3+bm5tz9uxZ/vnPf5ZYnpGRgbW1NXl5eSWWt2/fnv79+2tf29ra4u/vz+XLl7lz545esxkrOctqpBo3blyp9z2+dcjq1atZvXp1mdvcunVL5xsNm5ubEx0dzcmTJ0lISCAxMZGsrCyAUmNIFxeXUu9//I9KSkoKqampes1mjKSQRqoi9yAtKirS/lmtVgMwe/ZsunXrVub2ZRWnLBqNhgULFhAVFUX37t3x8vJizJgx9OjRg5deekmnfTzOY2JiotdsxkoKWYuYmJhQUFBQYllhYSEZGRnaI1Hr1q0BaNCgAX369CmxbWxsLFlZWVhaWur0886fP09UVBTTpk1j9uzZJX5mZmYmjo6OJbZPSUkptY/HN/dydHTU/sOhj2zGSsaQtUiTJk1ISEjg4cOH2mWnTp0iPz9f+9rd3Z2mTZsSHh6uvfUkFN+Wcs6cOSxatEjno29mZiYArq6uJZZ//vnnPHjwoNSDby5fvsyVK1e0r+/evctXX33Fs88+i52dnV6zGSs5QtYiI0aM4J133uGVV17hhRdeIDExkc8//1x7VITiMd+SJUuYM2cOo0aNIjAwEAsLCyIjI0lNTWXNmjWYmen2v4WXlxfW1tasWLGC1NRUGjZsyLlz5zhy5AgWFhYlSgXQqFEjJk+ezMSJEzE1NWX37t0UFhayaNEivWczVrX7t6tjgoODyczMZN++fbzzzjt06tSJjRs3EhYWVuKM5+DBgwkLC2Pz5s1s2rQJExMT2rdvz+bNmxkwYIDOP69JkyZs3bqVNWvWsGnTJurVq4ezszPr1q0jNjaWXbt2cffuXe3t/vv27YuHhwfbt28nMzOTrl278sEHH+Du7q73bMZKbnIlhILIGFIIBZFCCqEgUkghFEQKKYSCSCGFUBAppBAKIoUUQkGkkEIoiBRSCAX5/72rxA+1+fGVAAAAAElFTkSuQmCC\n"
},
"metadata": {}
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "Beautiful! That looks much better.\n\nYou can see the model gets confused (predicts the wrong label) relatively the same across both classes. In essence, there are 4 occasions where the model predicted 0 when it should've been 1 (false negative) and 3 occasions where the model predicted 1 instead of 0 (false positive)\n\n### Classification report\n\nWe can make a classification report using classification_report() and passing it the true labels as well as our models predicted labels.\n\nA classification report will also give us information of the precision and recall of our model for each class."
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "print(classification_report(y_test, y_preds))",
"execution_count": 89,
"outputs": [
{
"output_type": "stream",
"text": " precision recall f1-score support\n\n 0 0.89 0.86 0.88 29\n 1 0.88 0.91 0.89 32\n\n accuracy 0.89 61\n macro avg 0.89 0.88 0.88 61\nweighted avg 0.89 0.89 0.89 61\n\n",
"name": "stdout"
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "What's going on here?\n\nLet's get a refresh.\n* Precision - Indicates the proportion of positive Identifications (model predicted class 1) which were actually correct. A model which produces no false positives has a precision of 1.0.\n* Recall - Indicates the proportion of actual positives which were correctly classified. A model which produces no false negatives has a recall of 1.0.\n* F1 score - A combination of precision and recall. a perfect model achieves an F1 score of 1.0.\n* Support - The number of samples each metric was calculated on.\n* Accuracy - The accuracy of the model in decimal form. Perfect accuracy is equal to 1.0.\n* Macro avg - Short for macro average, the average precision, recall and F1 score between classes. Macro avg doesn’t class imbalance into effort, so if you do have class imbalances, pay attention to this metric.\n* Weighted avg - Short for weighted average, the weighted average precision, recall and F1 score between classes. Weighted means each metric is calculated with respect to how many samples there are in each class. This metric will favour the majority class (e.g. will give a high value when one class out performs another due to having more samples).\n\nOk, now we've got a few deeper insights on our model. But these where all calculated using a single training and test set.\n\nWhat we'll do to make them more solid is calculate them usingg cross-validation.\n\nHow?\n\nWe'll take the best model along with the best hyperparameters and use cross_val_score() along with various scoring
parameter values.\n\ncross_val_score()
works by taking an estimator (machine learning model) along with data and labels. It then evaluates the machine learning model on the data and labels using cross-validation and a defined scoring
parameter.\n\nLet's remind ourselves of the best hyperparameters and then see them in action."
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "#check best hyperparameters\ngs_log_reg.best_params",
"execution_count": 90,
"outputs": [
{
"output_type": "execute_result",
"execution_count": 90,
"data": {
"text/plain": "{'C': 0.20433597178569418, 'solver': 'liblinear'}"
},
"metadata": {}
}
]
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "# Create a new classifier with best parameters\nclf = LogisticRegression(C=0.20433597178569418,\n solver="liblinear")",
"execution_count": 91,
"outputs": []
},
{
"metadata": {},
"cell_type": "markdown",
"source": "Now we've got an instantiated classifier, let's find some cross-validated metrics."
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "# Cross-validates accuracy\ncv_acc = cross_val_score(clf,\n X,\n y,\n cv=5,\n scoring="accuracy")\ncv_acc",
"execution_count": 94,
"outputs": [
{
"output_type": "execute_result",
"execution_count": 94,
"data": {
"text/plain": "array([0.81967213, 0.90163934, 0.8852459 , 0.88333333, 0.75 ])"
},
"metadata": {}
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "Since there are 5 metrics here, we'll take the average."
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "cv_acc = np.mean(cv_acc)\ncv_acc",
"execution_count": 95,
"outputs": [
{
"output_type": "execute_result",
"execution_count": 95,
"data": {
"text/plain": "0.8479781420765027"
},
"metadata": {}
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "Now we'll do the same for other classification metrics."
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "# Cross-validated precision\ncv_precision = cross_val_score(clf,\n X,\n y,\n cv=5,\n scoring="precision")\ncv_precision = np.mean(cv_precision)\ncv_precision",
"execution_count": 98,
"outputs": [
{
"output_type": "execute_result",
"execution_count": 98,
"data": {
"text/plain": "0.8215873015873015"
},
"metadata": {}
}
]
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "# Cross-validated recall\ncv_recall = cross_val_score(clf, \n X,\n y,\n cv=5,\n scoring="recall")\ncv_recall = np.mean(cv_recall)\ncv_recall",
"execution_count": 99,
"outputs": [
{
"output_type": "execute_result",
"execution_count": 99,
"data": {
"text/plain": "0.9272727272727274"
},
"metadata": {}
}
]
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "# Cross_validated F1-score\ncv_f1 = cross_val_score(clf,\n X,\n y,\n cv=5,\n scoring="f1")\ncv_f1 = np.mean(cv_f1)\ncv_f1",
"execution_count": 100,
"outputs": [
{
"output_type": "execute_result",
"execution_count": 100,
"data": {
"text/plain": "0.8705403543192143"
},
"metadata": {}
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "Okay, we've got cross validated metrics, now what?\n\nLet's visualize them."
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "# Visualize cross-validated metrics\ncv_metrics = pd.DataFrame({"Accuracy": cv_acc,\n "Precision": cv_precision,\n "Recall": cv_recall,\n "F1": cv_f1},\n index=[0])\n\ncv_metrics.T.plot.bar(title="Cross-validated classification metrics",\n legend=False);",
"execution_count": 103,
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": "<Figure size 432x288 with 1 Axes>",
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAFJCAYAAAB3kv3qAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deVyN6f8/8NeJEmXLTkmWk6EiZW23RFmGQs0gS4NMPvaPFDNDfD4j+77LfJAQyVq2JgZj7GMYY5CclLVtylLpXL8//DpfR2kTR+7X8/HweOi67/vc73Od06v7XPd130cmhBAgIiJJ0NJ0AURE9PEw9ImIJIShT0QkIQx9IiIJYegTEUkIQ5+ISEIkG/oZGRkIDg6Gm5sbrKys0Lp1a/Tv3x87duyAUqnUdHkas3z5cpiamuL+/fsAgPDwcJiamuK3334rcLuirvcu8fHxJdouP7/99htMTU0RHh7+STzO++jcuTOGDBmi+lkIgfnz56N9+/Zo3bo1QkJCMGTIEHTu3PmD1ZCRkYHk5GTVz2+/Rz4nWVlZePToUaHrve/7XZPKa7oATYiNjcWYMWOQkJCA3r17w93dHZmZmTh+/Di+//57nD9/HvPnz4dMJtN0qRrXtm1bzJs3D02aNPlg+/D29katWrUwd+7cD7aPsiogIAAVK1ZU/RwTE4MNGzbA0dERXbt2hZWVFRo1aoQXL158kP1fu3YNY8aMwYIFC9C+fXsAQLdu3dCwYUMYGBh8kH1qSkJCAkaMGIHRo0fDzc2twHU/xu/FhyK50M/MzMS3336L1NRU7Nq1C82bN1ctGzFiBGbNmoVt27bBwsICXl5eGqz002BkZAQjI6MPuo9Tp06hX79+H3QfZVXXrl3Vfr558yYAYNKkSTA1NQUANG7c+IPt/++//8bjx4/V2po3b672e/O5uH//PuLi4oq07sf4vfhQJDe8s23bNty9exf+/v75vnH9/PxQtWpVbN++XQPVERUsOzsbAKCnp6fhSqisklzoHzx4EJUqVULPnj3zXa6rq4udO3ciIiJC1da5c2fMmDEDAQEBMDc3h729vWqM88KFCxg2bBgsLS1haWkJLy8vnD9/Xu0x09LSMG3aNDg6OsLMzAxdu3bFwoULkZmZqVonKysL//nPf9ClSxeYmZnBwcEBs2bNQlpaWoHP5/fff4epqSk2bdqUZ9m0adNgaWmp+uh//fp1/Otf/0KnTp3QsmVLdOzYEZMnT8bDhw/f+fj5jV0mJSXB398fHTp0gJWVFb7//ntkZWXl2fbevXvw8/ODvb09zMzM0K5dO/j4+ODWrVsAXh9Z5R6t7tmzR20/SqUSwcHB6NGjB8zMzGBnZ4c5c+YgIyNDbR/Pnz/Hf/7zH9ja2qJ169YYN24c0tPTC+yzXEIIbN68Gb169YKFhQU6d+6MBQsWFDhU8uTJEwQGBqpeJysrK3h5eeHixYtq6507dw6DBg2CtbU1LC0t4enpiejoaLV1bt68CW9vb3To0AGtWrVCv379sGvXLrV13hzT79y5M1asWAEA6NKli2ocP78x/Tt37mD8+PFo3749rKysMGTIEFy4cEFtnaioKAwePBhWVlYwMzND586dMW/ePNVruXz5cvj7+wMAvLy8VPvIb0w/JSUFM2fOhJ2dHczMzNC9e3esW7cOOTk5qnWWL18Oc3NzxMXFYfTo0bC0tETbtm3h5+eHlJSUd/Z57raWlpa4ffs2hg8fjtatW8POzg7r16+HEAIbN26Ek5MT2rRpA29v7zznGx4+fIipU6eiQ4cOMDc3R9++fbFv3z7V8vDwcNUne39/f9X7Mrfmo0ePwsbGBpaWlggLC8v39yIrKwvLly+Hs7MzLCws8u2Dw4cPw93dHZaWlrCyssLw4cPzvHc+NEkN7wghcOPGDbRp0wba2trvXK9Ro0Z52g4ePAgTExNMnz4dT58+hYGBAY4fP46xY8eiYcOGGDNmDAAgLCwMw4YNw7Jly9ClSxcAwIQJE/Dnn3/Cy8sLtWvXxuXLl7Fu3TqkpqZi9uzZAIDAwEAcOHAAXl5eMDIywq1btxASEoJ79+4hODj4nbW2atUKxsbGiIyMxPDhw1XtWVlZOHbsGLp27YqKFSvi5s2b+Prrr2FsbIxRo0ahYsWKuHTpEvbu3YvHjx9jy5YtRerDzMxMDB48GPfv34eXlxdq1aqFPXv24NChQ2rrPX36FAMHDoS+vj4GDx6M6tWr48aNG9i5cyfu3LmDw4cPw8DAAPPmzcPUqVNhbW2NgQMHqsZIp0+fjoiICPTr1w/Dhg3DnTt3EBoaikuXLiE0NBQVKlSAEAI+Pj44f/48Bg4ciGbNmiEqKgrTpk0r0nOZNWsWQkND4eTkhK+++gp3795FcHAw4uLiVOH6ppcvX2LQoEFIT0/HoEGDUKdOHcTFxSE0NBSjRo3CiRMnoK+vj9jYWIwePRpffPEFJk6cCADYuXMnvv32W2zduhXW1tZITk6Gt7c3qlevjjFjxqBChQo4ePAgpk+fjgoVKqB379559h8QEICIiAgcPXoU/v7+MDQ0zPd5xcXFYeDAgShfvjwGDx4MAwMDbN++HcOHD0dISAgsLCwQFhaGGTNmoHPnzpgyZQqys7Nx9OhRbNy4EZUqVcLYsWPRrVs3PHnyBDt27ICPjw/Mzc3z3V9aWho8PT2RkJAAT09PmJiY4PTp01i4cCH+/PNPLFmyRLWuUqmEl5cXrK2t4efnhz/++AO7du3Cy5cvsXTp0gJfr+zsbAwdOhRdu3aFs7Mzdu/ejQULFuDs2bNISEjA0KFDkZKSgg0bNsDf31/1nn706BEGDBgAIQSGDBmCqlWr4vjx4/j3v/+Nx48f45tvvkHbtm3h4+ODNWvWwMPDA1ZWVqr9vnr1CjNmzIC3tzeysrJgZWWFK1eu5KnP19cXJ0+eRO/evTF8+HBcvXoVCxcuVB0knTt3DhMnToS9vT0GDBiAFy9eYOvWrRg+fDgOHjz48YaLhIQkJSUJuVwuJk6cWKztnJycRPPmzcW9e/dUbdnZ2cLe3l44ODiI9PR0VXtaWpqws7MTdnZ2IisrSzx9+lTI5XKxYcMGtcecNm2aGDp0qOpnCwsLMWvWLLV1Fi9eLNzc3ERGRkaB9S1dulTI5XKRkJCgajt27JiQy+XixIkTQgghvv/+e9GqVSuRkpKitu3EiROFXC5XtS9btkzI5XIRHx8vhBBi9+7dQi6Xi7NnzwohhNiyZYuQy+Xi6NGjqsd49uyZcHV1VVtv7dq1Qi6Xi9u3b6vtb8GCBUIul4tr166p2uRyufDz81P9fPbsWSGXy0VoaKjatr/88ouQy+Xip59+EkIIER0dLeRyudi0aZNqnezsbDF06FAhl8vF7t2739lnt27dEqampmLGjBlq7YsWLRJyuVzcunVLVUfu4xw8eFDI5XJx8uRJtW1CQ0OFXC4Xhw8fFkIIsW7dOiGXy0VSUpJqneTkZOHs7Cw2b96s9lhXr15VrZOZmSn69esnFixYoGpzcnISgwcPVv389usjhBCDBw8WTk5Oqp/Hjx8vLCwsRFxcnNr+raysxLhx44QQQvTo0UN4eHgIpVKp1nf29vaiV69eqra3X//8apg/f36e94QQQsycOVPI5XIRExOjtt2PP/6otp63t7do0aKFeP78uXiX3G3nzp2rart165aQy+XC0tJSra8nT54sTE1NRWZmphBCCD8/P9GuXTvx6NEjtcecNGmSMDMzE0+fPhVCiDyv95v7XbZsmdq2b/dLTEyMkMvlYvXq1WrrTZ48WbRs2VKkpqaKH374QVhaWqr1+V9//SWcnZ1FZGTkO597aZPU8I6W1uun++bHraJq2LAhGjZsqPr5zz//xMOHDzFo0CDo6+ur2qtUqYLBgwfj0aNHuHbtGipXroxKlSph27ZtOHz4MJ4/fw4A+PHHH/HTTz+ptqtbty4OHTqE8PBw/PPPPwBef0LYvXt3oeO3uUeFUVFRqrZDhw6hRo0a6NSpEwBg5syZiI6ORrVq1VTrZGRkoEKFCgCgqqswJ0+eRM2aNdVOMFaqVAkDBgxQW2/UqFE4c+aM2uyGly9fql6DgvZ35MgRyGQyODg4IDk5WfWvRYsWqFWrFmJiYlS1aGlpqe27fPnyGDRoUKHPIyYmRnXk9yZvb2/s27dP7bXO5erqil9//RW2traqtjeHtXKfU926dQEAs2fPxrVr1wAA1atXx+HDh1X7y11n4cKFuHDhAnJycqCjo4Pw8HBMnjy50PrfRalU4sSJE3BwcICxsbGqvXr16ti2bRtmzJgBANi3bx/WrVunNkMtKSkJVapUKfJ7IVd0dDSaNGmS56Tzt99+CwA4fvy4WruLi4vaz1988QVevXqF1NTUQvf15j5yP5G3adNGbSaRoaEhhBB4+vQplEoljh07Bmtra5QvX17t/eTs7IysrCycPn260P2++ZrnJyYmBlpaWhg8eLBau5+fH/bu3Qt9fX3UrVsXz549w5w5c3Dnzh0AgKmpKQ4fPowePXoUWkNpkdTwTtWqVaGtra0257ioatSoofZz7pihiYlJnnVzZ1MkJibC0tISgYGB+O677zBu3Djo6OigXbt2cHZ2Rt++fVWhO3PmTEyYMAH+/v747rvv0Lp1a3Tr1g3u7u6oXLkycnJy8tStra2NatWqwcTEBC1btkRUVBRGjBiBly9fIjo6Gu7u7ihf/vVLLJPJkJKSgrVr1+LmzZtQKBRITEyE+P931i7qtQkJCQn5fgzNrx+ys7OxePFiXL9+HQqFAvfv31f9wS1ofwqFAkIIODo65rs8949gQkICatSokeePYlFmsyQkJADIO5RXpUoVVKlS5Z3byWQyrFu3DpcvX4ZCoYBCoVCdXM19Tj169MDRo0dx6NAhHDp0CLVq1YKDgwP69esHa2trAK+DasiQIdi6dSt+/fVXVKtWDba2tujdu/c7n3dRpKam4vnz52qBn0sul6v+r62tjfPnz+PAgQOIjY2FQqFAUlISAKBBgwbF2uf9+/dhZ2eXp71WrVqoUqWKqq9zvT3VU0dHB0DRDsZq1qyp+n/ue/vt381y5coBeP16pKSkID09HceOHcOxY8fyfcwHDx4Uut+39/G23PfimweAwOs+qFWrFgBg8ODBOHXqFLZu3YqtW7fC0NAQTk5O6N+//0edDSWp0JfJZLC0tMS1a9fw6tUr1ZvmbYsXL0Z8fDz8/f1VL1juGymXKOBrCHKX5Z436N27N+zs7HDs2DGcOHECZ86cwalTp7Bt2zaEhYVBR0cHHTt2xM8//6z6d/r0adWngfDwcDx//lx1jiBXu3btVOOWffr0wY8//oiEhAT88ccfeP78OXr16qVaNyYmBt9++y1q166NDh06qE6unjp1CmvXri1WH755Avpd/XHt2jUMGTIEurq66NSpE9zd3dGiRQsoFAoEBgYWuA+lUgk9Pb18x9UBqP5QymSyfE8gF+UPWEk+7SUkJMDDwwPPnz+Hra0tXF1d8cUXX0AIAV9fX9V62traWLZsGW7evImjR4/i5MmTCA8Px65duzB58mSMGjUKADBjxgx4eXnh8OHDOHnyJA4fPowDBw7Aw8Oj0D4q7HnlfqJ6l4ULF2LdunVo0aIFWrdujS+//BKWlpaYPXt2kULwTQX9LiiVyjznz97n+pe3fw8Le7zc/ujevTs8PT3zXacoY+mF9WdOTk6hz0tfXx9bt27FlStXcOzYMZw8eRJbtmxBSEgI5s2bl+95nA9BUqEPvL6w5Ny5czh06BD69OmTZ/nLly+xa9cu5OTkqA2FvC33aCg2NjbPsrt37wKA6uPcjRs30KxZM/Tv3x/9+/dHVlYW5s+fj82bN+PUqVOwtbXFjRs3ULduXfTs2RM9e/aEUqnEpk2bMG/ePBw8eBADBw7MM0PnzSNSV1dXBAUF4fjx47h48SKMjIzQunVr1fLZs2fD2NgYu3fvRqVKlVTt+/fvL2LPvWZoaIgLFy7k+aP59hW18+bNg46ODg4ePKh2ZLdmzZpC99GgQQOcOnUKZmZmeY66Dx8+rHpdjIyMEBMTg+TkZLV9FOXq3vr166vWfXMI6tGjR/jxxx/zfEwHgBUrViApKQmRkZFqnxDe7sPExEQkJibC2toapqamGDt2LB4+fIihQ4di48aNGDVqFJ4+fYpbt26hY8eOGDlyJEaOHImUlBT4+vpi586d+Pe//43KlSsX+jzeVr16dejq6uLevXt5lm3cuBFPnz7F4MGDsW7dOnz55ZeYN2+e2jpPnz4t9j4bNGiQ7+/BkydPkJGRgXr16hX7MUuLgYEBKlasiFevXqmGOnMlJibizz//VLv4raTq16+PM2fO4NmzZ2qfPK9fv47g4GCMGTMG5cqVQ3p6Olq3bo3WrVtjypQpuH37NgYNGoRNmzZ9tNCX1Jg+AHh4eKBBgwYICgrC33//rbYsJycHM2fOxNOnTzFy5MgCZ/i0bNkStWrVQmhoqNo0woyMDGzbtg21atWCmZkZbt26hUGDBqlNxdPR0UGLFi0AvD5ySU1NhYeHh9oRt5aWlmq2hJaWFipUqIBOnTqp/TMzM1Otn3sEn3tk+fYbKDU1FfXr11cL/AcPHuDIkSOq514Uzs7OSE9PR1hYmKotOzsbO3fuzLM/AwMDtTBOT0/Hnj178uxPS0tL7eg8d2rg6tWr1R4zOjoa48aNU4Vst27dAEBtdpMQAtu2bSv0eTg4OAAAQkND1drDw8MRGRmZ52N67nOqWLGi6g8G8HpMP/eajtzntGbNGgwbNkztcv66deuiTp06qiPG8PBwDBs2DH/88YdqnerVq8PY2BgymazQI8t3KV++PGxsbHDixAm1I/a0tDRs3LgRCoVCNQ24adOmatueOHECcXFxePXqlaott46CPj05OTkhNjY2z/DJunXrAOC9hqveV/ny5WFvb48TJ07gr7/+Uls2d+5c+Pr6qqaLvjksVFwODg5QKpVqvxfA6/dXZGQkatasiTlz5uDbb7/Fs2fPVMsbN26MKlWqlPj1LgnJHelXqFABK1aswIgRI9C/f3/07t0b5ubmSE1NRVRUFG7cuIEePXqoTX/Mj7a2Nr777jtMmDAB7u7u6N+/PwBg165dePz4MZYtWwYtLS20atUK1tbWWLx4MR48eABTU1M8ePAAW7duRePGjdGxY0fo6Oigd+/e2LZtG168eAFLS0ukpqZi69atqFmzZp4TX+/Su3dv1bzqN4d2AMDe3h6HDh3C999/D3Nzc9y/fx87d+5UzUl/841YkC+//BI7d+7E7NmzcefOHTRq1Aj79u3DkydP8uxv/fr1GD9+PGxtbfHkyRPs2rVLdST55v4MDAxw7tw57Ny5E7a2tnBwcECXLl0QHByM+/fvo1OnTkhISEBISAjq168Pb29vAED79u3h4uKC9evX48mTJ7CwsEB0dDSuX79e6PP44osvMGDAAGzZsgWPHz9Gx44dcfv2bWzfvh19+/ZF8+bN89xXxd7eHtHR0Rg9ejR69OiB9PR0REREQKFQqD2nQYMGYe/evRg0aBA8PDxQtWpVnD17Fr/99hvGjRsHAOjbty82bdoEHx8ffPXVV6hTpw6uXbummqb6PhdfTZ48GQMGDMCAAQNUEw127tyJ58+fY8KECTA2Nkb9+vWxZs0aZGZmom7durh69Sr27NmDChUq5HltgNfh9fTp03yPRkePHo0jR45gwoQJ+Oqrr9CoUSOcPXsWR44cgbOzs+oPrKZMmTIFv/32GwYNGoRBgwahfv36iImJwc8//wwPDw80a9YMwOs/usDrk9xCiGJdJd65c2fY2Nhg7ty5uHXrFszNzXH58mVERETA19cX1apVw/DhwzFy5EgMGjRIdT7v2LFjUCgUCAoK+iDPPV8fbZ7QJ+bhw4di7ty5wtXVVbRu3Vq0atVKDBw4UOzatUttSpUQeafNvenMmTNi8ODBolWrVsLKykqMGDFCnD9/Xm2dlJQUMXv2bNG5c2dhZmYmbGxsxPTp08Xjx49V67x48UIsXbpUODs7C3Nzc9GuXTsxfvx4tWl3hUlPTxfm5uaiX79+eZalpqaKgIAAYWNjIywsLISzs7OYO3euuHjxopDL5WLjxo1CiMKnbObuJzAwUHTq1Em0bt1ajBs3TjUFMXe9ly9firlz5wp7e3thbm4uunTpIqZPny7u3LkjmjdvLgIDA1WPFx4eLmxsbISZmZnYs2ePEEKIrKwssWrVKuHs7Cxatmwp7OzsxNSpU9Wmpeaut2TJEuHg4CDMzc3F8OHDxenTpwudsimEEDk5OWL9+vWqfXTr1k2sXLlSNdXv7Sl8SqVSrFmzRnTp0kWYmZkJR0dHMWHCBHH37l3Rvn17MXr0aNVjX7x4UYwYMUJ06NBBmJmZiV69eoktW7aovbf+/vtvMXbsWGFjYyNatmwpnJ2dxYoVK1T7F6JkUzZzH9vHx0e0adNGWFtbixEjRojr16+rLR8xYoSwtrYWVlZWol+/fiIkJET873//E3K5XPzxxx+q/s2dAtq2bVvx8uXLfGt48uSJmD59uujUqZMwMzMTLi4uYsOGDeLVq1cF1l5Qe1HWeXu677vWjYuLE5MmTRLt27cX5ubmwtXVVWzatEmtPiGEmD17trC0tBStW7cW9+7de+d+8/u9ePHihVi4cKFwdHQUZmZmwtXVVWzdulXk5OSo1vn555+Fp6enaNu2rbCwsBDu7u7iwIED73zeH4JMCH4xOhGRVEhuTJ+ISMoY+kREEsLQJyKSEIY+EZGEMPSJiCSEoU9EJCFl4uKslJRnUCo/3ZmlNWroIykpo/AVqUjYn6WL/Vm6PvX+1NKSoXr1d1/cVyZCX6kUn3ToA/jk6ytr2J+li/1Zuspyf3J4h4hIQhj6REQSwtAnIpIQhj4RkYQw9ImIJIShT0QkIQx9IiIJKRPz9ImkonKVitCtUPq/lrVqFf/7dgvyMvMV0v95UaqPSR8HQ5/oE6JboTx6T96r6TIKtX/hl0jXdBFUIhzeISKSEIY+EZGEMPSJiCSEoU9EJCEMfSIiCWHoExFJCEOfiEhCGPpERBLC0CcikhCGPhGRhDD0iYgkhPfeIaLPVlm4gd3HvnkdQ5+IPltl4QZ2H/vmdRzeISKSEIY+EZGEMPSJiCREkmP6H+LkDr+ZiIjKAkmGPk/uEJFUcXiHiEhCGPpERBLC0CcikpAih/6BAwfQs2dPWFhYwMXFBREREQWun5ycDH9/f9ja2qJdu3YYPXo04uLi3rdeIiJ6D0U6kRsZGYkpU6bAy8sLdnZ2OHbsGPz8/KCrq4sePXrkWV8IAV9fXygUCvz73/9GtWrVsGzZMnh5eWH//v2oWrVqqT8R0oyycJk7wNlQRLmK9Nu6aNEiuLi4ICAgAABgZ2eHtLQ0LF26NN/Qj4uLw6VLlxAUFIS+ffsCAJo0aYKuXbsiOjoa/fr1K8WnQJpUFmZCAZwNRZSr0OGd+Ph4KBQKODs7q7V3794dsbGxiI+Pz7NNZmYmAEBPT0/Vlnt0n5qa+l4FExFRyRUa+rGxsQAAExMTtXZjY2MAwN27d/Ns07x5c7Rv3x4rV67EnTt3kJycjDlz5qBSpUro2rVradRNREQlUOjwTnr66w/F+vr6au25R/EZGRn5bjdz5kx88803cHV1BQDo6Ohg5cqVMDIyKnaRNWroF77SZ6i0x7Wljv1Zutifpedj9mWhoS+EAADIZLJ827W08n5YuHPnDjw9PdGwYUMEBARAV1cXO3fuxLhx47BhwwZYW1sXq8ikpAwolaJY2xSkrLxZnzz59Eehy0pfAuzP0sb+LD2l2ZdaWrICD5QLDf3KlV932ttH9M+ePVNb/qaffvoJABAcHKway7exscHXX3+N//73vwgPDy9a9UREVKoKHdPPHctXKBRq7ffu3VNb/qbExEQ0adJEbWqmTCaDlZUVbt++/V4FExFRyRUa+sbGxjA0NERUVJRa+5EjR9CoUSPUr18/zzYmJia4desW0tLS1Np///13NGjQ4D1LJiKikirSPH1fX1/4+/ujatWqcHR0RHR0NCIjI7F48WIAr6++VSgUaNq0KfT19TFs2DDs27cP3t7eGDVqFHR1dbF3716cO3dOtQ0REX18RQp9Nzc3ZGVlITg4GGFhYTAyMkJQUJBqZk5MTAz8/f2xefNmtG/fHoaGhggNDcX8+fMxbdo0aGlpQS6XY9OmTejUqdMHfUJERPRuRb5+3tPTE56envkuc3Nzg5ubm1pbkyZNsGbNmverjoiIShXvsklEJCEMfSIiCWHoExFJCEOfiEhCGPpERBLC0CcikhCGPhGRhDD0iYgkhKFPRCQhDH0iIglh6BMRSQhDn4hIQhj6REQSwtAnIpIQhj4RkYQw9ImIJIShT0QkIQx9IiIJYegTEUkIQ5+ISEIY+kREEsLQJyKSEIY+EZGEMPSJiCSEoU9EJCEMfSIiCWHoExFJCEOfiEhCGPpERBLC0CcikhCGPhGRhDD0iYgkpMihf+DAAfTs2RMWFhZwcXFBREREgesrlUqsXr0aXbp0gYWFBXr37o2DBw++d8FERFRy5YuyUmRkJKZMmQIvLy/Y2dnh2LFj8PPzg66uLnr06JHvNv/973+xY8cOTJo0Cc2bN8fBgwcxefJk6Ovrw8HBoVSfBBERFU2RQn/RokVwcXFBQEAAAMDOzg5paWlYunRpvqGvUCgQEhKCwMBADBgwAADQsWNHxMXF4ZdffmHoExFpSKGhHx8fD4VCgUmTJqm1d+/eHZGRkYiPj4eRkZHasmPHjkFXVxd9+/ZVa9+6dWsplExERCVV6Jh+bGwsAMDExESt3djYGABw9+7dPNvcvHkTJiYmOHPmDPr06YMWLVrA2dkZhw4dKo2aiYiohAo90k9PTwcA6Ovrq7Xr6ekBADIyMvJsk5ycjAcPHiAgIADjx4+HoaEhwsLCMHHiRBgYGKBDhw7FKrJGDf3CV/oM1apVWdMlfFbYn6WL/Vl6PmZfFhr6QggAgEwmy7ddSyvvh4Xs7GwkJydjzZo1cHJyAvB6TD82NhYrVqwodugnJWVAqRTF2qYgZeXN+uRJuqZLKFRZ6UuA/Vna2GyacHAAAB5oSURBVJ+lpzT7UktLVuCBcqHDO5Urv+60t4/onz17prb8TXp6eihXrhxsbGxUbTKZDJ06dcLNmzeLVjkREZW6QkM/dyxfoVCotd+7d09t+ZuMjY2hVCrx6tUrtfbs7Ow8nxiIiOjjKTT0jY2NYWhoiKioKLX2I0eOoFGjRqhfv36ebezs7CCEQGRkpKrt1atX+OWXX2BlZVUKZRMRUUkUaZ6+r68v/P39UbVqVTg6OiI6OhqRkZFYvHgxgNcnbhUKBZo2bQp9fX107NgRDg4OmDNnDp4/f45GjRph27ZtSEhIwMKFCz/oEyIioncrUui7ubkhKysLwcHBCAsLg5GREYKCguDq6goAiImJgb+/PzZv3oz27dsDAJYtW4alS5di3bp1SEtLQ4sWLRAcHAwzM7MP92yIiKhARQp9APD09ISnp2e+y9zc3ODm5qbWpqurCz8/P/j5+b1fhUREVGp4l00iIglh6BMRSQhDn4hIQhj6REQSwtAnIpIQhj4RkYQw9ImIJIShT0QkIQx9IiIJYegTEUkIQ5+ISEIY+kREEsLQJyKSEIY+EZGEMPSJiCSEoU9EJCEMfSIiCWHoExFJCEOfiEhCGPpERBLC0CcikhCGPhGRhDD0iYgkhKFPRCQhDH0iIglh6BMRSQhDn4hIQhj6REQSwtAnIpIQhj4RkYQw9ImIJKTIoX/gwAH07NkTFhYWcHFxQURERJF38uDBA1hZWWHVqlUlKpKIiEpHkUI/MjISU6ZMgY2NDVauXIl27drBz88PUVFRhW4rhEBAQAAyMjLeu1giIno/5Yuy0qJFi+Di4oKAgAAAgJ2dHdLS0rB06VL06NGjwG23bduG2NjY96+UiIjeW6FH+vHx8VAoFHB2dlZr7969O2JjYxEfH1/gtgsWLMDs2bPfv1IiInpvhYZ+7lG6iYmJWruxsTEA4O7du/lup1QqMW3aNLi4uMDe3v596yQiolJQ6PBOeno6AEBfX1+tXU9PDwDeOVb/v//9D/Hx8VizZs371khERKWk0NAXQgAAZDJZvu1aWnk/LMTGxmLJkiVYtmwZKleu/N5F1qihX/hKn6Fatd6/7+j/sD9LF/uz9HzMviw09HND++0j+mfPnqktz5WTk4Np06ahR48esLGxwatXr1TLlEolXr16hfLli3T+WCUpKQNKpSjWNgUpK2/WJ0/SNV1CocpKXwLsz9LG/iw9pdmXWlqyAg+UCx3Tzx3LVygUau337t1TW57rwYMH+P333xEREYGWLVuq/gHA8uXLVf8nIqKPr9BDbmNjYxgaGiIqKgrdunVTtR85cgSNGjVC/fr11davXbs2du3aledx+vfvj6+++gru7u6lUDYREZVEkcZZfH194e/vj6pVq8LR0RHR0dGIjIzE4sWLAQDJyclQKBRo2rQp9PX1YW5unu/j1K5d+53LiIjowyvSFblubm6YNWsWTp06BV9fX5w7dw5BQUFwdXUFAMTExMDDwwPXr1//oMUSEdH7KfIZVU9PT3h6eua7zM3NDW5ubgVuf/PmzeJVRkREpY532SQikhCGPhGRhDD0iYgkhKFPRCQhDH0iIglh6BMRSQhDn4hIQhj6REQSwtAnIpIQhj4RkYQw9ImIJIShT0QkIQx9IiIJYegTEUkIQ5+ISEIY+kREEsLQJyKSEIY+EZGEMPSJiCSEoU9EJCEMfSIiCWHoExFJCEOfiEhCGPpERBLC0CcikhCGPhGRhDD0iYgkhKFPRCQhDH0iIglh6BMRSQhDn4hIQhj6REQSUuTQP3DgAHr27AkLCwu4uLggIiKiwPWfPHmCGTNmwMnJCZaWlnBzc0NkZOR7F0xERCVXvigrRUZGYsqUKfDy8oKdnR2OHTsGPz8/6OrqokePHnnWz8rKwjfffIP09HSMGzcOtWvXxuHDhzFhwgTk5OSgV69epf5EiIiocEUK/UWLFsHFxQUBAQEAADs7O6SlpWHp0qX5hv7Jkyfx119/ISwsDBYWFgAAGxsbJCYmYv369Qx9IiINKXR4Jz4+HgqFAs7Ozmrt3bt3R2xsLOLj4/Nso6enBw8PD5ibm6u1N27cGAqF4j1LJiKikir0SD82NhYAYGJiotZubGwMALh79y6MjIzUlnXs2BEdO3ZUa8vOzsaJEyfQrFmz9yqYiIhKrtDQT09PBwDo6+urtevp6QEAMjIyirSjBQsWIC4uDitXrixujahRQ7/wlT5DtWpV1nQJnxX2Z+lif5aej9mXhYa+EAIAIJPJ8m3X0ip4hEgIgfnz5+Onn36Ct7c3unbtWuwik5IyoFSKYm/3LmXlzfrkSbqmSyhUWelLgP1Z2tifpac0+1JLS1bggXKhoV+58utOe/uI/tmzZ2rL85OVlYVp06bh4MGD8Pb2xtSpU4tUNBERfRiFhn7uWL5CoYCpqamq/d69e2rL35aRkYHRo0fj0qVLCAgIwNChQ0ujXiIieg+Fzt4xNjaGoaEhoqKi1NqPHDmCRo0aoX79+nm2ycnJwZgxY/D7779j0aJFDHwiok9Ekebp+/r6wt/fH1WrVoWjoyOio6MRGRmJxYsXAwCSk5OhUCjQtGlT6OvrY/v27Th37hw8PDxQr149XLlyRfVYMpkMrVq1+jDPhoiIClSk0Hdzc0NWVhaCg4MRFhYGIyMjBAUFwdXVFQAQExMDf39/bN68Ge3bt8fhw4cBADt27MCOHTvUHqtcuXL4888/S/lpEBFRURQp9AHA09MTnp6e+S5zc3ODm5ub6ufNmze/f2VERFTqeJdNIiIJYegTEUkIQ5+ISEIY+kREEsLQJyKSEIY+EZGEMPSJiCSEoU9EJCEMfSIiCWHoExFJCEOfiEhCGPpERBLC0CcikhCGPhGRhDD0iYgkhKFPRCQhDH0iIglh6BMRSQhDn4hIQhj6REQSwtAnIpIQhj4RkYQw9ImIJIShT0QkIQx9IiIJYegTEUkIQ5+ISEIY+kREEsLQJyKSEIY+EZGEMPSJiCSEoU9EJCFFDv0DBw6gZ8+esLCwgIuLCyIiIgpc/9mzZ5g1axZsbGxgaWmJkSNHIi4u7n3rJSKi91Ck0I+MjMSUKVNgY2ODlStXol27dvDz80NUVNQ7t5k4cSKioqIwZcoUBAUF4dGjR/Dy8kJ6enqpFU9ERMVTvigrLVq0CC4uLggICAAA2NnZIS0tDUuXLkWPHj3yrH/hwgWcOHEC69evh729PQDA2toaXbp0QWhoKEaNGlWKT4GIiIqq0CP9+Ph4KBQKODs7q7V3794dsbGxiI+Pz7PN6dOnoaenBxsbG1WbgYEB2rZti5MnT5ZC2UREVBKFHunHxsYCAExMTNTajY2NAQB3796FkZFRnm2MjY1Rrlw5tfaGDRsiMjKy2EVqacmKvU1halevWOqPWdo+xPP+EMpCXwLsz9LG/iw9pdmXhT1WoaGfOwavr6+v1q6npwcAyMjIyLNNRkZGnvVzt8lv/cJUr65X7G0Ks3GGc+EraViNGnn78FNUFvoSYH+WNvZn6fmYfVno8I4QAgAgk8nybdfSyvsQucvy3WE+6xMR0cdRaAJXrlwZQN4j+mfPnqktf5O+vr5q+dvb5PcJgIiIPo5CQz93LF+hUKi137t3T23529vEx8fnOeK/d+9evusTEdHHUWjoGxsbw9DQMM+c/CNHjqBRo0aoX79+nm1sbW3xzz//4MyZM6q25ORkXLhwAZ06dSqFsomIqCTKzZw5c2ZhK1WuXBmrV69GSkoKZDIZNm3ahD179uCHH35As2bNkJycjJs3b0JfXx86Ojpo0KABzp07h23btqFatWpITExEQEAAhBD473//C11d3Y/w1IiI6G0yUdBZ1zds374dwcHBePDgAYyMjDBq1Cj07dsXABAeHg5/f39s3rwZ7du3BwCkpaVh7ty5OHbsGJRKJaysrDBt2jQ0btz4wz0bIiIqUJFDn4iIyj7OnyQikhCGPhGRhDD0iYgkhKFfTBMmTMDJkycLvOqYiOhTxdAvpsTERIwaNQr29vZYsGAB7ty5o+mSiIiKjLN3SuDu3buIiIjA/v378eDBA5ibm6Nfv37o1atXvreloKLJyMjAixcvoFQq8yyrU6eOBioi+vww9N/T2bNnERUVhZiYGKSkpKBr167o378/OnbsqOnSygyFQoGAgABcvHjxnevcuHHjI1ZU9owYMaLI68pkMmzcuPEDVkOfsiJ9cxa9W+XKlaGnpwcdHR1kZWXh77//hre3N5o3b4758+ejSZMmmi7xkxcYGIjbt29j7NixqFu3Lu/EWgLZ2dmaLuGz8ejRo2KtX9Y+hfJIvwQSExOxf/9+7Nu3D7GxsahXrx769euHfv36wdDQEPfv34ePjw+0tLSwb98+TZf7yWvdujXmzJmDXr16aboUIpiZmSEnJ6fI65e1T6E80i+mIUOG4OLFi9DW1kbXrl0xffp0dOzYUe37BgwNDdGjRw9s2rRJg5WWHXp6eqhataqmyyjTPvej048pLCwMo0ePRlZWFiZPnozy5T+vmOSRfjG5u7vD3d0dvXv3LvCk7Y0bN5CZmYnWrVt/xOrKpqCgINy9exerV6/O82U9VDTNmzcvVt+VtaPTjy02NhYDBgyAt7c3vv32W02XU6oY+iXw8uVLXL9+HVZWVgCAhw8f4ty5c3B2duYdREtg5cqV+Omnn1C9enVYWFigYkX17zSVyWQIDAzUUHVlQ3h4eLFCv1+/fh+wms9DSEgIFi5ciGPHjsHAwEDT5ZQahn4xxcfHY/jw4VAqlYiOjgYAnD59Gt988w2aNm2KDRs28KNzMXXu3LnA5TKZDMePH/9I1RC9lpOTg4sXL6Jp06YMfSnz9fXF/fv3sXTpUjRq1EjVnpCQgLFjx6JJkyZYsGCB5gokAvDHH3/g/PnzyM7OVl09rlQq8eLFC1y4cAGhoaEarvDT9euvv8LCwgJ6enqaLuWD+LzOUHwEFy5cQFBQkFrgA0CDBg3g6+uL77//XjOFfQbS0tJw5coVZGRkwMDAAObm5vxO5RIIDQ1FYGBgvrcK0dLSgq2trQaqKjtGjBiBHTt2wMLCQtW2Y8cOODs7o3r16hqsrHQw9ItJCIGsrKx3Ln/58uVHrObzsXr1aqxZswaZmZmqNh0dHYwaNQpjx47VYGVlz5YtW2Bvb4958+Zh7dq1yMjIQEBAAE6cOIFp06ahT58+mi7xk/b2H8ucnBzMnDkTZmZmn0Xo8yqYYmrbti1WrVqF1NRUtfZ//vkHa9euRbt27TRUWdm1c+dOLFu2DG5ubti2bRuOHDmCrVu3ws3NDatWrUJYWJimSyxT4uPj8fXXX6Nq1aowMzPDxYsXoauri+7du2PUqFHYvHmzpksscz6nUXAe6RfTlClTMHDgQHTu3Blt2rRBjRo1kJycjEuXLqF8+fKYO3eupkssc/73v/9hyJAhCAgIULU1bNgQ1tbW0NHRwZYtWzBgwAANVli2aGtrq2aRGRsb4969e8jOzoa2tjasrKx4/YjE8Ui/mExMTHDgwAF4eHggPT0dV65cQXJyMtzd3REREcHbLpRAfHw8HB0d813m6OiIe/fufdyCyrjmzZsjJiYGwOv3q1KpxO+//w6g+Bdx0eeHR/olUKdOHfj5+Wm6jM9GvXr1cOfOHXTq1CnPslu3bvFq3WIaOnQoxo8fj/T0dMyZMwddunTB1KlT4eLigr1796quL6Hi+VwuHGTol8Djx49x+fLld06H45TN4nF1dcXSpUtRt25ddOvWTdV+5MgRrFixAm5ubhqsruzp3r07Vq5cidjYWACvb2g3efJkhISEwNzcnDPMimD8+PHQ0dFRa/P19c3TBgCHDx/+WGWVCs7TL6YjR45g8uTJyM7OVv3lF0Ko/t+4cWMcPHhQkyWWOZmZmfjmm29w/vx56OjooEaNGkhKSkJ2djasra2xbt26PFfpUvHkzjjLL7RInb+/f7HW//HHHz9QJR8GQ7+Y3NzcoK2tjR9++AEhISHIycnByJEjceLECSxevBhr1qyBjY2Npsssk2JiYnD+/Hn8888/qFKlCtq1awd7e/vP5mP1x6JUKrFo0SJcvnwZISEhAF5fcOTr64sRI0ZwCqzEcXinmO7cuYOFCxeiRYsWaN++PYKDg9GkSRM0adIET58+Zei/B0dHx3ee0KWiW7lyJTZv3ozRo0er2uRyOby9vbF+/Xro6+tj2LBhmiuQNIqhX0xaWlqqE4vGxsaIjY2FUqmElpYW7OzssGfPHg1XWDaMGDECM2bMQOPGjQv91id+01Px7NmzB1OnTsXgwYNVbTVq1ICvry/09PQQGhrK0JcwTtksJhMTE1y+fBnA6/H7rKws/PXXXwCAZ8+eFXi1Lv2fN0+CZ2dnF/iPfVo8ycnJaNy4cb7LTE1N8eDBg49cEX1KeKRfTB4eHggMDMTz588xYcIEdOjQAdOnT8eAAQOwZcsWtGzZUtMllglbtmzJ9//0/kxMTHD06NF8p8BGR0ejYcOGGqiKPhXlZs6cOVPTRZQlZmZmqFy5MpKTk2FnZwcrKyuEh4dj79690NPTQ1BQEGrWrKnpMsuk58+fQ1tbGwBw9OhRnDlzBgYGBpynX0y6urpYsmQJYmNjkZOTg6SkJFy9ehXr1q3D7t27MXnyZHzxxReaLpM0hLN3iunSpUswNzdXhRPwespmSkrKZ3XP7Y8pNjYWPj4+cHV1xYQJE7BkyRKsWbMGwOsACw4ORps2bTRcZdkSEhKCVatWISkpSdVWrVo1jB07Vm2sn6SHoV9MdnZ2mDx5Mvr27avpUj4bvr6+iI2Nxbx582BqagpbW1vY2toiMDAQ06ZNQ1paGoeASkAIgbt37yI1NRWVK1dG48aNUa5cOU2XRRrGE7nFVL58ed7jvZSdP38ekyZNgrm5Oc6dO4f09HR4eHhAX18fnp6euHbtmqZLLJOysrKQlJSEBw8eoHbt2njy5ImmS6JPAE/kFtOYMWPw/fff4+bNm5DL5ahRo0aedTgUUTzZ2dmqcfuTJ0+iYsWKqvvD5OTkoHx5vk2LKyQkBEuXLsU///wDmUyGXbt2YenSpcjKysKqVatQqVIlTZdIGsLfpmLKvW/J8uXLAajfhCn3dgw3btzQSG1llVwux5EjR2BiYoKoqCjY2tqifPnyyM7ORkhICORyuaZLLFN27dqFOXPmwMvLC05OTqo5+f3794e/vz+WL1/OGwZKGEO/mPgFFKVv3Lhx8PX1RUhICHR0dDBy5EgAr28clpSUpDqpS0WzceNGDB8+HFOnTkVOTo6q3dnZGY8fP0ZwcDBDX8IY+sXEb8YqfTY2Nti/fz/++OMPtGrVCg0aNADw+qrdDh06oGnTphqusGy5f//+O78Ht1mzZhzblziGfjEV5ajTx8fnI1TyeTEyMoKRkZFaG6cWlkzdunVx9erVfC/OunHjBurWrauBquhTwdAvpiVLlrxzmb6+PmrXrs3QLwLee+fDcXd3x6pVq6CrqwsnJycAwMuXL3H8+HGsXr0aQ4YM0XCFpEkM/WLKvc/Om54/f44LFy5g5syZ+O677zRQVdnz9r13qPSMHj0aiYmJCAoKQlBQEIDXn5qEEOjZsyfGjBmj4QpJk3hxVinavXs3tm7dyjtt0ichLi4OZ8+eVV2cZW1tjWbNmiE0NBSDBg3SdHmkITzSL0X169fHnTt3NF1GmXTixAmcPXtWNavk6tWrWLx4MUaPHo0OHTpouLqy4eTJk9izZw9kMhm+/PJLODg4oFGjRqrlFy5cgJubG27evMnQlzBekVtKHj16hA0bNqhmnlDRHTp0CD4+Pmp/MCtWrAilUglvb2+cPHlSg9WVDfv27cOoUaNw/PhxnDhxAj4+Pjh69CgAIDU1FVOmTMGQIUNw+/ZtDB8+XMPVkiZxeKeYWrZsmefr+5RKJYQQEEJg3rx56NOnj4aqK5v69OmDdu3aYcaMGXmWzZ49G7///jt27dqlgcrKDnd3d2hra2PDhg3Q0dGBv78/YmNjsXjxYowYMQKJiYmws7NDQEAATExMNF0uaRCHd4rJx8cn3+9s1dfXh6Ojo9rHaSoahUKBgICAfJd17doV4eHhH7misicuLg5z5sxR3RfK19cXPXv2hK+vLzIzM7F06VJ0795dw1XSp4ChX0z/+te/ALyesZN7/5KMjAxkZGRw/nMJ1ahRA9evX8937P7mzZu8n34RPH/+HPXq1VP9bGhoCCEEypUrh3379uV7jyiSJo7pF9OLFy8wceJEDBw4UNV25coVODo6wt/fn1/tVwK9e/fGihUrsH37djx9+hRCCCQlJSEsLAzLly9H7969NV3iJ08IAS2t//t1zr2F8oQJExj4pIahX0wLFy7EmTNn1E6GWVpaYu7cuYiJicGqVas0WF3Z5OvrCzs7O8ycORN2dnZo0aIFbG1t8d1338HGxgbjxo3TdIllVp06dTRdAn1iOLxTTEeOHMG0adPQr18/VZuenh769u2L7OxsrF69GhMmTNBghWWPtrY2li1bhr///hsXL15EWloaKleuDCsrKzRv3lzT5ZVp+Z1/Imlj6BdTenr6Oz8u16tXT+3r6ah45HI5GjdujJSUFFSvXp330S+mN0/k5k7KmzVrFvT09NTW420tpI3DO8Vkamr6zitu9+7di2bNmn3kij4P165dg7e3NywtLeHg4ICbN29i2rRpWLlypaZLKxPatm2LChUqIDs7G9nZ2Xj16hXatm0LHR0dVVvuP553kjYeShXTmDFj4OPjg8TERHTp0gU1atRAcnIyfv75Z1y5coVj+iVw6dIlDBs2DM2aNcOoUaNUfVi3bl2sWLEC1atXx9dff63hKj9t/A5hKipenFUCP//8M5YvX44bN26oPkY3b94c48aNQ+fOnTVcXdnz9ddfo0qVKlizZg1evXoFMzMz7N69Gy1btsTcuXNx+vRp7N+/X9NlEn0WeKRfAk5OTnByckJmZiZSU1Ohp6fHL0t/D9evX8eyZcsA5D3x6OTkhO3bt2uiLKLPEsf0SyA0NBSTJk1ChQoVUKdOHdy4cQPOzs68u2YJ6enpvfME+KNHj/KciCSikmPoF9PWrVsRGBiodmRft25dWFtbY8aMGdi7d68GqyubOnfujCVLluDPP/9UtclkMjx58gRr166Fg4ODBqsj+rxwTL+Yunfvjj59+sDX1zfPshUrVuDw4cMcfy6m1NRUDB06FLdu3UKdOnXw4MEDNG3aFAkJCahduzZCQ0NhYGCg6TKJPgsc0y+mhw8fok2bNvkus7Kywvr16z9yRWVftWrVEBYWhoiICJw9exYmJibQ19eHp6cn3NzcVPc4IqL3x9Avpvr16+O3335Dx44d8yy7ePEiL3svgaCgILi6umLgwIFq9zQiotLH0C8mDw8PLFiwADk5OejatSsMDAyQkpKC6OhobNy4EePHj9d0iWXOzp07YWtrq+kyiCSBoV9Mw4YNw6NHj7Bp0yZs2LABwOtL3suXL48hQ4bgm2++0XCFZU/Lli1x5swZ2NjYaLoUos8eT+SWUHp6Oi5fvqy6OZiFhQUiIyOxc+dOzuAppqCgIGzZsgX16tVD06ZNUbNmTbXlMpkMgYGBGqqO6PPC0H9PV69exY4dO3Do0CG8ePECBgYGOHPmjKbLKlMKu4pZJpPh+PHjH6kaos8bQ78Enj17hv3792PHjh3466+/oK2tDScnJ/Tt2xf29vaqL7Cgwj19+hSJiYlo2LAhqlWrpulyiD57HNMvhuvXr2PHjh04cOAAXrx4gRYtWgAA1q5dm+9sHnq3zMxMBAQEIDIyUnX/IhcXF/zwww/8ekSiD4ihXwRhYWHYvn07rl+/jtq1a2PQoEFwc3NDjRo10K5dO973vQSWLFmCyMhIuLu7o0WLFrh79y527NgBpVKJJUuWaLo8os8W06oIvvvuO5iammL9+vWwtbVV3RQsPT1dw5WVXUePHoWvr6/alc2mpqb44YcfkJmZiQoVKmiwOqLPF++9UwTdunVDbGwsJk2ahEmTJiEmJgZKpVLTZZVpjx49Qrt27dTaHBwc8OrVK9y/f19DVRF9/nikXwTLly9HSkoK9u3bhz179sDHxwc1a9ZEt27dIJPJ+D2kJZCdnZ3naL569eoAXo/3E9GHwSP9IqpevTqGDh2KiIgI7NmzBz169FCdhJwxYwZWrFiBu3fvarrMzwInlBF9OAz9Evjiiy8wY8YM/PLLL1iyZAmMjY2xevVquLq6ws3NTdPllXn85ET04XCefil58uQJ9uzZg4iICBw6dEjT5XzymjdvDgsLC7XvJRBC4Ndff0WrVq3UvjhFJpNh48aNmiiT6LPD0CeNGDJkSLHW5xd/E5UOhj4RkYRwTJ+ISEIY+kREEsLQJyKSEIY+EZGEMPSJiCTk/wHb8XKi9wCPtQAAAABJRU5ErkJggg==\n"
},
"metadata": {}
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "Great! This looks like something we could share. An extension might be adding the metrics on top of each bar so someone can quickly tell they were.\n\nWhat now?\n\nThe final thing of the list of our model evaluation techniques is feature importance.\n\n### Feature importance\n\nFeature importance is another way of asking, "Which features contributing most of the outcomes of the model?"\n\nOr for our problem, trying to predict heart disease using a patient's medical characteristics, which characteristics contribute most to a model predicting whether someone has heart disease or not?\n\nUnlike some of the other functions we've seen, because how each model finds patterns in data is slightly different, how a model judges how important those patterns are is different as well. This means for each model, there's a slightly different way of finding which features were most important.\n\nYou can usually find an example via the Scikit-Learn documentation or via searching for something like "[MODEL TYPE] feature importance", such as, "random forest feature importance".\n\nSince we're using LogisticRegression
, we'll look at one way we can calculate feature importance for it.\n\nTo do so, we'll use the coef_
attribute. Looking at the Scikit-Learn documentation for LogisticRegression, the coef_
attribute is the coefficient of the features in the decision function.\n\nWe can access the coef_
attribute after we've fit an instance of LogisticRegression
."
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "# Fit an instance of LogisticRegression\nclf = LogisticRegression(C=0.20433597178569418,\n solver="liblinear")\n\nclf.fit(X_train, y_train);",
"execution_count": 104,
"outputs": []
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "# Check coef_\nclf.coef_",
"execution_count": 105,
"outputs": [
{
"output_type": "execute_result",
"execution_count": 105,
"data": {
"text/plain": "array([[ 0.00316728, -0.86044636, 0.66067051, -0.01156993, -0.00166374,\n 0.04386116, 0.31275829, 0.02459361, -0.60413071, -0.56862818,\n 0.45051626, -0.63609888, -0.67663381]])"
},
"metadata": {}
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "Looking at this it might not make much sense. But these values are how much each feature contributes to how a model makes a decision on whether patterns in a sample of patients health data leans more towards having heart disease or not.\n\nEven knowing this, in it's current form, this coef
_ array still doesn't mean much. But it will if we combine it with the columns (features) of our dataframe."
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "# Match coef's of feature to columns\nfeature_dict = dict(zip(df.columns, list(clf.coef_[0])))\nfeature_dict",
"execution_count": 106,
"outputs": [
{
"output_type": "execute_result",
"execution_count": 106,
"data": {
"text/plain": "{'age': 0.003167278990517721,\n 'sex': -0.8604463644626569,\n 'cp': 0.6606705054012705,\n 'trestbps': -0.011569931335912368,\n 'chol': -0.0016637438070300692,\n 'fbs': 0.04386116284216185,\n 'restecg': 0.31275829369964075,\n 'thalach': 0.02459361297137234,\n 'exang': -0.6041307139378419,\n 'oldpeak': -0.5686281825180214,\n 'slope': 0.450516263738603,\n 'ca': -0.6360988840661891,\n 'thal': -0.6766338062111971}"
},
"metadata": {}
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "Now we've match the feature coefficient to different features, let's visualize them."
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "# Visualize feature importance\nfeature_df = pd.DataFrame(feature_dict, index=[0])\nfeature_df.T.plot.bar(title="Feature Importance", legend=False);",
"execution_count": 107,
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": "<Figure size 432x288 with 1 Axes>",
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAFACAYAAABX87ByAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deVxN+f8H8NdtEy0UNdYSRqGNkjVbYx0GzWBmKMtXjD2yb4OZsSdLyDaWwmBmrA2jiTKTQdkZzBCyS5GK1vv5/dGj+3OdG+qeLvJ6Ph49HnzOuef9Oefee973fM7nfD4KIYQAERHRC/TedgWIiOjdw+RAREQSTA5ERCTB5EBERBJMDkREJMHkQEREEkwO9MaWLVsGe3v7V/5dunSp2OInJSXh2bNnxbb9wjh+/Djs7e2xbNmyt12VQnuXjiO9uwzedgXo/fPNN9+gRo0aGpdVrly5WGJGR0dj7Nix2LlzJ8qUKVMsMT4EPI70ppgcqNCaNm2KRo0a6TTmuXPn8PTpU53GLIl4HOlNsVmJiIgkeOVAxeb06dNYunQpzpw5AwCoX78+/P394ezsrFpHCIGffvoJv/zyC65du4acnBxUqVIF3t7e8PPzg0KhwMSJE7Fz504AgJeXFzw8PBAaGgofHx/cuXMHhw4dUov7crmPjw+MjIzg6OiITZs2wdjYGBs2bIC9vT2uXr2KoKAgHD9+HNnZ2ahTpw6GDRsGT0/PQu+vj48PypQpgx49emDp0qW4fv06bGxsMH78eLi5uWH+/Pk4cOAA9PX10bZtW0yePBnGxsYAgDZt2qBJkyZwdXVFSEgIkpKS4ODgAH9/fzRu3FgtTlxcHIKDg3H27FkAgJOTE0aMGIGGDRuq1mnTpg2aNm0KpVKJvXv3wsLCAg4ODoiOjpYcRwA4cOAAwsLCcOnSJWRmZsLa2hodOnSAv78/jIyM1I5j3759sXjxYvz3338oX748Pv/8cwwbNgx6ev//W/PatWtYunQpjh07hpycHNStWxejRo2Cu7u7ah05jz3Jj8mBCi01NRXJycmScjMzMxgaGgIAYmJiMHjwYDg4OGDUqFHIysrCr7/+it69e2P9+vWqk8TixYsREhKC7t27o2fPnkhPT8euXbsQGBgIKysrdO/eHb169UJaWhoiIiIwadIkfPzxx4Wu86lTp3Dz5k2MGzcOt2/fRq1atXDlyhV8/fXXqFChAgYPHgxDQ0Ps27cPgwYNQmBgIDp16lToOBcvXsTp06fh6+sLMzMzrFq1Cv7+/qhTpw5Kly6N0aNHIy4uDtu2bYO1tTWGDx+ueu3Ro0exZ88e+Pj4wMrKClu3bsXAgQPx448/wsPDAwAQGRmJ4cOHw8bGBkOGDAEA7NixA/369cPSpUvh5eWl2l54eDjs7OwwZcoUPHr0CM2aNYORkZHkOO7YsQNTp05FmzZtMHbsWGRnZyMiIgLr1q1DmTJl1Or477//wt/fH7169UKvXr2wb98+BAcHw9LSEr179wYA3LhxAz179oSBgQH69OkDS0tL/PTTT+jfvz82b94MZ2fnYjn2JDNB9IaWLl0qateuXeDfsWPHhBBC5ObmCi8vL/Hll1+KnJwc1evT09NF27ZtRdeuXYUQQmRlZYkGDRqI0aNHq8VJTU0Vjo6OYvDgwZLYt27dUpX16dNHtG7dWlLPl8v79OmjVr8Xyz/55BORnp6uKsvOzhZff/21aNq0qcjMzCzwWBw7dkzUrl1bLF26VBLn0KFDqrKwsDBRu3Zt0bNnT1WZUqkULVq0EL169VKVtW7dWtSuXVtERESoypKSkoS7u7vqtdnZ2aJFixaiZcuWIjU1VbVeSkqK8PT0FJ6eniIrK0u1PQcHB3Hz5k21ems6jh06dBC9evUSSqVS7Ti0aNFCdO7cWbJ/kZGRqrKMjAzRsGFDtX0ZNWqUcHZ2Fjdu3FCVJScnCzc3NzFy5EjVtop67Ek3eOVAhTZhwgQ4ODhIyvPL/vnnH9y6dQtfffUVUlJS1NZp3bo1NmzYgPv376NixYo4evQosrOz1dZ5/PgxTE1NZe1uaWxsrNbs8vjxY5w4cQI+Pj7IyMhARkaGalnbtm0xZ84cnD9/Hm5uboWKU6pUKbVmETs7OwBQ+0WvUChQpUoVPHjwQO21NWrUwCeffKL6v6WlJbp27YqwsDAkJSXhzp07uH//PsaOHQtTU1PVeubm5ujTpw8CAwNx4cIF1K9fHwBgY2MDGxub19Z5z549eP78ORQKhaosKSkJ5ubmkvegdOnSaNWqldr+2tnZ4dGjRwAApVKJ6OhotGzZEra2tqr1LCwssGXLFlhYWBTbsSd5MTlQodWrV++VvZUSEhIAAPPnz8f8+fM1rnPv3j1UrFgRhoaGiIqKQmRkJK5fv46bN2+qEoqQcTT5cuXKqbWJ37p1CwAQGhqqanfXVMeixDEw+P+vlb6+PgCgfPnyauvp6+tL9q9WrVqS7dna2kIIgTt37uD27dsA/j/hvCi/a/Hdu3dVyeHlmAUxNDREbGws9u3bh/j4eCQkJCApKQkAUKVKFcn+vXgcAcDIyAhKpRIA8OTJEzx79kwtMeSrXbs2gLweU4D8x57kxeRAsss/UYwaNQqurq4a16lRowaEEBg3bhz27dsHNzc31K9fH7169ULDhg3Rt2/fIsfPzc2VlOWfpF9ep3fv3mq/1l+k6WT9Oi8mhhe9+Ku8IPn3a16UX09NyeRF+cte3MbL+1yQwMBArF69GnXr1oWrqyu6du2K+vXr47vvvpOcpF9ODAXV91XrFdexJ3kxOZDs8n9tlilTBk2bNlVbdu7cOaSkpMDY2BhxcXHYt28fhg4dilGjRqnWycnJwZMnT1CtWrVXxtHT00NWVpakPL+J403qqK+vL6nj1atXcfv2bZQuXfq125FT/hXXi27evAl9fX1UrVpV1fwWHx8vWe/69esAgIoVKxYq5p07d7B69Wp07dpVcpX3JsfxZRYWFjA2NsbNmzcly9atW4dHjx6hf//+AN6tY09SfM6BZOfo6AgrKyuEhoYiPT1dVZ6WlgZ/f39MmjQJ+vr6ePLkCQDpr8Tt27fj+fPnyMnJUZXl/xJ98ddzhQoVkJSUpNZ2f+HCBY0nppdZW1vD0dERO3fuVHt9dnY2Jk+ejJEjR6rF14Xz58+ruv0CeSfnPXv2oHHjxihbtizq1aun6sWUlpamWi8tLQ1btmyBlZUVHB0dXxnj5eOY34T38nsQHR2NGzduFPoYGBgYoFmzZoiOjla76khJScG6deuQkJDwTh57kuKVA8nO0NAQ06ZNg7+/P7y9vfHFF1+gVKlS2LFjB+7evYuFCxfCwMAA9evXh6mpKebMmYO7d+/C3Nwcx48fx2+//YZSpUqpJRZLS0sAwNq1a9GiRQt4eXmhc+fO2LdvH/z8/PDVV18hKSkJoaGhqF69uuQmtyZTp05F37598fnnn+Orr75CuXLlEB4ejrNnzyIgIAAWFhbFdow0MTIygp+fH/r27QtjY2Ns2bIFSqUS48ePB6B+XD///HN88cUXAICff/4ZDx8+xNKlS1/b7PPycfT09ETlypUREhKCzMxMVKxYEefOncPOnTsl78GbCggIQI8ePdCjRw/07t0bpqam2L59O549ewZ/f38A796xJykmByoW7du3x48//oiVK1dixYoV0NPTw8cff4yVK1eidevWAPJ++a9evRoLFy7EihUrYGRkBDs7OyxatAjnzp3Dpk2b8OjRI1SoUAGffvopDh48iF9//RUnTpyAl5cXWrdujenTp2PTpk344YcfYGdnhxkzZiA2NhZRUVGvrWP9+vWxdetWLFu2DOvXr0dOTg7s7Owwd+5cdO/evZiPkJSrqys+/fRTrFixAqmpqXB3d0dAQIBaz7D847pixQosX74cBgYGcHFxwQ8//KD2gFlBNB3H1atXY+7cudi0aROEELCxscHkyZORk5ODH374ARcuXHjtFcmLatasiW3btmHRokVYu3Yt9PT04OzsjHnz5qmerXjXjj1JKYScXUKIqEjatGmDKlWqFNh7h0jXeM+BiIgkmByIiEiCyYGIiCR4z4GIiCR45UBERBJMDkREJFFinnN4/DgdSuWbt5CVL2+KpKS016+oJV3EKUn7UtLilKR9KWlxStK+FCWOnp4CFhYmBS4vMclBqRSFSg75r9EFXcQpSftS0uKUpH0paXFK0r7IHYfNSkREJMHkQEREEkwOREQkweRAREQSTA5ERCTB5EBERBJMDkREJFFinnP4EJiZl4ZxKc1vmZWVmcbyjMwcpD59XpzVIqISiMnhPWJcygBdAnYX6jV7A7sitZjqQ0QlF5uViIhIgsmBiIgkmByIiEiCyYGIiCSYHIiISILJgYiIJJgciIhIgsmBiIgkmByIiEiCyYGIiCSYHIiISILJgYiIJJgciIhIQtbksG/fPnz66adwdnZGx44dsWvXrleuv3v3btjb20v+Zs2aJWe1iIiokGQbsnv//v0YO3YsfH194enpiT/++AMTJkyAsbExOnTooPE1ly9fhq2tLebPn69WXqFCBbmqRURERSBbcli0aBE6duyIyZMnAwA8PT2RkpKCJUuWFJgcrly5gnr16sHV1VWuahARkQxkaVa6desWEhIS0K5dO7Xy9u3bIz4+Hrdu3dL4usuXL8Pe3l6OKhARkYxkSQ7x8fEAADs7O7VyW1tbAMD169clr3n48CGSkpLwzz//oEOHDqhXrx7at2//2vsURERU/GRpVkpNzZuI0tTUVK3cxMQEAJCWliZ5zeXLlwEAt2/fxrhx41CqVCns2rULEyZMQG5uLj7//HM5qkZEREUgS3IQQgAAFAqFxnI9PekFiqOjI0JCQtCwYUNVUmnevDmSkpKwZMmSQieH8uVNX7/SS6yszAr9mqLQVRxdxC9px0wXcUrSvpS0OCVpX+SOI0tyMDPLq9DLVwjp6elqy19kaWmJ1q1bS8pbtmyJo0ePIjk5GZaWlm9ch6SkNCiV4o3Xt7IyQ2Ji6huvX1RyxinqGy9n/PftmL3tOCVpX0panJK0L0WJo6eneOWPalnuOeTfa0hISFArv3nzptryF50+fRo7duyQlGdmZsLAwEBjQiEiIt2QJTnY2tqiatWqOHDggFr5wYMHUb16dVSuXFnymjNnzmDq1Kmqew8AoFQq8fvvv6NBgwYwNDSUo2pERFQEsj3nMGzYMEyaNAlly5ZFq1atcOjQIezfvx9BQUEAgOTkZCQkJKBWrVowNTWFt7c3QkNDMXz4cPj7+8PExARbtmzBv//+i82bN8tVLSIiKgLZhs/w9vbGzJkz8ddff2HYsGE4ceIE5s2bh06dOgEAoqKi0KtXL1y8eBEAULZsWYSGhsLZ2Rlz5syBv78/nj17hg0bNsDFxUWuahERURHIduUAAF9++SW+/PJLjcu8vb3h7e2tVlalShUsWrRIzioQEakxMy8N41KaT3UFdfLIyMxB6tPnxVmtd56syYGI6F1jXMoAXQJ2F+o1ewO7ovj7F73bOGQ3ERFJMDkQEZEEkwMREUkwORARkQSTAxERSTA5EBGRBJMDERFJMDkQEZEEkwMREUkwORARkQSTAxERSTA5EBGRBJMDERFJMDkQEZEEkwMREUkwORARkQQn+yGSAWcbo5KGyYFIBpxtjEoaNisREZEEkwMREUkwORARkQSTAxERSTA5EBGRBJMDERFJMDkQEZEEkwMREUkwORARkQSTAxERSTA5EBGRBJMDERFJMDkQEZEEkwMREUkwORARkQSTAxERSTA5EBGRBJMDERFJMDkQEZEEkwMREUkwORARkQSTAxERSTA5EBGRBJMDERFJMDkQEZEEkwMREUkwORARkQSTAxERSTA5EBGRBJMDERFJMDkQEZEEkwMREUkwORARkYSBnBvbt28fVq5ciVu3bqFKlSoYPHgwunXrVuD66enpWLhwIQ4ePIhnz57B3d0dU6ZMQfXq1eWsFn3AzMxLw7hUwR9zKyszSVlGZg5Snz4vzmoRvfNkSw779+/H2LFj4evrC09PT/zxxx+YMGECjI2N0aFDB42vGT16NM6fP4/x48fDxMQEwcHB8PX1RXh4OMzMpF9aosIyLmWALgG7C/WavYFdkVpM9SF6X8iWHBYtWoSOHTti8uTJAABPT0+kpKRgyZIlGpNDXFwcoqOjsWbNGrRo0QIA4O7uDi8vL2zduhWDBg2Sq2pERFRIstxzuHXrFhISEtCuXTu18vbt2yM+Ph63bt2SvCYmJgYmJiZo1qyZqszS0hINGzbEkSNH5KgWUYljZl4aVlZmkj8AGsutrMxgZl76Ldea3keyXDnEx8cDAOzs7NTKbW1tAQDXr19HtWrVJK+xtbWFvr6+WrmNjQ32798vR7WIShw2k5GuyJIcUlPzPnqmpqZq5SYmJgCAtLQ0yWvS0tIk6+e/RtP6RfWqG5K8GUlEpJksyUEIAQBQKBQay/X0pK1X+cs00bT+65QvL000AJCVnVuo7ejpKTQmjVfJys6FkaF+gcs1be91rykozt7AroV+jZz7U9C2irI/r1LYOhdEl8espMUpzGegqJ9nfm/kPW5yHTNApuSQ37Po5V/86enpastfZGpqitu3b0vK09PTNV5RvE5SUhqUyoITzsusrMyQmCjPxbaVlVmRLvXljC/XtgAUqY1aqRTv7P68zTjv674U9jNdlM9zSfve6Gp/5Hpv9PQUBf6oBmRKDvn3GhISEmBvb68qv3nzptryl1/z999/QwihdsVx8+ZNjeuT7qQ+fa6xjVpXJzoievtk6a1ka2uLqlWr4sCBA2rlBw8eRPXq1VG5cmXJa5o3b46nT5/i6NGjqrLk5GTExcWhadOmclSLiIiKSLbnHIYNG4ZJkyahbNmyaNWqFQ4dOoT9+/cjKCgIQN6JPyEhAbVq1YKpqSkaNmwIDw8PjBkzBmPHjkW5cuWwbNkymJmZ4auvvpKrWkREVASyJQdvb29kZWXhxx9/xI4dO1CtWjXMmzcPnTp1AgBERUVh0qRJ2LRpExo1agQACA4Oxty5czF//nwolUq4ublh8eLFKFu2rFzVIiKiIpB1bKUvv/wSX375pcZl3t7e8Pb2VisrW7Ys5syZgzlz5shZDSIi0hJHZSUiIgkmByIikmByICIiCSYHIiKSYHIgIiIJJgciIpJgciAiIgkmByIikmByICIiCSYHIiKSYHIgIiIJJgciIpJgciAiIgkmByIikmByICIiCSYHIiKSYHIgIiIJJgciIpJgciAiIgkmByIikmByICIiCSYHIiKSYHIgIiIJJgciIpJgciAiIgkmByIikmByICIiCSYHIiKSYHIgIiIJJgciIpJgciAiIgkmByIikmByICIiCSYHIiKSYHIgIiIJJgciIpJgciAiIgkmByIikmByICIiCSYHIiKSYHIgIiIJJgciIpJgciAiIgkmByIikmByICIiCSYHIiKSYHIgIiIJJgciIpJgciAiIgkmByIikmByICIiCSYHIiKSMJBrQ+np6Vi4cCEOHjyIZ8+ewd3dHVOmTEH16tVf+bp+/frh77//lpT//PPPcHJykqt6RERUCLIlh9GjR+P8+fMYP348TExMEBwcDF9fX4SHh8PMzKzA112+fBm+vr749NNP1cpr1qwpV9WIiKiQZEkOcXFxiI6Oxpo1a9CiRQsAgLu7O7y8vLB161YMGjRI4+sePHiAx48fw9PTE66urnJUhYiIZCDLPYeYmBiYmJigWbNmqjJLS0s0bNgQR44cKfB1ly9fBgDY29vLUQ0iIpKJLMkhPj4etra20NfXVyu3sbHB9evXC3zd5cuXYWRkhKVLl6JRo0ZwcnKCn5/fK19DRETF77XNSjk5OQgPDy9weYUKFZCWlgZTU1PJMhMTE6SlpRX42suXLyMrKwvGxsYIDg7GvXv3sHz5cvTu3Ru7d++GlZXVG+4GERHJ6bXJITMzE+PHjy9wuYeHBwwNDQtcrqdX8MXJkCFD0KtXLzRu3FhVVr9+fXTs2BFhYWEYPXr066qnUr68NDm9jpVVwTfKdUHO+LraF8Z5N2PoMs7bjv8+fm/edh2KEuO1ycHExARXrlx55TojR47E7du3JeXp6ekaryjy1a5dW1JWrVo11KxZU3U/4k0lJaVBqRRvvL6VlRkSE1MLFeNV2yoKOePLtS3Gef9iFEeconymCxu/pH1vdLU/cr03enqKV/6oluWeg52dHW7dugUh1E/ON2/ehJ2dncbXCCGwa9cuxMXFSZZlZGTAwsJCjqoREVERyJIcmjdvjqdPn+Lo0aOqsuTkZMTFxaFp06YaX6NQKLBu3TrMnj0bSqVSVX7x4kUkJCTAw8NDjqoREVERyJIcGjZsCA8PD4wZMwY7duxAREQE+vXrBzMzM3z11Veq9a5evYp//vlH9f8RI0bg4sWLGDt2LGJiYrBjxw4MHjwYderUQdeuXeWoGhERFYFsT0gHBwdj7ty5mD9/PpRKJdzc3LB48WKULVtWtc7MmTNx584dHDp0CADQrl07LF++HCEhIRg+fDiMjY3Rtm1bjBkzRtItloiIdEe25FC2bFnMmTMHc+bMKXCd0NBQSdknn3yCTz75RK5qEBGRDDgqKxERSTA5EBGRhGzNSkREhZGRmYO9gYXreJKRmVNMtaGXMTkQ0VuR+vQ5Cnr8S1cPD1LB2KxEREQSTA5ERCTBZiUikijs/QDeCyh5mByISKKg+wG8F/DhYLMSERFJMDkQEZEEm5WIiGRQ0p7bYHIgIpJBSbtPw2YlIiKSYHIgIiIJJgciIpJgciAiIgnekCYieo/o6ul1JgcioveIrnpFsVmJiIgkmByIiEiCyYGIiCSYHIiISILJgYiIJJgciIhIgsmBiIgkmByIiEiCyYGIiCSYHIiISILJgYiIJJgciIhIgsmBiIgkmByIiEiCyYGIiCSYHIiISILJgYiIJJgciIhIgsmBiIgkmByIiEiCyYGIiCSYHIiISMLgbVegJMjIzMHewK6Ffg0R0buKyUEGqU+fI7WAZVZWZkhMLGgpEdG7ic1KREQkweRAREQSTA5ERCTB5EBERBJMDkREJMHkQEREEiWmK6uenkInrykKXcQpSftS0uKUpH0paXFK0r4UNs7r1lUIIYS2FSIiopKFzUpERCTB5EBERBJMDkREJMHkQEREEkwOREQkweRAREQSTA5ERCTB5EBERBJMDkREJMHkQEREEkwOREQkweQgo/v3779yeXR0tI5qUjySk5Nx+fJlKJVKrbbz4MGDQv0Vh3/++QeRkZFIS0srlu3Tm7tx40aByzIyMjBv3jzdVYZUPpiB99LS0rB582bExMQgMTERS5cuxZEjR1C3bl00adJElhgeHh749ttv8emnn0pi//DDD9i1axcuXbokS6yMjAz8+uuvOHnyJFJSUlC+fHk0adIEnTt3hoGB9oPtpqWlYfbs2ahXrx569+6N/fv3Y9y4ccjNzUWNGjWwbt06VKxYsUjbdnBwgELx5qNHanvMHj58iHHjxqFRo0YYOnQowsLC8MMPP0AIAUtLS2zatAm1atXSKoaupaSk4OzZs0hNTdWYrLt06aJ1jFe9TwqFAmXKlIGNjQ18fX3RrVu3Isfx9PTExo0bUaNGDbXyP//8EzNmzMC9e/fwzz//FHn7Lzp9+jRiYmLw8OFDfPPNN7h27Rrq1q2L8uXLy7J9IO/zdvr0aWRnZyP/9KpUKvH8+XPExcVh4cKFRdrutGnT3nhdhUKBWbNmFSlOvhIzZPerPHjwAH369EFiYiKcnZ1x48YNZGVl4cyZMwgKCsKaNWtkSRBNmjRBQEAADh06hG+//Rbm5uaIjo7G9OnTkZKSgtGjR8uwN8CtW7fQt29f3Lt3D7a2tihfvjzOnDmDPXv24Mcff8TGjRthYWGhVYzAwEAcOHAATZs2BQAsXLgQDg4OGDJkCBYvXowFCxYgMDCwSNuePXt2oZKDthYsWIBr165h4MCBUCqVCAkJQdOmTTFu3Dh8//33WLhwIUJCQmSJFRsbW+AyhUIBExMTVKtWDaampkWOERMTg+HDhyMjIwOaftspFApZksPEiROxaNEi2Nraon379rCyssKjR48QGRmJy5cv47PPPkNSUhKmTJkCQ0NDyY+iN2VjYwMfHx9s2LABH3/8MZKTkzF79myEh4fD3t4eixYt0npfsrKyMHbsWERERMDAwAA5OTno1asX1q1bh6tXr2LLli2wsbHROs7BgwcREBCA7Oxs1WdcCKH698sJsDBiYmLeeF1Zvl/iAzBq1CjRsWNH8ejRI5GdnS3s7e3FhQsXRG5urhg4cKDo3bu3bLH27dsnGjVqJFq2bCn8/f2Fg4OD+N///icSEhJkizFo0CDh5eUlrly5olZ+6dIl0aZNGzF+/HitY3h6eopt27YJIYQ4f/68sLe3FwcOHBBCCHHgwAHRqFEjrWPoSpMmTcSePXuEEELExsYKe3t7ERUVJYQQ4vDhw8LNzU22WPb29sLBwUH19/L/HRwcRN26dcWkSZNETk5OkWJ07dpVfPbZZ+Lo0aMiISFB3L59W/Inh1GjRomhQ4cKpVIpWTZmzBgREBAghBBi4cKFwtvbu8hxMjIyxMCBA0Xjxo3FqlWrhIeHh2jQoIFYv369yM3NLfJ2XzRv3jzRoEEDERkZKTIyMlTngFu3bokOHToIf39/WeJ0795d9OzZU1y8eFFMnjxZTJgwQVy9elWsW7dOODo6ir/++kuWOLrwQSQHNzc38dtvvwkhhMjJyVF9MIQQIioqStaTgxBCREZGijp16gh7e3vRvXt3kZqaKuv269evL/bv369x2b59+0TDhg21juHk5CRiY2OFEEIEBweLevXqqfbj77//Fq6urlrHyHf16lUxatQo0aRJE+Ho6Cg8PT3F6NGjxdWrV2XZvrOzs2pfAgMDhZOTk8jIyBBCCBETEyMaNGggSxwh8t57FxcXMWPGDHHixAkRHx8vYmNjxZw5c4Sjo6PYsGGDCAsLEx4eHiI4OLhIMZycnFTJrTi5urqKI0eOaFz2559/qj4DcnwesrOzxciRI4WDg4Po06ePuH//vlbbe1nz5rlPh9IAACAASURBVM1FWFiYEEJ6DggPDxdNmzaVJY6zs7OIiIgQQgixe/du0bVrV9WyefPmiT59+sgS51UyMzNFTEyM1tv5IJqVcnNzUapUqQKXCZluu6SmpmLBggX4+eefUa9ePXTu3BkrVqzAp59+imnTpuGTTz6RJY6ZmRlycnI0LjM0NIShoaHWMapUqYIrV67A3d0df/zxB1xdXVVNIdHR0ahatarWMQDgypUr+Oqrr1C6dGl4eXmhfPnySExMxOHDh3H48GH89NNPsLe31ypG9erVERsbCxcXF/z+++/w8PBQfR727NmD6tWry7AneVavXg0fHx8EBASoyuzs7ODu7g4TExMcPHgQmzdvhkKhwIYNGzBs2LBCx6hUqRIyMjJkq3NBTExMEB8fD09PT8mya9euwdjYGEBek01B36+CnDp1SlLm4+OD27dv4/Llyzh37pzafYAGDRoUsvbqUlJSYGtrq3FZuXLlZOuYoKenh7JlywIAbG1tER8fD6VSCT09PXh6emLnzp2yxLl79y5mzpyJEydOIDs7W1WuVCpV5zNt79V9EMnB3d0dq1evRtOmTVUnzvw2ue3bt2v9wcvXoUMHpKamwt/fHwMHDoSenh46duyI6dOnY8SIEfDy8kJwcLDWcYYOHYoFCxbA1tYWTk5OqvKEhAQsWbIEQ4cO1TrGl19+iblz52Lz5s2Ij49XtfuOGDECf/zxR6Fujr3KwoULUaNGDWzatAllypRRlT979gz9+vXD4sWLsXLlSq1i+Pn5YcKECVi3bh2ePXuG6dOnAwB69OiBixcvFvneiSaXLl3CiBEjNC5zc3PDmjVrAAC1a9d+be+2gvj5+WHJkiWoW7cuqlWrVuS6vk7nzp2xePFiGBkZoV27drC0tERSUhIiIyOxZMkSeHt7Iy0tDWFhYWqfwzfx9ddfa2wXzz+xjRgxAgqFQtVer+2JrlatWggPD0fz5s0ly44cOYKaNWtqtf18dnZ2OH36NBo2bIgaNWogKysLly9fRt26dZGeno6srCxZ4sydOxdxcXH4/PPPcerUKZQuXRqurq6IiYnBv//+i2XLlmkd44NIDmPHjsXXX3+Ndu3aoXHjxlAoFNi4cSOuXr2Ka9euYfPmzbLEqVSpEjZu3KjW88Xa2hohISHYuXMn5syZI0ucAwcO4NmzZ+jZsyeqVasGa2trPHnyBDdu3EBubi42bdqETZs2qdb//fffCx2jb9++KF++PGJjYzF8+HB06tQJAGBkZITvvvsOX3zxhSz7EhcXhwULFqglBgAoU6YMBg4ciClTpmgdo3PnzqhUqRJOnjwJDw8PuLq6AgAaNWqE0aNHq266y6FSpUo4fPgwmjVrJll2+PBhfPTRRwCAxMRElCtX7o23265dO7WT6a1bt9CuXTtUqFBBcuyAor3nLwsICMDjx48xa9YstZ4venp66Nq1K8aNG4eIiAicPXsWP/74Y6G2/eLnUxeGDBmCESNGICUlBa1bt4ZCocCpU6ewZ88ebN68GfPnz5clTq9evTBr1iw8e/YM/v7+aNy4MaZMmYIePXogNDQU9erVkyXO8ePHMXr0aPTp0wdhYWE4dOgQxo0bhzFjxmDAgAGIjIyEl5eXVjE+mK6s169fR3BwMI4fP44nT57A1NQUHh4eGDp0KBwcHGSJkX/5WJCHDx/C2tpa6ziTJk0qVGy5klJxaNq0KWbNmqWxye2PP/7A+PHjNTZBFJVSqURycjLMzc1hZGQk23bz7dixA9OmTUPHjh3Rtm1bWFpaIjk5GZGRkfjtt98wbdo0NG/eHIMGDUKDBg0we/bsN9ruxIkTC9UDRc73PCEhAcePH8fjx49hbW0NNzc31RVLSkoKjI2NC92s9Dbs3bsXgYGBaldslpaW8Pf3R8+ePWWLExoaijt37mDixIlISEiAn58fEhISUKVKFSxfvlzrZlIAcHR0xPr169GwYUMcPXoUY8aMwbFjxwDk/TCYN28eDh06pFWMD+LKAci73JOz+UATPT09KJVK/Pbbb6rnKaZOnYozZ87A0dFRtr70c+bMwZYtWxAXF6dq7omNjcWUKVPwzTffwNvbW5Y4//zzD1avXo24uDg8ffoU5cuXR+PGjTFkyBBZuv0BgKurK9asWQNPT0+1E0xGRgbWrFmD+vXryxInOjoaK1aswMWLF5Gbmwt9fX24uLhg1KhR8PDwkCUGkNdUpaenh+XLl2P//v2q8qpVq2LOnDno1q0bwsPDUbVqVYwdO/aNtzt37txXLhcvdJeUm42NTYHvd377urbOnz+P2NjYAp8N2Lp1q9YxunTpgi5duiA+Ph5PnjyBmZkZatas+cofdEWhr6+Phw8fAsg7dt999x0mTZqEoUOHypIYgLwWiUePHgHIu7eRkpKCxMREWFlZoVy5ckhKStI6xgeRHHTR9xzIuyE9cOBAnDt3DpUrV8bdu3eRnp6OvXv3YtasWQgLC0PdunW1igFA9RBXjx49VGUVK1aEu7s7pk2bBn19fXTt2lWrGEePHsWgQYNQvnx5tGnTBuXLl0dSUhIOHz6M33//HVu2bJHliisgIABffPEFvLy80KZNG1SoUAGPHj3CoUOHkJ6eLkuT3549ezB+/Hi4uLhg5MiRsLS0RGJiIn7//XcMGDAAISEhGtuii+rzzz/H559/joSEBCQnJ+Ojjz5CpUqVVMs//fTTIj8TkG/r1q2IjY1V/TiIi4vDlClTMGTIEHTv3l2rbefLzMzEqlWrEBUVhWfPnmnsuCFH89XWrVsxa9YsjdvX09OT9b25du0a4uLiVA+OlilTBlWqVJFt+5q+m5UqVUKjRo1k+24CeQ8OLl26FJUrV4aLiwsqVqyI9evXY8SIEdi1a5eq+VIrWvd3eg+82Nfc3t5e9Sdn33MhhJg6dapo3ry5uHTpktrzFKmpqaJnz57Cz89Plv1p165dgd0gly1bJjp37qx1DG9vbzFgwACRmZmpVp6RkSH69u0r+vbtq3WMfJcvXxYjRowQTZs2FfXq1RNNmjQRI0eOlDzHUVSdOnUSEyZM0LhsxIgRat0N5bBlyxYxevRo1f9PnDgh2rZtK3799VdZth8aGiocHBzEtGnTVGUJCQli0qRJom7dumLXrl2yxJk2bZpwcHAQvr6+Yvz48WLixImSPzl07NhRDBo0SDx58kTMmzdPTJs2TTx//lwcOHBAuLq6ir1792od4/nz52LEiBGSc0CdOnXEtGnTND7LURS6+G4KIURSUpLo3r278PX1FULkdZutU6eO6ny2ZcsWrWN8EMlBF33PhRCiUaNGqhPAy32pIyIihIeHhyz74+zsLI4ePapx2dGjR4Wzs7MsMaKjozUui4qKkvU5h+Lm5ORU4MNHf/31l3BycpItli5O3Lo6AXl4eIhVq1bJsq1XcXR0VD23ER4eLjp16qRatmLFCtGjRw+tY8yYMUO4urqKsLAw8fDhQ5GTkyMePHggNm7cKFxdXcWyZcu0jiGEbr6bL3rxeZDY2FixZs0acfz4cVm2/UE0K+mi7zmQ105uaWmpcVmpUqVk68ZWuXJlHD9+XOOQHydPnpTlktLW1hb//vsvWrRoIVl2586dIo+rpElxjxNUt25dxMbGauxB9N9//8k6rlJoaCiGDx+u9hmqVq0aZs+ejcqVK2Pt2rVaNyvcv3+/wO7XL3aX1VZWVhacnZ1l2darGBoaqp6ZsLW1xc2bN5GdnQ1DQ0O4ublh/fr1Wsc4cOAAxowZg969e6vKrK2t4evrC6VSifXr12P48OFax9HFd/NFL27P3d0d7u7usm37g0gOuuh7DuT1INi6dStatmwpWfbbb7/Jcr8ByOsut3DhQuTm5uKTTz6BpaUlHj9+jEOHDmHdunUYNWqU1jFmzJiB4cOHQ6FQoHPnzrCyssKTJ08QFRWFJUuWYNq0aWojphb1Q19c4wS92MOpa9eumD17Np4/f4727dujQoUKSElJwZ9//omNGzdi5syZRaq7Jro4cevqBNS8eXMcOXIEjRs3lmV7BXFwcEBUVBQaNWoEOzs7KJVKnD17Fu7u7rKNypudnV3gMyE1a9ZEamqqLHF08d0E8jogbN++HceOHcPTp08l3x2FQoF169ZpFeODSA7F1ff8ZaNGjUL//v3h7e2Nli1bQqFQYP/+/Vi5cqXqwyGHfv364cGDB1i/fj3Wrl2rKtfX14ePjw8GDhyodQwfHx8olUosWLBAbRTJ/A/huHHj1NYv6kNKCxYsgI2NDSZOnIiqVavK1nPk5YeshBDYuHGjWv/6/H0ZPXo0OnToIEtcXZy4dXUC+uyzzzB16lQ8fvwYDRo0UP26f5EcA/z17dsXo0aNQmpqKr7//nt4eXlh/Pjx6NixI3bv3g03NzetY3Tr1g1r1qxB48aN1fZDqVRi69at6Ny5s9YxAN18NwFg8eLFWLVqFSpVqoQqVaoUS0+1D+I5h1f1PQ8PD8f06dOL1Pdck9jYWAQGBuLcuXNQKpVQKBSoW7cuRo0apbGJRhupqak4c+aMqlues7Nzgc1ahVXYx/yL2kPG2dkZy5Yt03i1pY1mzZrB19cXrq6uCA4ORo8ePV7ZFCZXd9YNGzZg4cKF6N+/f4EnbjlOEPPmzUNoaChyc3NVZfknoAkTJmi9fQCv7Y0mx5PL+SIjIxEfHw8/Pz88fvwYAQEBOHXqFJycnDB//ny13l5FsWzZMoSGhsLAwABt2rRRPTj6559/4s6dO+jSpYvquRc5hrsuzu8mkHdV16FDB0ydOlW2bb7sg0gOAPDLL79g+fLluHv3rqqsatWqGD58uKrv+c6dOzF//nyt3sT09HSkpaWhbNmySE5Oxq+//orU1FS0bdtW1vbAkqJ9+/YYM2YM2rdvL+t2nZ2dsWDBArRv3x516tTBtm3bdNJ+DujmxA3knYBOnz6NlJSUYjkB3blz57XryNkNtDi1adPmjddVKBSIjIwsxtpoz8XFBatXr0ajRo2KLcYHkxyAvJnMrl27BoVCgapVq0IIgefPn+PkyZNq/ZKL6uzZs/Dz80OvXr0QEBCAb7/9Ftu3b4eZmRnS09OxbNmyQn1I37akpCRs2LAB586dw8OHD2FhYYFGjRrBx8dHtpPQzz//jB9//BGrVq2SdZwgHx8fnD59GtbW1rh79y6srKwKfCJaoVDgjz/+kC02UPy/HF/l5s2bBQ4y9y7TxUQ8JcWAAQPQuHFjDBo0qNhifBDJ4cqVKxg7diyuXr2qcblCoZBlpqn+/fvj+fPnWLBgASpUqIAmTZrA29sb06dPx/Tp03Hp0iXs2LFD6zi6cP78efTr1w9KpRINGjRA+fLl8ejRI5w8eRLm5uYICwsr8glI0zhBQghZxwl6+PAhQkND8eTJE+zYsQNt2rR55cn5+++/L3SM17l79y4ePnyI2rVrQ6FQoHTp0rJsNzU1FUFBQYiNjVXrAZf/RHFSUpJszT0HDhwo8Mnl06dP4/Dhw1rHyJ+I5+DBgzA0NEROTg5+/vlnLFiwQNaJeIC8nnFnzpxBWloaLC0t4eTkpPXDr7ryYieLq1evYvbs2RgwYADc3Nw0fra0HVD0g7ghPX/+fDx58gQTJkzA4cOHYWRkhNatW+PIkSOIjo6WbRCws2fPIigoCNWqVcMff/yBzMxMVbfFTp06Yc+ePbLE0YX58+ejZs2aWL16tdpN+sTERAwcOBDff/99kXveNGjQQC05vPwhft0YVW/C2tpa1XU5JiYGI0eOlG0Mrdc5dOgQ5s+fj5s3b0KhUGDHjh1YsWIFypYti++++w76+vpabX/27NnYu3cvPD09ER8fj9KlS6N69eo4efIkkpOTtW4vz7d8+XIsW7ZMNUS8oaEhDAwMkJycDD09PVmutoG8m6sxMTFYsWIFmjVrBhcXFwB5CdvPzw9BQUEICgrSOs7KlSsREhKCzMxMVZmRkREGDRokSzfW4vZiJ4v8RL1ixQoAkHS+kOV+kCxPS7zjGjRoIHbs2CGEEOKnn35Sm/ltxIgRYuTIkbLEcXNzE3///bcQQojp06cLDw8P1ZOXERER79Xsac7OzuLQoUMal0VERAgXFxfZYm3evFnjE8W//PKLbDF0JTIyUjg4OIhhw4aJn3/+WfUgZFhYmKhXr55Yvny51jGaNm0qQkJChBBCrFu3TgwePFgIIURaWpro1q2bmD17ttYxhBDCy8tLTJgwQSiVShEUFKSaYfD8+fOiSZMmIjQ0VJY4upiIZ9u2bcLBwUHMmDFDnDx5Uty8eVPExsaKb7/9VtSpU0ds375d6xjF7fjx42p/kZGRkrLjx4+LiIgI1eRm2pB3xKl3VFZWlmpCl+rVq+Py5cuqZd7e3jhz5owscRwdHbFjxw6cOXMGBw4cQKtWraBQKJCUlIQ1a9bA0dFRlji68NFHH6kG9npZdna2bO3nYWFh+O6779Qu7V8cJ2r37t2yxNGVZcuWoXv37ggODka3bt1U5b1798awYcNk2Z+UlBTVgIS1atXChQsXAORNztO/f39ERUVpHQPIe2ajS5cuUCgUqFevHk6fPg0g73P+zTffyNZEqouJeDZu3AgfHx98++23aNCgAWxsbODu7o4ZM2agT58+CA0N1TpGcfPw8FD99e3bFxUqVFAry/8zNTWVpePDB9GsVLlyZdy+fRvu7u6oXr060tLScOfOHVSpUgWlSpVCSkqKLHHGjRuHgQMHIjw8HJaWlhgyZAiAvPkEhBCFHvP+bQoICMCMGTNgaWmpNi58XFwcAgMDMXr0aFni6OKJYl26du2a2pP4L3Jzc9N64iJA/YRpa2uLpKQkPHnyBOXKlUOlSpVke3CsTJkyquY9Gxsb3L59GxkZGTA2NkadOnVw+/ZtWeLoYiKeW7duoVWrVhqXtWrVCtu2bdM6RnGbMGEC7t27ByCv6WjGjBka75fcuHEDFSpU0DreB5EcPvnkEyxcuBAmJiZo27YtatSogSVLlmDw4MHYsGGDbL1k6tWrh4iICFy7dg0ff/yx6ubqd999hwYNGuist0pR1atXT63tMicnB8OHD4eBgQEsLS3x9OlTZGRkwMDAAIGBgbI8AKWroSB0xcLCAjdu3NB4ortx4wYsLCy0jtGkSROsWrUKderUgY2NDcqWLYtdu3ahX79+iIqKkiUGADg5OWH37t1o0qQJ7OzsoK+vj2PHjqFVq1a4fv26bPNh6GIinkqVKuHatWsaJ3b677//ZBt6vDh17NgRGzduVP1fX19fcv9KT08Pbm5u6N+/v9bxPojkMHz4cNy8eRPbt29H27ZtMWnSJAwfPhx79+6Fvr6+athjOZiamqpuqOWTa+7o4vbNN98U25wABdH1WDTFrVOnTliyZAkqVqyoShAKhQKXL1/GihUrZHkSe+TIkejTpw/GjRuHsLAwDB48GHPnzsXq1avx+PHjIo8N9rJBgwbhf//7H1JSUrBy5Up89tlnmDBhApo0aYLo6GjZPtdt27bFggULEBgYqJqg5ocffoClpSWmT5+umoVQGy++L23btlWVHzx4EMHBwbLNgVKcWrVqpbr68fHxwYwZM2Sb3lSTD6Ira76srCzVr51bt27hwoULqFevnmzd5D4kDx48kOXErasninUlMzMTw4YNw19//QUDAwPk5OTA3NwcqampqF+/PtauXauxu25hZWRkID4+XjVe1969e3Hq1Ck4OzvLNp8DkDfh07///otu3bohMzMT33//vSrOxIkTZf/FXVwT8WRmZmLgwIGIjY2FkZGRan6S7Oxs1RzzcnU1Lik+qORAb+5VTxXHxcXBz89PdYNSW7p6oliXYmJicOzYMbUpafM7KLwvdDXtbb6XJ+Jp3Lix7E9gR0VFITY2Fk+fPoW5uTk8PDzQokWL9+p90RUmB1L58ccf8ezZMwBQjUek6erg9OnTuHDhAo4fPy5b7Lf5RLGcdu3ahZYtW2ps909MTMTevXsxYMCAQm932rRpb7yuHGMDAUDPnj2xcOFCjVfWu3btwpw5c2T5DGRkZGD8+PGIiIhQG11UT08PX3zxBWbOnMmT91vwQdxzoDeTk5Oj6k2jUCjw66+/StbR09ODubk5xowZI2tsMzMzeHp6yrrNt2HSpEnYtm2bxuRw6dIlBAUFFSk5xMTEvPG6cp1Ik5KS0LVrV0ycOBG9evUCkHe1MH36dERFRck2fee8efPw559/YurUqWjXrh0sLS2RlJSEAwcOICgoCNbW1kV6SK0wx1mOIa5LGl45kEYODg7Yvn27zgare58NHjxYNTTLnTt3ChzHKSkpCdWqVcPevXt1XcUiefbsGebOnYvt27ejVatWaNmyJYKCgmBkZIRJkyZpPQ92viZNmmDo0KHw8fGRLNuwYQPWr1+P6OjoQm9X0/Ze5X141kGXeOVAGr34oGC+3NxcPH/+/L0Zi0ZXhgwZgp9//hlA3kCCTk5Okiax/Cuu/F/gxSUrKwtxcXEau2wWVpkyZTBr1ix4enpi1KhRiI6ORp06dbBp0yZZPwPFNRHPyyf7/BGTP/roI2RlZWHz5s24f/8+R0wuAJMDaZSbm4uQkBDY2NigS5cuOH78OEaOHImnT5+iadOmCAoKgrm5+duu5jvB1dUVrq6uAPKO29ChQ2UdYfZld+/excyZM3HixAlkZ2erypVKparNXq6B98LDwzFnzhyUKVMGjRo1QmRkJMaOHYsZM2bINlWsLibieXnE5NmzZ2Pbtm0wMzNDWFjYezdisi6wWYk0CgoKwtq1azF58mT07t0bXbp0QVZWFnr37o3169ejRYsWsk6vWdI8e/ZM1WU1IiIC9+7dQ+vWrWVJGiNHjkRMTAy6d++OU6dOoXTp0nB1dUVMTAz+/fdfLFu2TO2p9qL63//+h6NHj6JZs2b4/vvvUbFiRURHR2P69OlITU2Fv78/fH19tY6ji4l4StKIyTqj9ehMVCK1adNGrF27VgghxNWrV4W9vb3YuXOnEEKI3bt3i2bNmr3N6r2zrl27Jtq2bSuCgoKEEEIEBQUJe3t7YW9vL1xcXMTJkye1juHh4aEa9C40NFT0799fCJE3aJ2vr6+YNGmS1jGEyBtIUtOAdKmpqWLixInCwcFBljitW7d+4782bdoUKUb9+vVFVFSUECJv4EgHBwdx5swZIYQQf//9t6wDSZYUbFYijR4+fKh60jsqKgp6enqqaU4rVqwo24TsJU1gYCD09fXh5eWFrKwsbNmyBZ06dcKsWbMwceJEBAUFaX3jMz09Hfb29gCAGjVqIDg4GEDesyFff/015s2bp/V+AMC+fftQsWJF3L9/H8eOHcPDhw/RvXt3JCYmYubMmbLNu53/VHRx0tPTQ6lSpQAAf/75J8zNzVWdLdLS0jTOj/2hY3IgjaytrVWDFR46dAh16tRR3WQ9ffq0bO3NJU1sbCx++OEHODk54a+//kJqaip69eoFU1NTfPnllxgxYoTWMaytrVUj5tra2iIlJQWJiYmwsrJCuXLlkJSUpHUMIO9HQP4Dijk5OVAoFGjWrBkWLVqEBw8eqI3z867LHzHZ2Nj4vR8xWVeYHEijzp07Y86cOdi7dy9OnjyJ6dOnA8gb82br1q2qEWdJXXZ2tmpIiSNHjqB06dJwc3MDkHez2sBA+6+cp6cnli5disqVK8PFxQUVK1bE+vXrMWLECOzatUu28ahWr16N0NBQjB8/Hq1bt1aNSTR8+HCMHDkSQUFBRZ5B7+XZAF+nKLMBvqgkjZisK0wOpJG/vz/KlCmD2NhYBAQE4OuvvwaQN9aOn58fhg4d+pZr+G6qXbs2Dh48CDs7Oxw4cADNmzeHgYEBsrOzsXnzZtSuXVvrGPnjTS1atAgbN27E6NGjMXHiRKxfvx4AVIlcW9u2bcOIESPg6+urNrRJ/fr14e/vjyVLlhR52y/OBqhUKhEeHg4zMzO0bNkSVlZWePLkCWJiYpCcnCxL99/3fcTkt+Jt3/QgKkn++usv4eLiIhwcHISzs7M4d+6cECLvpquzs7M4evSobLHu37+v+ndsbKxYs2aNOH78uGzbd3R0FDExMUII6Qxtf//9t3BycpIlzoIFC8RXX30lnj17plaemZkpBgwYIKZOnSpLHCocXjlQgZRKJX777TfExMQgMTERU6dOxZkzZ+Do6IhatWq97eq9k5o1a4a9e/fi/PnzcHFxUQ0cN2DAADRu3FjW4/Zi85G7u7vsD3LZ2Njgzz//1PhAXVxcnGzPcuzYsQNz586VjIpqZGQEX19fjBkzBt99950ssejNMTmQRqmpqRg4cCDOnTuHypUr4+7du0hPT8fevXsxa9YshIWFqYaLJnXVqlVDtWrVkJOTg8TERFhYWKBPnz6ybV8Ige3bt+PYsWN4+vSp2mB1gHzjBPXt2xfffvstcnJy0KZNGygUCty6dQsnT57EunXrMHbsWK1j5CtoNsb79++rehmRbjE5kEbz58/H3bt3sXPnTtSqVUvVm2PJkiX43//+h8WLF2P16tVvuZbvpgsXLiAoKAgnTpxAbm4uduzYgdDQUFSrVk2WiXgWL16MVatWoVKlSqhSpUqxjVjas2dPPH78GCEhIQgLC4MQAv7+/jA0NMSAAQPQu3dvWeK0adMGCxcuhLW1tdpVyqFDh7Bo0SJZZhykwmNyII0iIiIwYcIEODg4qN2MNDU1hZ+fH6ZMmfIWa/fuOnXqFPr164ePP/4YgwYNwooVKwDkdQsNDg6GhYWF6uZ+Uf3yyy/o06cPpk6dKkeVX2nw4MHo3bs3Tp8+rRpO3cXFRbapSIG8kWyvXr2KAQMGwNjYGBYWFkhOTkZWVhaaNWuGcePGyRaL3hyTA2mUkZFRYA+OUqVKISsrS8c1ej8sXLgQTZs2RUhICHJycrB8+XIAeb2/MjIysHXrVq2TQ2pqqtpUl8XN1NS0WIdTNzc3x/bt2xEdHY24uDg8ffoUFhYWaNy4scbpY0k3oHV/fgAABCxJREFUmBxII0dHR2zduhUtW7aULPvtt994v6EAFy9exNKlSwFI51Vo3bo1fvrpJ61juLm54ezZs2jUqJHW23pXKBQKtTmS6e1jciCNRo0ahf79+8Pb2xstW7aEQqHA/v37sXLlShw+fBhr165921V8J5mYmBT4hPKDBw9gYmJSpO2eOnVK9e8OHTpg9uzZyMjIgJubm8a5jxs0aFCkOLryNma2o8LhqKxUoNjYWAQGBuLcuXNQKpVQKBSoU6cO/P39VeMskbqpU6fiyJEjCAkJgb29PerVq4dff/0VVlZW6NevH1xcXDB79uxCb9fBwUF1JaKpd1I+IQQUCoVsQ3YXl8IMj61QKBAZGVmMtSFNmBxIo1OnTsHJyQmGhobIyMhASkoKTE1Ni/zL90Px5MkT9O3bF//99x8++ugj3Lt3D7Vq1cKdO3dgbW2NrVu3Fulp3BMnTqj9Py0tTeOEO0+fPkV2djY6duxY5H14W/777z+cOHEC6enpsLCwQIMGDVCzZs23Xa0PFpMDaeTp6YmAgAB069btbVflvTJv3jx4eXkhPj4ex44dw5MnT2BqagoPDw94e3urhm3QRp06dbBt2zaNU7geO3YMgwYNwrlz57SOoytCCEybNg2//PKL2lWRQqFAt27dMHv27GLrrksF4z0H0sjAwIDTgRbB9u3b0bx5c/Ts2RM9e/aUbbsTJkzAvXv3AOSdTGfMmKHx/blx4wYqVKggW1xdWLVqFXbt2oWAgAB06dIFFSpUQGJiIvbu3YulS5eiRo0a8PPze9vV/ODoz5gxY8bbrgS9e8qUKYM5c+aompRSUlJw7949tb9KlSq97Wq+c2JiYpCZmYlmzZrJul1DQ0OcO3cOenp6uH37NipWrAhjY2Po6emp/gwMDGBnZ4dJkybJNjKrLkycOBF9+vTBN998A1NTU+jp6cHU1BRubm5QKpXYtWsXfHx83nY1PzhsViKNHBwc1P7/Pt70fBvy5z+oVKkSatWqJfkVL0fPGx8fH8yYMaPEtMe7uLggJCRE4zMNR48exTfffPNeNZOVFGxWIo02bdr02pueJPX777/D2toaubm5uHLlCq5cuaK2XI62c21nknvXVKtWDadPn9aYHE6fPg0rK6u3UCticiCN+vbt+9qbnu9jj5jipospL0uaL774AosWLUKZMmXQqVMnVKhQAY8ePUJ4eDhWrVqFwYMHv+0qfpDYrEQqL970PHHiBOrWrVvgTU8DAwOeCEkWubm5mDx5Mnbv3i1pvvzss88wd+5c6OnpvcUafpiYHEglKipKNS/w33//DScnJ0ly0NPTg7m5Ofr376/xqoKoqK5evYrY2FikpKTA3NwcHh4enDfkLWJyII1K2k1PIiocJgciIpJgQx4REUkwORARkQSTAxERSTA5EBGRBJMDERFJ/B9gtrxPVARGPQAAAABJRU5ErkJggg==\n"
},
"metadata": {}
}
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": "You'll notice some are negative and some are positive.\n\nThe larger the value (bigger bar), the more the feature contributes to the models decision.\n\nIf the value is negative, it means there's a negative correlation. And vice versa for positive values.\n\nFor example, the sex
attribute has a negative value of -0.904, which means as the value for sex
increases, the target value decreases.\n\nWe can see this by comparing the sex
column to the target column."
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "pd.crosstab(df["sex"], df["target"])",
"execution_count": 108,
"outputs": [
{
"output_type": "execute_result",
"execution_count": 108,
"data": {
"text/plain": "target 0 1\nsex \n0 24 72\n1 114 93",
"text/html": "(pd.crosstab(df[\"slope\"], df[\"target\"])
it is. As slope goes up, so does target.\n\nWhat can you do with this information?\n\nThis is something you might want to talk to a subject matter expert about. They may be interested in seeing where machine learning model is finding the most patterns (highest correlation) as well as where it's not (lowest correlation).\n\nDoing this has a few benefits:\n\n1. Finding out more - If some of the correlations and feature importances are confusing, a subject matter expert may be able to shed some light on the situation and help you figure out more.\n2. Redirecting efforts - If some features offer far more value than others, this may change how you collect data for different problems. See point 3.\n3. Less but better - Similar to above, if some features are offering far more value than others, you could reduce the number of features your model tries to find patterns in as well as improve the ones which offer the most. This could potentially lead to saving on computation, by having a model find patterns across less features, whilst still achieving the same performance levels.\n\n## 6. Experimentation\n\nWell we've completed all the metrics your boss requested. You should be able to put together a great report containing a confusion matrix, a handful of cross-valdated metrics such as precision, recall and F1 as well as which features contribute most to the model making a decision.\n\nBut after all this you might be wondering where step 6 in the framework is, experimentation.\n\nWell the secret here is, as you might've guessed, the whole thing is experimentation.\n\nFrom trying different models, to tuning different models to figuring out which hyperparameters were best.\n\nWhat we've worked through so far has been a series of experiments.\n\nAnd the truth is, we could keep going. But of course, things can't go on forever.\n\nSo by this stage, after trying a few different things, we'd ask ourselves did we meet the evaluation metric?\n\nRemember we defined one in step 3.\n\n> If we can reach 95% accuracy at predicting whether or not a patient has heart disease during the proof of concept, we'll pursure this project.\n\nIn this case, we didn't. The highest accuracy our model achieved was below 90%.\n\nWhat next?\n\nYou might be wondering, what happens when the evaluation metric doesn't get hit?\n\nIs everything we've done wasted?\n\nNo.\n\nIt means we know what doesn't work. In this case, we know the current model we're using (a tuned version of LogisticRegression
) along with our specific data set doesn't hit the target we set ourselves.\n\nThis is where step 6 comes into its own.\n\nA good next step would be to discuss with your team or research on your own different options of going forward.\n\n* Could you collect more data?\n* Could you try a better model? If you're working with structured data, you might want to look into CatBoost or XGBoost.\n* Could you improve the current models (beyond what we've done so far)?\n* If your model is good enough, how would you export it and share it with others? (Hint: check out Scikit-Learn's documentation on model persistance)\n\nThe key here is to remember, your biggest restriction will be time. Hence, why it's paramount to minimise your times between experiments.\n\nThe more you try, the more you figure out what doesn't work, the more you'll start to get a hang of what does."
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "",
"execution_count": null,
"outputs": []
}
],
"metadata": {
"_draft": {
"nbviewer_url": "https://gist.github.com/c676d6d237931116001b7280dc8ba5c4"
},
"celltoolbar": "Initialization Cell",
"gist": {
"id": "c676d6d237931116001b7280dc8ba5c4",
"data": {
"description": "end-to-end-heart-disease-classification.ipynb",
"public": true
}
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3",
"language": "python"
},
"language_info": {
"name": "python",
"version": "3.8.2",
"mimetype": "text/x-python",
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"pygments_lexer": "ipython3",