homework/HW1.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:d70152474cb236c2ba795571cbec8aea6182dc4faf74613b4705b605c2c87592"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Homework 1. Exploratory Data Analysis\n",
      "\n",
      "Due: Thursday, September 18, 2014 11:59 PM\n",
      "\n",
      "<a href=https://raw.githubusercontent.com/cs109/2014/master/homework/HW1.ipynb download=HW1.ipynb> Download this assignment</a>\n",
      "\n",
      "---"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Introduction\n",
      "\n",
      "In this homework we ask you three questions that we expect you to answer using data. For each question we ask you to complete a series of tasks that should help guide you through the data analysis. Complete these tasks and then write a short (100 words or less) answer to the question.\n",
      "\n",
      "**Note**: We will briefly discuss this homework assignment on Thursday in class.\n",
      "\n",
      "#### Data\n",
      "For this assignment we will use two databases: \n",
      "\n",
      "1. The [Sean Lahman's Baseball Database](http://seanlahman.com/baseball-archive/statistics) which contains the \"complete batting and pitching statistics from 1871 to 2013, plus fielding statistics, standings, team stats, managerial records, post-season data, and more. For more details on the latest release, please [read the documentation](http://seanlahman.com/files/database/readme2012.txt).\"\n",
      "\n",
      "2. [Gapminder](http://www.gapminder.org) is a great resource that contains over [500 data sets](http://www.gapminder.org/data/) related to world indicators such as income, GDP and life expectancy. \n",
      "\n",
      "\n",
      "#### Purpose\n",
      "\n",
      "In this assignment, you will learn how to: \n",
      "\n",
      "a. Load in CSV files from the web. \n",
      "\n",
      "b. Create functions in python. \n",
      "\n",
      "C. Create plots and summary statistics for exploratory data analysis such as histograms, boxplots and scatter plots. \n",
      "\n",
      "\n",
      "#### Useful libraries for this assignment \n",
      "\n",
      "* [numpy](http://docs.scipy.org/doc/numpy-dev/user/index.html), for arrays\n",
      "* [pandas](http://pandas.pydata.org/), for data frames\n",
      "* [matplotlib](http://matplotlib.org/), for plotting\n",
      "\n",
      "---"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# special IPython command to prepare the notebook for matplotlib\n",
      "%matplotlib inline \n",
      "\n",
      "import numpy as np\n",
      "import pandas as pd\n",
      "import matplotlib.pyplot as plt"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 1
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Problem 1\n",
      "\n",
      "In Lecture 1, we showed a plot that provided evidence that the 2002 and 2003 Oakland A's, a team that used data science, had a competitive advantage. Since, others teams have started using data science as well. Use exploratory data analysis to determine if the competitive advantage has since disappeared. "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### Problem 1(a) \n",
      "Load in [these CSV files](http://seanlahman.com/files/database/lahman-csv_2014-02-14.zip) from the [Sean Lahman's Baseball Database](http://seanlahman.com/baseball-archive/statistics). For this assignment, we will use the 'Salaries.csv' and 'Teams.csv' tables. Read these tables into a pandas `DataFrame` and show the head of each table. \n",
      "\n",
      "**Hint** Use the [requests](http://docs.python-requests.org/en/latest/), [StringIO](http://docs.python.org/2/library/stringio.html) and [zipfile](https://docs.python.org/2/library/zipfile.html) modules to get from the web.  "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#your code here"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 2
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### Problem 1(b)\n",
      "\n",
      "Summarize the Salaries DataFrame to show the total salaries for each team for each year. Show the head of the new summarized DataFrame. "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#your code here"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 3
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### Problem 1(c)\n",
      "\n",
      "Merge the new summarized Salaries DataFrame and Teams DataFrame together to create a new DataFrame\n",
      "showing wins and total salaries for each team for each year year. Show the head of the new merged DataFrame.\n",
      "\n",
      "**Hint**: Merge the DataFrames using `teamID` and `yearID`."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#your code here"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### Problem 1(d)\n",
      "\n",
      "How would you graphically display the relationship between total wins and total salaries for a given year? What kind of plot would be best? Choose a plot to show this relationship and specifically annotate the Oakland baseball team on the on the plot. Show this plot across multiple years. In which years can you detect a competitive advantage from the Oakland baseball team of using data science? When did this end? \n",
      "\n",
      "**Hints**: Use a `for` loop to consider multiple years. Use the `teamID` (three letter representation of the team name) to save space on the plot.  "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#your code here"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 4
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### Problem 1(e):\n",
      "\n",
      "**For AC209 Students**: Fit a linear regression to the data from each year and obtain the residuals. Plot the residuals against time to detect patterns that support your answer in 1(d). "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#your code here"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 5
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Discussion for Problem 1\n",
      "\n",
      "*Write a brief discussion of your conclusions to the questions and tasks above in 100 words or less.*\n",
      "\n",
      "---\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Problem 2\n",
      "\n",
      "Several media reports have demonstrated the income inequality has increased in the US during this last decade. Here we will look at global data. Use exploratory data analysis to determine if the gap between Africa/Latin America/Asia and Europe/NorthAmerica has increased, decreased or stayed the same during the last two decades. "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### Problem 2(a)\n",
      "\n",
      "Using the list of countries by continent from [World Atlas](http://www.worldatlas.com/cntycont.htm) data, load in the `countries.csv` file into a pandas DataFrame and name this data set as `countries`.  This data set can be found on Github in the 2014_data repository [here](https://github.com/cs109/2014_data/blob/master/countries.csv). "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#your code here"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 6
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Using the [data available on Gapminder](http://www.gapminder.org/data/), load in the [Income per person (GDP/capita, PPP$ inflation-adjusted)](https://spreadsheets.google.com/pub?key=phAwcNAVuyj1jiMAkmq1iMg&gid=0) as a pandas DataFrame and name this data set as `income`.\n",
      "\n",
      "**Hint**: Consider using the pandas function `pandas.read_excel()` to read in the .xlsx file directly."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#your code here"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 7
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Transform the data set to have years as the rows and countries as the columns. Show the head of this data set when it is loaded. "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#your code here"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 8
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### Problem 2(b)\n",
      "\n",
      "Graphically display the distribution of income per person across all countries in the world for any given year (e.g. 2000).  What kind of plot would be best? "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#your code here"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 9
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### Problem 2(c)\n",
      "\n",
      "Write a function to merge the `countries` and `income` data sets for any given year. "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "\"\"\"\n",
      "Function\n",
      "--------\n",
      "mergeByYear\n",
      "\n",
      "Return a merged DataFrame containing the income, \n",
      "country name and region for a given year. \n",
      "\n",
      "Parameters\n",
      "----------\n",
      "year : int\n",
      "    The year of interest\n",
      "\n",
      "Returns\n",
      "-------\n",
      "a DataFrame\n",
      "   A pandas DataFrame with three columns titled \n",
      "   'Country', 'Region', and 'Income'. \n",
      "\n",
      "Example\n",
      "-------\n",
      ">>> mergeByYear(2010)\n",
      "\"\"\"\n",
      "#your code here"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### Problem 2(d) \n",
      "\n",
      "Use exploratory data analysis tools such as histograms and boxplots to explore the distribution of the income per person by region data set from 2(c) for a given year. Describe how these change through the recent years?\n",
      "\n",
      "**Hint**: Use a `for` loop to consider multiple years.  "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#your code here"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 11
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Discussion for Problem 2\n",
      "\n",
      "*Write a brief discussion of your conclusions to the questions and tasks above in 100 words or less.*\n",
      "\n",
      "---\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Problem 3\n",
      "\n",
      "In general, if group A has larger values than group B on average, does this mean the largest values are from group A? Discuss after completing each of the problems below. "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### Problem 3(a)\n",
      "\n",
      "Assume you have two list of numbers, X and Y, with distribution approximately normal. X and Y have standard deviation equal to 1, but the average of X is different from the average of Y.  If the difference in the average of X and the average of Y is larger than 0, how does the proportion of X > a compare to the proportion of Y > a? "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Write a function that analytically calculates the ratio of these two proportions: Pr(X > a)/Pr(Y > a) as function of the difference in the average of X and the average of Y. \n",
      "\n",
      "**Hint**: Use the `scipy.stats` module for useful functions related to a normal random variable such as the probability density function, cumulative distribution function and survival function. \n",
      "\n",
      "**Update**: Assume Y is normally distributed with mean equal to 0. "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Show the curve for different values of a (a = 2,3,4 and 5)."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "\"\"\"\n",
      "Function\n",
      "--------\n",
      "ratioNormals\n",
      "\n",
      "Return ratio of these two proportions: \n",
      "    Pr(X > a)/Pr(Y > a) as function of \n",
      "    the difference in the average of X \n",
      "    and the average of Y. \n",
      "\n",
      "Parameters\n",
      "----------\n",
      "diff : difference in the average of X \n",
      "    and the average of Y. \n",
      "a : cutoff value\n",
      "\n",
      "Returns\n",
      "-------\n",
      "Returns ratio of these two proportions: \n",
      "    Pr(X > a)/Pr(Y > a)\n",
      "    \n",
      "Example\n",
      "-------\n",
      ">>> ratioNormals(diff = 1, a = 2)\n",
      "\"\"\"\n",
      "#your code here"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#your code here"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 13
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### Problem 3(b)\n",
      "\n",
      "Now consider the distribution of income per person from two regions: Asia and South America. Estimate the average income per person across the countries in those two regions. Which region has the larger average of income per person across the countries in that region? \n",
      "\n",
      "**Update**: Use the year 2012. "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#your code here"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 14
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### Problem 3(c) \n",
      "\n",
      "Calculate the proportion of countries with income per person that is greater than 10,000 dollars. Which region has a larger proportion of countries with income per person greater than 10,000 dollars? If the answer here is different from the answer in 3(b), explain why in light of your answer to 3(a).\n",
      "\n",
      "**Update**: Use the year 2012. "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#your code here"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 15
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#### Problem 3(d)\n",
      "\n",
      "**For AC209 Students**: Re-run this analysis in Problem 3 but compute the average income per person for each region, instead of the average of the reported incomes per person across countries in the region. Why are these two different? Hint: use this [data set](https://spreadsheets.google.com/pub?key=phAwcNAVuyj0XOoBL_n5tAQ&gid=0). "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#your code here"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 16
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Discussion for Problem 3\n",
      "\n",
      "*Write a brief discussion of your conclusions to the questions and tasks above in 100 words or less.*\n",
      "\n",
      "---\n"
     ]
    }
   ],
   "metadata": {}
  }
 ]
}