3. ConvertRDDtoCSV.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# NYC Taxi Dataset Project - Convert RDD output to CSV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overall Steps\n",
    "\n",
    "**Step 0:** Prerequisites\n",
    "\n",
    "**Step 1:** Load data, reformat, save data to csv\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: You must complete DataPrepNew.ipynb, and download the single file containing the RDD output saved as a file called taxi_data.\n",
    "\n",
    "### Step 0: Prerequisites\n",
    "You will need a directory containing the following files:\n",
    "    \n",
    "    a) a subdirectory called data containing:\n",
    "    \n",
    "        1) taxi_data\n",
    "    \n",
    "    b) ConvertRDDtoCSV.ipynb (this notebook)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Load data, reformat, save data to csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Enter below which file containing taxi data you would like to convert from RDD to csv."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "file_to_convert = 'taxi_data_test'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "num_records_dropped = 0\n",
    "num_valid_records = 0\n",
    "\n",
    "# Open file to read from\n",
    "with open('./data/' + file_to_convert,'r') as in_fp:\n",
    "    # Open file to write to\n",
    "    with open('./data/' + file_to_convert + '.csv','w') as out_fp:\n",
    "        # Each line is of the format:\n",
    "        # ((time_cat, time_num, time_cos, time_sin, day_cat, day_num, day_cos, day_sin, weekend, geohash), number of pickups)\n",
    "        for line_str in in_fp:\n",
    "            # Bool indicating if record is valid\n",
    "            write_line_out = False\n",
    "            \n",
    "            # Get rid of parens, quotes, spaces\n",
    "            line_str = re.sub('[()\\'\\s]', '', line_str)\n",
    "            line = line_str.split(',')\n",
    "            \n",
    "            # Need to check if the datetime info is valid:\n",
    "            # (i.e. if all the data for is records was present in the original datafile)\n",
    "            #     If it is valid, line should have length 11\n",
    "            #     If it is not valid, line should be [None, geohash, number of taxis]\n",
    "            if len(line) == 15:\n",
    "            # Need to check the geohash is valid:\n",
    "            # (i.e. if latitude/longitude was present in the original datafile)\n",
    "                if line[12] is not None:\n",
    "                    csv_line = \",\".join(line[0:14]) + '\\n'\n",
    "                    write_line_out = True\n",
    "            \n",
    "            # Check if the line is valid\n",
    "            if write_line_out:\n",
    "                out_fp.write(csv_line)\n",
    "                num_valid_records += 1\n",
    "            else:\n",
    "                num_records_dropped += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of records dropped 0\n",
      "Number of valid records 7238532\n"
     ]
    }
   ],
   "source": [
    "# See how many records were dropped\n",
    "print \"Number of records dropped\",num_records_dropped\n",
    "# See how many records were valid\n",
    "print \"Number of valid records\",num_valid_records"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}