Skip to content

Commit 5bacede

Browse files
mebauermebauer
committed
testing all notebooks and source datasets
1 parent ea8b6d5 commit 5bacede

7 files changed

+200
-196
lines changed

‎1-reading-writing-files.ipynb

Lines changed: 70 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@
4747
"cell_type": "markdown",
4848
"metadata": {},
4949
"source": [
50-
"**Goal:** In this notebook, we review various ways to read (load) and write (save) data from NYC Open Data. Specifically, we will focus on reading our data into a pandas dataframe.\n",
50+
"**Goal:** In this notebook, we review various ways to read (load) and write (save) data from NYC Open Data. Specifically, we focus on reading our data into a pandas dataframe.\n",
5151
"\n",
5252
"**Main Library:** [pandas](https://pandas.pydata.org/) is a fast, powerful, flexible and easy to use open source data analysis and manipulation tool, built on top of the Python programming language."
5353
]
@@ -112,6 +112,7 @@
112112
}
113113
],
114114
"source": [
115+
"# watermark\n",
115116
"%reload_ext watermark\n",
116117
"%watermark -v -p numpy,pandas,geopandas,matplotlib,json,requests,sodapy"
117118
]
@@ -399,7 +400,7 @@
399400
"\n",
400401
"2) We use pandas `.shape` method to print the dimensions of the dataframe (i.e. number of rows, number of columns).\n",
401402
"\n",
402-
"We will use these two methods throughout the examples."
403+
"We will use these two methods extensively throughout the examples."
403404
]
404405
},
405406
{
@@ -833,7 +834,7 @@
833834
"metadata": {},
834835
"source": [
835836
"## 1.4 Reading in a Shapefile\n",
836-
"Information about [Shapefiles](https://desktop.arcgis.com/en/arcmap/latest/manage-data/shapefiles/what-is-a-shapefile.htm#:~:text=A%20shapefile%20is%20a%20simple,%2C%20or%20polygons%20(areas).)."
837+
"Information about [Shapefiles](https://desktop.arcgis.com/en/arcmap/latest/manage-data/shapefiles/what-is-a-shapefile.htm#:~:text=A%20shapefile%20is%20a%20simple,%2C%20or%20polygons%20(areas).). A popular Geospatial file format."
837838
]
838839
},
839840
{
@@ -1310,7 +1311,7 @@
13101311
"cell_type": "markdown",
13111312
"metadata": {},
13121313
"source": [
1313-
"## 2.1 Unzipping and reading in data as CSV to local folder\n",
1314+
"## 2.1 Unzipping and Reading In Data as CSV to Local Folder\n",
13141315
"\n",
13151316
"We will retrieve, unzip and read data in our downloads folder. Note: I'm using a Mac.\n",
13161317
"\n",
@@ -1614,7 +1615,7 @@
16141615
"cell_type": "markdown",
16151616
"metadata": {},
16161617
"source": [
1617-
"## 2.2 Unzipping and reading in data as CSV from local folder"
1618+
"## 2.2 Unzipping and Reading In Data as CSV from Local Folder"
16181619
]
16191620
},
16201621
{
@@ -1648,7 +1649,7 @@
16481649
}
16491650
],
16501651
"source": [
1651-
"# list files in this file path\n",
1652+
"# list files in this folder path\n",
16521653
"%ls data/unzipped-data/"
16531654
]
16541655
},
@@ -1862,7 +1863,7 @@
18621863
"cell_type": "markdown",
18631864
"metadata": {},
18641865
"source": [
1865-
"## 2.3 Unzipping and reading in data as a CSV in-memory"
1866+
"## 2.3 Unzipping and Reading In Data as a CSV In-Memory"
18661867
]
18671868
},
18681869
{
@@ -2106,7 +2107,7 @@
21062107
}
21072108
],
21082109
"source": [
2109-
"# read our csv data into a dataframe from our ZIP file\n",
2110+
"# read our CSV data into a dataframe from our ZIP file\n",
21102111
"file = zf.open('pluto_20v1.csv')\n",
21112112
"pluto_data = pd.read_csv(file, low_memory=False)\n",
21122113
"\n",
@@ -2136,14 +2137,14 @@
21362137
"cell_type": "markdown",
21372138
"metadata": {},
21382139
"source": [
2139-
"# 3. Reading in data from NYC Open Data"
2140+
"# 3. Reading In Data from NYC Open Data"
21402141
]
21412142
},
21422143
{
21432144
"cell_type": "markdown",
21442145
"metadata": {},
21452146
"source": [
2146-
"## 3.1 Reading in data as CSV in static form"
2147+
"## 3.1 Reading In Data as CSV in Static Form"
21472148
]
21482149
},
21492150
{
@@ -2365,8 +2366,8 @@
23652366
"cell_type": "markdown",
23662367
"metadata": {},
23672368
"source": [
2368-
"## 3.2 Reading in data as JSON in static form\n",
2369-
"Note: I do not read data on NYC Open Data this way, but reading JSON in static form does come up, especially when you're working with JSON data from the web. Here, I demonstrate a sample workflow.\n",
2369+
"## 3.2 Reading In Data as JSON in Static Form\n",
2370+
"Note: I do not read data on NYC Open Data this way, but reading JSON in static form does come up, especially when you're working with JSON data from the web. Understanding the structure of JSON data is a good skill. Here, I demonstrate a sample workflow.\n",
23702371
"![building_footprints_csv](images/building-footprints-json.png)"
23712372
]
23722373
},
@@ -2636,6 +2637,8 @@
26362637
"\n",
26372638
"# loop through columns and save to list\n",
26382639
"for col in cols:\n",
2640+
" \n",
2641+
" # sanity check\n",
26392642
" print(col['fieldName'])\n",
26402643
" \n",
26412644
" # append column name to list\n",
@@ -3072,7 +3075,7 @@
30723075
}
30733076
],
30743077
"source": [
3075-
"# sanity check\n",
3078+
"# column info sanity check\n",
30763079
"df_json.info()"
30773080
]
30783081
},
@@ -3100,7 +3103,7 @@
31003103
"cell_type": "markdown",
31013104
"metadata": {},
31023105
"source": [
3103-
"## 3.3 Reading in Shapefile data \n",
3106+
"## 3.3 Reading In Shapefile Data \n",
31043107
"![borough-boundaries-shp](images/borough-boundaries-shp.png)"
31053108
]
31063109
},
@@ -3236,14 +3239,32 @@
32363239
"cell_type": "code",
32373240
"execution_count": 38,
32383241
"metadata": {},
3242+
"outputs": [
3243+
{
3244+
"name": "stdout",
3245+
"output_type": "stream",
3246+
"text": [
3247+
"<class 'geopandas.geodataframe.GeoDataFrame'>\n"
3248+
]
3249+
}
3250+
],
3251+
"source": [
3252+
"# preview object type\n",
3253+
"print(type(gdf))"
3254+
]
3255+
},
3256+
{
3257+
"cell_type": "code",
3258+
"execution_count": 39,
3259+
"metadata": {},
32393260
"outputs": [
32403261
{
32413262
"data": {
32423263
"text/plain": [
32433264
"<AxesSubplot: >"
32443265
]
32453266
},
3246-
"execution_count": 38,
3267+
"execution_count": 39,
32473268
"metadata": {},
32483269
"output_type": "execute_result"
32493270
},
@@ -3296,7 +3317,7 @@
32963317
},
32973318
{
32983319
"cell_type": "code",
3299-
"execution_count": 39,
3320+
"execution_count": 40,
33003321
"metadata": {},
33013322
"outputs": [
33023323
{
@@ -3311,7 +3332,7 @@
33113332
"output_type": "stream",
33123333
"text": [
33133334
"client.__dict__:\n",
3314-
"{'domain': 'data.cityofnewyork.us', 'session': <requests.sessions.Session object at 0x1675f24d0>, 'uri_prefix': 'https://', 'timeout': 10}\n"
3335+
"{'domain': 'data.cityofnewyork.us', 'session': <requests.sessions.Session object at 0x160c734d0>, 'uri_prefix': 'https://', 'timeout': 10}\n"
33153336
]
33163337
}
33173338
],
@@ -3331,7 +3352,7 @@
33313352
},
33323353
{
33333354
"cell_type": "code",
3334-
"execution_count": 40,
3355+
"execution_count": 41,
33353356
"metadata": {},
33363357
"outputs": [
33373358
{
@@ -3497,14 +3518,14 @@
34973518
"4 {525F2C24-616B-4F29-98A3-8FEA5D4B1A7D} "
34983519
]
34993520
},
3500-
"execution_count": 40,
3521+
"execution_count": 41,
35013522
"metadata": {},
35023523
"output_type": "execute_result"
35033524
}
35043525
],
35053526
"source": [
3506-
"# we set the limit at 1,000 rows\n",
3507-
"# increase limit if you'd like, but there might be a limit if you haven't signed up for an app token\n",
3527+
"# we manually set the limit at 1,000 rows\n",
3528+
"# increase limit if you'd like, but there *might* be a limit if you haven't signed up for an app token\n",
35083529
"limit = 1_000\n",
35093530
"\n",
35103531
"# get data by passing dataset id and limit number\n",
@@ -3522,7 +3543,7 @@
35223543
},
35233544
{
35243545
"cell_type": "code",
3525-
"execution_count": 41,
3546+
"execution_count": 42,
35263547
"metadata": {},
35273548
"outputs": [
35283549
{
@@ -3542,13 +3563,13 @@
35423563
"cell_type": "markdown",
35433564
"metadata": {},
35443565
"source": [
3545-
"# 4. Writing out data\n",
3566+
"# 4. Writing Out Data\n",
35463567
"Save data to specified destination."
35473568
]
35483569
},
35493570
{
35503571
"cell_type": "code",
3551-
"execution_count": 42,
3572+
"execution_count": 43,
35523573
"metadata": {},
35533574
"outputs": [
35543575
{
@@ -3720,7 +3741,7 @@
37203741
"4 Other (Man {BB58FD7B-CC22-4896-901D-F8BAFF4AC129} "
37213742
]
37223743
},
3723-
"execution_count": 42,
3744+
"execution_count": 43,
37243745
"metadata": {},
37253746
"output_type": "execute_result"
37263747
}
@@ -3738,25 +3759,25 @@
37383759
"cell_type": "markdown",
37393760
"metadata": {},
37403761
"source": [
3741-
"## 4.1 Writing to a CSV file"
3762+
"## 4.1 Writing to a CSV File"
37423763
]
37433764
},
37443765
{
37453766
"cell_type": "code",
3746-
"execution_count": 43,
3767+
"execution_count": 44,
37473768
"metadata": {},
37483769
"outputs": [
37493770
{
37503771
"name": "stdout",
37513772
"output_type": "stream",
37523773
"text": [
3753-
"README.md sample-data.geojson\r\n",
3754-
"building-footprints-pluto.csv sample-data.gpkg\r\n",
3774+
"README.md sample-data.csv\r\n",
3775+
"building-footprints-pluto.csv sample-data.geojson\r\n",
3776+
"nta-shape.geojson sample-data.gpkg\r\n",
37553777
"output.csv sample-data.json\r\n",
37563778
"output.json sample-data.xlsx\r\n",
37573779
"output.xlsx \u001b[34mshapefile\u001b[m\u001b[m/\r\n",
3758-
"sample-buildings.zip \u001b[34munzipped-data\u001b[m\u001b[m/\r\n",
3759-
"sample-data.csv\r\n"
3780+
"sample-buildings.zip \u001b[34munzipped-data\u001b[m\u001b[m/\r\n"
37603781
]
37613782
}
37623783
],
@@ -3773,25 +3794,25 @@
37733794
"cell_type": "markdown",
37743795
"metadata": {},
37753796
"source": [
3776-
"## 4.2 Writing to an Excel (xlsx) file"
3797+
"## 4.2 Writing to an Excel (xlsx) File"
37773798
]
37783799
},
37793800
{
37803801
"cell_type": "code",
3781-
"execution_count": 44,
3802+
"execution_count": 45,
37823803
"metadata": {},
37833804
"outputs": [
37843805
{
37853806
"name": "stdout",
37863807
"output_type": "stream",
37873808
"text": [
3788-
"README.md sample-data.geojson\r\n",
3789-
"building-footprints-pluto.csv sample-data.gpkg\r\n",
3809+
"README.md sample-data.csv\r\n",
3810+
"building-footprints-pluto.csv sample-data.geojson\r\n",
3811+
"nta-shape.geojson sample-data.gpkg\r\n",
37903812
"output.csv sample-data.json\r\n",
37913813
"output.json sample-data.xlsx\r\n",
37923814
"output.xlsx \u001b[34mshapefile\u001b[m\u001b[m/\r\n",
3793-
"sample-buildings.zip \u001b[34munzipped-data\u001b[m\u001b[m/\r\n",
3794-
"sample-data.csv\r\n"
3815+
"sample-buildings.zip \u001b[34munzipped-data\u001b[m\u001b[m/\r\n"
37953816
]
37963817
}
37973818
],
@@ -3813,20 +3834,20 @@
38133834
},
38143835
{
38153836
"cell_type": "code",
3816-
"execution_count": 45,
3837+
"execution_count": 46,
38173838
"metadata": {},
38183839
"outputs": [
38193840
{
38203841
"name": "stdout",
38213842
"output_type": "stream",
38223843
"text": [
3823-
"README.md sample-data.geojson\r\n",
3824-
"building-footprints-pluto.csv sample-data.gpkg\r\n",
3844+
"README.md sample-data.csv\r\n",
3845+
"building-footprints-pluto.csv sample-data.geojson\r\n",
3846+
"nta-shape.geojson sample-data.gpkg\r\n",
38253847
"output.csv sample-data.json\r\n",
38263848
"output.json sample-data.xlsx\r\n",
38273849
"output.xlsx \u001b[34mshapefile\u001b[m\u001b[m/\r\n",
3828-
"sample-buildings.zip \u001b[34munzipped-data\u001b[m\u001b[m/\r\n",
3829-
"sample-data.csv\r\n"
3850+
"sample-buildings.zip \u001b[34munzipped-data\u001b[m\u001b[m/\r\n"
38303851
]
38313852
}
38323853
],
@@ -3848,7 +3869,7 @@
38483869
},
38493870
{
38503871
"cell_type": "code",
3851-
"execution_count": 46,
3872+
"execution_count": 47,
38523873
"metadata": {},
38533874
"outputs": [
38543875
{
@@ -4033,7 +4054,7 @@
40334054
"4 POLYGON ((-73.84769 40.87912, -73.84784 40.879... "
40344055
]
40354056
},
4036-
"execution_count": 46,
4057+
"execution_count": 47,
40374058
"metadata": {},
40384059
"output_type": "execute_result"
40394060
}
@@ -4049,16 +4070,17 @@
40494070
},
40504071
{
40514072
"cell_type": "code",
4052-
"execution_count": 47,
4073+
"execution_count": 48,
40534074
"metadata": {},
40544075
"outputs": [
40554076
{
40564077
"name": "stdout",
40574078
"output_type": "stream",
40584079
"text": [
4059-
"output.cpg output.shp sample-data.dbf sample-data.shx\r\n",
4060-
"output.dbf output.shx sample-data.prj\r\n",
4061-
"output.prj sample-data.cpg sample-data.shp\r\n"
4080+
"nta-shape.cpg nta-shape.shx output.shp sample-data.prj\r\n",
4081+
"nta-shape.dbf output.cpg output.shx sample-data.shp\r\n",
4082+
"nta-shape.prj output.dbf sample-data.cpg sample-data.shx\r\n",
4083+
"nta-shape.shp output.prj sample-data.dbf\r\n"
40624084
]
40634085
}
40644086
],

0 commit comments

Comments
 (0)