Problem setting up local MySQL database to store airlines data

[trant22t]

I tried to set up a local MySQL database to store the airlines data. When I ran:

library(etl)
library(airlines)
db <- src_mysql_cnf(dbname = "airlines")
ontime <- etl("airlines", db = db, dir = "~/dumps/airlines")
ontime %>%
  etl_init() %>%
  etl_update(years = 2016)

R gave me the error message:
Error in download.file(url, method = method, ...) :
cannot download all files
In addition: Warning message:
In download.file(url, method = method, ...) :
URL 'http://tsdata.bts.gov/PREZIP/On_Time_On_Time_Performance_2016_10.zip': status was '404 Not Found'

I'm on a Mac and my R version is 3.3.1.

[nicholasjhorton]

I hope that there are ways to make the data loading a bit more robust. I worry that this command would download 30 years of data then bomb out because of the missing months in 2016.

I also wonder if the instructions might be modified to start with a less aggressive data load:

ontime %>%
etl_init() %>%
etl_update(years = 1987:2016)

Perhaps showing how to install a single (complete year, e.g., 2015) then add in the other years?

ontime %>%
etl_init() %>%
etl_update(years = 2015)

takes a long time!

ontime %>%
etl_update(years= c(1987:2014, 2016))

Additional note from NJH: the example now has more than 24GB!

There are over 300 months worth of files to download, and they will occupy more than 21 GB in their zipped and unzipped states.

[beanumber]

This is a known bug (#54). The problem is that we don't know which months of data are available on the server until we try to download them. I'm happy to field a pull request if you have a way for this function to fail gracefully.

Note that while the error breaks etl_extract(), it occurs after you have downloaded all of the valid data. So you can proceed by running etl_transform() and etl_load() in succession and these should still work.

[nicholasjhorton]

On Dec 3, 2016, at 10:19 AM, Ben Baumer ***@***.***> wrote: This is a known bug (#54). The problem is that we don't know which months of data are available on the server until we try to download them. I'm happy to field a pull request if you have a way for this function to fail gracefully.

MY RESPONSE: Is there a way to check the results of download.file() (or embed this within a call to try())?

Note that while the error breaks etl_extract(), it occurs after you have downloaded all of the valid data. So you can proceed by running etl_transform() and etl_load() in succession and these should still work.

MY RESPONSE: Is it possible to add a note about this to the vignette? This process isn't wholly clear to me (and likely to others as well).

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

Nicholas Horton Professor of Statistics Department of Mathematics and Statistics, Amherst College PO Box 5000, AC #2239 Amherst, MA 01002-5000