Dealing with big files

[scottsfarley93]

There are two issues related to the UI when processing big responses:

1. Showing download/server process. Since the Neotoma folks enabled compression, most of the time is actually spent on the server itself, not in the download (yay!). It would be nice to know when the download begins (readystatechange). This is hard to do with Pace.

2. Merging the dataset and sample data metadata is time consuming. Then, once they've been merged, creating the crossfilter is really time intensive. Optimizing this so the time-to-display after download is faster would be ideal.

Related issue is #11 -- will be fixed when the second item is done.

[scottsfarley93]

Punting this issue because it relies on NeotomaDB API modifications.

Specifically:
NeotomaDB/Neotoma-API#10 -- if implemented, would require one loop instead of two in processing the return, releasing interface lock during processing

Other relevant issues open in @NeotomaDB:
NeotomaDB/Neotoma-API#12 -- caching requests would make large returns nearly instant.
NeotomaDB/Neotoma-API#14 -- content length header would allow a download progress bar to be truly in sync with download.

Note, if NeotomaDB/Neotoma-API#16 (return total abundance fields to new endpoint) is implemented, the whole structure of the requests may be changed.