Chunked Map Uploading for Maps larger than 100MB

[Gocnak]

Our file uploading process is currently just a single POST request to the backend. It'd be great to get resume-able uploads with multi part chunking it, especially when we move to cloud storage.

The frontend should update visually to respect this!

[KJelles]

I looked into this @Gocnak but it feels like Uppy is a bit wack with Angular, their latest package isn't even bundled correctly :) Might need to just improve on the file-upload component visually instead

[Gocnak]

Yep that's fair! I'll rename the issue to better fit then.

[tsa96]

Do we really this? Most modern internet should be able to handle a single POST request of ~100mb perfectly fine, and 0.10.0 deploy shouldn't have an issue with files > 100mb, even though it still streams through the backend. As me and Nick discussed recently, if our backend instances are started to use too much bandwidth and we need to do fancier upload stuff we can, but until we actually run into that problem (with the 0.10.0 deploy), we're fighting with a (often extremely complex) problem before it's actually affected us.

@Gocnak if you agree, then #869 closes this.

[Gocnak]

Note that if we're protecting our backend via Cloudflare, we will need to have multipart uploading to get any file > 100 MB uploaded, unless we use presigned URLs.

[tsa96]

Sorry, what Cloudflare system requires this? Is there really no way to configure this?

[tsa96]

We discussed this a few weeks ago, our free Cloudflare DDoS stuff limits to maximum of 100MB.

Chunking with current system is really annoying given our only state atm is in Postgres.

Since we don't do anything with BSPs (currently) on the backend the easiest approach is probably presigned URLs. But this is definitely low priority, maps should really try to stay under 100MB. Not sure it's completely necessary for 0.10.0.

[tsa96]

Okay we can chunking and use disk but needs a spec, don't have time now, ping me if interested in doing this.
Will need to implement some separate disk-based io stuff since not doing anything currently.

[Gocnak]

Chunking impl: https://medium.com/@selieshjksofficial/streamlining-large-file-uploads-with-chunk-uploads-in-node-js-and-express-js-e40d00c26c2d