S3 Pre-signed URLs for upload endpoints

[tsa96]

We want to keep traffic through the actual API to a minimum. There shouldn't be any need to stream data like map uploads through the API itself, instead, just generate pre-signed S3 URLs and pass those to the client.

We should go through every endpoint that does anything upload/download related and ensure that where possible, we avoid file streams. The game seems like it's already using it for map downloads, not sure about images and runs though - I'll look at that myself.

[Gocnak]

Images will be fun because they need to be resized to ensure they're good. We could honestly use something like Cloudflare Workers to resize it and upload to the S3 bucket instead.

Runs will be a separate pipeline in the future, due to the extra processing required for AC purposes.

[tsa96]

Yeah, good point re images. Though the case of map image upload is rare enough that probably we can just let the API handle it? If total size of images for a map is like 20MB of images (rough estimate, for 4k). CF Workers sounds like quite a lot of fun though.

[tsa96]

Blocking for now; we're going to see how much bandwidth we get through after 0.10.0. S3 presigned gives less control, nice to avoid if poss.

[tsa96]

Closing, we're going down this route #318

[GordiNoki]

After a little conversation with Tom, we decided to yet again use pre-signed urls, since you can actually control an object being put in bucket and bandwidth isn't really an issue there.

[tsa96]

Main reason against chunking being we don't have a way to guarantee requests with chunked data hit the same server, so using local disk doesn't work.

[Gocnak]

Note that nginx (and others) support session stickiness, something like ip_hash defined with our multiple instances would direct the same IP address to the same server: https://nginx.org/en/docs/http/load_balancing.html#nginx_load_balancing_with_ip_hash

[tsa96]

Note that nginx (and others) support session stickiness, something like ip_hash defined with our multiple instances would direct the same IP address to the same server: https://nginx.org/en/docs/http/load_balancing.html#nginx_load_balancing_with_ip_hash

Right, but if devops stuff gets more complicated in the future we have to do the work to ensure we keep that, plus we have to rely on writing to disk (which we don't do currently), seems like a lot of extra headache for quite a small feature, vs just using S3