Unable to restore etcd snapshot from Netapp StorageGrid S3 endpoint

[brandond]

From rancher/rke2#6285 (reply in thread)

NetApp replied that they were able to reproduce the error in a lab environment. They inspected the network traffic and determined that the issue is caused by "the minio client library in RKE2 having a bug where gzip streamed data is terminated prematurely". They say that RKE2 sets Accept-Encoding: gzip but that the header is not honored by AWS or MinIO and thus they send non-compressed data back to RKE2. NetApp say that this is why it works on AWS but not in StorageGRID, and that StorageGRID always honors the Accept-Encording header.

It looks like minio has their own default transport which disables compression to work around this issue. We don't observe that same behavior when setting up the transports that was pass in to minio - which we do so that we can configure TLS options.
https://github.com/minio/minio-go/blob/v7.0.83/transport.go#L41-L64

We should disable compression in the http.Transport that we are passing to minio, as they do the same internally when constructing their default transports.

[fmoral2]

this is validate here, since it was performed snapshots uploading and restoring from S3
#11609

[lindhe]

@fmoral2 Just to clarify: did you validate that it worked with NetApp StorageGRID? Because it worked on other S3 implementations already before, this fix is just for StorageGRID.

[fmoral2]

@fmoral2 Just to clarify: did you validate that it worked with NetApp StorageGRID? Because it worked on other S3 implementations already before, this fix is just for StorageGRID.

so we cant really validate there right since its a payed solution?

as per comments we added the fix from minio
https://github.com/k3s-io/k3s/pull/11604/files#diff-225890fd4fbceafdd32b01f99473cad767cfcd77f03ded1731c44166c0c7db18R175

and after the fix/update still working

[lindhe]

That's right, it's an enterprise solution with custom hardware and a paid license, so I don't expect you to be able to validate it.

I'm happy to try and validate this for you (since I posted the original issue), as soon as there's a build ready. I would prefer to have an RKE2 build, so I can keep my config the same as when I tested before. As soon as that has been released, I'll let you know.

[lindhe]

I tried it, and it works now!! 🚀

I first tried with v1.30.9 and it didn't work, then I tried v1.30.10 and it worked! At least for a single-node cluster. Will try out a HA cluster later today.

Thank you so much for this fix, @brandond !