Nix copy to s3 skips upload when another endpoint have the same bucket name

[csernazs]

Describe the bug

When copying nix paths to two different s3 servers which have the same bucket name, the second copy does not copy the paths.

Steps To Reproduce

Copy a store path to foo.example.com:

nix copy -v /nix/store/...  --to s3://nix?profile=foo&scheme=https&endpoint=foo.example.com

Then, for the same path, copy it to bar.example.com:

nix copy -v /nix/store/...  --to s3://nix?profile=bar&scheme=https&endpoint=bar.example.com

This second command won't copy anything to the server, it will print copying 0 paths... and no connection attempts are made to the bar.example.com (according to strace and more verbose output).

The workaround is to specify --option narinfo-cache-positive-ttl 0, but it reduces the speed of copy dramatically if the closure is large.

Expected behavior

The path is copied to both servers.

Metadata

nix-env (Nix) 2.31.3

Additional context

I think the issue is that in ~/.cache/nix/binary-cache-v7.sqlite, nix registers the target as s3://nix (so only the bucket name is stored there), without the hostname or other parameters. That makes the second nix copy believe that the paths have been already copied.

Checklist

• checked [latest Nix manual] ([source])
• checked [open bug issues and pull requests] for possible duplicates

[xokdvium]

I think the issue is that in ~/.cache/nix/binary-cache-v7.sqlite, nix registers the target as s3://nix (so only the bucket name is stored there), without the hostname or other parameters.

Yeah, your diagnosis is correct. This has been on our radar for a bit, but we haven't yet come around to fixing that. That should just be a matter of refactoring how the cache key is calculated.

[csernazs]

hi @xokdvium ,

Thanks for the quick answer! If there's already an issue for this, feel free to close this with a dup.

Do you accept PRs or you want to do this as part of a bigger refactor? (I'm not a C++ dev and not trying to fix it but may consider this later)

I think having the URL (what the user provided) in the db as-is would be a simple fix for this, although ideally there could be some normalization on that (eg, parameters sorted, some parameters removed perhaps). I'm also not sure if compression parameter can be taken into processing, as the nar path will be different (having a .xz suffix or not, depending on this).

Thanks,
Zsolt

[xokdvium]

If there's already an issue for this

Don't think we had one, let's keep this one.

compression parameter can be taken into processing

Compression parameter only really affects the upload code path, nix gets the compression from a narinfo.

I think for S3 the cache key has to include the profile/region/bucket name/endpoint.

having a .xz suffix or not

The file extension has no semantic difference, it's just a convention. NARs can be compressed differently in the same cache and it's recorded in a .narinfo

Do you accept PRs or you want to do this as part of a bigger refactor

Sure, the trail of FIXMEs is easy to follow to the responsible code. But if you'd rather not write C++, I'll try to get around to that soonish.