Skip to content

Conversation

@szlend
Copy link

@szlend szlend commented Apr 24, 2025

The symlinkJoin builder might be lighter on dependencies, but it actually creates a very inefficient directory structure, symlinking literally every file in the toolchain. This results in ~47k symlinks created with the default toolchain and takes a while on slower file systems.

The buildEnv builder depends on perl and is a bit awkward to use with propagated inputs, but it creates a much more efficient directory structure.

With symlinkJoin:

find ./result/ | wc -l
47006

With buildEnv:

find ./result/ | wc -l
80

@oxalica
Copy link
Owner

oxalica commented Apr 26, 2025

The symlinkJoin builder might be lighter on dependencies, but it actually creates a very inefficient directory structure, symlinking literally every file in the toolchain. This results in ~47k symlinks created with the default toolchain and takes a while on slower file systems.

That's definitely a problem I want to fix for a long time. But I don't know the "proper" method to produce the minimal symlink tree. It's good to know that buildEnv does a better job here.

The buildEnv builder depends on perl and is a bit awkward to use with propagated inputs

I agree. Since the aggregated drv is almost always built on users' machine, this change will force all users to download perl to build any toolchain, which is less ideal. But I think build-only dependency to perl is probably still better than persistent thousands of symlinks after-build.

Ideally, we would like to optimize upstream implementation of symlinkJoin.

@lyynd
Copy link

lyynd commented Jul 24, 2025

The approach in this PR has an additional benefit of fixing #221 (just tested).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants