We're switching from hermetic_cc_toolchain to hermetic-llvm in dfinity/ic#10991 and it works like a charm!
We're also thinking about switching to rules_rs as well. However, before we do that I would like to report the following issue which I first reported at hermeticbuild/hermetic-llvm#697. But I was asked to report it here instead.
One thing we had to do was patch this line in the pkg-config rust crate using bazel/pkg-config.patch which sets the default print_system_libs from true to false, which causes it to not set PKG_CONFIG_ALLOW_SYSTEM_LIBS=1 when probing.
With that variable set (the crate's upstream default), pkg-config prints -L flags even for directories on the system library search path, so a plain probe of a host library emits the host library dir (e.g. -L/usr/lib/x86_64-linux-gnu) as a rustc link-search path. That dir ends up in the link line ahead of the hermetic toolchain's library search paths, which makes -lc resolve to the HOST glibc while the crt objects come from the hermetic glibc — an incompatible mix (undefined__libc_csu_init/__libc_csu_fini, wrong GLIBC_ symbol versions).
The following is an example of the failure that landed on @crate_index__devicemapper-0.34.4//:_bs_ — the build script of the downstream devicemapper crate, not the final binary, since devicemapper-sys's build script emits cargo:rustc-link-search=/usr/lib/x86_64-linux-gnu and rules_rust propagates it to everything downstream:
ERROR: .../crate_index__devicemapper-0.34.4/BUILD.bazel:84:19: Compiling Rust bin _bs_ (30 files) [for tool] failed
error: linking with `external/llvm++llvm_toolchain_minimal+.../bin/clang++` failed: exit status: 1
= note: ... "-L" "/usr/lib/x86_64-linux-gnu" ...
= note: ld.lld: error: undefined symbol: __libc_csu_fini
>>> referenced by init.c
>>> bazel-out/.../external/llvm+/runtimes/crt_objects_directory_linux/crt1.o:(_start)
ld.lld: error: undefined symbol: __libc_csu_init
>>> referenced by init.c
>>> bazel-out/.../external/llvm+/runtimes/crt_objects_directory_linux/crt1.o:(_start)
clang++: error: linker command failed with exit code 1
The mechanism is exactly as described in the patch comment: the host -L /usr/lib/x86_64-linux-gnu sits ahead of the hermetic glibc_library_search_directory in the link line, so -lc resolves to the host glibc (2.39-ish, which dropped __libc_csu_init/__libc_csu_fini in glibc 2.34) while crt1.o comes from the hermetic glibc 2.31 and still references them.
That last point is what makes it a good example for rules_rs: the symptom is an obscure link error inside a transitively built build script, with no hint that a cargo:rustc-link-search from a different crate caused it
Without the variable, pkg-config's own system-dir filtering kicks in: the probe keeps its version checks, -l libs, -I include paths (for bindgen) and -D defines, but no host -L dir leaks into the link line. The -l libs resolve against cc_import dependencies that link the host library by explicit path.
We're switching from
hermetic_cc_toolchaintohermetic-llvmin dfinity/ic#10991 and it works like a charm!We're also thinking about switching to rules_rs as well. However, before we do that I would like to report the following issue which I first reported at hermeticbuild/hermetic-llvm#697. But I was asked to report it here instead.
One thing we had to do was patch this line in the
pkg-configrust crate usingbazel/pkg-config.patchwhich sets the defaultprint_system_libsfromtruetofalse, which causes it to not setPKG_CONFIG_ALLOW_SYSTEM_LIBS=1when probing.With that variable set (the crate's upstream default), pkg-config prints
-Lflags even for directories on the system library search path, so a plain probe of a host library emits the host library dir (e.g.-L/usr/lib/x86_64-linux-gnu) as a rustc link-search path. That dir ends up in the link line ahead of the hermetic toolchain's library search paths, which makes-lcresolve to the HOST glibc while the crt objects come from the hermetic glibc — an incompatible mix (undefined__libc_csu_init/__libc_csu_fini, wrong GLIBC_ symbol versions).The following is an example of the failure that landed on
@crate_index__devicemapper-0.34.4//:_bs_— the build script of the downstreamdevicemappercrate, not the final binary, sincedevicemapper-sys's build script emitscargo:rustc-link-search=/usr/lib/x86_64-linux-gnuand rules_rust propagates it to everything downstream:The mechanism is exactly as described in the patch comment: the host
-L /usr/lib/x86_64-linux-gnusits ahead of the hermeticglibc_library_search_directoryin the link line, so-lcresolves to the host glibc (2.39-ish, which dropped__libc_csu_init/__libc_csu_finiin glibc 2.34) whilecrt1.ocomes from the hermetic glibc 2.31 and still references them.That last point is what makes it a good example for rules_rs: the symptom is an obscure link error inside a transitively built build script, with no hint that a
cargo:rustc-link-searchfrom a different crate caused itWithout the variable, pkg-config's own system-dir filtering kicks in: the probe keeps its version checks,
-llibs,-Iinclude paths (for bindgen) and-Ddefines, but no host-Ldir leaks into the link line. The-llibs resolve against cc_import dependencies that link the host library by explicit path.