Bugfix: Generated binaries should not depend on devel package paths

[hpkfft]

Prefix library paths specified by -L are converted to canonical paths to improve the library runpath stored in the binary output. For example, if pkg-config --libs add returns the following:

-L/usr/local/lib/pkgconfig/../../lib -ladd

we convert the path so that it is as if it had returned:

-L/usr/local/lib -ladd

Motivation

Consider a third-party math library that is distributed as two packages for installation: libadd and libadd-devel.
The first installs the following files (under /opt or /usr/local as the system administrator chooses):

lib/libadd.so.0 -> libadd.so.0.1.0
lib/libadd.so.0.1.0

The second, which is the development package, installs the following files:

include/add/add.h
lib/libadd.so -> libadd.so.0
lib/pkgconfig/add.pc

The file add.pc is as follows:

prefix=${pcfiledir}/../..
includedir=${prefix}/include
libdir=${prefix}/lib

Name: add
Description: addition: add
Version: 0.1.0
Libs: -L${libdir} -ladd
Cflags: -I${includedir}

On a build machine, the sysadmin has installed both packages. But the application that uses this math library will be deployed on hundreds of machines (virtual machines in the cloud, nodes on a supercomputer, docker containers, etc.). The system administrator only installs libadd on those machines, not libadd-devel, nor meson, ninja, etc.

The problem is that, without this patch, a meson.build file containing

add_dep = dependency('add', method: 'pkg-config')
executable('myapp', 'myapp.c', dependencies: add_dep)

will create an application binary with the following library runpath:

$ ldd myapp
    ...
    libadd.so.0 => /usr/local/lib/pkgconfig/../../lib/libadd.so.0
    ...

and this won't run on the deployment nodes since the directory /usr/local/lib/pkgconfig does not exist there.

With this patch:

$ ldd myapp
    ...
    libadd.so.0 => /usr/local/lib/libadd.so.0
    ...

and all is well.

[eli-schwartz]

The motivation here sounds quite wrong. If you're distributing libadd to hundreds of machines and the administrator of each machine can choose where to install it, because it's a relocatable package and uses ${pcfiledir}, then it would be factually incorrect to use /usr/local/lib as the rpath. It would also be factually incorrect to use /usr/local/lib/pkgconfig/../../lib.

The only correct option is to evaluate the pcfiledir pkg-config variable in the text used for rpath, with the rpath token ${ORIGIN} (which is evaluated at runtime by ld.so)

[hpkfft]

In this scenario, there's only one administrator, who creates two disk images (one for the build machine and one for the deployment machines). The only difference between the two images is whether or not libadd-devel is installed. The choice of installation location (e.g., /usr/local) is made once and is applied to both the build and deployment nodes. The idea is to save time and bandwidth when spinning up the deployment nodes by having a smaller image for them.

Sorry, I'm not really understanding your suggestion. The ${ORIGIN} would refer to the location of myapp, which does not seem helpful for locating libadd.so.0.
The ${pcfiledir} in the third-party math library's pc file is evaluated by pkg-config at the time meson setup is run for myapp. Meson just receives the output string -L/usr/local/lib/pkgconfig/../../lib -ladd

As the developer of myapp, I would like my source code to build correctly on a supercomputer that has installed the math library at any arbitrary (known) location. I hope to use the meson flag --pkg-config-path to allow the same source code to work regardless of the sysadmin's choice.

The dependency code for pkgconfig already converts relative paths to absolute paths.
Is there a reason not to use the canonical path returned by os.path.realpath(path)?
Even without the deployment motivation I have described, isn't it nicer to set the RUNPATH to /usr/local/lib?

[hpkfft]

With this PR:

$ readelf -d myapp

Dynamic section at offset 0x2dc0 contains 28 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libadd.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000001d (RUNPATH)            Library runpath: [/usr/local/lib]
....

The executable myapp is built on a build machine provided by (and dedicated for) a given compute cluster and is to be run on that same cluster's many deployment machines.
The executable was built using meson setup --pkg-config-path /usr/local/lib/pkgconfig

[virtuald]

Even without the deployment motivation I have described, isn't it nicer to set the RUNPATH to /usr/local/lib?

No, if the pkg-config path is relocatable, then the library is intended to be moved around. Turning that into an absolute path would break lots of usages. Instead, your app should make sure that it lives where the library expects you to be (by looking at $ORIGIN). If you can't do that, then you need to use LD_LIBRARY_PATH.

[hpkfft]

The meson code in pkgconfig.py already converts the path provided by pkg-config into an absolute path as follows:

if not os.path.isabs(path):
    path = os.path.join(self.env.get_build_dir(), path)

According to the comments, it does this because, "We need the full path of the library to calculate RPATH values" and because, "De-dup of libraries is easier when we have absolute paths."
My patch is to also canonicalize the path using Path(path).resolve().as_posix() before adding the string to the set of library paths.
For example, this will convert /something/lib/pkgconfig/../../lib to /something/lib.

If the library moves around, it seems to me that my patch doesn't make the problem any worse. Do you see a use case for which the absolute path with the ../ works but the canonicalized path without it does not? (In fact, my patch seems like it might help with library path de-duplication.)

Certainly, if the library is moved around from one machine to the next, then whoever is running the application would need to use LD_LIBRARY_PATH.

I do not see how ${ORIGIN} is useful here. The code I wish to build with meson, myapp, is made available as source code. It needs to use a third-party math library, say libadd.so.0. Some computing facilities install libadd.so.0 in a directory based on the math library vendor, e.g., /opt/intel/, others install it based on the department that requested it, e.g., /opt/astrophysics/, others may use /usr/local/, etc.

your app should make sure that it lives where the library expects you to be

The third-party math library has no idea that myapp exists. The myapp binary would be somewhere like /home/paul/cosmology/myapp. So, ${ORIGIN} is /home/paul/cosmology/. I am not a system admin, and I cannot install myapp in /opt/ or /usr/local/.

It seems common for third-party libraries to ship their product in two pieces, e.g., libadd_0.1.2 and libadd-dev_0.1.2.
For example, on stackoverflow: Linux dev packages
It seems desirable that I should be able to build myapp on a machine with both libadd_0.1.2 and libadd-dev_0.1.2 installed and be able to run it on a machine with only the former installed, assuming of course that libadd_0.1.2 was installed into the same location on both machines. It seems that is the whole point of the packaging convention.

I would like to run

meson setup --pkg-config-path=/opt/vendor/lib/pkgconfig:/opt/astrophysics/lib/pkgconfig:/usr/local/lib/pkdconfig mybuild

on the build machine to have pkg-config tell meson setup where to find libadd.so.0.
Then, I would like to build myapp and be able to run it on any of the compute nodes that have installed libadd_0.1.2 but not libadd-dev_0.1.2 .

I think the same thing happens with Kubernetes deployments--a minimized container image without -dev packages installed is used for deployment to save time and money.

[hpkfft]

Rebased on top of master.
Note that the masos failures are unrelated to this PR: "ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied"

[hpkfft]

Rebased on top of master so the CI checks pass.
No other changes made.

[hpkfft]

Rebased on top of master.
No other changes made.

[hpkfft]

Rebased.

[hpkfft]

Rebased.

[hpkfft]

Rebased.

[hpkfft]

Rebased.

[hpkfft]

Rebased.

[bonzini]

Summing up the various comments:

• you're distributing libadd to hundreds of machines and the administrator of each machine can choose where to install it, because it's a relocatable package and uses ${pcfiledir}, then it would be factually incorrect to use /usr/local/lib as the rpath. It would also be factually incorrect to use /usr/local/lib/pkgconfig/../../lib.
  • True. However, the fact that a package uses ${pcfiledir} does not imply that the package will be moved around, only that it can be moved around. Right now Meson does not use $ORIGIN, it uses /usr/local/lib/pkgconfig/../../lib.
• isn't it nicer to set the RUNPATH to /usr/local/lib?
  • Niceness doesn't count much, but I am quite convinced by the fact that more things can go wrong with .. in your rpath.
• if the pkg-config path is relocatable, then the library is intended to be moved around. Turning that into an absolute path would break lots of usages
  • Meson already does it.
• Instead, your app should make sure that it lives where the library expects you to be (by looking at $ORIGIN).
  • First, I don't understand it. Isn't it the app that expects the library to be "somewhere" through its $ORIGIN?
  • Second, relocatability of apps and libraries are two separate things. You can build an app as relocatable (i.e. it is able to find its own datadir for example) while requiring a given absolute path for a library.
  • Third, even if using $ORIGIN, moving around an executable might very well end up falling back to /usr/lib-like paths for system shared libraries such as the one found by pkgconfig, because you might be moving around the app but all of its dependencies are still installed via rpm/deb/...

So I am going to merge this in a few days, but leaving time to comment.

[eli-schwartz]

To be perfectly clear, no meson does not do that.

(Unless you are running the binary uninstalled, inside the build directory, in which case meson adds many rpaths that don't make any sense at all, because it's more reliable than knowing which rpaths are required and which ones aren't.)

Any rpaths that meson doesn't remove during meson install are:

• rpaths manually specified by hand in install_rpath: kwargs in a meson.build file
• passed via LDFLAGS

There has been a serious suggestion or three, to have meson install actually genuinely support automatically detecting needed install-time rpaths for dependencies installed to custom locations. But we would probably need a glibc ld.so.conf file format parser, among other things. It's on my list of things to look into "someday".

[eli-schwartz]

General commentary on something that repeatedly bugs me across many PRs.

[bonzini]

To be perfectly clear, no meson does not do that.

Right - more precisely, this patch only affects which absolute paths are created and not which files (whether uninstalled or installed) have absolute paths.

Even for uninstalled binaries, I guess you could build one a machine that has the dev package, and run t on another machine that mounts the same home directory at the same location but only has the runtime package.

So it boils to:

• @hpkfft should confirm whether he's using installed or uninstalled binaries
• using ../../ paths is clearer because it is obvious that it comes from ${pcfiledir}
• canonicalizing paths produces more readable command lines

And anyway I'm moving it to 1.13 given the other comments.

[eli-schwartz]

• canonicalizing paths produces more readable command lines

Yes, that is the one argument I personally find tempting. But it feels slightly weak if it's not really fixing anything by construction (albeit perhaps is more convenient for specific individuals), while adding even more filesystem stat calls and adding workarounds to the testsuite whose purpose we will never remember down the line.

[hpkfft]

@hpkfft should confirm whether he's using installed or uninstalled binaries

Uninstalled. Supercomputers have an interactive front-end node (a build machine) used for code development. Researchers develop and compile their code, myapp, on this machine. To run their code, researchers use a command such as qsub from PBS to submit their job to a run queue. For example, you can request 8 nodes for your batch job. Batch jobs run on compute nodes, not on the front-end node. A PBS scheduler allocates blocks of compute nodes to jobs to provide exclusive access. When the resources (i.e., the 8 nodes having the properties you requested (e.g., memory size)) become available, your job will execute on the compute nodes. When the job is complete, the PBS standard output and standard error of the job will be returned in files available to you. Then, you fix some more bugs and try again.

If meson install removes the RUNPATH that we are discussing, then this PR does not affect the results produced by meson install. We do not need to consider it any further.

canonicalizing paths produces more readable command lines

Yes, that is the one argument I personally find tempting. But it feels slightly weak if it's not really fixing anything by construction (albeit perhaps is more convenient for specific individuals), while adding even more filesystem stat calls

It is fixing something by construction, as described above.

It adds filesystem stat calls to meson setup, not to meson compile, which is presumably run more frequently than the former. Moreover, it removes filesystem stat calls from running myapp, since its RUNPATH will contain /whatever/lib rather than /whatever/lib/pkgconfig/../../lib. The quality of the meson-generated binary should be the primary goal.

[eli-schwartz]

If meson install removes the RUNPATH that we are discussing, then this PR does not affect the results produced by meson install. We do not need to consider it any further.

That is exactly the problem. :( Making meson install correct is a very compelling argument as a matter of principle. The results of it are supposed to run on many machines, by design.

If we discard that from consideration then the uninstalled binaries aren't designed to run on many machines, just the one where the build is running. Even the rpaths we do add are a private implementation detail, solely so that meson test and custom_target(command: [fooprog]) and meson devenv can run the program. We could rely on LD_LIBRARY_PATH set as an environment variable in all those scenarios, but setting environment variables is awkward for custom targets (needs to be wrapped in meson --internal exe --unpickle foo.dat), and doesn't play nice with additional user-defined environment variables, and RPATH is easy and has no downsides and even makes running outside of meson devenv work a lot of the time.

But they are a private implementation detail, and we could someday change it if we felt there were compelling advantages for our supported use cases. I don't anticipate that happening, to be fair.

There are absolutely some projects that only work inside of meson devenv, where RPATH isn't enough, so we can't provide a generalized guarantee that running ./builddir/src/fooprog will work, nor that copying it out of the build directory will work. Although for standalone single-file executables, they probably will.

The rpaths are a security vulnerability by default because other machines may have /home/eschwartz/git/fooproject/builddir/src belong to someone else, who can sideload malware and have you run it with your unix permissions / ACLs / selinux contexts.

I have no idea whether this matters to workflows that involve submitting jobs to a compute node! If the user accounts and uids and homedirs are guaranteed to be identical and consistent across nodes then likely it's fine.

It is fixing something by construction, as described above.

?

Again, it's not fixing it in the sense that meson is designed to support the use case of copying binaries onto another machine without installing them. Changing the style of RPATH entries doesn't make that part of the design.

If it's not a designed scenario then we are in the realm of "maybe it coincidentally works, and we don't mind that it works and presumably won't go out of our way to break it with no cause".

It adds filesystem stat calls to meson setup, not to meson compile, which is presumably run more frequently than the former. Moreover, it removes filesystem stat calls from running myapp, since its RUNPATH will contain /whatever/lib rather than /whatever/lib/pkgconfig/../../lib.

Maybe, maybe not. :D

"meson setup is slow" is a real problem that some people have. I could see this being a problem for example in projects that have many dependencies and are already feeling that "setup time would be good to optimize pretty please".

And adding stat calls to every external library path in order to check if it can be resolved, will affect every dependency lookup for all users of meson. The vast majority of binaries won't have anything to resolve in the generated RPATH, so, running your myapp may have fewer system calls but running someone else's otherapp will see no difference.

[eli-schwartz]

AFAICT, os.path.realpath isn't even an interesting scenario for your use case, you only need os.path.normpath (that is, collapse ../)?

As I already pointed out I don't like the use of pathlib.Path concrete methods, can we seize the opportunity in moving to os.path, to also skip symlink canonicalization and just flatten the use of ../ backtracking? That should still handle ${pcfiledir}/../../.

It will look prettier, avoid encoding dependencies on pkgconfig/ directories, and as an abstract op the standard library implementation won't need to stat the filesystem.

[hpkfft]

[can we] just flatten the use of ../ backtracking

Yes, that's all I want to accomplish!

Again, it's not fixing it in the sense that meson is designed to support the use case of copying binaries onto another machine without installing them.

Yeah, I see your point. But qsub does copy binaries onto other machines, so it'd be nice if it worked. And container deployments are kinda like copying. Or maybe a better mental model is that we want the binary to continue to run on the build machine after the libadd-dev_0.1.2 package has been removed. Yes, I like that model. I'd like meson to support the scenario that (1) libadd_0.1.2, libadd-dev_0.1.2, ninja, meson are installed, then (2) myapp is built, then (3) libadd-dev_0.1.2, ninja, meson are removed, then (4) myapp is run. [This is equivalent to qsub copying the myapp binary to the compute node and running it there.]

[hpkfft]

The Gentoo test failure appears unrelated:

   return get_active_overrides().subprocess_timeout
                    ERROR    fatal: detected dubious ownership in     _git.py:32
                             repository at '/__w/meson/meson'                   
                             To add an exception for this directory,            
                             call:                                              
                                                                                
                                     git config --global --add                  
                             safe.directory /__w/meson/meson                    
git introspection failed: fatal: detected dubious ownership in repository at '/__w/meson/meson'

Please take another look at this PR.
Thanks,
Paul

[bonzini]

Or maybe a better mental model is that we want the binary to continue to run on the build machine after the libadd-dev_0.1.2 package has been removed

I think the best mental model is that of shared (for example, NFS-mounted) home directories, where you build on a large machine but then test on something that has the hardware you need.

[hpkfft]

Rebased.