Installation target missing?

[matthewfeickert]

From trying to run a build with

$ cmake -DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX -Ditch="DarkBit;ColliderBit;CosmoBit;FlavBit;PrecisionBit;NeutrinoBit;ObjectivesBit;DecayBit;SpecBit" -DPYTHON_EXECUTABLE=$CONDA_PREFIX/bin/python -S gambit_2.6 -B build
$ cmake -LH build
$ cmake --build build --target scanners --parallel 10
$ cmake --build build --target gambit --parallel 10
$ cmake --install build

and getting back

-- Install configuration: "None"

it seems that GAMBIT has no install targets. What are all the necessary shared libraries and headers that need to be distributed with the gambit binary under a directory tree to work as a standalone installation?

I'm also confused as to why the build directory target isn't being used given that built binaries are showing up under the source directory, which should not be involved in the build after the initial cmake command.

$ find . -type f -iname "gambit"
./gambit_2.6/gambit
$ find . -type f -iname "libscan*"
./gambit_2.6/ScannerBit/lib/libscanner_jswarm.so
./gambit_2.6/ScannerBit/lib/libscanner_diver_1.0.0.so
./gambit_2.6/ScannerBit/lib/libscanner_ais.so
./gambit_2.6/ScannerBit/lib/libscanner_python.so
./gambit_2.6/ScannerBit/lib/libscanner_polychord_1.18.2.so
./gambit_2.6/ScannerBit/lib/libscanner_simple.so
./gambit_2.6/ScannerBit/lib/libscanner_polychord_1.20.1.so
./gambit_2.6/ScannerBit/lib/libscanner_multinest_3.11.so
./gambit_2.6/ScannerBit/lib/libscanner_diver_1.0.2.so
./gambit_2.6/ScannerBit/lib/libscanner_multinest_3.10.so
./gambit_2.6/ScannerBit/lib/libscanner_twalk.so
./gambit_2.6/ScannerBit/lib/libscanner_multinest_3.12.so
./gambit_2.6/ScannerBit/lib/libscanner_minuit2_6.23.01.so
./gambit_2.6/ScannerBit/lib/libscanner_diver_1.0.4.so
./gambit_2.6/ScannerBit/lib/libscanner_diver_1.0.5.so
./gambit_2.6/ScannerBit/lib/libscanner_postprocessor_2.0.0.so
./gambit_2.6/ScannerBit/lib/libscanner_postprocessor_1.0.0.so
./gambit_2.6/ScannerBit/lib/libscanner_polychord_1.17.1.so

[patscott]

As you've observed, GAMBIT's cmake setup is old school, and will need significant revision to make what you're trying to do in hep-packaging-coordination/packaging-hep-simulation-stack#9 work (but of course we're happy to help, as we see the need even independent of conda-forge).

There is no install target. The build is in the source dir. Most HEP and sampling dependencies ('backends' and 'scanners') are managed by GAMBIT itself as a sort of package manager that also fixes a million bugs and build deficiencies in them along the way. GAMBIT has never been installed as a standalone binary with headers, let alone rolled up as a package for a real package manager.

Don't assume it conforms to modern cmake or packaging practices. You'll need to take the time to dig into and understand the build for yourself, including in particular how gambit deals with shared libs of backends and scanners. If you'd like some of us to Zoom with you to try to walk you through it / answer questions, we'd be more than happy.

[matthewfeickert]

The build is in the source dir. Most HEP and sampling dependencies ('backends' and 'scanners') are managed by GAMBIT itself as a sort of package manager that also fixes a million bugs and build deficiencies in them along the way.

Ah, thanks for this information. I think this alone means GAMBIT isn't a good target to distribute as a package at the moment. As you point out, it could be with perhaps a fair amount of work, but that also raises the question of does the current GAMBIT team want this change?

If this work were to happen then this would fundamentally shift the way that things work from a development practice as it would mean moving to the concept of builds having at minimum a non-source directory build directory, which would also be relocatable (as cmake --install doesn't execute arbitrary code, it copies directory trees). I think that from what you've explained to me so far that the GAMBIT team would probably be open to this, but I also appreciate that any changes in development workflow can bring friction and might realistically not be a high priority compared to other areas of development and use cases (for very good reasons). There's also the reality that this work wouldn't be something that could be done super quickly, as it sounds like this would touch most things in the build system.

Are these changes things the GAMBIT dev team would be open to (with the understanding this might be something that wouldn't get finished until after Summer 2025)?

[patscott]

I expect they would if there were someone putting their hand up to take lead responsibility for that work. Friction and timelines are unlikely to be a concern, but sourcing the necessary dev time to do the work from within the GAMBIT Core Workgroup is. @ChrisJChang and @anderkve should really comment given that they're significantly more engaged in GAMBIT dev work these days than I am.

[ChrisJChang]

Please correct me if I have misunderstood the discussion, but as I understand the question is about rather than compiling a gambit executable in the source directory as we currently do, whether we can move to providing a shared library, and some headers (like many other HEP packages do).

One reason that I imagine this was not thought of/attempted with gambit in the past is that (as Pat says), gambit acts as a package manager for many different dependencies, and is not necessarily practical to imagine running from within someone else's code (given that this was not how it was imagined to be run).

However, gambit also ships with standalone versions of each of its physics modules (with more in development / able to be added in the future). I think that fits much nicer with this idea of shipping as a bunch of headers and shared libraries. These standalone are a way of running the physics calculations in gambit outside of a full global fit, and could certainly be called from within other codes.

These could be a first target for changes in the build system.

I can second what Pat says that this sounds like something we are open to (depending on person power in the core group).

@anderkve can also probably give his opinion.

[matthewfeickert]

Please correct me if I have misunderstood the discussion, but as I understand the question is about rather than compiling a gambit executable in the source directory as we currently do, whether we can move to providing a shared library, and some headers (like many other HEP packages do).

The question/suggestion would be a typical (include/, lib/, bin/) directory tree layout of headers, (shared) libraries (that are depended on), and executables that are built under a given build directory without polluting the source tree and then can be installed to a relocatable path via CMake's cmake --install command.

One reason that I imagine this was not thought of/attempted with gambit in the past is that (as Pat says), gambit acts as a package manager for many different dependencies, and is not necessarily practical to imagine running from within someone else's code (given that this was not how it was imagined to be run).

Regardless of if anyone would ever build against a GAMBIT library there are still advantages to having an application/executable that follows standard installation conventions and is relocatable (e.g. you can actually package it). There's also more immediate benefits of not building in your source tree, as having the source tree directory and the build directory be separate and distinct means that you can trivially build multiple configurations across multiple build dirs which you can then test and/or use during development.

When you refer to GAMBIT as a

sort of package manager that also fixes a million bugs and build deficiencies

this makes me think of the vendoring and unreproducible build issues that come from approaches like mg5amcnlo's. I don't know what your use cases and motivations have been for trying to be a package manager for people, so I'm not in a position to say anything, but other tools that have reported to do this raise the question of why not just use a real package management tool at this point. You obviously don't need to defend your choices, but I guess I'd be curious to learn more about why this is necessary and if GAMBIT allows you the choice of using actual package management/pointing to existing builds, or if it forces installation regardless.

Also RE: the bugs you've identified that your system fixes, have you reported them and/or upstreamted your fixes for them to the projects? Or are these only bugs for GAMBIT's particular use cases/modifications and not actually bugs in the projects?

However, gambit also ships with standalone versions of each of its physics modules (with more in development / able to be added in the future). I think that fits much nicer with this idea of shipping as a bunch of headers and shared libraries. These standalone are a way of running the physics calculations in gambit outside of a full global fit, and could certainly be called from within other codes.

I think I will need to take the time to understand how the modules/Bits interact and if there is any codependence, or if they are all fully isolated libraries. (I should also listen to the docs and reread the 2017 GAMBIT paper.)

---

Also thanks to all of you for taking the time to give clear and constructive information as you have been so far. I very much appreciate it, but I also don't expect that you will have the time to be able to explain the decision making history of why some things exist the way they do (or that you should be expected to just because someone showed up and started asking questions on GitHub).

[patscott]

A couple of quick answers where they are possible:

why not just use a real package management tool at this point

Because approximately none of the HEP theory projects needed for BSM pheno exists as a package in a real package manager. Might not be as true now as it was when we started GAMBIT a decade ago, but I'd wager it's still mostly true, especially when you factor in that we need theory packages from astro/cosmo and DM as well as traditional collider pheno.

if GAMBIT allows you the choice of using actual package management/pointing to existing builds

It does.

Also RE: the bugs you've identified that your system fixes, have you reported them and/or upstreamted your fixes for them to the projects?

Yes. Pythia, SUSY-HIT, SuperIso, DarkSUSY, nulike, higgbounds, multimodecode, and lots of others. Some of the time they got incorporated upstream, often enough the authors didn't bother or no longer really maintain them.

Or are these only bugs for GAMBIT's particular use cases/modifications and not actually bugs in the projects?

There are a lot of those too. For example, many of the packages we use as backends were never designed to be built and used as shared libraries, only as executables or static libraries, so there are a lot of mods to allow that.

[patscott]

tbh, looking at how GAMBIT patches a lot of the theory codes is probably a very sensible shortcut to getting started on packaging them up for inclusion in https://github.com/hep-packaging-coordination (most are not there).