[feature] Multiple CMake builds from single conanfile

[s-geiger-si]

What is your suggestion?

Extracting this issue from #10220 after @memsharded suggested in a comment that it should be its own issue.

I would like to be able to build code that is being generated with the OpenAPI generator which I invoke via Conan. The issue is that the openapi-generator produces a folder tree that contains a pretty-much self contained CMake project.

I tried to explore several alternative options:

1. Moving openapi generation and built from Conan into CMake
2. Generating the openapi code via Conan and using add_subdirectory to build everything from a single CMake project.
3. Building the openapi generated code from conan as a separate project (this is what this issue is about).

Option 1 would imply that we move everything into CMake and use add_custom_command to invoke the openapi-generator. This is not possible, since we cannot load additional CMakeLists.txt files during a CMake configure call.
Option 2 is generally possible, but it has the disadvantage, that install() directives from the openapi modules are being leaked into the main project and we end up installing headers, static libs and other stuff into the main install prefix and then need to filter them out later on.

A better solution would be to explore option 3 and to configure and build the openapi generated code as a separate sub-step via Conan, install it into a CMAKE_PREFIX_PATH and then use find_package from the main project to resolve the dependencies. The openapi generated code only produces static libraries which are being consumed in the main project, so I do not need to create a Conan package from them. Currently I have not found a way to do this, because CMakeToolchain, and CMake assume that a single build is happening.

What I would like to do is:

1. Install Conan dependencies
2. Generate OpenAPI stubs with the openapi-generator
3. Build each of the generated folders and install them via a separate CMake build tree (which can be private to Conan)
4. Configure the main project and add the install tree to the CMAKE_PREFIX_PATH
5. Build the main project and proceed to install/package it.

Further background can also be found here: https://stackoverflow.com/questions/78361333/integrate-openapi-generate-with-conan-2-and-cmake-with-proper-caching

Question from @memsharded

@s-geiger-si it would be good to understand why a different toolchain would be necessary. How this would be managed without Conan? Isn't it possible to do it with CMake only then?

Have you tried just calling self.run("cmake .... ) directly for the subprojects?

Not yet, I was hoping that I could reuse the Conan classes for CMake and CMakeToolchain, since I had the impression that this is the best practice approach.

The CMake(self) is a relatively thin wrapper, and regarding the toolchain, still need to understand what toolchain would be use[d] to build these libraries, why the same toolchain wouldn't be the right one.

I think I could reuse the same toolchain, since I do not need a different dependency subset, but as I understand it, the toolchain seems to also control details such as the CMAKE_PREFIX_PATH and the CMAKE_INSTALL_PREFIX or the build folder for the CMake build tree and I am not able to pass this as an overwrite to CMake(self) or to cmake.configure().

What I would effectively like to have is some structure like the following (if that makes sense):

<build-root>/     # as set via --output-folder to conan install/build (i.e., --output-folder ./build)
   main-build/`   # build tree for main project
     install/     # install tree for main project
   openapi-1/     # source tree (generated via conan using openapi-generator)
      build/      # build tree for openapi generated code
        install/  # install tree for openapi generated and built code
   openapi-2/`    # source tree (generated via conan using openapi-generator)
      build/
        install/
   cmake_toolchain.json
   [ other files generated by conan ]

For the main project build tree, I would then like to configure CMake to set CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:./openapi-1/build/install:./openapi-2/build/install.

So basically, I was hoping I could do something like:

def build(self):
  build_openapi_stubs(...)

  cmake = CMake(self, main-tc) # set build tree to ./main-build and update CMAKE_PREFIX_PATH
  cmake.configure()
  cmake.build()
  
def build_openapi_stubs(...):
   generate_stubs(...)
   cmake = CMake(self) # set build tree to ./openapi-N and install target to ./openapi-N/install
   cmake.configure()
   cmake.build()
   cmake.install()

Maybe I am missing something and this is totally the wrong approach?

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[memsharded]

Hi @s-geiger-si

Thanks for your detailed feedback.

I'd like to discuss first a bit more option2. Because this honestly sound like the one that I would choose.

Generating the openapi code via Conan and using add_subdirectory to build everything from a single CMake project.

Yes, this looks very easy, like the recipe can also generate or add a CMakeLists.txt that does the add_subdirectory() to the source and to the generated subfolders.

Option 2 is generally possible, but it has the disadvantage, that install() directives from the openapi modules are being leaked into the main project and we end up installing headers, static libs and other stuff into the main install prefix and then need to filter them out later on.

But I think this is a very small disadvantage. Doing a shutil.rmtree() or the like in the package() method after the cmake.install() would most likely be enough if I understood it correctly. Also, the cmake.install(..., component=xxx) has the syntax to install only some specific component, so maybe that is even a better approach than having to remove it later.

So this approach would be much simpler to implement in the recipe, but moreover it will also have consistent input of the same CMake toolchain very easily. Please let me know what you think.

I have also checked a bit the implementation of CMake helper, and it is completely tied to the layout and different folders like self.source_folder, self.build_folder, etc. it seems extremely challenging to change this to allow this case.

[s-geiger-si]

I agree that a workaround is possible and I do have a working workaround which involves using a custom cmake template for the openapi-generator that works better with the add_subdirectory() approach. Alternatively as you write, I could also delete the unwanted files with a python function from Conan. My main issue with this is that it involves additional maintenance overhead (keeping a custom cmake template in the code, documenting that and missing out on upstream changes to that template). It also does not feel like the right approach to build a module that is intended as a standalone project via add_subdirectory.

My main point here is that there is no satisfactory solution that involves a seamless integration of the tree tools involved (i.e., openapi-generator, cmake and conan) and that the solution that I believe is envisioned by the openapi-generator is not by default compatible with Conan unless we employ some workarounds. I want to understand if I am just the only one who is encountering this kind of problem and wants to build two different cmake project from a single Conan file.

Given that Conan "understands" cmake (via the CMakeToolchain, CMake, CMakeDeps integrations) it would be great if I could say: "Hey Conan, generate and build this other cmake project first, use a separate build tree and install everything into a different install prefix and then consider that when building the main project." and all of that with a relatively clear declarative style that does not involve writing a bunch of python code to achieve the integration.

Given that I have a workaround this does not have the highest priority for me. But it would be great if I could come back to this at some point (even if I have to wait a bit) and replace my workaround with a cleaner approach.

Btw. I have seen other issues that go in a similar direction and ask about changing the CMake install prefix (which seems to default to the build_folder rather than a subfolder like <build-folder>/install). I think it could be helpful to remove the implicit assumption that everything happens at the root of the build folder of the current conan context. CMake already provides very nice mechanisms to select cleanly separate source, build and install trees. Maybe first step could be to enable the cmake layout to insert additional subfolders (relative to self.build-folder) to customize the relative build and install tree location. These could default to . so that they don't need to be managed for most users, but could be overwritable by advanced users with a constructor argument of the different cmake classes.

[valgur]

I have also missed this capability when working on some multi-project CMake libraries and ended up using the "root CMakeLists" workaround.

Could the AutotoolsToolchain(self, namespace="...") feature be extended to CMakeToolchain and others, perhaps? It worked very elegantly in a fairly complex mingw-w64 recipe, for example: https://github.com/conan-io/conan-center-index/blob/a13772c47c67b01383bb858846ce29adf23cf537/recipes/mingw-w64/linux/conanfile.py#L130-L281 I would love to have something similar for CMake.

[memsharded]

I have had a look to check the complexity of this.
It happens that the CMake helper is extremely coupled with the layout and folders definition, which is a single one.
Splitting and making the layout definitions "multiple" sounds too much for this very niche use case, so the way to go would be parameterizing the CMake helper. But given the complexity and the use case, it seems it will be challenging to prioritize this at the moment.

[memsharded]

This has been implemented in #18905, that now allows this flow.
It will be in next Conan 2.21 release soon, many thanks for the feedback!