[feature] New cmake_find_mode - Copy Project provided Config files.

[tstrutz]

What is your suggestion?

We have been integrating conan into our C++ workflows for a number of weeks now and have noticed what feels like an oversight in the way that the build system integrates with the cmake-conan workflow.

One of our applications generates <PROJECT_NAME>-Config.cmake and <PROJECT_NAME>-ConfigVersion.cmake files, and Conan suggests the "none" cmake_find_mode" should be sufficient to use these. However, we have run into major issues with the way Conan separates configurations into separate packages. With the current "none" feature, there will be TWO <PROJECT_NAME>-Config.cmake files, one for Release and Debug. This is a problem, because there actually needs to be one file that tells you where to locate both configurations. The real issue with the current implementation is because of the way find_package works, it will never find both especially when using a Multi-Configuration such as Ninja-Multi-Config generator, and so the project typically finds only the configuration that happens to be located first.

Additionally, we can't use the generated Config files Conan makes as ours define find_dependency calls for a bunch of dependencies that can't be managed by Conan such as Matlab, and other non-public projects. We also set some of our project's data paths in these files, so that a downstream project (sorta like a plugin) can import these paths.

I have a pull request here that implements a new "copy" mode that will essentially copy the <PROJECT_NAME>-Config.cmake and <PROJECT_NAME>-ConfigVersion.cmake and CMake-generated targets files for each built package to the conan generators folder when using the Conan CMake integration.

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[memsharded]

Hi @tstrutz

I already did a review in #18816

I have started to investigate this a bit further, and the first issue I have found is that the CMake generated files with something like:

install(TARGETS hello EXPORT helloConfig)
        export(TARGETS hello
            NAMESPACE hello::
            FILE "${CMAKE_CURRENT_BINARY_DIR}/helloConfig.cmake"
        )
        install(EXPORT helloConfig
            DESTINATION "${CMAKE_INSTALL_PREFIX}/hello/cmake"
            NAMESPACE hello::
        )

contains by default absolute paths.
So these files are not relocatable and won't work in general. Our tests for this feature do not cover the important relocation case, so I'll add some test that illustrate this issue.

[tstrutz]

Hi @tstrutz

I already did a review in #18816

I have started to investigate this a bit further, and the first issue I have found is that the CMake generated files with something like:

install(TARGETS hello EXPORT helloConfig)
export(TARGETS hello
NAMESPACE hello::
FILE "${CMAKE_CURRENT_BINARY_DIR}/helloConfig.cmake"
)
install(EXPORT helloConfig
DESTINATION "${CMAKE_INSTALL_PREFIX}/hello/cmake"
NAMESPACE hello::
)

contains by default absolute paths. So these files are not relocatable and won't work in general. Our tests for this feature do not cover the important relocation case, so I'll add some test that illustrate this issue.

I'm not sure if this creates a difference, but we pass relative paths into the corresponding install paths, therefore I believe these should be relative to the value of CMAKE_INSTALL_PREFIX. Example here:

# Create the installation rules
install ( TARGETS ${${PROJECT_NAME}_TARGETS}
          EXPORT ${PROJECT_NAME}Targets
          PERMISSIONS OWNER_READ OWNER_WRITE GROUP_READ GROUP_WRITE WORLD_READ
          LIBRARY DESTINATION lib/$<CONFIG>
          RUNTIME DESTINATION bin/$<CONFIG>
          PUBLIC_HEADER DESTINATION include/${PROJECT_NAME}
          ARCHIVE DESTINATION lib/$<CONFIG>
          INCLUDES DESTINATION include/${PROJECT_NAME}
          )

install( EXPORT ${PROJECT_NAME}Targets
         FILE ${PROJECT_NAME}Targets.cmake
         NAMESPACE ${PROJECT_NAME}::
         DESTINATION cmake )

I do know for us on (CMake 3.27) the files do contain an ${_IMPORT_PREFIX} which should be relocatable in some sense. It points to the parent directory that the configuration file is present in typically.

[tstrutz]

Following up on some of your comments as you mentioned going to this thread for them @memsharded.

The problem with this is that the files that projects provide might have different names, the "Targets" files is mostly a convention, but not mandatory. This will not work in many cases, and it is necessary a more general approach

Should we provide the ability to pass in the corresponding names of these files then? I tried to use mostly existing code to generate these filenames.

Or just leverage the existing conan variables such as cmake_file_name etc?

A generator shouldn't be copying artifacts like headers or libs around, that is more the responsibility of a deployer.

Would you recommend moving this somewhere else in the code base, or just not doing this step entirely? This is how I attempted to make the files relocatable using the ${_IMPORT_PREFIX} which points to parent directory of the folder.

Another alternative might be to edit or replace the ${_IMPORT_PREFIX} so that include/lib references for each target point back to the conan package.

[memsharded]

I'm not sure if this creates a difference, but we pass relative paths into the corresponding install paths, therefore I believe these should be relative to the value of CMAKE_INSTALL_PREFIX. Example here:

Thanks for the hint.
Yes, I made my PR in #18819 to pass green by making paths relative instead of using CMAKE_INSTALL_PREFIX, in the different tests in the suite that did that.

Should we provide the ability to pass in the corresponding names of these files then? I tried to use mostly existing code to generate these filenames.

Or just leverage the existing conan variables such as cmake_file_name etc?

Probably leveraging conan properties like cmake_file_name, yes, but I am concerned this might even not be enough, for projects that generate files outside of the typical conventions, for example by not creating xxx-config-release.cmake.

Would you recommend moving this somewhere else in the code base, or just not doing this step entirely? This is how I attempted to make the files relocatable using the ${_IMPORT_PREFIX} which points to parent directory of the folder.

Another alternative might be to edit or replace the ${_IMPORT_PREFIX} so that include/lib references for each target point back to the conan package.

The idea of generators is that they should be able to use the artifacts from their installed location. Copying artifacts like headers or libraries sounds more like the deployer concept in https://blog.conan.io/2023/05/23/Conan-agnostic-deploy-dependencies.html, which could be a different approach, forgetting about using the artifacts from the cache and use them from an aggregated deployment in user folder, where the debug and release artifacts might be collected under the same folder.

At this moment, I'd like to explore first the concept of generator for this use case, what can be done by generating helper CMake files that could use the artifacts directly from the Conan cache. I'll try to brainstorm a bit tomorrow with the team.

Additionally, we can't use the generated Config files Conan makes as ours define find_dependency calls for a bunch of dependencies that can't be managed by Conan such as Matlab, and other non-public projects. We also set some of our project's data paths in these files, so that a downstream project (sorta like a plugin) can import these paths.

Just in case you haven't considered, there is the cmake_build_modules approach that allows to inject any arbitrary functionality to the Conan CMakeDeps generated files, maybe that could also provide the functionality you are looking for while leveraging the CMakeDeps multi-config functionality.

[tstrutz]

The cmake_build_modules thing is a good possible candidate for this. However, one of the dependencies we use is actually conditional. So it may be very difficult to inject this in an expected way with Conan.

It looks like a lot of these could be moved to a seperate file that is then included in the Conan file.

If we were to wrap these find_dependency calls, will conan generate these with the appropriate cmake configuration calls to enable/disable the booleans? It looks like we could also just use multiple cmake.in files that generate the corresponding stuff too.

The project Config.cmake.in file that we are trying to use, though this is a sanitized version:

include(CMakeFindDependencyMacro)
find_dependency(OpenMP REQUIRED)
find_dependency(midasio REQUIRED)
find_dependency(MatlabIO REQUIRED)
find_dependency(vclMath REQUIRED)
find_dependency(fmt CONFIG REQUIRED)
find_dependency(stduuid REQUIRED)
find_dependency(spdlog REQUIRED)
find_dependency(nlohmann_json REQUIRED)

if (@MY_PROJECT_CREATE_PYTHON_BINDINGS@)
    find_dependency(Python3 REQUIRED COMPONENTS Development.Module)
endif()

if (@MY_PROJECT_CMAKE_USE_AM6K@)
    find_dependency(am6k REQUIRED)
endif()

if (@MY_PROJECT_CMAKE_USE_BOOST@)
    find_dependency(Boost REQUIRED)
endif()

if (@MY_PROJECT_CMAKE_USE_ASIO@)
    find_dependency(asio REQUIRED)
endif()

if (@MY_PROJECT_CMAKE_USE_DATACUBE@)
    find_dependency(xtensor REQUIRED)
endif()

if (@MY_PROJECT_CMAKE_USE_DRSLAM@)
    find_dependency( DRSLAMAPI )
endif()

if(@MY_PROJECT_CMAKE_USE_BLUEMAX@)
    find_dependency( Bluemax6 REQUIRED )
endif()

if (@MY_PROJECT_CMAKE_USE_MATLAB_ENGINE@)
    find_dependency( Matlab REQUIRED)
endif()

if(@MY_PROJECT_CMAKE_BUILD_MFA@)
    find_dependency(Boost COMPONENTS random REQUIRED)
    find_dependency(websocketpp REQUIRED)
endif()

if (@MY_PROJECT_CMAKE_USE_CUDA@) # <---- This may not work with the `cmake_build_modules` code as only FFTW is managed by conan here.
    find_dependency( CUDAToolkit REQUIRED COMPONENTS cufft cublas cudart cuda_driver )
else()
    find_dependency(FFTW3 REQUIRED)
endif()

if (@MY_PROJECT_CMAKE_USE_OPTIX@)
    find_dependency( OptiX REQUIRED )
endif()

if (@MY_PROJECT_CMAKE_USE_QT5_NETWORK@)
    find_dependency( Qt5 COMPONENTS Core Network REQUIRED )
endif()

list(APPEND CMAKE_MODULE_PATH ${CMAKE_CURRENT_LIST_DIR})
include("${CMAKE_CURRENT_LIST_DIR}/MY_PROJECTConfigVersion.cmake")

set(MY_PROJECT_VERSION ${PACKAGE_VERSION})
set(MY_PROJECT_DATA_PATH @MY_PROJECT_DATA_PATH@)
set(MY_PROJECT_CONFIG_PATH @MY_PROJECT_CONFIG_PATH@)

include("${CMAKE_CURRENT_LIST_DIR}/MY_PROJECTTargets.cmake")

@PACKAGE_INIT@

message(STATUS "Found MY_PROJECT cmake installation at ${CMAKE_CURRENT_LIST_FILE}")
message(STATUS "The loaded MY_PROJECTData Path is ${MY_PROJECT_DATA_PATH}")
message(STATUS "The loaded MY_PROJECTConfig Path is ${MY_PROJECT_CONFIG_PATH}")

set_and_check(MY_PROJECT_INSTALL_PATH ${PACKAGE_PREFIX_DIR})

check_required_components(MY_PROJECT)

[tstrutz]

Probably leveraging conan properties like cmake_file_name, yes, but I am concerned this might even not be enough, for projects that generate files outside of the typical conventions, for example by not creating xxx-config-release.cmake.

Would it make sense to provide a list of the cmake files that your project directly provides to be placed with the other CMake files instead of searching for the specific names like we do now?

New field, let's call it cmake_modules_provided or similar. It'd be a direct list of the files that we should search for in the installed packaged, instead of just assuming the names like I am doing.