Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BLD/RLS: update wheels to include GDAL 3.10.2 #524

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

jorisvandenbossche
Copy link
Member

Updating to GDAL 3.10.2 (microsoft/vcpkg@d24ce43)

@jorisvandenbossche
Copy link
Member Author

The issue with the linux wheels is the build of GDAL failing with (got this message from running the docker image locally):

...
-- Looking for sqlite3ext.h
-- Looking for sqlite3ext.h - found
CMake Error at cmake/helpers/CheckDependentLibraries.cmake:276 (message):
  /opt/vcpkg/installed/x64-linux-dynamic-release/lib/libsqlite3.a lacks the
  RTree extension! GPKG will not behave properly.  Define the
  ACCEPT_MISSING_SQLITE3_RTREE:BOOL=ON CMake variable if you want to build
  despite this limitation.
Call Stack (most recent call first):
  ogr/ogrsf_frmts/gpkg/CMakeLists.txt:19 (check_sqlite3_rtree)

For some reason, it thinks that sqlite is not built with rtree support, although vcpkg clearly indicates that it did do that (this can also bee seen in the vcpkg logs in CI):

# vcpkg install --overlay-triplets=opt/vcpkg/custom-triplets \
>     --overlay-ports=opt/vcpkg/custom-ports \
>     --feature-flags="versions,manifests" \
>     --x-manifest-root=opt/vcpkg \
>     --x-install-root=opt/vcpkg/installed
Detecting compiler hash for triplet x64-linux...
Compiler found: /opt/rh/devtoolset-10/root/usr/bin/c++
Detecting compiler hash for triplet x64-linux-dynamic-release...
Compiler found: /opt/rh/devtoolset-10/root/usr/bin/c++
The following packages will be built and installed:
  * curl[core,non-http,openssl,ssl]:[email protected] -- /opt/vcpkg/buildtrees/versioning_/versions/curl/faa9e75a688eeb7144584f9a86c014cefffddb98
  * expat:[email protected] -- /opt/vcpkg/buildtrees/versioning_/versions/expat/97dd7e516104f330e1bb5eafff10852211dbd2df
    gdal[core,curl,expat,geos,iconv,jpeg,lerc,openssl,png,qhull,recommended-features,sqlite3]:[email protected] -- /opt/vcpkg/buildtrees/versioning_/versions/gdal/0bf0a27bf465e7c06645b60990e8d67ed6546f51
  * geos:[email protected]#1 -- /opt/vcpkg/buildtrees/versioning_/versions/geos/331bb2a4ee2ca09a1d85f801bf3eb52a0ebb2acf
  * json-c:[email protected] -- /opt/vcpkg/buildtrees/versioning_/versions/json-c/786a0712b7019e33da008008e17fb920932e9e8d
  * lerc:[email protected] -- /opt/vcpkg/buildtrees/versioning_/versions/lerc/6220867a42fbfbe091a676ac82582a9969788178
  * libgeotiff:[email protected]#1 -- /opt/vcpkg/buildtrees/versioning_/versions/libgeotiff/160d4b272c93184c34fdb69402484ac856ad10f5
  * libiconv:[email protected] -- /opt/vcpkg/buildtrees/versioning_/versions/libiconv/e6b1ec6194b4adf7c08d114be8dcc848950094e1
  * libjpeg-turbo:[email protected]#1 -- /opt/vcpkg/buildtrees/versioning_/versions/libjpeg-turbo/6180688844449e5724d6dc8eb49ab90124438ce2
  * liblzma:[email protected] -- /opt/vcpkg/buildtrees/versioning_/versions/liblzma/b5e5694620b41a4d668390e5d14fa2326e0afdc3
  * libpng:[email protected] -- /opt/vcpkg/buildtrees/versioning_/versions/libpng/0074ace75112e0a4ff3465de21e692c07d6e6b5f
  * nlohmann-json:[email protected]#1 -- /opt/vcpkg/buildtrees/versioning_/versions/nlohmann-json/324d32ddba70bc49650d1a067ddcf3d0bfc6c9b6
  * openssl:[email protected] -- /opt/vcpkg/buildtrees/versioning_/versions/openssl/d740be74dc7940c2b88e69db0b2d181024e1028c
  * pkgconf:[email protected] -- /opt/vcpkg/buildtrees/versioning_/versions/pkgconf/ae3886d8a627ec99dd18890389b6d5d331e29799
  * proj[core,net,tiff]:[email protected] -- /opt/vcpkg/buildtrees/versioning_/versions/proj/18367089a6c705c372a3d3ed2f5d631420c80340
  * qhull:[email protected]#5 -- /opt/vcpkg/buildtrees/versioning_/versions/qhull/0c30770c608574944db1c98437d356af3e64fe5b
  * sqlite3[core,json1,rtree]:[email protected] -- /opt/vcpkg/buildtrees/versioning_/versions/sqlite3/943b28204d93333f225d1f79486f7ab6f3e66295
  * sqlite3[core,json1,tool]:[email protected] -- /opt/vcpkg/buildtrees/versioning_/versions/sqlite3/943b28204d93333f225d1f79486f7ab6f3e66295
  * tiff[core,jpeg,lzma,zip]:[email protected] -- /opt/vcpkg/buildtrees/versioning_/versions/tiff/54c1c045d21157ce32df8a1c8b7b9d57b0d5d6ce
  * vcpkg-cmake:x64-linux@2024-04-23 -- /opt/vcpkg/buildtrees/versioning_/versions/vcpkg-cmake/e74aa1e8f93278a8e71372f1fa08c3df420eb840
  * vcpkg-cmake-config:x64-linux@2024-05-23 -- /opt/vcpkg/buildtrees/versioning_/versions/vcpkg-cmake-config/97a63e4bc1a17422ffe4eff71da53b4b561a7841
  * vcpkg-cmake-get-vars:x64-linux@2024-09-22 -- /opt/vcpkg/buildtrees/versioning_/versions/vcpkg-cmake-get-vars/f23148add155147f3d95ae622d3b0031beb25acf
  * vcpkg-pkgconfig-get-modules:x64-linux@2024-04-03 -- /opt/vcpkg/buildtrees/versioning_/versions/vcpkg-pkgconfig-get-modules/6845369c8cb7d3c318e8e3ae92fd2b7570a756ca
  * vcpkg-tool-meson:[email protected] -- /opt/vcpkg/buildtrees/versioning_/versions/vcpkg-tool-meson/dc948c67d7f1359319f801078422e996b0a89fd0
  * zlib:[email protected] -- /opt/vcpkg/custom-ports/zlib

So it installs sqlite3[core,json1,rtree]:[email protected] and also sqlite3[core,json1,tool]:[email protected] (this is coming as a dependency for proj[tools]).
I am wondering if the fact that there are those two versions of sqlite (one native to the target and one for the host) that causes some issue and the wrong one to be found by gdal. But, we already have been building those two versions of sqlite3 for quite a while, and so only now this started to give some issue ...

Locally I tried to then checkout an older commit for GDAL 3.10.1 or 3.10.0, but actually running in the same issue ..

@theroggy
Copy link
Member

theroggy commented Mar 10, 2025

I'm not sure if it is related, but at least it is a "coincidence"... but I and other users started noticing installation issues when installing geofileops via conda since the beginning of last week.

The issue is that a 4 year old version (3.32) of the "sqlite" package is occasionally being installed while the "libsqlite" package is a recent one. The "sqlite" conda package is the one being depended on by gdal, and when this older version is installed, gdal has issues to load because some entry points build against in sqlite were only introduced in sqlite version 3.7.

I don't know why this old version is getting installed though :-(...

Related issues: geofileops/geofileops#653, geofileops/geofileops#645, OSGeo/gdal#9021

@jorisvandenbossche
Copy link
Member Author

I am assuming that that is not related. Here it is a build issue (not failing to find some symbol at import), and both sqlite versions installed are a recent version (3.49.0). Unless there might be another sqlite that is installed on the base docker image that for some reason gets found instead.

@jorisvandenbossche
Copy link
Member Author

Success! (at least for now, by using an older manylinux image, will see in the future when we have to update that ..).

In the verbose error logs, there was something about "ld: cannot find -l@LDFLAGS_MATH@" when trying detect the features of sqlite using CMake's check_symbol_exists (check_symbol_exists(sqlite3_rtree_query_callback sqlite3.h SQLite3_HAS_RTREE)).
Potentially related message: https://sqlite.org/forum/info/e40b9b424a. I noticed that in the manylinux tracker, there was also some mention of problems after a recent sqlite update: pypa/manylinux#1752. Both point to some recent build configuration changes in sqlite.

And because the manylinux image comes with libsqlite as well (since python uses that), that might mess with how cmake is finding the incorrect sqlite lib when detecting the features in the GDAL build step.
Tested this by using a slightly older manylinux image, that still has sqlite 3.47 (the build configuration changes were introduced in 3.48). And now it seems to work.

So good for now, but we will still need to figure out how to fix this issue later this year when we have to update the manylinux images ... (for example, when updating to include newer python versions)

Logs from check_symbol_exists (the full logs of this can be seen at cat /opt/vcpkg/buildtrees/gdal/x64-linux-dynamic-release-rel/CMakeFiles/CMakeConfigureLog.yaml in the docker image) in the failure case:

...
  -
    kind: "try_compile-v1"
    backtrace:
      - "/opt/_internal/pipx/venvs/cmake/lib/python3.12/site-packages/cmake/data/share/cmake-3.31/Modules/CheckSymbolExists.cmake:163 (try_compile)"
      - "/opt/_internal/pipx/venvs/cmake/lib/python3.12/site-packages/cmake/data/share/cmake-3.31/Modules/CheckSymbolExists.cmake:68 (__CHECK_SYMBOL_EXISTS_IMPL)"
      - "cmake/modules/packages/FindSQLite3.cmake:138 (check_symbol_exists)"
      - "/opt/vcpkg/scripts/buildsystems/vcpkg.cmake:859 (_find_package)"
      - "cmake/helpers/CheckDependentLibrariesCommon.cmake:150 (find_package)"
      - "cmake/helpers/CheckDependentLibraries.cmake:246 (gdal_check_package)"
      - "gdal.cmake:69 (include)"
      - "CMakeLists.txt:225 (include)"
    checks:
      - "Looking for sqlite3_rtree_query_callback"
    directories:
      source: "/opt/vcpkg/buildtrees/gdal/x64-linux-dynamic-release-rel/CMakeFiles/CMakeScratch/TryCompile-9i14rR"
      binary: "/opt/vcpkg/buildtrees/gdal/x64-linux-dynamic-release-rel/CMakeFiles/CMakeScratch/TryCompile-9i14rR"
    cmakeVariables:
      CMAKE_CXX_SCAN_FOR_MODULES: "0"
      CMAKE_C_FLAGS: "-fPIC  "
      CMAKE_C_FLAGS_DEBUG: "-g"
      CMAKE_EXE_LINKER_FLAGS: ""
      CMAKE_MODULE_PATH: "/opt/vcpkg/buildtrees/gdal/src/v3.10.2-f928bb402d.clean/cmake/modules/packages;/opt/vcpkg/buildtrees/gdal/src/v3.10.2-f928bb402d.clean/cmake/modules/thirdparty;/opt/vcpkg/buildtrees/gdal/src/v3.10.2-f928bb402d.clean/cmake/modules;/opt/vcpkg/buildtrees/gdal/src/v3.10.2-f928bb402d.clean/cmake/modules/../helpers"
      VCPKG_CHAINLOAD_TOOLCHAIN_FILE: "/opt/vcpkg/scripts/toolchains/linux.cmake"
      VCPKG_CRT_LINKAGE: "dynamic"
      VCPKG_CXX_FLAGS: ""
      VCPKG_CXX_FLAGS_DEBUG: ""
      VCPKG_CXX_FLAGS_RELEASE: ""
      VCPKG_C_FLAGS: ""
      VCPKG_C_FLAGS_DEBUG: ""
      VCPKG_C_FLAGS_RELEASE: ""
      VCPKG_HOST_TRIPLET: "x64-linux"
      VCPKG_INSTALLED_DIR: "/opt/vcpkg/installed"
      VCPKG_LINKER_FLAGS: ""
      VCPKG_LINKER_FLAGS_DEBUG: ""
      VCPKG_LINKER_FLAGS_RELEASE: ""
      VCPKG_PREFER_SYSTEM_LIBS: "OFF"
      VCPKG_TARGET_ARCHITECTURE: "x64"
      VCPKG_TARGET_TRIPLET: "x64-linux-dynamic-release"
      Z_VCPKG_ROOT_DIR: "/opt/vcpkg"
    buildResult:
      variable: "SQLite3_HAS_RTREE"
      cached: true
      stdout: |
        Change Dir: '/opt/vcpkg/buildtrees/gdal/x64-linux-dynamic-release-rel/CMakeFiles/CMakeScratch/TryCompile-9i14rR'
        
        Run Build Command(s): /opt/vcpkg/downloads/tools/ninja/1.12.1-linux/ninja -v cmTC_4fdd2
        [1/2] /opt/rh/devtoolset-10/root/usr/bin/cc  -I/opt/vcpkg/installed/x64-linux-dynamic-release/include -fPIC    -std=gnu99 -o CMakeFiles/cmTC_4fdd2.dir/CheckSymbolExists.c.o -c /opt/vcpkg/buildtrees/gdal/x64-linux-dynamic-release-rel/CMakeFiles/CMakeScratch/TryCompile-9i14rR/CheckSymbolExists.c
        [2/2] : && /opt/rh/devtoolset-10/root/usr/bin/cc -fPIC  CMakeFiles/cmTC_4fdd2.dir/CheckSymbolExists.c.o -o cmTC_4fdd2  -Wl,--enable-new-dtags,-rpath=/usr/local/lib  -L/usr/local/lib  -lsqlite3  -l@LDFLAGS_MATH@  -l@LDFLAGS_ZLIB@  -l@LDFLAGS_ICU@ && :
        FAILED: cmTC_4fdd2 
        : && /opt/rh/devtoolset-10/root/usr/bin/cc -fPIC  CMakeFiles/cmTC_4fdd2.dir/CheckSymbolExists.c.o -o cmTC_4fdd2  -Wl,--enable-new-dtags,-rpath=/usr/local/lib  -L/usr/local/lib  -lsqlite3  -l@LDFLAGS_MATH@  -l@LDFLAGS_ZLIB@  -l@LDFLAGS_ICU@ && :
        /opt/rh/devtoolset-10/root/usr/libexec/gcc/x86_64-redhat-linux/10/ld: cannot find -l@LDFLAGS_MATH@
        /opt/rh/devtoolset-10/root/usr/libexec/gcc/x86_64-redhat-linux/10/ld: cannot find -l@LDFLAGS_ZLIB@
        /opt/rh/devtoolset-10/root/usr/libexec/gcc/x86_64-redhat-linux/10/ld: cannot find -l@LDFLAGS_ICU@
        collect2: error: ld returned 1 exit status
        ninja: build stopped: subcommand failed.
        
      exitCode: 1
...

and in the working case now:

...
  -
    kind: "try_compile-v1"
    backtrace:
      - "/opt/_internal/pipx/venvs/cmake/lib/python3.12/site-packages/cmake/data/share/cmake-3.31/Modules/CheckSymbolExists.cmake:163 (try_compile)"
      - "/opt/_internal/pipx/venvs/cmake/lib/python3.12/site-packages/cmake/data/share/cmake-3.31/Modules/CheckSymbolExists.cmake:68 (__CHECK_SYMBOL_EXISTS_IMPL)"
      - "cmake/modules/packages/FindSQLite3.cmake:138 (check_symbol_exists)"
      - "/opt/vcpkg/scripts/buildsystems/vcpkg.cmake:859 (_find_package)"
      - "cmake/helpers/CheckDependentLibrariesCommon.cmake:150 (find_package)"
      - "cmake/helpers/CheckDependentLibraries.cmake:246 (gdal_check_package)"
      - "gdal.cmake:69 (include)"
      - "CMakeLists.txt:225 (include)"
    checks:
      - "Looking for sqlite3_rtree_query_callback"
    directories:
      source: "/opt/vcpkg/buildtrees/gdal/x64-linux-dynamic-release-rel/CMakeFiles/CMakeScratch/TryCompile-xlESup"
      binary: "/opt/vcpkg/buildtrees/gdal/x64-linux-dynamic-release-rel/CMakeFiles/CMakeScratch/TryCompile-xlESup"
    cmakeVariables:
      CMAKE_CXX_SCAN_FOR_MODULES: "0"
      CMAKE_C_FLAGS: "-fPIC  "
      CMAKE_C_FLAGS_DEBUG: "-g"
      CMAKE_EXE_LINKER_FLAGS: ""
      CMAKE_MODULE_PATH: "/opt/vcpkg/buildtrees/gdal/src/v3.10.2-f928bb402d.clean/cmake/modules/packages;/opt/vcpkg/buildtrees/gdal/src/v3.10.2-f928bb402d.clean/cmake/modules/thirdparty;/opt/vcpkg/buildtrees/gdal/src/v3.10.2-f928bb402d.clean/cmake/modules;/opt/vcpkg/buildtrees/gdal/src/v3.10.2-f928bb402d.clean/cmake/modules/../helpers"
      VCPKG_CHAINLOAD_TOOLCHAIN_FILE: "/opt/vcpkg/scripts/toolchains/linux.cmake"
      VCPKG_CRT_LINKAGE: "dynamic"
      VCPKG_CXX_FLAGS: ""
      VCPKG_CXX_FLAGS_DEBUG: ""
      VCPKG_CXX_FLAGS_RELEASE: ""
      VCPKG_C_FLAGS: ""
      VCPKG_C_FLAGS_DEBUG: ""
      VCPKG_C_FLAGS_RELEASE: ""
      VCPKG_HOST_TRIPLET: "x64-linux"
      VCPKG_INSTALLED_DIR: "/opt/vcpkg/installed"
      VCPKG_LINKER_FLAGS: ""
      VCPKG_LINKER_FLAGS_DEBUG: ""
      VCPKG_LINKER_FLAGS_RELEASE: ""
      VCPKG_PREFER_SYSTEM_LIBS: "OFF"
      VCPKG_TARGET_ARCHITECTURE: "x64"
      VCPKG_TARGET_TRIPLET: "x64-linux-dynamic-release"
      Z_VCPKG_ROOT_DIR: "/opt/vcpkg"
    buildResult:
      variable: "SQLite3_HAS_RTREE"
      cached: true
      stdout: |
        Change Dir: '/opt/vcpkg/buildtrees/gdal/x64-linux-dynamic-release-rel/CMakeFiles/CMakeScratch/TryCompile-xlESup'
        
        Run Build Command(s): /opt/vcpkg/downloads/tools/ninja/1.12.1-linux/ninja -v cmTC_5e2e1
        [1/2] /opt/rh/devtoolset-10/root/usr/bin/cc  -I/opt/vcpkg/installed/x64-linux-dynamic-release/include -fPIC    -std=gnu99 -o CMakeFiles/cmTC_5e2e1.dir/CheckSymbolExists.c.o -c /opt/vcpkg/buildtrees/gdal/x64-linux-dynamic-release-rel/CMakeFiles/CMakeScratch/TryCompile-xlESup/CheckSymbolExists.c
        [2/2] : && /opt/rh/devtoolset-10/root/usr/bin/cc -fPIC  CMakeFiles/cmTC_5e2e1.dir/CheckSymbolExists.c.o -o cmTC_5e2e1  -Wl,--enable-new-dtags,-rpath=/usr/local/lib  -L/usr/local/lib  -lsqlite3  -lz  -lm  -ldl  -lpthread && :
        
      exitCode: 0
...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants