external_deps: revamp lib build, add cmake and configure wrapper, update lib feature disablement, update lib versions #1433

illwieckz · 2024-11-11T16:57:00Z

This is WIP, current external_deps build status:

system	deps build	static build	test
linux-amd64-default	✅️	✅️	✅️
linux-arm64-default	✅️
linux-i686-default	✅️	✅️	✅️
linux-armhf-default	✅️
windows-amd64-mingw	✅️	N/A	✅️
windows-i686-mingw	✅️ Buster ❌️ GLEW Noble	N/A	✅️
windows-amd64-msvc	✅️
windows-i686-msvc	✅️
macos-amd64-default	✅️	N/A	✅️

~~I haven't tested if the engine builds and runs properly with those.~~ WIP

What this PR does:

Use CMake when packages provide a CMakeLists.txt file
Factorize much code
Make a nice list of package base url, so it's easy to click all of them and check for new versions
Update package versions
Update package build options

illwieckz · 2024-11-11T17:00:47Z

For windows-i686-mingw I get this with GLEW:

i686-w64-mingw32-ld -nostdlib -shared -soname libglew32.dll --out-implib lib/libglew32.dll.a
     -o lib/glew32.dll tmp/linux-mingw32/default/shared/glew.o
     -Lexternal_deps/build-windows-i686-mingw_10/prefix/lib -L/usr/i686-w64-mingw32/lib
     -lopengl32 -lgdi32 -luser32 -lkernel32 
i686-w64-mingw32-ld: tmp/linux-mingw32/default/shared/glew.o:glew.c:(.text+0x141ab):
 undefined reference to `strlen'

Everything else build.

illwieckz · 2024-11-11T17:23:07Z

For windows-amd64-msvc I get this with Vorbis:

[ 84%] Linking C shared library libvorbis.dll
/usr/bin/x86_64-w64-mingw32-ld:
 external_deps/build-windows-amd64-msvc_10/vorbis/libvorbis-1.3.7/win32/vorbis.def:3: syntax error
/usr/bin/x86_64-w64-mingw32-ld:
 external_deps/build-windows-amd64-msvc_10/vorbis/libvorbis-1.3.7/win32/vorbis.def: file format not recognized; treating as linker script
/usr/bin/x86_64-w64-mingw32-ld:
 external_deps/build-windows-amd64-msvc_10/vorbis/libvorbis-1.3.7/win32/vorbis.def:2: syntax error

Everything else build.

illwieckz · 2024-11-11T18:28:39Z

OGG has a CMakeLists.txt file but no install target, so I returned back to configure for this one.
Opus requires at least CMake 3.16.

illwieckz · 2024-11-11T18:32:24Z

When building newer Opus on Debian buster (the distro we use for our releases), I get this:

opus/opus-1.5.2/silk/x86/NSQ_del_dec_avx2.c:959:43: warning: implicit declaration of function '_mm_loadu_si64'; did you mean '_mm_loadu_si32'? [-Wimplicit-function-declaration]
         __m256i x = _mm256_cvtepi16_epi64(_mm_loadu_si64(&x16[i]));
                                           ^~~~~~~~~~~~~~
                                           _mm_loadu_si32

Debian Buster provides GCC 8, but GCC 11.3 or GCC 12 may be required:
https://stackoverflow.com/questions/72837929/mm-loadu-si32-not-recognized-by-gcc-on-ubuntu

illwieckz · 2024-11-11T18:41:29Z

When building newer Opus on Debian buster (the distro we use for our releases), I get this: […]

Telling Opus to not assume more than SSE2 fixed that.

illwieckz · 2024-11-11T18:54:40Z

So, the MinGW GLEW and MSVC Vorbis errors are the only ones.

illwieckz · 2024-11-11T19:26:25Z

So the Vorbis build error is actually a bug:

I added a workaround.

illwieckz · 2024-11-11T19:31:40Z

I don't get why I get that GLEW build error with MinGW, the build function has not been modified, and the version has not been updated.

illwieckz · 2024-11-11T19:35:07Z

I don't get why I get that GLEW build error with MinGW, the build function has not been modified, and the version has not been updated.

Also the code is the same for both windows-amd64-msvc, windows-i686-msvc, windows-amd64-mingw and windows-i686-mingw, but it only fails with windows-i686-mingw… That doesn't make sense.

illwieckz · 2024-11-11T19:40:17Z

So, I don't know what happened, now I don't reproduce the MinGW GLEW error… Maybe I gorgot to prune the prefix folder and some stray files messed-up…

illwieckz · 2024-11-11T19:41:54Z

Ah, I now see something: I reproduce the bug with MinGW from Ubuntu 24.04 Noble, not with MinGW from Debian 10 Buster. So, since we produce release builds with Debian Buster, it's not a big problem, but it should be fixed for the future…

slipher · 2024-11-11T21:24:14Z

What's the purpose of migrating things to build with CMake?

illwieckz · 2024-11-16T21:31:33Z

So with this a static build for linux-amd64 completes and runs.

slipher · 2024-11-16T21:47:57Z

So with this a static build for linux-amd64 completes and runs.

Huh? It worked before, so I have a hard time believing that is suddenly necessary to change the build system of 8 packages.

illwieckz · 2024-11-16T21:49:58Z

I never said this is a response to “What's the purpose of migrating things to build with CMake?”, I'm just reporting the status of me testing that branch. I said in first post:

I haven't tested if the engine builds and runs properly with those.

So now I'm running those tests.

DolceTriade · 2024-11-16T22:04:08Z

What's the motivation behind this change?

illwieckz · 2024-11-18T01:00:22Z

Purposes:

Update packages to latest versions,
Update packages build options for latest versions (for example curl adds protocols, so we have to disable them),
Mutualize cmake/configure code as generic reused functions and avoid copy/paste boilerplate,
Rely on CMake as much as possible to make it easy to spot new CMake build options to disable them and other changes.

When using configure it is hard to compare the list of already used options with the new options, one has to read configure's whole output and compare with what's currently used. On the contrary with cmake, one just runs the existing command, then go to the build dir and run ccmake, and see what's enabled and should not, and report the difference to the build script.

illwieckz · 2024-11-18T03:06:40Z

windows-amd64-mingw and windows-i686-mingw engine both build and run.

slipher · 2025-04-21T10:55:14Z

cmake/cross-toolchain-mingw32.cmake

@@ -8,7 +8,7 @@ set( CMAKE_CXX_COMPILER i686-w64-mingw32-g++ )
 set( CMAKE_RC_COMPILER i686-w64-mingw32-windres )

 # Target prefix
-set( CMAKE_FIND_ROOT_PATH /usr/i686-w64-mingw32 )
+set( CMAKE_FIND_ROOT_PATH /usr/i686-w64-mingw32;${CMAKE_PREFIX_PATH} )


Surely this doesn't do anything, as the toolchain file is executed before CMakeLists.txt, so CMAKE_PREFIX_PATH wouldn't be set (unless you set it on the command line.)

Oh it's for building external_deps, not compiling the game. Could use a comment

slipher · 2025-04-21T11:02:00Z

external_deps/build.sh

+	case "${PLATFORM}" in
+	windows-*-*)
+		# CMake looks for libSDL2.a and aborts if missing if this file exists:
+		rm -rf "${PKG_PREFIX}/lib/cmake/SDL2/SDL2staticTargets.cmake"


What's this? I don't see a file like this anywhere, and if I revert this commit it does not appear.

For me, this file came up and sabotaged the build not for Windows, but for Linux targets by injecting a -lsamplerate flag. Anyway on the slipher/deps-combined branch I reverted that commit and did a proper fix by configuring the SDL build not to use the dependency.

slipher · 2025-04-21T11:09:54Z

external_deps/build.sh

+		-DCURL_USE_MBEDTLS=OFF \
+		-DCURL_USE_NSS=OFF \
+		-DCURL_USE_OPENSSL=OFF \
+		-DCURL_USE_WOLFSSL=ON \


Confusion with USE/DISABLE, should be OFF.

slipher · 2025-04-21T22:47:44Z

The zlib build is not usable for MSVC because it produces a zconf.h including <unistd.h>.

slipher · 2025-04-22T05:33:33Z

For MSVC platforms, the resulting JPEG, WebP, and Curl DLLs have a libgcc DLL dependency. This problem is actually already present with JPEG in the version 10 deps package.
So you currently cannot run a client built with Visual Studio if you do not have MinGW in your PATH. With webp and curl it is a new regression.

slipher · 2025-04-22T22:37:20Z

I have pushed a branch named slipher/deps-combined which fixes all issues known to me. It contains the commits from #1642, #1643, this PR, and several more on top of that.

illwieckz · 2025-05-01T23:49:02Z

Note to myself: There may be some absolute path to rewrite in some installed .cmake files using ${CMAKE_CURRENT_LIST_DIR} instead. Especially with SDL2.

slipher · 2025-05-01T23:51:13Z

I know about the absolute path issue and fixed all cases in the slipher/deps-combined branch.

slipher · 2025-05-30T00:31:54Z

Rebased illwieckz/deps and slipher/deps-combined following merge of #1642.

illwieckz · 2025-05-30T03:18:45Z

Thanks

illwieckz · 2025-07-23T15:12:01Z

external_deps/build.sh


 base_linux_armhf_default_packages="${base_linux_arm64_default_packages}"
+release_linux_armhf_default_packages="${release_linux_arm64_default_packages}"


@slipher the release is here.

Responding to:

docker: bump Debian version from buster to bullseye, and rebuild most Linux libraries Unvanquished/release-scripts#40 (comment)

slipher · 2025-07-23T17:01:37Z

What's the status of this, is it ready for the next round of review yet?

illwieckz added 2 commits November 11, 2024 00:01

external_deps: use -O2 and -fPIC for all packages

118291c

cmake: make MinGW toolchains also look for CMAKE_PREFIX_PATH

01b7a8c

illwieckz marked this pull request as draft November 11, 2024 16:57

external_deps: mutualize some MinGW and MSVC code

269447d

illwieckz force-pushed the illwieckz/deps branch from cc59996 to bd1e318 Compare November 11, 2024 18:24

illwieckz force-pushed the illwieckz/deps branch from 03961c7 to c537fe9 Compare November 13, 2024 03:03

illwieckz added 2 commits November 16, 2024 20:51

cmake: download test deps archive from test/deps if not released yet

517ca98

external_deps: bump lib versions

144727f

illwieckz force-pushed the illwieckz/deps branch 2 times, most recently from ac1c6e1 to 5653846 Compare November 18, 2024 01:26

external_deps: refactor and unify cmake and configure usage

75501a3

illwieckz force-pushed the illwieckz/deps branch from 5860e20 to c73312c Compare November 18, 2024 02:37

slipher reviewed Apr 21, 2025

View reviewed changes

slipher force-pushed the illwieckz/deps branch from caf3648 to 21d2e2b Compare May 30, 2025 00:31

illwieckz force-pushed the illwieckz/deps branch from 21d2e2b to 612a195 Compare July 17, 2025 21:23

external_deps: naclruntime is only buildable on linux-amd64

a464205

illwieckz force-pushed the illwieckz/deps branch 7 times, most recently from 6b6439a to d72b466 Compare July 19, 2025 20:35

illwieckz added 3 commits July 19, 2025 23:12

external_deps: add the missing “ncurses” to “all” linux dependencies

48abfaf

cmake,external_deps: bump deps version

a5e1f3b

external_deps: add the “release” meta package

47ecb1d

illwieckz force-pushed the illwieckz/deps branch 2 times, most recently from 9c6042a to 4fd2627 Compare July 19, 2025 22:03

illwieckz mentioned this pull request Jul 19, 2025

docker: bump Debian version from buster to bullseye, and rebuild most Linux libraries Unvanquished/release-scripts#40

Open

illwieckz force-pushed the illwieckz/deps branch from 4fd2627 to 47ecb1d Compare July 19, 2025 22:28

illwieckz commented Jul 23, 2025

View reviewed changes

illwieckz mentioned this pull request Aug 1, 2025

The Compiler support topic #1720

Open


		base_linux_armhf_default_packages="${base_linux_arm64_default_packages}"
		release_linux_armhf_default_packages="${release_linux_arm64_default_packages}"

external_deps: revamp lib build, add cmake and configure wrapper, update lib feature disablement, update lib versions #1433

Are you sure you want to change the base?

external_deps: revamp lib build, add cmake and configure wrapper, update lib feature disablement, update lib versions #1433

Uh oh!

Conversation

illwieckz commented Nov 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

illwieckz commented Nov 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

illwieckz commented Nov 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

illwieckz commented Nov 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

illwieckz commented Nov 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

illwieckz commented Nov 11, 2024

Uh oh!

illwieckz commented Nov 11, 2024

Uh oh!

illwieckz commented Nov 11, 2024

Uh oh!

illwieckz commented Nov 11, 2024

Uh oh!

illwieckz commented Nov 11, 2024

Uh oh!

illwieckz commented Nov 11, 2024

Uh oh!

illwieckz commented Nov 11, 2024

Uh oh!

slipher commented Nov 11, 2024

Uh oh!

illwieckz commented Nov 16, 2024

Uh oh!

slipher commented Nov 16, 2024

Uh oh!

illwieckz commented Nov 16, 2024

Uh oh!

DolceTriade commented Nov 16, 2024

Uh oh!

illwieckz commented Nov 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

illwieckz commented Nov 18, 2024

Uh oh!

slipher Apr 21, 2025

Choose a reason for hiding this comment

Uh oh!

slipher Apr 21, 2025

Choose a reason for hiding this comment

Uh oh!

slipher Apr 21, 2025

Choose a reason for hiding this comment

Uh oh!

slipher Apr 23, 2025

Choose a reason for hiding this comment

Uh oh!

slipher Apr 21, 2025

Choose a reason for hiding this comment

Uh oh!

illwieckz May 30, 2025

Choose a reason for hiding this comment

Uh oh!

slipher commented Apr 21, 2025

Uh oh!

slipher commented Apr 22, 2025

Uh oh!

slipher commented Apr 22, 2025

Uh oh!

illwieckz commented May 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

slipher commented May 1, 2025

Uh oh!

slipher commented May 30, 2025

Uh oh!

illwieckz commented Nov 11, 2024 •

edited

Loading

illwieckz commented Nov 11, 2024 •

edited

Loading

illwieckz commented Nov 11, 2024 •

edited

Loading

illwieckz commented Nov 11, 2024 •

edited

Loading

illwieckz commented Nov 11, 2024 •

edited

Loading

illwieckz commented Nov 18, 2024 •

edited

Loading

illwieckz commented May 1, 2025 •

edited

Loading