Unable to build due to multiple errors

[DNikolaevAtRocket]

Thanks for the port.

The pre-built version seems to work fine; but when we try to build it, it fails with multiple errors:

• thread-local storage is not supported for the current target
• use of undeclared identifier 'mlock' / 'munlock'
• use of undeclared identifier '__uint128_t'

$ zopen build -vv

...
[  3%] Building CXX object third_party/zstd/CMakeFiles/duckdb_zstd.dir/compress/zstd_ldm.cpp.o
cd /u/xxxxxxx/duckdbport/duckdb/src/optimizer/compressed_materialization && /usr/lpp/IBM/cnw/v2r1/openxl/bin/ibm-clang++64 -x c++ -DDUCKDB -DDUCKDB_BUILD_LIBRARY -DDUCKDB_EXTENSION_CORE_FUNCTIONS_LINKED=1 -DDUCKDB_EXTENSION_PARQUET_LINKED=1 -DDUCKDB_MAIN_LIBRARY -D_SILENCE_ALL_MS_EXT_DEPRECATION_WARNINGS=1 -I/u/xxxxxxx/duckdbport/duckdb/src/include -I/u/xxxxxxx/duckdbport/duckdb/third_party/fsst -I/u/xxxxxxx/duckdbport/duckdb/third_party/fmt/include -I/u/xxxxxxx/duckdbport/duckdb/third_party/hyperloglog -I/u/xxxxxxx/duckdbport/duckdb/third_party/fastpforlib -I/u/xxxxxxx/duckdbport/duckdb/third_party/skiplist -I/u/xxxxxxx/duckdbport/duckdb/third_party/ska_sort -I/u/xxxxxxx/duckdbport/duckdb/third_party/fast_float -I/u/xxxxxxx/duckdbport/duckdb/third_party/re2 -I/u/xxxxxxx/duckdbport/duckdb/third_party/miniz -I/u/xxxxxxx/duckdbport/duckdb/third_party/utf8proc/include -I/u/xxxxxxx/duckdbport/duckdb/third_party/concurrentqueue -I/u/xxxxxxx/duckdbport/duckdb/third_party/pcg -I/u/xxxxxxx/duckdbport/duckdb/third_party/pdqsort -I/u/xxxxxxx/duckdbport/duckdb/third_party/tdigest -I/u/xxxxxxx/duckdbport/duckdb/third_party/mbedtls/include -I/u/xxxxxxx/duckdbport/duckdb/third_party/jaro_winkler -I/u/xxxxxxx/duckdbport/duckdb/third_party/vergesort -I/u/xxxxxxx/duckdbport/duckdb/third_party/yyjson/include -I/u/xxxxxxx/duckdbport/duckdb/third_party/zstd/include -I/u/xxxxxxx/duckdbport/duckdb/extension -I/u/xxxxxxx/duckdbport/duckdb/extension/core_functions/include -I/u/xxxxxxx/duckdbport/duckdb/extension/parquet/include -DNSIG=42 -D_XOPEN_SOURCE=600 -D_ALL_SOURCE -D_OPEN_SYS_FILE_EXT=1 -D_AE_BIMODAL=1 -D_ENHANCED_ASCII_EXT=0xFFFFFFFF  -DZOSLIB_OVERRIDE_CLIB=1 -DZOSLIB_OVERRIDE_CLIB_GETENV=1 -DZOSLIB_OVERRIDE_CLIB_LOCALE_FORCE=0  -fzos-le-char-mode=ascii -mnocsect -fno-short-enums -m64 -mzos-target=zosv2r5 -march=z13  -O3 	-I/u/xxxxxxx/.local/zopen/usr/local/zopen/curl/curl-8.15.0.20250728_192752.zos/include	-I/u/xxxxxxx/.local/zopen/usr/local/zopen/libpsl/libpsl-master.20250811_092633.zos/include -I/u/xxxxxxx/.local/zopen/usr/local/zopen/libssh2/libssh2-1.11.0.20240103_144102.zos/include -I/u/xxxxxxx/.local/zopen/usr/local/zopen/openssl/openssl-3.5.2.20250806_162036.zos/include -I/u/xxxxxxx/.local/zopen/usr/local/zopen/ncurses/ncurses-6.5.20250819_155443.zos/include -I/u/xxxxxxx/.local/zopen/usr/local/zopen/ncurses/ncurses-6.5.20250819_155443.zos/include/ncurses -isystem /u/xxxxxxx/.local/zopen/usr/local/zopen/zoslib/zoslib-zopen2.20250825_142938.zos/include -include /u/xxxxxxx/.local/zopen/usr/local/zopen/zoslib/zoslib-zopen2.20250825_142938.zos/include/zos-v2r5-symbolfixes.h -O3 -DNDEBUG -O3 -DNDEBUG   -std=c++11 -fPIC -o CMakeFiles/duckdb_optimizer_compressed_materialization.dir/ub_duckdb_optimizer_compressed_materialization.cpp.o -c /u/xxxxxxx/duckdbport/duckdb/src/optimizer/compressed_materialization/ub_duckdb_optimizer_compressed_materialization.cpp
cd /u/xxxxxxx/duckdbport/duckdb/third_party/zstd && /usr/lpp/IBM/cnw/v2r1/openxl/bin/ibm-clang++64 -x c++ -DDUCKDB_BUILD_LIBRARY -D_SILENCE_ALL_MS_EXT_DEPRECATION_WARNINGS=1 -I/u/xxxxxxx/duckdbport/duckdb/src/include -I/u/xxxxxxx/duckdbport/duckdb/third_party/fsst -I/u/xxxxxxx/duckdbport/duckdb/third_party/fmt/include -I/u/xxxxxxx/duckdbport/duckdb/third_party/hyperloglog -I/u/xxxxxxx/duckdbport/duckdb/third_party/fastpforlib -I/u/xxxxxxx/duckdbport/duckdb/third_party/skiplist -I/u/xxxxxxx/duckdbport/duckdb/third_party/ska_sort -I/u/xxxxxxx/duckdbport/duckdb/third_party/fast_float -I/u/xxxxxxx/duckdbport/duckdb/third_party/re2 -I/u/xxxxxxx/duckdbport/duckdb/third_party/miniz -I/u/xxxxxxx/duckdbport/duckdb/third_party/utf8proc/include -I/u/xxxxxxx/duckdbport/duckdb/third_party/concurrentqueue -I/u/xxxxxxx/duckdbport/duckdb/third_party/pcg -I/u/xxxxxxx/duckdbport/duckdb/third_party/pdqsort -I/u/xxxxxxx/duckdbport/duckdb/third_party/tdigest -I/u/xxxxxxx/duckdbport/duckdb/third_party/mbedtls/include -I/u/xxxxxxx/duckdbport/duckdb/third_party/jaro_winkler -I/u/xxxxxxx/duckdbport/duckdb/third_party/vergesort -I/u/xxxxxxx/duckdbport/duckdb/third_party/yyjson/include -I/u/xxxxxxx/duckdbport/duckdb/third_party/zstd/include -DNSIG=42 -D_XOPEN_SOURCE=600 -D_ALL_SOURCE -D_OPEN_SYS_FILE_EXT=1 -D_AE_BIMODAL=1 -D_ENHANCED_ASCII_EXT=0xFFFFFFFF  -DZOSLIB_OVERRIDE_CLIB=1 -DZOSLIB_OVERRIDE_CLIB_GETENV=1 -DZOSLIB_OVERRIDE_CLIB_LOCALE_FORCE=0  -fzos-le-char-mode=ascii -mnocsect -fno-short-enums -m64 -mzos-target=zosv2r5 -march=z13  -O3 	-I/u/xxxxxxx/.local/zopen/usr/local/zopen/curl/curl-8.15.0.20250728_192752.zos/include	-I/u/xxxxxxx/.local/zopen/usr/local/zopen/libpsl/libpsl-master.20250811_092633.zos/include -I/u/xxxxxxx/.local/zopen/usr/local/zopen/libssh2/libssh2-1.11.0.20240103_144102.zos/include -I/u/xxxxxxx/.local/zopen/usr/local/zopen/openssl/openssl-3.5.2.20250806_162036.zos/include -I/u/xxxxxxx/.local/zopen/usr/local/zopen/ncurses/ncurses-6.5.20250819_155443.zos/include -I/u/xxxxxxx/.local/zopen/usr/local/zopen/ncurses/ncurses-6.5.20250819_155443.zos/include/ncurses -isystem /u/xxxxxxx/.local/zopen/usr/local/zopen/zoslib/zoslib-zopen2.20250825_142938.zos/include -include /u/xxxxxxx/.local/zopen/usr/local/zopen/zoslib/zoslib-zopen2.20250825_142938.zos/include/zos-v2r5-symbolfixes.h -O3 -DNDEBUG -O3 -DNDEBUG   -std=c++11 -fPIC -w -o CMakeFiles/duckdb_zstd.dir/compress/zstd_ldm.cpp.o -c /u/xxxxxxx/duckdbport/duckdb/third_party/zstd/compress/zstd_ldm.cpp
In file included from /u/xxxxxxx/duckdbport/duckdb/src/optimizer/ub_duckdb_optimizer.cpp:22:
In file included from /u/xxxxxxx/duckdbport/duckdb/src/optimizer/regex_range_filter.cpp:14:
In file included from /u/xxxxxxx/duckdbport/duckdb/src/include/duckdb/function/scalar/regexp.hpp:12:
/u/xxxxxxx/duckdbport/duckdb/third_party/re2/re2/re2.h:1002:8: error: thread-local storage is not supported for the current target
 1002 | extern thread_local const RE2* context;
      |        ^

...

[ 21%] Building CXX object src/common/adbc/nanoarrow/CMakeFiles/duckdb_adbc_nanoarrow.dir/ub_duckdb_adbc_nanoarrow.cpp.o
cd /u/xxxxxxx/duckdbport/duckdb/src/common/adbc/nanoarrow && /usr/lpp/IBM/cnw/v2r1/openxl/bin/ibm-clang++64 -x c++ -DDUCKDB -DDUCKDB_BUILD_LIBRARY -DDUCKDB_EXTENSION_CORE_FUNCTIONS_LINKED=1 -DDUCKDB_EXTENSION_PARQUET_LINKED=1 -DDUCKDB_MAIN_LIBRARY -D_SILENCE_ALL_MS_EXT_DEPRECATION_WARNINGS=1 -I/u/xxxxxxx/duckdbport/duckdb/src/include -I/u/xxxxxxx/duckdbport/duckdb/third_party/fsst -I/u/xxxxxxx/duckdbport/duckdb/third_party/fmt/include -I/u/xxxxxxx/duckdbport/duckdb/third_party/hyperloglog -I/u/xxxxxxx/duckdbport/duckdb/third_party/fastpforlib -I/u/xxxxxxx/duckdbport/duckdb/third_party/skiplist -I/u/xxxxxxx/duckdbport/duckdb/third_party/ska_sort -I/u/xxxxxxx/duckdbport/duckdb/third_party/fast_float -I/u/xxxxxxx/duckdbport/duckdb/third_party/re2 -I/u/xxxxxxx/duckdbport/duckdb/third_party/miniz -I/u/xxxxxxx/duckdbport/duckdb/third_party/utf8proc/include -I/u/xxxxxxx/duckdbport/duckdb/third_party/concurrentqueue -I/u/xxxxxxx/duckdbport/duckdb/third_party/pcg -I/u/xxxxxxx/duckdbport/duckdb/third_party/pdqsort -I/u/xxxxxxx/duckdbport/duckdb/third_party/tdigest -I/u/xxxxxxx/duckdbport/duckdb/third_party/mbedtls/include -I/u/xxxxxxx/duckdbport/duckdb/third_party/jaro_winkler -I/u/xxxxxxx/duckdbport/duckdb/third_party/vergesort -I/u/xxxxxxx/duckdbport/duckdb/third_party/yyjson/include -I/u/xxxxxxx/duckdbport/duckdb/third_party/zstd/include -I/u/xxxxxxx/duckdbport/duckdb/extension -I/u/xxxxxxx/duckdbport/duckdb/extension/core_functions/include -I/u/xxxxxxx/duckdbport/duckdb/extension/parquet/include -DNSIG=42 -D_XOPEN_SOURCE=600 -D_ALL_SOURCE -D_OPEN_SYS_FILE_EXT=1 -D_AE_BIMODAL=1 -D_ENHANCED_ASCII_EXT=0xFFFFFFFF  -DZOSLIB_OVERRIDE_CLIB=1 -DZOSLIB_OVERRIDE_CLIB_GETENV=1 -DZOSLIB_OVERRIDE_CLIB_LOCALE_FORCE=0  -fzos-le-char-mode=ascii -mnocsect -fno-short-enums -m64 -mzos-target=zosv2r5 -march=z13  -O3 	-I/u/xxxxxxx/.local/zopen/usr/local/zopen/curl/curl-8.15.0.20250728_192752.zos/include	-I/u/xxxxxxx/.local/zopen/usr/local/zopen/libpsl/libpsl-master.20250811_092633.zos/include -I/u/xxxxxxx/.local/zopen/usr/local/zopen/libssh2/libssh2-1.11.0.20240103_144102.zos/include -I/u/xxxxxxx/.local/zopen/usr/local/zopen/openssl/openssl-3.5.2.20250806_162036.zos/include -I/u/xxxxxxx/.local/zopen/usr/local/zopen/ncurses/ncurses-6.5.20250819_155443.zos/include -I/u/xxxxxxx/.local/zopen/usr/local/zopen/ncurses/ncurses-6.5.20250819_155443.zos/include/ncurses -isystem /u/xxxxxxx/.local/zopen/usr/local/zopen/zoslib/zoslib-zopen2.20250825_142938.zos/include -include /u/xxxxxxx/.local/zopen/usr/local/zopen/zoslib/zoslib-zopen2.20250825_142938.zos/include/zos-v2r5-symbolfixes.h -O3 -DNDEBUG -O3 -DNDEBUG   -std=c++11 -fPIC -o CMakeFiles/duckdb_adbc_nanoarrow.dir/ub_duckdb_adbc_nanoarrow.cpp.o -c /u/xxxxxxx/duckdbport/duckdb/src/common/adbc/nanoarrow/ub_duckdb_adbc_nanoarrow.cpp
In file included from /u/xxxxxxx/duckdbport/duckdb/src/common/ub_duckdb_common.cpp:13:
/u/xxxxxxx/duckdbport/duckdb/src/common/encryption_key_manager.cpp:35:2: error: use of undeclared identifier 'mlock'; did you mean 'Block'?
   35 |         mlock(key, key_len);
      |         ^
/u/xxxxxxx/duckdbport/duckdb/src/include/duckdb/storage/block.hpp:20:7: note: 'Block' declared here
   20 | class Block : public FileBuffer {
      |       ^
In file included from /u/xxxxxxx/duckdbport/duckdb/src/common/ub_duckdb_common.cpp:13:
/u/xxxxxxx/duckdbport/duckdb/src/common/encryption_key_manager.cpp:44:2: error: use of undeclared identifier 'munlock'
   44 |         munlock(key, key_len);
      |         ^
make[2]: Leaving directory '/u/xxxxxxx/duckdbport/duckdb'

...

[ 23%] Built target duckdb_common_enums
make  -f src/common/progress_bar/CMakeFiles/duckdb_progress_bar.dir/build.make src/common/progress_bar/CMakeFiles/duckdb_progress_bar.dir/depend
In file included from /u/xxxxxxx/duckdbport/duckdb/src/common/operator/ub_duckdb_common_operators.cpp:2:
In file included from /u/xxxxxxx/duckdbport/duckdb/src/common/operator/cast_operators.cpp:24:
/u/xxxxxxx/duckdbport/duckdb/third_party/fast_float/fast_float/fast_float.h:252:3: error: unknown type name '__uint128_t'
  252 |   __uint128_t r = ((__uint128_t)a) * b;
      |   ^
make[2]: Entering directory '/u/xxxxxxx/duckdbport/duckdb'
cd /u/xxxxxxx/duckdbport/duckdb && /u/xxxxxxx/.local/zopen/usr/local/zopen/cmake/cmake-DEV.20250402_145954.zos/bin/cmake -E cmake_depends "Unix Makefiles" /u/xxxxxxx/duckdbport/duckdb /u/xxxxxxx/duckdbport/duckdb/src/common/progress_bar /u/xxxxxxx/duckdbport/duckdb /u/xxxxxxx/duckdbport/duckdb/src/common/progress_bar /u/xxxxxxx/duckdbport/duckdb/src/common/progress_bar/CMakeFiles/duckdb_progress_bar.dir/DependInfo.cmake "--color="
/u/xxxxxxx/duckdbport/duckdb/third_party/fast_float/fast_float/fast_float.h:252:21: error: use of undeclared identifier '__uint128_t'
  252 |   __uint128_t r = ((__uint128_t)a) * b;
      |                     ^
Dependee "/u/xxxxxxx/duckdbport/duckdb/src/common/progress_bar/CMakeFiles/duckdb_progress_bar.dir/DependInfo.cmake" is newer than depender "/u/xxxxxxx/duckdbport/duckdb/src/common/progress_bar/CMakeFiles/duckdb_progress_bar.dir/depend.internal".
Dependee "/u/xxxxxxx/duckdbport/duckdb/src/common/progress_bar/CMakeFiles/CMakeDirectoryInformation.cmake" is newer than depender "/u/xxxxxxx/duckdbport/duckdb/src/common/progress_bar/CMakeFiles/duckdb_progress_bar.dir/depend.internal".
Scanning dependencies of target duckdb_progress_bar
make[2]: Leaving directory '/u/xxxxxxx/duckdbport/duckdb'
make  -f src/common/progress_bar/CMakeFiles/duckdb_progress_bar.dir/build.make src/common/progress_bar/CMakeFiles/duckdb_progress_bar.dir/build
make[2]: Entering directory '/u/xxxxxxx/duckdbport/duckdb'

...

Not sure what we are doing wrong.
Could you please help us to fix these?

[IgorTodorovskiIBM]

@v1gnesh have you come across this?

One difference I can spot is the compiler. We use the OEL clang (https://epwt-www.mybluemix.net/software/support/trial/cst/programwebsite.wss?siteId=1803) vs ibm-clang++, however it probably shouldn't make a difference in this case.

[DNikolaevAtRocket]

@IgorTodorovskiIBM,

I've tried the OEL clang, now it finds __uint128_t. Apparently Open XL C/C++ doesn't support the extension.
The other two are still there.

Thanks.

[v1gnesh]

@IgorTodorovskiIBM - don't think I've seen these.

mlock, munlock - these have had upgrades via newly added primitives in z/OS 3.1 and 3.2 - could be worth revisiting what's available now from the docs.

thread-local - I vaguely remember seeing a thread-local related TechDocs, or something in 3.2 notes.

Apart from that, yes, DuckDB has changed a lot since the latest available zopen release.
This'll need looking into..

[v1gnesh]

Found it - PH62468: NEW FUNCTION - Additional Support for C/C++ Thread Local Storage.

In addition to the above note (on 3.1 or 3.2 adding primitives), zopen (via zoslib) itself got some upgrades - ibmruntimes/zoslib#58.

Depending on how important this is, it is possible to revive this work 🤔

[DNikolaevAtRocket]

@v1gnesh,

Thanks a lot for the input. I didn't realize zopen build doesn't use a locked DuckDB version but pulls the default branch.

I was able to build it with these two tiny patches:

• mlock / munlock: I don't see anything that can do the trick, and added empty stubs for z/OS.
  We don't need the code path anyway, so swappable encryption keys is not a security issue for us.
  In case you have time to figure it out and point out the direction, we appreciate it.

• thread_local: added z/OS to the "unsupported" list (joining iOS and MinGW).
  Making TLS work is interesting in general, but not sure if it's worth it here.

Regarding importance - well, it's a part of a research project at the moment, we'll see how it goes.
I think we have all we need now, we may need official patches later. If you accept pull requests, we can provide them too.

[IgorTodorovskiIBM]

Regarding importance - well, it's a part of a research project at the moment, we'll see how it goes. I think we have all we need now, we may need official patches later. If you accept pull requests, we can provide them too.

Yes, you can submit a PR in this repo and we can review it. Thank you!

[v1gnesh]

mlock and munlock aren't there but there are workarounds for it.
https://github.com/zopencommunity/prometheusport/blob/dc6d2bdfe9c84852774d1f3e0ed474abf989725a/patches/wharf_patch#L561
https://github.com/zopencommunity/opensslport/blob/76454eff779e4e46aaa89da2b783f9fbe5e608fb/hardware_patches/0008-007_zos.patch#L84

These are invocations from Go, but the source at /usr/lpp/IBM/cvg/go/src/cmd/vendor/golang.org/x/sys/unix/syscall_zos_s390x.go shows what to use instead... __mlockall().

[DNikolaevAtRocket]

I hesitated to use __mlockall() since making the entire address space unswappable is not quite the same...
And it's out of question for our project.

I will use it in our PR to the repo though since apparently it's a common approach.
Thanks.

[DNikolaevAtRocket]

Hi @v1gnesh,

There were another couple of issues I had to fix (common for all big-endian platforms): see https://duckdb.org/docs/stable/dev/building/unofficial_and_unsupported_platforms#big-endian-architectures

I've noticed that you did a big deal of porting work a couple of years ago (thanks for that!), and it was merged upstream.
Probably I'll submit a PR there and ask you for review if that's okay.

[v1gnesh]

Hi @DNikolaevAtRocket,

Yup, they don't support it officially.
However, it is now possible to give them access to a z/OS system, for them to also test/verify if they want to, in addition to them accepting PRs (based on contributor testing results).

At the time, I think we ran through the test suite and it was mostly passing (didn't record stats).
@IgorTodorovskiIBM also helped with testing some basic SQL (which worked).

Didn't try long enough to deliberately try to surface endianness issues.

Yes, sure, go for it, thank you!
Even from my PRs, I think a portion of that work could be undone, because some of that functionality is now available in zoslib (removing the setting of constants in favour of adding zoslib to the build dependencies).

[v1gnesh]

duckdb/duckdb#19641 Nice one, @DNikolaevAtRocket. Thank you!

Are any updates required to the buildenv ?

[DNikolaevAtRocket]

Hey @v1gnesh,

Yes, I'd add flags to build some extra in-tree extensions (at least autocomplete, json, icu), and we need -fvisibility=default to make the DLL work.

Also, there's big endian support to finish and merge it upstream.
With my current patches, it passes 96% of the tests. Taking care of the rest.

[DNikolaevAtRocket]

An update:

Have submitted the PR to fix the major big-endian issues.
When / if it's merged, I'll make another one to address the rest.

[DNikolaevAtRocket]

The patch is merged, but new Block Allocator feature breaks compilation on z/OS:

In file included from /u/tsdxn/db_engines/duckdbport/duckdb/src/storage/ub_duckdb_storage.cpp:8:
/u/tsdxn/db_engines/duckdbport/duckdb/src/storage/block_allocator.cpp:30:77: error: use of undeclared identifier 'MAP_ANONYMOUS'
   30 |         const auto ptr = mmap(nullptr, size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
      |                                                                                    ^
/u/tsdxn/db_engines/duckdbport/duckdb/src/storage/block_allocator.cpp:71:35: error: use of undeclared identifier 'MADV_DONTNEED'
   71 |         success = madvise(pointer, size, MADV_DONTNEED) == 0;
      |                                          ^
/u/tsdxn/db_engines/duckdbport/duckdb/src/storage/block_allocator.cpp:190:2: error: thread-local storage is not supported for the current target
  190 |         thread_local BlockAllocatorThreadLocalState local_state(block_allocator);
      |         ^
/u/tsdxn/db_engines/duckdbport/duckdb/src/storage/block_allocator.cpp:192:9: warning: reference to stack memory associated with local variable 'local_state' returned [-Wreturn-stack-address]
  192 |         return local_state;
      |                ^~~~~~~~~~~

[DNikolaevAtRocket]

The last compilation issue has been taken care of: duckdb/duckdb#19977