Skip to content

highs_bindings defining PYBIND11_DETAILED_ERROR_MESSAGES breaks ODR for other sources using pybind #3144

Description

@NoLuckNoWin

Hi!

I encountered a crash during import in our repo that uses HiGHS vendored in SciPy. Crash happens during import.

The problem is that HiGHS defines PYBIND11_DETAILED_ERROR_MESSAGES unconditionally in highs_bindings.cpp: https://github.com/ERGO-Code/HiGHS/blob/master/highspy/highs_bindings.cpp#L1

  • With define enabled, pybind11::arg_v adds additional type member: https://github.com/pybind/pybind11/blob/master/include/pybind11/cast.h#L2041.
  • When multiple sources are built with this defined enabled and disabled we get ODR violation since pybind::arg_v::~arg_v has 2 differing implementations (one destroys type member, one does not).
  • This leads to crash when in resulting executable constructor creates "smaller" version without type (since this implementation was selected) but destructor tries to destroy it and crashes trying to free some garbage memory.

Unfortunately, I am not allowed to give full example for this issue we encountered, but i have smaller reproducer: https://github.com/NoLuckNoWin/highs-odr-repro.
It builds HiGHS + pypocketfft (I originally encountered this crash trying to import it) into one shared library and then tries to import it. pypocketfft uses pybind11 and is built without PYBIND11_DETAILED_ERROR_MESSAGES.
Stacktrace from repro:

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/nix/store/8kvxvr3pmsypxiypq4g8zy13glnfr7nx-glibc-2.42-67/lib/libthread_db.so.1".
free(): invalid pointer

Program received signal SIGABRT, Aborted.
0x00007ffff749fdcc in __pthread_kill_implementation () from /nix/store/8kvxvr3pmsypxiypq4g8zy13glnfr7nx-glibc-2.42-67/lib/libc.so.6
#0  0x00007ffff749fdcc in __pthread_kill_implementation () from /nix/store/8kvxvr3pmsypxiypq4g8zy13glnfr7nx-glibc-2.42-67/lib/libc.so.6
#1  0x00007ffff744265e in raise () from /nix/store/8kvxvr3pmsypxiypq4g8zy13glnfr7nx-glibc-2.42-67/lib/libc.so.6
#2  0x00007ffff7429350 in abort () from /nix/store/8kvxvr3pmsypxiypq4g8zy13glnfr7nx-glibc-2.42-67/lib/libc.so.6
#3  0x00007ffff742a3c8 in __libc_message_impl.cold () from /nix/store/8kvxvr3pmsypxiypq4g8zy13glnfr7nx-glibc-2.42-67/lib/libc.so.6
#4  0x00007ffff74aaa15 in malloc_printerr () from /nix/store/8kvxvr3pmsypxiypq4g8zy13glnfr7nx-glibc-2.42-67/lib/libc.so.6
#5  0x00007ffff74aaa38 in malloc_printerr_tail () from /nix/store/8kvxvr3pmsypxiypq4g8zy13glnfr7nx-glibc-2.42-67/lib/libc.so.6
#6  0x00007ffff661a275 in std::__new_allocator<char>::deallocate (this=0x7fffffffb480, __p=0x7ffff6cb6f6a "out", __n=140737333915499) at /nix/store/sq0nrnfwhkc5ljvklnrk5ps4358g4nbj-gcc-15.2.0/include/c++/15.2.0/bits/new_allocator.h:172
#7  0x00007ffff661a236 in std::allocator_traits<std::allocator<char> >::deallocate (__a=..., __p=0x7ffff6cb6f6a "out", __n=140737333915499) at /nix/store/sq0nrnfwhkc5ljvklnrk5ps4358g4nbj-gcc-15.2.0/include/c++/15.2.0/bits/alloc_traits.h:649
#8  std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_destroy (this=0x7fffffffb480, __size=140737333915498) at /nix/store/sq0nrnfwhkc5ljvklnrk5ps4358g4nbj-gcc-15.2.0/include/c++/15.2.0/bits/basic_string.h:305
#9  0x00007ffff661a0aa in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_dispose (this=0x7fffffffb480) at /nix/store/sq0nrnfwhkc5ljvklnrk5ps4358g4nbj-gcc-15.2.0/include/c++/15.2.0/bits/basic_string.h:299
#10 0x00007ffff6614489 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string (this=0x7fffffffb480) at /nix/store/sq0nrnfwhkc5ljvklnrk5ps4358g4nbj-gcc-15.2.0/include/c++/15.2.0/bits/basic_string.h:896
#11 0x00007ffff66703cd in pybind11::arg_v::~arg_v (this=0x7fffffffb460) at /nix/store/8q6fpv5zjwcvd9d94rdjjkzvwij0saz5-python3.13-pybind11-3.0.4/include/pybind11/cast.h:1992
#12 0x00007ffff67fb9d2 in pybind11_init_pypocketfft (m=...) at /nix/store/jk3pi2j31y197wrczjh80imcsrdhlijj-source/scipy/fft/_pocketfft/pypocketfft.cxx:767
#13 0x00007ffff67fac07 in pybind11_exec_pypocketfft (pm=0x7ffff73ded40) at /nix/store/jk3pi2j31y197wrczjh80imcsrdhlijj-source/scipy/fft/_pocketfft/pypocketfft.cxx:760
#14 0x00007ffff7a225ee in PyModule_ExecDef () from /nix/store/l9k0anq0z7zz81zcwy035jfwap9ga6rl-python3-3.13.13/lib/libpython3.13.so.1.0
#15 0x00007ffff7a2299d in _imp_exec_dynamic () from /nix/store/l9k0anq0z7zz81zcwy035jfwap9ga6rl-python3-3.13.13/lib/libpython3.13.so.1.0
#16 0x00007ffff7937893 in cfunction_vectorcall_O () from /nix/store/l9k0anq0z7zz81zcwy035jfwap9ga6rl-python3-3.13.13/lib/libpython3.13.so.1.0
#17 0x00007ffff7bd9f34 in _PyEval_EvalFrameDefault () from /nix/store/l9k0anq0z7zz81zcwy035jfwap9ga6rl-python3-3.13.13/lib/libpython3.13.so.1.0
#18 0x00007ffff7a517ef in object_vacall () from /nix/store/l9k0anq0z7zz81zcwy035jfwap9ga6rl-python3-3.13.13/lib/libpython3.13.so.1.0
#19 0x00007ffff7a51a5b in PyObject_CallMethodObjArgs () from /nix/store/l9k0anq0z7zz81zcwy035jfwap9ga6rl-python3-3.13.13/lib/libpython3.13.so.1.0
#20 0x00007ffff7b47e7c in PyImport_ImportModuleLevelObject () from /nix/store/l9k0anq0z7zz81zcwy035jfwap9ga6rl-python3-3.13.13/lib/libpython3.13.so.1.0
#21 0x00007ffff7be5bd1 in _PyEval_EvalFrameDefault () from /nix/store/l9k0anq0z7zz81zcwy035jfwap9ga6rl-python3-3.13.13/lib/libpython3.13.so.1.0
#22 0x00007ffff7bebf73 in PyEval_EvalCode () from /nix/store/l9k0anq0z7zz81zcwy035jfwap9ga6rl-python3-3.13.13/lib/libpython3.13.so.1.0
#23 0x00007ffff7bf305b in run_eval_code_obj () from /nix/store/l9k0anq0z7zz81zcwy035jfwap9ga6rl-python3-3.13.13/lib/libpython3.13.so.1.0
#24 0x00007ffff7bf3277 in run_mod () from /nix/store/l9k0anq0z7zz81zcwy035jfwap9ga6rl-python3-3.13.13/lib/libpython3.13.so.1.0
#25 0x00007ffff7bf6cfc in Py_RunMain () from /nix/store/l9k0anq0z7zz81zcwy035jfwap9ga6rl-python3-3.13.13/lib/libpython3.13.so.1.0
#26 0x00007ffff742b285 in __libc_start_call_main () from /nix/store/8kvxvr3pmsypxiypq4g8zy13glnfr7nx-glibc-2.42-67/lib/libc.so.6
#27 0x00007ffff742b338 in __libc_start_main_impl () from /nix/store/8kvxvr3pmsypxiypq4g8zy13glnfr7nx-glibc-2.42-67/lib/libc.so.6
#28 0x0000555555555085 in _start ()

As you can see, pypocketfft.cxx crashes while trying to free its std::string in pybind11::arg_v::~arg_v even though it was compiled without this define (ODR).

This define was added in #1447

Obvious fix would be removing this define (it is enabled in debug build). Could you, please, help me, can it be removed from default build?

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions