Namespacing shared libraries for python to avoid name clashes

[jonaslb]

Is your feature request related to a problem? Please describe.

eckitlib (python) bundles certain libraries like libldap in its eckitlib.libs folder. This is great under the assumption that no other libraries are doing the same.

Unfortunately, psycopg2-binary (driver for postgres databases) is doing the same thing, with what seems to be a considerable overlap in library names.

Here's a small MRE that causes a segfault:

docker run --rm -it python:3.13 sh -c 'pip install cfgrib psycopg2-binary; python3 -c "import gribapi;im port psycopg2"'

There's a uv script approach as an alternative MRE:

#!/usr/bin/env -S uv run

# /// script
# requires-python = ">=3.11"
# dependencies = [
#   "psycopg2-binary==2.9.11",
#   "cfgrib==0.9.15.1",
# ]
# ///

import gribapi
import psycopg2

print("gribapi module:", getattr(gribapi, "__version__", "unknown"))
print("psycopg2 version:", psycopg2.__version__)
print("about to exit normally, but watch for invalid pointer or segfault messages")

This script seems to give free(): invalid pointer when run with my system python but always segfault in docker.

Describe the solution you'd like

Some ideas (ai-assisted):

• build vendored libs with private SONAMEs (e.g. libldap_eckit.so), and link own extension modules against that SONAME.
• Or, build them as static libraries and link them into a single private .so that doesn’t leak generic symbols.
  • Use symbol visibility controls / version scripts so that: nothing like ldap_initialize, SSL_CTX_new, etc. is exported globally from the wheel’s private copy.

I know this is probably a very significant effort. However, it seems a real issue at the moment that different python "stacks" are growing that are fundamentally incompatible due to "claiming" the same shared library names in an incompatible manner.

Describe alternatives you've considered

No response

Additional context

I've posted this on psycopg2 issue tracker as well. Possibly there will be some insight there: psycopg/psycopg2#1824

Organisation

No response

[Ozaq]

@jonaslb thanks for raising this issue, this is certainly nothing we can "just" fix.

As suggested in the linked issue your best bet is compiling from source vs. system dependencies.

crgrib depends on eccodes which depends on eccodeslib. You would need to compile eccodes from source and the use environment variables as described here - findlibs doc - to force eccodes to stop searching for the shared objects in site-packages.

This should be FINDLIBS_DISABLE_PACKAGE=yes and ECCODES_HOME=<path-to-your-eccodes-build-folder> (expects lib|lib64/libeccodes.so at this path)

Let me know If you have any questions / issues if you are setting this up.

[jonaslb]

Thanks for proposing a workaround, although as far as workarounds go, the simplest in this particular case is just to ensure psycopg2 is imported first, as it seems (possibly) to have a newer library to load.

I hope this issue can remain open, because I strongly feel it "should" be as simple as uv add ec-libraries (or another project tool) and it'd "just work", without requiring any custom fiddling with compilation arguments and prefixes and whatnot. Especially since eccodes is usually not a direct dependency, but somewhere down a dependency chain. Maybe on a slightly longer timescale I (or someone else) could find/think of a concrete way forward to reach a good solution.