Skip to content

Commit d52a81b

Browse files
rgommersrgommers
committed
WIP: DOC: add documentation about using shared libraries
1 parent 9faca83 commit d52a81b

File tree

2 files changed

+199
-0
lines changed

2 files changed

+199
-0
lines changed

‎docs/how-to-guides/shared-libraries.rst

+198
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,198 @@
1+
.. SPDX-FileCopyrightText: 2024 The meson-python developers
2+
..
3+
.. SPDX-License-Identifier: MIT
4+
5+
.. _shared-libraries:
6+
7+
**********************
8+
Using shared libraries
9+
**********************
10+
11+
Python projects may build shared libraries as part of their project, or link
12+
with shared libraries from a dependency. This tends to be a common source of
13+
issues, hence this page aims to explain how to include shared libraries in
14+
wheels, any limitations and gotchas, and how support is implemented in
15+
``meson-python`` under the hood.
16+
17+
We distinguish between *internal* shared libraries, i.e. they're built as part
18+
of the build executed by ``meson-python``, and *external* shared libraries that
19+
are only linked against from targets (usually Python extension modules) built
20+
by ``meson-python``. For internal shared libraries, we also distinguish whether
21+
the shared library is being installed to its default location (i.e. ``libdir``,
22+
usually something like ``<prefix>/lib/``) or to a location in ``site-packages``
23+
within the Python package install tree. All these scenarios are (or will be)
24+
supported, with some caveats:
25+
26+
+-----------------------+------------------+---------+-------+-------+
27+
| shared library source | install location | Windows | macOS | Linux |
28+
+=======================+==================+=========+=======+=======+
29+
| internal | libdir | no (1) | ✓ | ✓ |
30+
+-----------------------+------------------+---------+-------+-------+
31+
| internal | site-packages | ✓ | ✓ | ✓ |
32+
+-----------------------+------------------+---------+-------+-------+
33+
| external | n/a | ✓ (2) | ✓ | ✓ |
34+
+-----------------------+------------------+---------+-------+-------+
35+
36+
.. TODO: add subproject as a source
37+
38+
1: Internal shared libraries on Windows cannot be automaticall handled
39+
correctly, and currently ``meson-python`` therefore raises an error for them.
40+
`PR meson-python#551 <https://github.com/mesonbuild/meson-python/pull/551>`__
41+
may improve that situation in the near future.
42+
43+
2: External shared libraries require ``delvewheel`` usage on Windows (or
44+
some equivalent way, like amending the DLL search path to include the directory
45+
in which the external shared library is located). Due to the lack of RPATH
46+
support on Windows, there is no good way around this.
47+
48+
49+
Internal shared libraries
50+
=========================
51+
52+
A shared library produced by ``library()`` or ``shared_library()`` built like this
53+
54+
.. code-block:: meson
55+
56+
example_lib = shared_library(
57+
'example',
58+
'examplelib.c',
59+
install: true,
60+
)
61+
62+
is installed to ``libdir`` by default. If the only reason the shared library exists
63+
is to be used inside the Python package being built, then it is best to modify
64+
the install location to be within the Python package itself:
65+
66+
.. code-block:: python
67+
68+
install_path: py.get_install_dir() / 'mypkg/subdir'
69+
70+
Then an extension module in the same install directory can link against the
71+
shared library in a portable manner by using ``install_rpath``:
72+
73+
.. code-block:: meson
74+
75+
py3.extension_module('_extmodule',
76+
'_extmodule.c',
77+
link_with: example_lib,
78+
install: true,
79+
subdir: 'mypkg/subdir',
80+
install_rpath: '$ORIGIN'
81+
)
82+
83+
The above method will work as advertised on macOS and Linux; ``meson-python`` does
84+
nothing special for this case. On Windows, due to the lack of RPATH support, we
85+
need to preload the shared library on import to make this work by adding this
86+
to ``mypkg/subdir/__init__.py``:
87+
88+
.. code-block:: python
89+
90+
def _load_sharedlib():
91+
"""Load the `example_lib.dll` shared library on Windows
92+
93+
This shared library is installed alongside this __init__.py file. Due to
94+
lack of rpath support, Windows cannot find shared libraries installed
95+
within wheels. So pre-load it.
96+
"""
97+
if os.name == "nt":
98+
import ctypes
99+
try:
100+
from ctypes import WinDLL
101+
basedir = os.path.dirname(__file__)
102+
except:
103+
pass
104+
else:
105+
dll_path = os.path.join(basedir, "example_lib.dll")
106+
if os.path.exists(dll_path):
107+
WinDLL(dll_path)
108+
109+
_load_sharedlib()
110+
111+
If an internal shared library is not only used as part of a Python package, but
112+
for example also as a regular shared library in a C/C++ project or as a
113+
standalone library, then the method shown above won't work - the library has to
114+
be installed to the default ``libdir`` location. In that case, ``meson-python``
115+
will detect that the library is going to be installed to ``libdir`` - which is
116+
not a recommended install location for wheels, and not supported by
117+
``meson-python``. Instead, ``meson-python`` will do the following *on platforms
118+
other than Windows*:
119+
120+
1. Install the shared library to ``<project-name>.mesonpy.libs`` (i.e., a
121+
top-level directory in the wheel, which on install will end up in
122+
``site-packages``).
123+
2. Rewrite RPATH entries for install targets that depend on the shared library
124+
to point to that new install location instead.
125+
126+
This will make the shared library work automatically, with no other action needed
127+
from the package author. *However*, currently an error is raised for this situation
128+
on Windows. This is documented also in :ref:`reference-limitations`.
129+
130+
131+
External shared libraries
132+
=========================
133+
134+
External shared libraries are installed somewhere on the build machine, and
135+
usually detected by a ``dependency()`` or ``compiler.find_library()`` call in a
136+
``meson.build`` file. When a Python extension module or executable uses the
137+
dependency, the shared library will be linked against at build time. On
138+
platforms other than Windows, an RPATH entry is then added to the built
139+
extension modulo or executable, which allows the shared library to be loaded at
140+
runtime.
141+
142+
.. note::
143+
144+
An RPATH entry alone is not always enough - if the directory that the shared
145+
library is located in is not on the loader search path, then it may go
146+
missing at runtime. See, e.g., `meson#2121 <https://github.com/mesonbuild/meson/issues/2121>`__
147+
and `meson#13046 <https://github.com/mesonbuild/meson/issues/13046>`__ for
148+
issues this can cause.
149+
150+
TODO: describe workarounds, e.g. via ``-Wl,-rpath`` or setting ``LD_LIBRARY_PATH``.
151+
152+
On Windows, the shared library can either be preloaded, or vendored with
153+
``delvewheel`` in order to make the built Python package usable locally.
154+
155+
156+
Publishing wheels which depend on external shared libraries
157+
-----------------------------------------------------------
158+
159+
On all platforms, wheels which depend on external shared libraries usually need
160+
post-processing to make them usable on machines other than the one on which
161+
they were built. This is because the RPATH entry for an external shared library
162+
contains a path specific to the build machine. This post-processing is done by
163+
tools like ``auditwheel`` (Linux), ``delvewheel`` (Windows), ``delocate``
164+
(macOS) or ``repair-wheel`` (any platform, wraps the other tools).
165+
166+
Running any of those tools on a wheel produced by ``meson-python`` will vendor
167+
the external shared library into the wheel and rewrite the RPATH entries (it
168+
may also do some other things, like symbol mangling).
169+
170+
On Windows, the package author may also have to add the preloading like shown
171+
above with ``_load_sharedlib()`` to the main ``__init__.py`` of the package,
172+
``delvewheel`` may or may not take care of this (please check its documentation
173+
if your shared library goes missing at runtime).
174+
175+
176+
Using libraries from a Meson subproject
177+
=======================================
178+
179+
TODO
180+
181+
- describe ``--skip-subprojects`` install option and why it's usually needed
182+
- describe how to default to a static library and fold it into an extension module
183+
- write and link to a small example project (also for internal and external
184+
shared libraries; may be a package in ``tests/packages/``)
185+
- what if we really have a ``shared_library()`` in a subproject which can't be
186+
built as a static library?
187+
188+
- this works on all platforms but Windows (for the same reason as internal
189+
shared libraries work on all-but-Windows)
190+
- one then actually has to install the *whole* subproject, which is likely
191+
to include other (unwanted) targets. It's possible to restrict to the
192+
``'runtime'`` install tag, but that may still install for example an
193+
``executable()``.
194+
195+
- mention the more complex case of an external dependency with a subproject as
196+
a fallback
197+
198+

‎docs/index.rst

+1
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,7 @@ the use of ``meson-python`` and Meson for Python packaging.
8282
how-to-guides/config-settings
8383
how-to-guides/meson-args
8484
how-to-guides/debug-builds
85+
how-to-guides/shared-libraries
8586
reference/limitations
8687
projects-using-meson-python
8788

0 commit comments

Comments
 (0)