- Linux
- OSX
- Windows
When multiple wheels depend on each other and share some .so
files, it is possible to rely on auditwheel --exclude
to make sure the .so
is included in only one of the wheels to avoid duplicating the file in every package.
That allows Python wheels to share the same .so
as far as at least one of them is loading them in memory. See pypa/auditwheel#76 and pypa/auditwheel#368 for discussions on the topic.
The problem with this approach is that the package where the .so
is included will have the library name mangled by auditwheel
, while the ones where it's excluded will reference to the library with its original name.
Suppose you have libone.whl
and libtwo.whl
both depending on libfoo.so
, and libone
is a dependency of libtwo
so you already know you can include libfoo.so
only in libone.whl
, you could do:
auditwheel repair dist/libone.whl
auditwheel repair dist/libtwo.whl --exclude libfoo.so
In such case you would end up with the following wheels:
venv/lib/python3.10/site-packages/libone
├── __init__.py
├── _libone.cpython-310-aarch64-linux-gnu.so
venv/lib/python3.10/site-packages/libone.libs
└── libfoo-ef63151d.so
venv/lib/python3.10/site-packages/libtwo
├── __init__.py
├── _libtwo.cpython-310-aarch64-linux-gnu.so
The problem would be that while _libone.cpython-310-aarch64-linux-gnu.so
was patched by auditwheel
to know about the libfoo-ef63151d.so
name, _libtwo.cpython-310-aarch64-linux-gnu.so
was not, and so still refers to the original name:
$ ldd venv/lib/python3.10/site-packages/libone/_libone.cpython-310-aarch64-linux-gnu.so
libfoo-ef63151d.so => ../libone.libs/libfoo-ef63151d.so (0x0000ffff8f8f0000)
$ ldd venv/lib/python3.10/site-packages/libtwo/_libtwo.cpython-310-aarch64-linux-gnu.so
libfoo.so => not found
Which means that trying to import libtwo
will fail with:
ImportError: libfoo.so: cannot open shared object file: No such file or directory
Which makes sense, because we actually provided libfoo-ef63151d.so
and not libfoo.so
.
To solve this problem consolidatewheels
will patch all provided wheels to make sure that they share a single naming convention for libraries that were mangled.
After consolidatewheels
is used, the final result would be:
$ ldd venv/lib/python3.10/site-packages/libone/_libone.cpython-310-aarch64-linux-gnu.so
libfoo-ef63151d.so => ../libone.libs/libfoo-ef63151d.so (0x0000ffff8f8f0000)
$ ldd venv/lib/python3.10/site-packages/libtwo/_libtwo.cpython-310-aarch64-linux-gnu.so
libfoo-ef63151d.so => not found
which would work correctly as far as libone
is imported _before libtwo
as they will both look for libfoo-ef63151d.so
which was loaded already by libone
.
consolidatewheels
works also in conjunction with auditwheel
, consolidating all libraries embedded by auditwheel
. But on Linux duplicates won't be removed automatically, so you need to make sure to use auditwheel --exclude
to ensure libraries are not embedded twice.
consolidatewheels
works also in conjunction with delocate
, consolidating all libraries embedded by delocate
and removing duplicates of the embedded libraries when they are provided in multiple wheels.
consolidatewheels
works also in conjunction with delvewheel
, consolidating all libraries embedded by delvewheel
and removing duplicates of the embedded libraries when they are provided in multiple wheels.
Install with:
$ pip install consolidatewheels
Note that consolidatewheels
requires patchelf
to be available in the system, and it only works on Linux
systems. But those are the same requirements that auditwheel
has, so you are probably already satisfying them if you use auditwheel
.
Usage instructions:
consolidatewheels --help
Example:
consolidatewheels libone.whl libtwo.whl --dest=./consolidated_wheels
For a more complex example and a testing environment, you can take a look at https://github.com/amol-/wheeldeps which uses consolidatewheels