pypa / auditwheel Goto Github PK
View Code? Open in Web Editor NEWAuditing and relabeling cross-distribution Linux wheels.
License: Other
Auditing and relabeling cross-distribution Linux wheels.
License: Other
To help in situations like #28, it would be nice if auditwheel could check for impure files in purelib and complain if they're found.
I used the manylinux docker infrastructure to build:
http://travis-wheels.scikit-image.org/netCDF4-1.2.3.1-cp34-cp34m-manylinux1_x86_64.whl
Upon import, I see:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/stefan/envs/py3/lib/python3.4/site-packages/netCDF4/__init__.py", line 3, in <module>
from ._netCDF4 import *
ImportError: libhdf5_hl-9c7ba457.so.10.0.2: cannot open shared object file: No such file or directory
Looking at the vendored libraries, the RPATH is invalid for libnetcdf (not for libhdf5)--possibly has something to do with original libnetcdf library already having an RPATH entry.
Build recipe can be found here:
https://github.com/stefanv/manylinux-builds/blob/build_netcdf/build_netcdfs.sh
It was run on yesterday's version of the manylinux docker image.
After building some python-kadmin wheels on the manylinux docker image, I encountered an exception when running auditwheel on the wheels:
$ auditwheel -vv repair wheelhouse/python_kadmin-0.1.1-cp26-cp26m-linux_x86_64.whl -w /io/wheelhouse
...<snip>
Traceback (most recent call last):
File "/usr/local/bin/auditwheel", line 11, in <module>
sys.exit(main())
File "/opt/_internal/cpython-3.5.1/lib/python3.5/site-packages/auditwheel/main.py", line 49, in main
rval = args.func(args, p)
File "/opt/_internal/cpython-3.5.1/lib/python3.5/site-packages/auditwheel/main_repair.py", line 60, in execute
wheel_abi = analyze_wheel_abi(args.WHEEL_FILE)
File "/opt/_internal/cpython-3.5.1/lib/python3.5/site-packages/auditwheel/wheel_abi.py", line 59, in analyze_wheel_abi
get_wheel_elfdata(wheel_fn)
File "/opt/_internal/cpython-3.5.1/lib/python3.5/site-packages/auditwheel/wheel_abi.py", line 48, in get_wheel_elfdata
max_versioned_symbol(versioned_symbols), uses_ucs2_symbols)
File "/opt/_internal/cpython-3.5.1/lib/python3.5/site-packages/auditwheel/policy/versioned_symbols.py", line 22, in max_versioned_symbol
set_if_greater(max_required_ver, name, Version(ver))
File "/opt/_internal/cpython-3.5.1/lib/python3.5/site-packages/pip/_vendor/distlib/version.py", line 32, in __init__
self._parts = parts = self.parse(s)
File "/opt/_internal/cpython-3.5.1/lib/python3.5/site-packages/pip/_vendor/distlib/version.py", line 273, in parse
result = _normalized_key(s)
File "/opt/_internal/cpython-3.5.1/lib/python3.5/site-packages/pip/_vendor/distlib/version.py", line 194, in _pep_440_key
raise UnsupportedVersionError('Not a valid version: %s' % s)
pip._vendor.distlib.version.UnsupportedVersionError: Not a valid version: MIT
Taking a look at the version symbols inside krb5, we find:
$ readelf -V ./lib/krb5/libkrb5.so
...<snip>
Version definition section '.gnu.version_d' contains 3 entries:
Addr: 0x000000000000bcc0 Offset: 0x00bcc0 Link: 4 (.dynstr) 000000: Rev: 1 Flags: BASE Index: 1 Cnt: 1 Name: libkrb5.so.3
0x001c: Rev: 1 Flags: none Index: 2 Cnt: 1 Name: krb5_3_MIT
0x0038: Rev: 1 Flags: none Index: 3 Cnt: 1 Name: HIDDEN
Version needs section '.gnu.version_r' contains 5 entries:
Addr: 0x000000000000bd18 Offset: 0x00bd18 Link: 4 (.dynstr)
000000: Version: 1 File: libcom_err.so.3 Cnt: 1
0x0010: Name: com_err_3_MIT Flags: none Version: 10
0x0020: Version: 1 File: libresolv.so.2 Cnt: 2
0x0030: Name: GLIBC_2.2.5 Flags: none Version: 13
0x0040: Name: GLIBC_2.9 Flags: none Version: 9
0x0050: Version: 1 File: libk5crypto.so.3 Cnt: 1
0x0060: Name: k5crypto_3_MIT Flags: none Version: 7
0x0070: Version: 1 File: libkrb5support.so.0 Cnt: 1
0x0080: Name: krb5support_0_MIT Flags: none Version: 6
0x0090: Version: 1 File: libc.so.6 Cnt: 7
0x00a0: Name: GLIBC_2.14 Flags: none Version: 15
0x00b0: Name: GLIBC_2.8 Flags: none Version: 14
0x00c0: Name: GLIBC_2.16 Flags: none Version: 12
0x00d0: Name: GLIBC_2.4 Flags: none Version: 11
0x00e0: Name: GLIBC_2.3.4 Flags: none Version: 8
0x00f0: Name: GLIBC_2.2.5 Flags: none Version: 5
0x0100: Name: GLIBC_2.3 Flags: none Version: 4
So I believe the issue here is that we are attempting to determine the max symbol version on all included version symbols, even though we only care about the ones in policy.json (GLIBC, CXXABI, etc.). While there are conventions for the numeric versioning for the GLIBC etc. symbols, this is not generally true, which is why we're running into the issue with e.g. k5crypto_3_MIT having a version read as MIT. A fix would be to filter out any symbols we don't care about per policy before running max_versioned_symbol
on them?
To implement pypa/manylinux#31:
--disable-shared
python and/or uses patchelf --remove-needed
.But you should never have a wheel with a name like:: PKG-VERSION-cp27-none-manylinux1_x86_64.whl ^^^^ BAD! Don't do this!
auditwheel uses patchelf to rename vendored libraries with unique names. This works wonderfully until something links against a library with the same symbol names and loads them before we get there. This scenario occurs when using uwsgi
with cryptography
's new manylinux1 wheels. Since uwsgi
links against libssl/libcrypto itself we're seeing segfaults where it's calling between two different versions of the library (the one uwsgi
loads globally and the one cryptography
has vendored). You can see more discussion (and a better explanation from @njsmith) here: pyca/cryptography#3804 (comment)
One potential solution that was put forth would be to make it so patchelf is capable of name mangling individual symbols so that even in a scenario like the one above there's no issue with namespace collision. This sounds very challenging but as more and more manylinux1 wheels start being shipped it may be the right long term solution.
I was hoping to get a full blown example using the setup.py
that built
cffi-1.5.0-cp35-cp35m-linux_x86_64.whl
I'm not sure how to include the .so
files as dependencies when building the initial wheel i.e:
{
"libc.so.6": "/lib64/libc-2.5.so",
"libffi.so.5": "/usr/lib64/libffi.so.5.0.6",
"libpthread.so.0": "/lib64/libpthread-2.5.so"
}
Thanks, Mike.
Running auditwheel show on a wheel of https://github.com/anntzer/mpl_cairo compiled with gcc7.1 (which is definitely not manylinux1-compatible, I was just doing some tests) gives the following traceback:
Traceback (most recent call last):
File "/home/antony/.local/bin/auditwheel", line 11, in <module>
sys.exit(main())
File "/home/antony/.local/lib/python3.6/site-packages/auditwheel/main.py", line 49, in main
rval = args.func(args, p)
File "/home/antony/.local/lib/python3.6/site-packages/auditwheel/main_show.py", line 28, in execute
winfo = analyze_wheel_abi(args.WHEEL_FILE)
File "/home/antony/.local/lib/python3.6/site-packages/auditwheel/wheel_abi.py", line 73, in analyze_wheel_abi
get_wheel_elfdata(wheel_fn)
File "/home/antony/.local/lib/python3.6/site-packages/auditwheel/wheel_abi.py", line 56, in get_wheel_elfdata
ctx.path)
File "/home/antony/.local/lib/python3.6/site-packages/auditwheel/policy/external_references.py", line 53, in lddtree_external_references
whitelist) # type: List[str]
File "/home/antony/.local/lib/python3.6/site-packages/auditwheel/policy/external_references.py", line 39, in get_req_external
whitelist) for lib in libs), libs)
File "/home/antony/.local/lib/python3.6/site-packages/auditwheel/policy/external_references.py", line 39, in <genexpr>
whitelist) for lib in libs), libs)
<elided>
File "/home/antony/.local/lib/python3.6/site-packages/auditwheel/policy/external_references.py", line 39, in <genexpr>
whitelist) for lib in libs), libs)
RecursionError: maximum recursion depth exceeded while calling a Python object
(Python 3.6.1 ArchLinux, auditwheel from pypi).
First of all I think this initiative is amazing and really hope this will make portable Linux binaries trivial.
I've had some experience with this on the company I work for. Basically we came up with the same solution, except that we compile under a CentOS 5 chroot.
But one of the problems we faced was libstdc++. We also use gcc 4.8.5, which means the libstdc++ version is 3.4.19.
CentOS default libstdc++ is 3.4.12, therefore if you are compiling C++ code using this approach, you might have to run the executable only on CentOS 7 and earlier.
So the most obvious solution would be to just ship libstdc++
. This is what we did, but then we got missing symbols on recent Linux distributions because they have a libstdc++ newer than 3.4.19, and some crucial system libraries (specially OpenGL implementations) were linked against it.
Not the most beautiful solution, but we ended up having to create an initialization script that would check which version is newer (the shipped or system one), and set LD_LIBRARY_PATH to the shipped one if necessary.
I know that most of base libraries are written in C, but even though I think this is an important issue, specially if someone is using OpenGL (I know that at least Mesa and Nvidia proprietary implementation use C++).
So have anyone else thought about this problem?
When compiling a manylinux1 wheel against OpenSSL 1.1.0 the following exception is raised:
File "/usr/local/bin/auditwheel", line 11, in <module>
sys.exit(main())
File "/opt/_internal/cpython-3.6.0/lib/python3.6/site-packages/auditwheel/main.py", line 49, in main
rval = args.func(args, p)
File "/opt/_internal/cpython-3.6.0/lib/python3.6/site-packages/auditwheel/main_repair.py", line 60, in execute
wheel_abi = analyze_wheel_abi(args.WHEEL_FILE)
File "/opt/_internal/cpython-3.6.0/lib/python3.6/site-packages/auditwheel/wheel_abi.py", line 82, in analyze_wheel_abi
symbol_policy = versioned_symbols_policy(versioned_symbols)
File "/opt/_internal/cpython-3.6.0/lib/python3.6/site-packages/auditwheel/policy/versioned_symbols.py", line 28, in versioned_symbols_policy
sym_name, _ = symbol.split("_", 2)
ValueError: too many values to unpack (expected 2)
Looking in PDB the issue is that the symbol
is expected to look something like GLIBC_2.2.5
, but OpenSSL 1.1.0 is OPENSSL_1_1_0
, which results in too many values.
For Py < 3.2, the manylinux1 policy requires building against UCS4 Python, so we have to check for this in extensions modules by looking through the symbol table for the use of any PyUnicodeUCS2_*
functions, and give a helpful error message.
$pip show auditwheel
Name: auditwheel
Version: 1.5.0
Summary: Cross-distribution Linux wheels
Home-page: https://github.com/pypa/auditwheel
Author: Robert T. McGibbon
Author-email: [email protected]
License: MIT
Location: /usr/local/lib/python2.7/dist-packages
Requires: setuptools, wheel, typing, pyelftools
$auditwheel
Traceback (most recent call last):
File "/usr/local/bin/auditwheel", line 6, in <module>
from auditwheel.main import main
File "/usr/local/lib/python2.7/dist-packages/auditwheel/main.py", line 8, in <module>
from . import main_addtag
File "/usr/local/lib/python2.7/dist-packages/auditwheel/main_addtag.py", line 3, in <module>
from .policy import (load_policies, get_policy_name, get_priority_by_name,
File "/usr/local/lib/python2.7/dist-packages/auditwheel/policy/__init__.py", line 26
print('Error: This tool only supports Linux', file=sys.stderr)
^
SyntaxError: invalid syntax
$uname -a
Linux A10092495U 4.4.0-57-generic #78-Ubuntu SMP Fri Dec 9 23:50:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Working on creating wheels for VTK, I found out that libXt.so.6
wasn't specified in policy.json and was copied in .libs
directory.
Would it make sense to add it ?
What could I do to check that is it reasonable to add this library to the whitelist ? Is it mater of having a consistent API/API across all versions of linux we support ?
I receive the following error:
ValueError: Cannot repair wheel, because required library "libarb.so.2" could not be located
However, I know where this library is located. How can I point auditwheel to that location?
auditwheel
has a command named addtag
to rename a wheel file to insert the manylinux1_x86_64
platform tag into its filename (in addition to the existing linux_x86_64
) and similarly for manylinux1_i686
/ linux_i686
.
This is an issue because uploading wheel files with the linux_x86_64
/ linux_i686
platform tags to warehouse or PyPI is forbidden (while manylinux1_x86_64
/ manylinux1_i686
are authorized or will soon be).
Instead auditwheel repair
should probably replace the linux
based tags by manylinux1
tags whenever the repair
operation succeeds.
CFFI extensions compiled for PyPy do not export 'init' + modname
nor 'PyInit_' + modname
symbols but instead they export '_cffi_pypyinit_' + modname
and thus they are skipped by auditwheel
. However actually auditwheel
can repair them. As an example one may take a look at python-snappy
- it uses cffi for PyPy. And proof for using _cffi_pypyinit_
prefix may be found at cffi/recompiler.py
.
The tests are full of the following traceback
Traceback (most recent call last):
File "/home/travis/virtualenv/python3.4.3/bin/auditwheel", line 10, in <module>
sys.exit(main())
File "/home/travis/virtualenv/python3.4.3/lib/python3.4/site-packages/auditwheel/main.py", line 49, in main
rval = args.func(args, p)
File "/home/travis/virtualenv/python3.4.3/lib/python3.4/site-packages/auditwheel/main_repair.py", line 81, in execute
update_tags=args.UPDATE_TAGS)
File "/home/travis/virtualenv/python3.4.3/lib/python3.4/site-packages/auditwheel/repair.py", line 81, in repair_wheel
new_soname, new_path = copylib(src_path, dest_dir)
File "/home/travis/virtualenv/python3.4.3/lib/python3.4/site-packages/auditwheel/repair.py", line 132, in copylib
verify_patchelf()
File "/home/travis/virtualenv/python3.4.3/lib/python3.4/functools.py", line 472, in wrapper
result = user_function(*args, **kwds)
File "/home/travis/virtualenv/python3.4.3/lib/python3.4/site-packages/auditwheel/repair.py", line 37, in verify_patchelf
version)
ValueError: patchelf found. auditwheel repair requirespatchelf >= 0.9.
Interestingly, the auditwheel tests are not failing, despite the tracebacks thrown.
The problem is that patchelf --version
doesn't output anything on version 0.9. I've heard @geofft has submitted a patch to fix this upstream, and I've tested & confirmed the patch fixes the bug.
The PEP 513 whitelist includes libpanelw.so.5
and libncursesw.so.5
, both of which are part of ncurses version 5.
But apparently the new hostness is ncurses 6, which breaks ABI with ncurses 5. I'm told that Fedora 24 and successors have already switched to ncurses 6, with the ncurses 5 libraries being available in a compatibility package, but not installed by default. So... I guess these have to be dropped from the whitelist.
Is it possible to have a configurable policy document, perhaps a command line option to specify a different policy document that would bundle only the libraries specified in this user-specified policy doc.
Use case:
For organizations with self-hosted PyPI repositories, the process for bundling shared libraries can conform to a standard set by the organization itself. For instance, for orgs that run Alpine Linux, auditwheel repair
can omit shared libraries common to Alpine Linux, e.g. musl c.
By allowing a user to specify their own policy for determining what to dependencies bundle they can reduce wheel sizes.
I'm sorry not to have done a better job of debugging this but ..
I'm trying to build h5py wheels.
I think I correctly built and repaired the wheel, and it installs and tests OK on a Debian sid machine I have.
However, when I test the wheel on the docker image itself, I get:
python -c 'import h5py; h5py.run_tests()'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/opt/3.4.4/lib/python3.4/site-packages/h5py/__init__.py", line 24, in <module>
from . import _errors
ImportError: libsz.so.2: cannot open shared object file: No such file or directory
Script to generate hdf5 and libsz libraries: https://gist.github.com/9279c5c368d61118cf1c
Script to build h5py wheel : https://gist.github.com/matthew-brett/0dbd3747cbd76ea8ca49
Unfixed wheel : http://nipy.bic.berkeley.edu/scipy_installers/tmp/unfixed/h5py-2.4.0-cp34-cp34m-linux_x86_64.whl
Fixed wheel : http://nipy.bic.berkeley.edu/scipy_installers/tmp/fixed/h5py-2.4.0-cp34-cp34m-linux_x86_64.whl
Script to run wheel test : https://gist.github.com/41eaad61fbf02072b2ba
libsz.so.2
is in the .libs
directory of the h5py install, and rpath for the _errors
extension does contain $ORIGIN/.libs
.
I don't know why, auditwheel does not graft the shared library dependencies to my manylinux wheel.
Below is the screen output.
[root@d52c3a246527 dist]# auditwheel -v show ecell-4.1.0-cp35-cp35m-linux_x86_64.whl
INFO:auditwheel.wheel_abi:processing: ecell4/bd.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.lddtree:linker ldpaths: {'conf': ['/lib', '/lib64/', '/usr/lib', '/usr/lib64'], 'env': ['/opt/rh/devtoolset-2/root/usr/lib64', '/opt/rh/devtoolset-2/root/usr/lib', '/usr/local/lib64', '/usr/local/lib'], 'interp': []}
INFO:auditwheel.wheel_abi:processing: ecell4/core.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.wheel_abi:processing: ecell4/egfrd.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.wheel_abi:processing: ecell4/gillespie.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.wheel_abi:processing: ecell4/meso.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.wheel_abi:processing: ecell4/ode.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.wheel_abi:processing: ecell4/spatiocyte.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.wheel_abi:{
"linux_x86_64": {
"libs": {
"libm.so.6": "/lib64/libm-2.5.so",
"libgcc_s.so.1": "/lib64/libgcc_s-4.1.2-20080825.so.1",
"libhdf5_cpp.so.11": "/usr/local/lib/libhdf5_cpp.so.11.0.0",
"libc.so.6": "/lib64/libc-2.5.so",
"libhdf5.so.10": "/usr/local/lib/libhdf5.so.10.1.0",
"librt.so.1": "/lib64/librt-2.5.so",
"libgsl.so.0": "/usr/lib64/libgsl.so.0.14.0",
"libpthread.so.0": "/lib64/libpthread-2.5.so",
"libgslcblas.so.0": "/usr/lib64/libgslcblas.so.0.0.0",
"libdl.so.2": "/lib64/libdl-2.5.so",
"libstdc++.so.6": "/usr/lib64/libstdc++.so.6.0.8"
},
"priority": 0
},
"manylinux1_x86_64": {
"libs": {
"libgsl.so.0": "/usr/lib64/libgsl.so.0.14.0",
"libhdf5_cpp.so.11": "/usr/local/lib/libhdf5_cpp.so.11.0.0",
"libgslcblas.so.0": "/usr/lib64/libgslcblas.so.0.0.0",
"libhdf5.so.10": "/usr/local/lib/libhdf5.so.10.1.0"
},
"priority": 100
}
}
ecell-4.1.0-cp35-cp35m-linux_x86_64.whl is consistent with the
following platform tag: "linux_x86_64".
The wheel references external versioned symbols in these system-
provided shared libraries: libgcc_s.so.1 with versions {'GCC_3.0'},
libpthread.so.0 with versions {'GLIBC_2.2.5'}, libc.so.6 with versions
{'GLIBC_2.2.5', 'GLIBC_2.4', 'GLIBC_2.3.4'}, libm.so.6 with versions
{'GLIBC_2.2.5'}, libstdc++.so.6 with versions {'GLIBCXX_3.4',
'CXXABI_1.3.1', 'CXXABI_1.3'}
The following external shared libraries are required by the wheel:
{
"libc.so.6": "/lib64/libc-2.5.so",
"libdl.so.2": "/lib64/libdl-2.5.so",
"libgcc_s.so.1": "/lib64/libgcc_s-4.1.2-20080825.so.1",
"libgsl.so.0": "/usr/lib64/libgsl.so.0.14.0",
"libgslcblas.so.0": "/usr/lib64/libgslcblas.so.0.0.0",
"libhdf5.so.10": "/usr/local/lib/libhdf5.so.10.1.0",
"libhdf5_cpp.so.11": "/usr/local/lib/libhdf5_cpp.so.11.0.0",
"libm.so.6": "/lib64/libm-2.5.so",
"libpthread.so.0": "/lib64/libpthread-2.5.so",
"librt.so.1": "/lib64/librt-2.5.so",
"libstdc++.so.6": "/usr/lib64/libstdc++.so.6.0.8"
}
In order to achieve the tag platform tag "manylinux1_x86_64" the
following shared library dependencies will need to be eliminated:
libgsl.so.0, libgslcblas.so.0, libhdf5.so.10, libhdf5_cpp.so.11
[root@d52c3a246527 dist]# auditwheel -v repair ecell-4.1.0-cp35-cp35m-linux_x86_64.whl
Repairing ecell-4.1.0-cp35-cp35m-linux_x86_64.whl
INFO:auditwheel.wheel_abi:processing: ecell4/bd.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.lddtree:linker ldpaths: {'conf': ['/lib', '/lib64/', '/usr/lib', '/usr/lib64'], 'env': ['/opt/rh/devtoolset-2/root/usr/lib64', '/opt/rh/devtoolset-2/root/usr/lib', '/usr/local/lib64', '/usr/local/lib'], 'interp': []}
INFO:auditwheel.wheel_abi:processing: ecell4/core.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.wheel_abi:processing: ecell4/egfrd.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.wheel_abi:processing: ecell4/gillespie.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.wheel_abi:processing: ecell4/meso.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.wheel_abi:processing: ecell4/ode.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.wheel_abi:processing: ecell4/spatiocyte.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.wheel_abi:{
"linux_x86_64": {
"libs": {
"libhdf5.so.10": "/usr/local/lib/libhdf5.so.10.1.0",
"libpthread.so.0": "/lib64/libpthread-2.5.so",
"libc.so.6": "/lib64/libc-2.5.so",
"libstdc++.so.6": "/usr/lib64/libstdc++.so.6.0.8",
"libgsl.so.0": "/usr/lib64/libgsl.so.0.14.0",
"libm.so.6": "/lib64/libm-2.5.so",
"libdl.so.2": "/lib64/libdl-2.5.so",
"librt.so.1": "/lib64/librt-2.5.so",
"libhdf5_cpp.so.11": "/usr/local/lib/libhdf5_cpp.so.11.0.0",
"libgcc_s.so.1": "/lib64/libgcc_s-4.1.2-20080825.so.1",
"libgslcblas.so.0": "/usr/lib64/libgslcblas.so.0.0.0"
},
"priority": 0
},
"manylinux1_x86_64": {
"libs": {
"libhdf5_cpp.so.11": "/usr/local/lib/libhdf5_cpp.so.11.0.0",
"libgsl.so.0": "/usr/lib64/libgsl.so.0.14.0",
"libgslcblas.so.0": "/usr/lib64/libgslcblas.so.0.0.0",
"libhdf5.so.10": "/usr/local/lib/libhdf5.so.10.1.0"
},
"priority": 100
}
}
Grafting: /usr/local/lib/libhdf5_cpp.so.11.0.0 -> ecell4/.libs/libhdf5_cpp-44aff6ad.so.11.0.0
Setting RPATH: ecell4/.libs/libhdf5_cpp-44aff6ad.so.11.0.0 to "$ORIGIN/."
Grafting: /usr/lib64/libgsl.so.0.14.0 -> ecell4/.libs/libgsl-b885fd68.so.0.14.0
Grafting: /usr/lib64/libgslcblas.so.0.0.0 -> ecell4/.libs/libgslcblas-15b2d0fe.so.0.0.0
Grafting: /usr/local/lib/libhdf5.so.10.1.0 -> ecell4/.libs/libhdf5-669db1ee.so.10.1.0
Setting RPATH: ecell4/bd.cpython-35m-x86_64-linux-gnu.so to "$ORIGIN/.libs"
Setting RPATH: ecell4/core.cpython-35m-x86_64-linux-gnu.so to "$ORIGIN/.libs"
Setting RPATH: ecell4/egfrd.cpython-35m-x86_64-linux-gnu.so to "$ORIGIN/.libs"
Setting RPATH: ecell4/gillespie.cpython-35m-x86_64-linux-gnu.so to "$ORIGIN/.libs"
Setting RPATH: ecell4/meso.cpython-35m-x86_64-linux-gnu.so to "$ORIGIN/.libs"
Setting RPATH: ecell4/ode.cpython-35m-x86_64-linux-gnu.so to "$ORIGIN/.libs"
Setting RPATH: ecell4/spatiocyte.cpython-35m-x86_64-linux-gnu.so to "$ORIGIN/.libs"
Previous filename tags: linux_x86_64
New filename tags: manylinux1_x86_64
Previous WHEEL info tags: cp35-cp35m-linux_x86_64
New WHEEL info tags: cp35-cp35m-manylinux1_x86_64
Fixed-up wheel written to /root/ecell4/python/dist/wheelhouse/ecell-4.1.0-cp35-cp35m-manylinux1_x86_64.whl
[root@d52c3a246527 dist]# auditwheel -v show wheelhouse/ecell-4.1.0-cp35-cp35m-manylinux1_x86_64.whl
INFO:auditwheel.wheel_abi:processing: ecell4/core.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.lddtree:linker ldpaths: {'conf': ['/lib', '/lib64/', '/usr/lib', '/usr/lib64'], 'env': ['/opt/rh/devtoolset-2/root/usr/lib64', '/opt/rh/devtoolset-2/root/usr/lib', '/usr/local/lib64', '/usr/local/lib'], 'interp': []}
INFO:auditwheel.wheel_abi:processing: ecell4/gillespie.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.wheel_abi:processing: ecell4/bd.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.wheel_abi:processing: ecell4/ode.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.wheel_abi:processing: ecell4/spatiocyte.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.wheel_abi:processing: ecell4/meso.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.wheel_abi:processing: ecell4/egfrd.cpython-35m-x86_64-linux-gnu.so
INFO:auditwheel.wheel_abi:{
"linux_x86_64": {
"libs": {
"libstdc++.so.6": "/usr/lib64/libstdc++.so.6.0.8",
"libgcc_s.so.1": "/lib64/libgcc_s-4.1.2-20080825.so.1",
"libpthread.so.0": "/lib64/libpthread-2.5.so",
"libm.so.6": "/lib64/libm-2.5.so",
"libdl.so.2": "/lib64/libdl-2.5.so",
"librt.so.1": "/lib64/librt-2.5.so",
"libc.so.6": "/lib64/libc-2.5.so"
},
"priority": 0
},
"manylinux1_x86_64": {
"libs": {},
"priority": 100
}
}
ecell-4.1.0-cp35-cp35m-manylinux1_x86_64.whl is consistent with the
following platform tag: "manylinux1_x86_64".
The wheel references external versioned symbols in these system-
provided shared libraries: libgcc_s.so.1 with versions {'GCC_3.0'},
libpthread.so.0 with versions {'GLIBC_2.2.5'}, libc.so.6 with versions
{'GLIBC_2.2.5', 'GLIBC_2.3.4', 'GLIBC_2.4'}, libm.so.6 with versions
{'GLIBC_2.2.5'}, libstdc++.so.6 with versions {'CXXABI_1.3.1',
'CXXABI_1.3', 'GLIBCXX_3.4'}
The following external shared libraries are required by the wheel:
{
"libc.so.6": "/lib64/libc-2.5.so",
"libdl.so.2": "/lib64/libdl-2.5.so",
"libgcc_s.so.1": "/lib64/libgcc_s-4.1.2-20080825.so.1",
"libm.so.6": "/lib64/libm-2.5.so",
"libpthread.so.0": "/lib64/libpthread-2.5.so",
"librt.so.1": "/lib64/librt-2.5.so",
"libstdc++.so.6": "/usr/lib64/libstdc++.so.6.0.8"
}
In my manylinux wheel, it seems that hdf5 and gsl are not grafted by auditwheel.
I would be grateful if you could give me some clues on this.
This can be reproduced with https://github.com/ecell/ecell4-manylinux/blob/v4.1.0/.travis.yml
I used the latest quay.io/pypa/manylinux1_x86_64
image to build my wheel.
The auditwheel version is
[root@d52c3a246527 ~]# auditwheel -V
auditwheel 1.5.0 installed at /opt/_internal/cpython-3.6.0/lib/python3.6/site-
packages (python 3.6)
Bug originally filed by @lpsinger at pypa/manylinux#156:
If you have a package that contains multiple C extensions that depend on the same shared libraries, then auditwheel repair
saves duplicate copies of all of the shared libraries. An example is shown below.
This makes for packages that are several times larger than necessary. Could auditwheel repair
sweep up all of the dependencies and put them in a single .libs
directory, rather than creating duplicates?
$ tar -tf lalsuite-6.48-cp27-cp27m-manylinux1_x86_64.whl
lalxml/git_version.py
lalxml/_lalxml.so
lalxml/__init__.py
lalxml/lalxml.py
lalxml/.libs/libgsl-b885fd68.so.0.14.0
lalxml/.libs/libhdf5_hl-04af5206.so.100.0.1
lalxml/.libs/libhdf5-db4e48f9.so.101.0.0
lalxml/.libs/liblalsupport-bd9338fc.so.11.0.0
lalxml/.libs/liblalxml-a864f854.so.2.0.1
lalxml/.libs/libxml2-8ed442f7.so.2.6.26
lalxml/.libs/libz-a147dcb0.so.1.2.3
lalxml/.libs/libfftw3-c8dcb815.so.3.2.4
lalxml/.libs/liblal-8b0d5843.so.14.0.0
lalxml/.libs/libgslcblas-15b2d0fe.so.0.0.0
lalxml/.libs/libfftw3f-044fdcc5.so.3.2.4
lalmetaio/git_version.py
lalmetaio/lalmetaio.py
lalmetaio/_lalmetaio.so
lalmetaio/__init__.py
lalmetaio/.libs/libgsl-b885fd68.so.0.14.0
lalmetaio/.libs/libhdf5_hl-04af5206.so.100.0.1
lalmetaio/.libs/libhdf5-db4e48f9.so.101.0.0
lalmetaio/.libs/liblalsupport-bd9338fc.so.11.0.0
lalmetaio/.libs/liblalmetaio-f4e58690.so.5.0.1
lalmetaio/.libs/libz-a147dcb0.so.1.2.3
lalmetaio/.libs/libfftw3-c8dcb815.so.3.2.4
lalmetaio/.libs/liblal-8b0d5843.so.14.0.0
lalmetaio/.libs/libmetaio-c02fec4d.so.1.1.0
lalmetaio/.libs/libgslcblas-15b2d0fe.so.0.0.0
lalmetaio/.libs/libfftw3f-044fdcc5.so.3.2.4
lal/_lal.so
lal/gpstime.py
lal/lal.py
lal/git_version.py
lal/__init__.py
lal/rate.py
lal/series.py
lal/.libs/libgsl-b885fd68.so.0.14.0
lal/.libs/libhdf5_hl-04af5206.so.100.0.1
lal/.libs/libhdf5-db4e48f9.so.101.0.0
lal/.libs/liblalsupport-bd9338fc.so.11.0.0
lal/.libs/libz-a147dcb0.so.1.2.3
lal/.libs/libfftw3-c8dcb815.so.3.2.4
lal/.libs/liblal-8b0d5843.so.14.0.0
lal/.libs/libgslcblas-15b2d0fe.so.0.0.0
lal/.libs/libfftw3f-044fdcc5.so.3.2.4
lal/utils/cache.py
lal/utils/__init__.py
lal/utils/series.py
lal/spectrum/distributions.py
lal/spectrum/averagespectrum.py
lal/spectrum/__init__.py
lalsuite-6.48.dist-info/WHEEL
lalsuite-6.48.dist-info/METADATA
lalsuite-6.48.dist-info/DESCRIPTION.rst
lalsuite-6.48.dist-info/metadata.json
lalsuite-6.48.dist-info/RECORD
lalsuite-6.48.dist-info/top_level.txt
lalsimulation/git_version.py
lalsimulation/__init__.py
lalsimulation/_lalsimulation.so
lalsimulation/lalsimulation.py
lalsimulation/.libs/libgsl-b885fd68.so.0.14.0
lalsimulation/.libs/libhdf5_hl-04af5206.so.100.0.1
lalsimulation/.libs/liblalsimulation-f35d2299.so.16.0.3
lalsimulation/.libs/libhdf5-db4e48f9.so.101.0.0
lalsimulation/.libs/liblalsupport-bd9338fc.so.11.0.0
lalsimulation/.libs/libz-a147dcb0.so.1.2.3
lalsimulation/.libs/libfftw3-c8dcb815.so.3.2.4
lalsimulation/.libs/liblal-8b0d5843.so.14.0.0
lalsimulation/.libs/libgslcblas-15b2d0fe.so.0.0.0
lalsimulation/.libs/libfftw3f-044fdcc5.so.3.2.4
lalframe/git_version.py
lalframe/frread.py
lalframe/__init__.py
lalframe/lalframe.py
lalframe/_lalframe.so
lalframe/.libs/libgsl-b885fd68.so.0.14.0
lalframe/.libs/libhdf5_hl-04af5206.so.100.0.1
lalframe/.libs/libhdf5-db4e48f9.so.101.0.0
lalframe/.libs/liblalsupport-bd9338fc.so.11.0.0
lalframe/.libs/libFrame-0c783110.so.1.5.0
lalframe/.libs/liblalframe-2600de1c.so.10.0.2
lalframe/.libs/libz-a147dcb0.so.1.2.3
lalframe/.libs/libfftw3-c8dcb815.so.3.2.4
lalframe/.libs/liblal-8b0d5843.so.14.0.0
lalframe/.libs/libgslcblas-15b2d0fe.so.0.0.0
lalframe/.libs/libfftw3f-044fdcc5.so.3.2.4
lalframe/utils/frtools.py
lalframe/utils/__init__.py
I applied auditwheel to a project named E-Cell.
But there was NameError for this.
If you have any suggestions in this regard, please let me know.
[root@db4ffc7bd536 python]# auditwheel show dist/ecell-4.0.0-cp27-cp27mu-linux_x86_64.whl
ecell-4.0.0-cp27-cp27mu-linux_x86_64.whl is consistent with the
following platform tag: "linux_x86_64".
The wheel references the following external versioned symbols in
system-provided shared libraries: CXXABI_1.3.1, GLIBC_2.4,
GLIBCXX_3.4, GCC_3.0.
The following external shared libraries are required by the wheel:
{
"libc.so.6": "/lib64/libc-2.5.so",
"libgcc_s.so.1": "/lib64/libgcc_s-4.1.2-20080825.so.1",
"libgsl.so.0": "/usr/lib64/libgsl.so.0.14.0",
"libgslcblas.so.0": "/usr/lib64/libgslcblas.so.0.0.0",
"libhdf5.so.9": null,
"libhdf5_cpp.so.9": null,
"libm.so.6": "/lib64/libm-2.5.so",
"libpthread.so.0": "/lib64/libpthread-2.5.so",
"libstdc++.so.6": "/usr/lib64/libstdc++.so.6.0.8"
}
In order to achieve the tag platform tag "manylinux1_x86_64" the
following shared library dependencies will need to be eliminated:
libgsl.so.0, libgslcblas.so.0, libhdf5.so.9, libhdf5_cpp.so.9
[root@db4ffc7bd536 python]# auditwheel repair dist/ecell-4.0.0-cp27-cp27mu-linux_x86_64.whl
Repairing ecell-4.0.0-cp27-cp27mu-linux_x86_64.whl
Traceback (most recent call last):
File "/usr/local/bin/auditwheel", line 11, in <module>
sys.exit(main())
File "/opt/_internal/cpython-3.5.1/lib/python3.5/site-packages/auditwheel/main.py", line 49, in main
rval = args.func(args, p)
File "/opt/_internal/cpython-3.5.1/lib/python3.5/site-packages/auditwheel/main_repair.py", line 81, in execute
update_tags=args.UPDATE_TAGS)
File "/opt/_internal/cpython-3.5.1/lib/python3.5/site-packages/auditwheel/repair.py", line 79, in repair_wheel
libname)
NameError: name 'libname' is not defined
... IMHO it should just do nothing in this case.
I experienced this while using the manylinux docker images on Travis here https://travis-ci.org/pombreda/thirdparty-manylinux/jobs/152792412#L697
The auditwheel step fails because a six wheels had been pulled in
Because since that version symbol and section names are strings in elftools and it mentioned in CHANGES.
Here is commit changed behaviour - eliben/pyelftools@108eaea
Auditwheel 1.6 seems to be failing to detect and graft the dependencies of the wheels.
For new failure example see:
https://api.travis-ci.org/jobs/227872764/log.txt?deansi=true
For failure fixed by downgrading auditwheel to 1.5:
https://api.travis-ci.org/jobs/227948047/log.txt?deansi=true
With the downgrade, I see this in the log (second link, just above):
Grafting: /usr/local/lib/libopenblasp-r0.2.18.so -> numpy/.libs/libopenblasp-r0-39a31c03.2.18.so
Grafting: /usr/lib64/libgfortran.so.3.0.0 -> numpy/.libs/libgfortran-ed201abd.so.3.0.0
Setting RPATH: numpy/core/multiarray.so to "$ORIGIN/../.libs"
Setting RPATH: numpy/linalg/_umath_linalg.so to "$ORIGIN/../.libs"
Setting RPATH: numpy/linalg/lapack_lite.so to "$ORIGIN/../.libs"
These are missing in the run with new auditwheel 1.6 (first link above).
Seems to be the same problem for matplotlib:
https://api.travis-ci.org/jobs/227868822/log.txt?deansi=true
Here it looks like the libpng library was not grafted into the matplotlib wheel.
I've been playing with manylinux builds and can't seem to get auditwheel to ever find shared libraries and fix them.
For example, I've tried the following by cloning https://github.com/pypa/python-manylinux-demo
, navigating to its root directory and running
docker run -v `pwd`:/io -it --rm quay.io/pypa/manylinux1_x86_64 /bin/bash
From inside the container, I then build the wheels via
/io/travis/build-wheels.sh
Everything completes successfully, but the auditwheel repair
doesn't seem to have packaged anything up and the wheels have no .libs
directory containing atlas.
From inside the container, running auditwheel -vv show python_manylinux_demo-1.0-cp27-cp27mu-linux_x86_64.whl
gives the following:
DEBUG:auditwheel.wheel_abi:{}
DEBUG:auditwheel.policy.versioned_symbols:Required symbol versions: {}
INFO:auditwheel.wheel_abi:{
"manylinux1_x86_64": {
"priority": 100,
"libs": {}
},
"linux_x86_64": {
"priority": 0,
"libs": {}
}
}
DEBUG:auditwheel.wheel_abi:external referene info
DEBUG:auditwheel.wheel_abi:{
"manylinux1_x86_64": {
"priority": 100,
"libs": {}
},
"linux_x86_64": {
"priority": 0,
"libs": {}
}
}
python_manylinux_demo-1.0-cp27-cp27mu-linux_x86_64.whl is consistent
with the following platform tag: "manylinux1_x86_64".
The wheel references no external versioned symbols from system-
provided shared libraries.
The wheel requires no external shared libraries! :)
but unzipping that wheel and runnning ldd
tells a different story
[root@1965501e43ea pymanylinuxdemo]# ldd extension.so
linux-vdso.so.1 => (0x00007ffc52bd8000)
libcblas.so.3 => /usr/lib64/atlas/libcblas.so.3 (0x00007fdcd467e000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fdcd4461000)
libc.so.6 => /lib64/libc.so.6 (0x00007fdcd4108000)
libatlas.so.3 => /usr/lib64/atlas/libatlas.so.3 (0x00007fdcd3813000)
libm.so.6 => /lib64/libm.so.6 (0x00007fdcd358f000)
/lib64/ld-linux-x86-64.so.2 (0x00005619467b3000)
Any help would be greatly appreciated! I'm experiencing the same thing across the board with other builds I've been trying to manylinuxize.
Update for the weary traveler: You may be using a too old version of patchelf
, try compiling it from source and using that instead.
I'm getting this error when I try to import the module after install my repaired wheel in a fresh VM (or trying to import it directly from the .so on the machine where it was built).
Inconsistency detected by ld.so: dl-version.c: 205: _dl_check_map_versions: Assertion `needed != NULL' failed!
The wheel is build for this package (https://github.com/safijari/pyOpenKarto/blob/python-devel/setup.py) in python2.7. I run auditwheel and get:
(auditwheel_pipenv-zl3561H8) > $ auditwheel repair py_openkarto-0.0.1-cp27-cp27mu-linux_x86_64.whl --plat linux_x86_64
Repairing py_openkarto-0.0.1-cp27-cp27mu-linux_x86_64.whl
Grafting: /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25 -> .libsopenkarto/libstdc++-637c843a.so.6.0.25
Grafting: /usr/lib/x86_64-linux-gnu/libboost_system.so.1.65.1 -> .libsopenkarto/libboost_system-ca31776d.so.1.65.1
Grafting: /lib/x86_64-linux-gnu/libm-2.27.so -> .libsopenkarto/libm-2-24310d2a.27.so
Grafting: /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.65.1 -> .libsopenkarto/libboost_thread-4e4bd896.so.1.65.1
Grafting: /lib/x86_64-linux-gnu/libc-2.27.so -> .libsopenkarto/libc-2-cd7c1a03.27.so
warning: working around a Linux kernel bug by creating a hole of 2101248 bytes in ‘.libsopenkarto/libc-2-cd7c1a03.27.so’
Grafting: /lib/x86_64-linux-gnu/libpthread-2.27.so -> .libsopenkarto/libpthread-2-1032040b.27.so
warning: working around a Linux kernel bug by creating a hole of 2076672 bytes in ‘.libsopenkarto/libpthread-2-1032040b.27.so’
Grafting: /lib/x86_64-linux-gnu/libgcc_s.so.1 -> .libsopenkarto/libgcc_s-b10d5179.so.1
Grafting: /lib/x86_64-linux-gnu/librt-2.27.so -> .libsopenkarto/librt-2-110b6497.27.so
Setting RPATH: openkarto.so to "$ORIGIN/.libsopenkarto"
Previous filename tags: linux_x86_64
No filename tags change needed.
Previous WHEEL info tags: cp27-cp27mu-linux_x86_64
No WHEEL info change needed.
Fixed-up wheel written to /home/jari/Pipenvs/karto/pyOpenKarto/dist/wheelhouse/py_openkarto-0.0.1-cp27-cp27mu-linux_x86_64.whl
Nothing seems wrong with that output as far as I can tell. Pls halp :(
Update: Hmm, I moved all the .so files elsewhere and moved them back one by one, once I moved back this file...
Grafting: /usr/lib/x86_64-linux-gnu/libboost_system.so.1.65.1 -> .libsopenkarto/libboost_system-ca31776d.so.1.65.1
that's when I get the crash... hmm. Unsure if this is to blame as I have another package like this which doesn't use boost and gives the same crash after auditwheel.
Using the latest manylinux image (and so hopefully the latest auditwheel version) I have the problem that repair
does not correctly relink all libraries. The dependency hierarchy is as follows:
parquet.so
is a Python module (Cython generated) that depends on the C++ libraries libpyarrow.so
and libarrow.so
directly.libpyarrow.so
is a pure C++ helper library but also depends on libarrow.so
directly.parquet.so
has the correct links after the repair but libpyarrow.so
does not.
Both libs parquet.so
and libpyarrow.so
are part of the wheel, libarrow.so
is in this case an external one).
Console output:
Repairing pyarrow-0.1.0.dev0-cp27-cp27mu-linux_x86_64.whl
Grafting: /usr/lib/libarrow.so -> pyarrow/./libarrow-17b4bbba.so
Grafting: /usr/lib/libboost_filesystem.so.1.60.0 -> pyarrow/./libboost_filesystem-d4710d70.so.1.60.0
Grafting: /usr/lib/libarrow_parquet.so -> pyarrow/./libarrow_parquet-745cd113.so
Grafting: /usr/lib/libparquet.so -> pyarrow/./libparquet-3ec6b6fd.so
Grafting: /usr/lib/libboost_system.so.1.60.0 -> pyarrow/./libboost_system-17a2ee5a.so.1.60.0
Grafting: /usr/lib/libarrow_io.so -> pyarrow/./libarrow_io-f94b9eae.so
Setting RPATH: pyarrow/schema.so to "$ORIGIN/."
Setting RPATH: pyarrow/scalar.so to "$ORIGIN/."
Setting RPATH: pyarrow/config.so to "$ORIGIN/."
Setting RPATH: pyarrow/table.so to "$ORIGIN/."
Setting RPATH: pyarrow/array.so to "$ORIGIN/."
Setting RPATH: pyarrow/error.so to "$ORIGIN/."
Setting RPATH: pyarrow/io.so to "$ORIGIN/."
Setting RPATH: pyarrow/parquet.so to "$ORIGIN/."
Previous filename tags: linux_x86_64
New filename tags: manylinux1_x86_64
Previous WHEEL info tags: cp27-cp27mu-linux_x86_64
New WHEEL info tags: cp27-cp27mu-manylinux1_x86_64
Library linkages:
% readelf --dynamic pyarrow/parquet.so | grep NEEDED
0x0000000000000001 (NEEDED) Shared library: [libpyarrow.so]
0x0000000000000001 (NEEDED) Shared library: [libarrow-17b4bbba.so]
0x0000000000000001 (NEEDED) Shared library: [libarrow_io-f94b9eae.so]
0x0000000000000001 (NEEDED) Shared library: [libarrow_parquet-745cd113.so]
0x0000000000000001 (NEEDED) Shared library: [libstdc++.so.6]
0x0000000000000001 (NEEDED) Shared library: [libm.so.6]
0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1]
0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
% readelf --dynamic pyarrow/libpyarrow.so | grep NEEDED
0x0000000000000001 (NEEDED) Shared library: [libarrow.so]
0x0000000000000001 (NEEDED) Shared library: [libarrow_io.so]
0x0000000000000001 (NEEDED) Shared library: [libarrow_parquet.so]
0x0000000000000001 (NEEDED) Shared library: [libstdc++.so.6]
0x0000000000000001 (NEEDED) Shared library: [libm.so.6]
0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1]
0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
Happy to fix this on my own but I would need some pointers where I should start to investigate!
Using the manylinux docker image, I get this error message:
auditwheel: error: cannot repair "wheelhouse/pygame-1.9.2.dev1-cp34-cp34m-linux_x86_64.whl" to "manylinux1_x86_64" ABI because of the presence of too-recent versioned symbols. You'll need to compile the wheel on an older toolchain.
It would be helpful to know which symbols it doesn't like!
Auditwheel is a great tool and is used by a lot of common libraries to build their wheel packages. I'm most interested in numpy and they mentioned that they use this library to package their wheels. Unfortunately somewhere in their build process the binaries are modified by patchelf such that strip doesn't work.
Is it possible to add the option to strip binaries directly within the auditwheel process?
Stripped binaries are critical for deployment, particularly of lambda functions and I'm unable to do the strip myself once installed.
A lot of people would benefit from this change in terms of saved space in deployment.
In turbodbc
we link against pyarrow
in such a fashion that turbodbc
picks up the Arrow libraries via a relative RPATH in the same virtualenv. Thus we don't need to include them in the manylinux1
wheel of Turbodbc. With auditwheel's current behaviour they are sadly grafted into turbodbc's wheel. To fix this, two options come to my mind:
I'm happy to implement one of these approaches but I would like to hear what the preferred one would be.
Is there a specific reason why auditwheel
does not support Python 2.7 (besides it requiring more work to make the code Python2/3 portable)?
I got a simple error from a print call in main_addtag.py, line 49.
Using auditwheel
on a wheel that:
Linux dhcp-7-187 4.9.0-1-amd64 #1 SMP Debian 4.9.6-3 (2017-01-28) x86_64 GNU/Linux
)the output is:
auditwheel show dist/dd-0.5.2.dev0+f9ea13bad6ef42cccbe0df9dafd66bea265a490d-cp35-cp35m-linux_x86_64.whl
dd-0.5.2.dev0+f9ea13bad6ef42cccbe0df9dafd66bea265a490d-cp35-cp35m-linu
x_x86_64.whl is consistent with the following platform tag:
"manylinux1_x86_64".
The wheel references external versioned symbols in these system-
provided shared libraries: libc.so.6 with versions {'GLIBC_2.14',
'GLIBC_2.7', 'GLIBC_2.2.5'}
The following external shared libraries are required by the wheel:
{
"libc.so.6": "/lib/x86_64-linux-gnu/libc-2.24.so",
"libpthread.so.0": "/lib/x86_64-linux-gnu/libpthread-2.24.so"
}
I am confused by the fact that GLIBC 2.14 and 2.7 are listed, but PEP 513 requires that GLIBC <= 2.5
(which agrees with policy.json
). Looking closer at the extension module:
ldd -r -v build/lib.linux-x86_64-3.5/dd/cudd.cpython-35m-x86_64-linux-gnu.so
linux-vdso.so.1 (0x00007ffe1328e000)
libpython3.5m.so.1.0 => /some/path/python35/lib/libpython3.5m.so.1.0 (0x00007f3d50ed5000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f3d50c75000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3d508d7000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f3d506d3000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f3d504d0000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f3d501ca000)
/lib64/ld-linux-x86-64.so.2 (0x00005580a0e8a000)
Version information:
build/lib.linux-x86_64-3.5/dd/cudd.cpython-35m-x86_64-linux-gnu.so:
libc.so.6 (GLIBC_2.14) => /lib/x86_64-linux-gnu/libc.so.6
libc.so.6 (GLIBC_2.7) => /lib/x86_64-linux-gnu/libc.so.6
libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
/some/path/python35/lib/libpython3.5m.so.1.0:
libdl.so.2 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libdl.so.2
libutil.so.1 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libutil.so.1
libm.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libm.so.6
libpthread.so.0 (GLIBC_2.3.2) => /lib/x86_64-linux-gnu/libpthread.so.0
libpthread.so.0 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libpthread.so.0
libc.so.6 (GLIBC_2.7) => /lib/x86_64-linux-gnu/libc.so.6
libc.so.6 (GLIBC_2.14) => /lib/x86_64-linux-gnu/libc.so.6
libc.so.6 (GLIBC_2.3.4) => /lib/x86_64-linux-gnu/libc.so.6
libc.so.6 (GLIBC_2.9) => /lib/x86_64-linux-gnu/libc.so.6
libc.so.6 (GLIBC_2.17) => /lib/x86_64-linux-gnu/libc.so.6
libc.so.6 (GLIBC_2.6) => /lib/x86_64-linux-gnu/libc.so.6
libc.so.6 (GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.6
libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
libc.so.6 (GLIBC_2.3) => /lib/x86_64-linux-gnu/libc.so.6
/lib/x86_64-linux-gnu/libpthread.so.0:
ld-linux-x86-64.so.2 (GLIBC_2.2.5) => /lib64/ld-linux-x86-64.so.2
ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
libc.so.6 (GLIBC_2.14) => /lib/x86_64-linux-gnu/libc.so.6
libc.so.6 (GLIBC_2.3.2) => /lib/x86_64-linux-gnu/libc.so.6
libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
libc.so.6 (GLIBC_PRIVATE) => /lib/x86_64-linux-gnu/libc.so.6
/lib/x86_64-linux-gnu/libc.so.6:
ld-linux-x86-64.so.2 (GLIBC_2.3) => /lib64/ld-linux-x86-64.so.2
ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
/lib/x86_64-linux-gnu/libdl.so.2:
ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
libc.so.6 (GLIBC_PRIVATE) => /lib/x86_64-linux-gnu/libc.so.6
libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
/lib/x86_64-linux-gnu/libutil.so.1:
libc.so.6 (GLIBC_PRIVATE) => /lib/x86_64-linux-gnu/libc.so.6
libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
/lib/x86_64-linux-gnu/libm.so.6:
ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
libc.so.6 (GLIBC_PRIVATE) => /lib/x86_64-linux-gnu/libc.so.6
The various GLIBC dependencies are on /lib/x86_64-linux-gnu/libc.so.6
. Running the library says:
/lib/x86_64-linux-gnu/libc.so.6
GNU C Library (Debian GLIBC 2.24-9) stable release version 2.24, by Roland McGrath et al.
Copyright (C) 2016 Free Software Foundation, Inc.
...
So, does auditwheel
not check for the referenced versions?
The source code suggests that the version sets are checked, but there is also a whitelist that seems to be processed in parallel with versioned symbols, inside wheel_abi
.
It would be nice if auditwheel will support checking if path is closed against dynamic linking. What do you think? I know of https://github.com/matthew-brett/delocate but it's for MacOS and I'd like to have similar tool for Linux.
I believe there is a typo in the policy for the CXXABI symbol version. While the file lists "3.4.8" as the max permissible version, I dumped all the .gnu.version_d
info from all the whitelisted libs inside the manylinux docker image and found that the max version was actually "1.3.1".
Even on a Debian jessie image (much more recent than the CentOS 5 manylinux image), I cannot find a CXXABI symbol with a version greater than 1.3.8.
Hi,
Running auditwheel repair may generate subtly different versions of the wheel, depending on the exact version of so libraries being included.
When looking at the wheel or otherwise running pip freeze there is no externally visible difference though.
I was wondering if it would make sense to have auditwheel repair accept a new optional "local version identifier" [1] argument that would be appended to the repaired wheel version.
This would enable user to encode some information about the embedded dependencies in the repaired version.
[1] https://www.python.org/dev/peps/pep-0440/#local-version-identifiers
Wheel 0.32.0 was released two days ago and has refactored its (private) API, breaking auditwheel
:
Traceback (most recent call last):
File "/usr/local/bin/auditwheel", line 11, in <module>
sys.exit(main())
File "/opt/_internal/cpython-3.6.6/lib/python3.6/site-packages/auditwheel/main.py", line 49, in main
rval = args.func(args, p)
File "/opt/_internal/cpython-3.6.6/lib/python3.6/site-packages/auditwheel/main_repair.py", line 43, in execute
from .repair import repair_wheel
File "/opt/_internal/cpython-3.6.6/lib/python3.6/site-packages/auditwheel/repair.py", line 13, in <module>
from .wheeltools import InWheelCtx, add_platforms
File "/opt/_internal/cpython-3.6.6/lib/python3.6/site-packages/auditwheel/wheeltools.py", line 14, in <module>
from wheel.util import urlsafe_b64encode, open_for_csv, native # type: ignore
ImportError: cannot import name 'open_for_csv'
https://travis-ci.org/python-pillow/pillow-wheels/jobs/435427307#L4860
The same thing hit multibuild and delocate: pypa/wheel#255.
Note that wheel officially has no public API:
It should be noted that wheel is not intended to be used as a library, and as such there is no stable, public API.
(We're working around the issue by pinning to the previous wheel==0.31.1.)
Currently auditwheel repair
command supports only one WHEEL_FILE
argument, I thought it would be nice if it will be possible to give multiple wheels to repair, what do you think? It will be useful in my use case when I build a set of wheels some of which should be repaired. And if you consider it to be useful addition, I'll submit a patch.
Hi
I'm developing a library that has several C++ extensions, some of which are linked internally. For each extension library that has a dependency, I add the relative path (with $ORIGIN) to the linked library, as such:
runtime_library_dirs.append("\$ORIGIN/%s" % <rel-path-between modA.so and modB.so>)
This gives me .so files with dependencies:
0x0000000000000001 (NEEDED) Shared library: [libptf77blas.so.3]
0x0000000000000001 (NEEDED) Shared library: [libptcblas.so.3]
0x0000000000000001 (NEEDED) Shared library: [libatlas.so.3]
0x0000000000000001 (NEEDED) Shared library: [modA.so]
0x0000000000000001 (NEEDED) Shared library: [libstdc++.so.6]
0x0000000000000001 (NEEDED) Shared library: [libm.so.6]
0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1]
0x0000000000000001 (NEEDED) Shared library: [libpthread.so.0]
0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
0x000000000000000e (SONAME) Library soname: [modB.so]
0x000000000000001d (RUNPATH) Library runpath: [$ORIGIN/relpath/to/modA]
However, when using the auditwheel tool, the runpath is stripped and replaced with $ORIGIN/../../../.libs. As such, my library (here modA.so) can no longer be found:
0x0000000000000001 (NEEDED) Shared library: [libptf77blas-536e8f3e.so.3.0]
0x0000000000000001 (NEEDED) Shared library: [libptcblas-f940d3f1.so.3.0]
0x0000000000000001 (NEEDED) Shared library: [libatlas-dd66cbe8.so.3.0]
0x0000000000000001 (NEEDED) Shared library: [modA.so]
0x0000000000000001 (NEEDED) Shared library: [libstdc++.so.6]
0x0000000000000001 (NEEDED) Shared library: [libm.so.6]
0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1]
0x0000000000000001 (NEEDED) Shared library: [libpthread.so.0]
0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
0x000000000000000e (SONAME) Library soname: [modB.so]
0x000000000000001d (RUNPATH) Library runpath: [$ORIGIN/../../.libs]
It would be really nice if there was a way to have the runpath appended with the path to the grafted libraries, and not replaced. Is there some solution to this? Or, am I missing something?
Please consider checking that a module library has undefined python symbols to verify that it has not been statically linked against libpython.a
I am using CMake as a build tool so it's a bit more manual than distutils for linking. I had previously been linking my module to libpython as a shared library. In testing with the recent manylinux docker images, I created a 32-bit wheel what was linked against it's static libpyton. When I ran audit wheel is said it was compliant.
I think that if the module is statically linked it should raise a flag with auditwheel.
In a project that auditwheel show
reports is manylinux1
compatible and that auditwheel repair
successfully repairs, adding the namespace_packages
argument to the setuptools.setup
call results in auditwheel show
still reporting that the project is manylinux1
compatible, however, auditwheel repair
breaks:
Traceback (most recent call last):
File "/usr/local/bin/auditwheel", line 11, in <module>
sys.exit(main())
File "/opt/_internal/cpython-3.5.1/lib/python3.5/site-packages/auditwheel/main.py", line 49, in main
rval = args.func(args, p)
File "/opt/_internal/cpython-3.5.1/lib/python3.5/site-packages/auditwheel/main_repair.py", line 81, in execute
update_tags=args.UPDATE_TAGS)
File "/opt/_internal/cpython-3.5.1/lib/python3.5/site-packages/auditwheel/repair.py", line 68, in repair_wheel
raise RuntimeError('Is this wheel malformatted? Or a bug?')
RuntimeError: Is this wheel malformatted? Or a bug?
This is with auditwheel 1.3.0 within the manylinux1 x64 docker image.
Here is a minimal testcase (n.b. requires cython).
It would be great if this package would offer an API at the same level of the command line interface. My concrete use case is that I'd like to run auditwheel straight from setup.py as a build step after bdist_wheel within a custom command (the same could be done by subclassing build_ext). Currently I'm doing something like:
class bdist_wheel_and_repair(Command):
user_options = []
def initialize_options(self):
pass
def finalize_options(self):
pass
def run(self):
self.run_command('bdist_wheel')
if sys.platform.startswith('linux'):
import auditwheel.main
import unittest.mock
wheel_path = next(item[2] for item in self.distribution.dist_files if item[0] == 'bdist_wheel')
with unittest.mock.patch('sys.argv', [sys.argv[0], 'repair', wheel_path]):
auditwheel.main.main()
How about something like auditwheel.repair(path)
etc. to avoid mocking sys.argv
?
Not a bug report, rather a note on a nice feature. I found that auditwheel can also be used to audit a conda package for external shared libraries. I have not tested if any of the other sub-commands work. Thanks for this tool, it is really helpful.
$ auditwheel show linux-64-tensorflow-0.7.1-py35_1.tar.bz2
linux-64-tensorflow-0.7.1-py35_1.tar.bz2 is consistent with the
following platform tag: "linux_x86_64".
The wheel references the following external versioned symbols in
system-provided shared libraries: GLIBC_2.14, CXXABI_1.3.5, GCC_3.0,
GLIBCXX_3.4.19.
This constrains the platform tag to "linux_x86_64". In order to
achieve a more compatible tag, you would to recompile a new wheel from
source on a system with earlier versions of these libraries, such as
CentOS 5.
The following external shared libraries are required by the wheel:
{
"libc.so.6": "/lib64/libc-2.5.so",
"libdl.so.2": "/lib64/libdl-2.5.so",
"libgcc_s.so.1": "/lib64/libgcc_s-4.1.2-20080825.so.1",
"libm.so.6": "/lib64/libm-2.5.so",
"libpthread.so.0": "/lib64/libpthread-2.5.so",
"libstdc++.so.6": "/usr/lib64/libstdc++.so.6.0.8",
"libz.so.1": "/lib64/libz.so.1.2.3"
}
In order to achieve the tag platform tag "manylinux1_x86_64" the
following shared library dependencies will need to be eliminated:
libz.so.1
Hi all,
I'd like to propose @ehashman for a commit bit. See #43, #73 for some previous contributions, and she's interested in getting more involved with auditwheel maintenance.
Any objections? @rmcgibbo?
And mechanically, I think we would need someone with admin rights to the pypa organization to invite her to the manylinux team? @dstufft?
I found some bugs trying to build and repair cffi
. fixes in the works.
This library is part of glibc (similarly to libc, libm, librt, libpthread etc which are already whitelisted) and uses the same symbol versioning. I wonder if perhaps it is better not to bundle it?
Context: I ran into some trouble with the psycopg2 wheel, please see the details here: psycopg/psycopg2-wheels#2. If I replace the bundled version with the system version it works fine.
Should this be src_path instead of src_name?
flake8 testing of https://github.com/pypa/auditwheel on Python 3.6.3
$ flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics
./auditwheel/repair.py:120:22: F821 undefined name 'src_name'
new_soname = src_name
^
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.