adang1345/delvewheel

Self-contained Python wheels for Windows

delvewheel

delvewheel is a command-line tool for creating self-contained Python wheel packages for Windows that have DLL dependencies that may not be present on the target system. It is functionally similar to auditwheel (for Linux) and delocate (for macOS).

Suppose that you have built a Python wheel for Windows containing an extension module, and the wheel depends on DLLs that are present in the build environment but may not be present on the end user's machine. This tool determines which DLLs a wheel depends on (aside from system libraries) and copies those DLLs into the wheel. This tool also takes extra steps to avoid DLL hell and to ensure that the DLLs are properly loaded at runtime.

Installation

delvewheel can be installed using pip.

pip install delvewheel

You can also install from the source code by opening a command-line shell at the repository root and running

pip install .

Supported Platforms

delvewheel can be run using Python 3.7+ on any platform.

delvewheel can repair wheels targeting Python 2.6+ for win32, win_amd64, or win_arm64.

The environment used to run delvewheel does not need to match the target environment of the wheel being repaired. For example, you can run delvewheel using 32-bit Python 3.7 to repair a wheel for 64-bit Python 2.6. You can even run delvewheel with PyPy3.6 on 32-bit x86 Linux to repair a wheel whose target environment is CPython 3.11 on Windows arm64.

Usage

delvewheel show: show external DLLs that the wheel depends on

delvewheel repair: copy external DLL dependencies into the wheel and patch the wheel so that these libraries are loaded at runtime

delvewheel needed: list the direct DLL dependencies of a single executable

delvewheel uses the PATH environment variable to search for DLL dependencies. To specify an additional directory to search for DLLs, add the location of the DLL to the PATH environment variable or use the --add-path option.

For a summary of additional command-line options, use the -h option (delvewheel -h, delvewheel show -h, delvewheel repair -h, delvewheel needed -h).

Additional Options

The path separator to use in the following options is ';' on Windows and ':' on Unix-like platforms.

delvewheel show

• --add-path: additional path(s) to search for DLLs, path-separator-delimited. These paths are searched before those in the PATH environment variable.
• --add-dll: name(s) of additional DLL(s) to vendor into the wheel, path-separator-delimited. We do not automatically search for dependencies of these DLLs unless another included DLL depends on them. If you use this option, it is your responsibility to ensure that the additional DLL is found at load time.
• --no-dll: name(s) of DLL(s) to specifically exclude from the wheel, path-separator-delimited. Dependencies of these DLLs are also automatically excluded if no other included DLL depends on them.
• --ignore-existing: don't search for or vendor in DLLs that are already in the wheel. We still search for and vendor in dependencies of these DLLs if they are not in the wheel. This flag is meant for simpler integration with other DLL bundling tools/techniques but is not a catch-all. If you use this flag, it is your responsibility to ensure that the DLLs that are already in the wheel are loaded correctly.
• --analyze-existing: analyze and vendor in dependencies of DLLs that are already in the wheel. If you use this option, it is your responsibility to ensure that these dependencies are found at load time.
• -v: verbosity
  • -v: level 1, some diagnostic information
  • -vv: level 2, include warnings from pefile
• --extract-dir: directory to store extracted contents of wheel for debug use (default is a temp directory)

delvewheel repair

• --add-path: additional path(s) to search for DLLs, path-separator-delimited. These paths are searched before those in the PATH environment variable.
• --add-dll: name(s) of additional DLL(s) to vendor into the wheel, path-separator-delimited. We do not automatically search for or vendor in dependencies of these DLLs unless another included DLL depends on them. We do not mangle the names of these DLLs or their direct dependencies. If you use this option, it is your responsibility to ensure that the additional DLL is found at load time.
• --no-dll: name(s) of DLL(s) to specifically exclude from the wheel, path-separator-delimited. Dependencies of these DLLs are also automatically excluded if no other included DLL depends on them.
• --ignore-existing: don't search for or vendor in DLLs that are already in the wheel. Don't mangle the names of these DLLs or their direct dependencies. We still search for and vendor in dependencies of these DLLs if they are not in the wheel. This flag is meant for simpler integration with other DLL bundling tools/techniques but is not a catch-all. If you use this flag, it is your responsibility to ensure that the DLLs that are already in the wheel are loaded correctly.
• --analyze-existing: analyze and vendor in dependencies of DLLs that are already in the wheel. These dependencies are name-mangled by default. If you use this option, it is your responsibility to ensure that these dependencies are found at load time.
• -v: verbosity
  • -v: level 1, some diagnostic information
  • -vv: level 2, include warnings from pefile
• --extract-dir: directory to store extracted contents of wheel for debug use (default is a temp directory)
• -w,--wheel-dir: directory to write the repaired wheel (default is wheelhouse relative to current working directory)
• --no-mangle: name(s) of DLL(s) not to mangle, path-separator-delimited
• --no-mangle-all: don't mangle any DLL names
• --strip: strip DLLs that contain an overlay when name-mangling. The GNU strip utility must be present in PATH.
• -L,--lib-sdir: subdirectory suffix to store vendored DLLs (default .libs). For example, if your wheel is named mywheel-0.0.1-cp310-cp310-win_amd64.whl, then the vendored DLLs are stored in mywheel.libs by default. If your wheel contains a top-level extension module that is not in any package, then this setting is ignored, and vendored DLLs are instead placed directly into site-packages when the wheel is installed.
• --namespace-pkg: namespace packages, specified in case-sensitive dot notation and delimited by the path separator. Normally, we patch or create __init__.py in each top-level package to add the vendored DLL location to the DLL search path at runtime. If you have a top-level namespace package that requires __init__.py to be absent or unmodified, then this technique can cause problems. This option tells delvewheel to use an alternate strategy that does not create or modify __init__.py at the root of the given namespace package(s). For example,
  • --namespace-pkg package1 declares package1 as a namespace package.
  • On Windows, --namespace-pkg package1.package2;package3 declares package1, package1\package2, and package3 as namespace packages.
• --include-symbols: include .pdb symbol files with the vendored DLLs. To be included, a symbol file must be in the same directory as the DLL and have the same filename before the extension, e.g. example.dll and example.pdb.
• --include-imports: include .lib import library files with the vendored DLLs. To be included, an import library file must be in the same directory as the DLL and have the same filename before the extension, e.g. example.dll and example.lib.

Version Scheme

Semantic versioning is used.

Name Mangling

This section describes in detail how and why delvewheel mangles the vendored DLL filenames by default. It is fairly technical, so feel free to skip it if it's not relevant to you.

Suppose you install two Python extension modules A.pyd and B.pyd into a single Python environment, where the modules come from separate projects. Each module depends on a DLL named C.dll, so each project ships its own C.dll. Because of how the Windows DLL loader works, if A.pyd is loaded before B.pyd, then both modules end up using A.pyd's version of C.dll. Windows does not allow two DLLs with the same name to be loaded in a single process (unless you have a private SxS assembly, but that's a complicated topic that's best avoided in my opinion). This is a problem if B.pyd is not compatible with A.pyd's version of C.dll. Maybe B.pyd requires a newer version of C.dll than A.pyd. Or maybe the two C.dlls are completely unrelated, and the two project authors by chance chose the same DLL name. This situation is known as DLL hell.

To avoid this issue, delvewheel renames the vendored DLLs. For each DLL, delvewheel computes a hash based on the DLL contents and the wheel distribution name and appends the hash to the DLL name. For example, if the authors of A.pyd and B.pyd both decided to use delvewheel as part of their projects, then A.pyd's version of C.dll could be renamed to C-a55e90393a19a36b45c623ef23fe3f4a.dll, while B.pyd's version of C.dll could be renamed to C-b7f2aeead421653280728b792642e14f.dll. Now that the two DLLs have different names, they can both be loaded into a single Python process. Even if only one of the two projects decided to use delvewheel, then the two DLLs would have different names, and DLL hell would be avoided.

Simply renaming the DLLs is not enough, though because A.pyd is still looking for C.dll. To fix this, delvewheel goes into A.pyd and finds its import directory table, which tells the Windows loader the names of the DLL dependencies. This table contains an entry with a pointer to the string "C.dll", which is embedded somewhere in A.pyd. delvewheel then finds a suitable location in A.pyd to write the string "C-a55e90393a19a36b45c623ef23fe3f4a.dll" and edits the import directory table entry to point to this string. Now, when A.pyd is loaded, it knows to look for C-a55e90393a19a36b45c623ef23fe3f4a.dll.

So far, we have described the simplest possible example where there exists one Python extension module with one DLL dependency. In the real world, DLL dependency relationships are often more complicated, and delvewheel can handle them as well. For example, suppose a project has the following properties.

• There are two extension modules D.pyd and E.pyd.
• D.pyd depends on F.dll and G.dll.
• F.dll depends on G.dll and H.dll.
• E.pyd depends on I.dll.
• I.dll depends on H.dll and J.dll.

delvewheel would execute the following when name-mangling.

• Edit the import directory table of D.pyd to point to F-c070a14b5ebd1ef22dc434b34bcbb0ae.dll and G-38752d7e43f7175f4f5e7e906bbeaac7.dll.
• Edit the import directory table of E.pyd to point to I-348818deee8c8bfbc462c6ba9c8e1898.dll.
• Edit the import directory table of F.dll to point to G-38752d7e43f7175f4f5e7e906bbeaac7.dll and H-43c80d2389f603a00e22dd9862246dba.dll.
• Edit the import directory table of I.dll to point to H-43c80d2389f603a00e22dd9862246dba.dll and J-9f50744ed67c3a6e5b24b39c08b2b207.dll.
• Rename F.dll to F-c070a14b5ebd1ef22dc434b34bcbb0ae.dll.
• Rename G.dll to G-38752d7e43f7175f4f5e7e906bbeaac7.dll.
• Rename H.dll to H-43c80d2389f603a00e22dd9862246dba.dll.
• Rename I.dll to I-348818deee8c8bfbc462c6ba9c8e1898.dll.
• Rename J.dll to J-9f50744ed67c3a6e5b24b39c08b2b207.dll.

Limitations

• delvewheel reads DLL file headers to determine which libraries a wheel depends on. DLLs that are loaded at runtime using ctypes/cffi (from Python) or LoadLibrary (from C/C++) will be missed. Support for runtime-loaded DLLs is limited; however, the following options are available.

  • Specify additional DLLs to vendor into the wheel using the --add-dll option.
  • Include the runtime-loaded DLL into the wheel yourself, and use the --analyze-existing option.

  If you use any of these options, it is your responsibility to ensure that the runtime-loaded DLLs are found at load time.

• Wheels created using delvewheel are not guaranteed to work on systems older than Windows 7 SP1. We avoid vendoring system libraries that are provided by Windows 7 SP1 or later. If you intend to create a wheel for an older Windows system that requires an extra DLL, use the --add-dll flag to vendor additional DLLs into the wheel.

• Due to a limitation in how name-mangling is performed, delvewheel is unable to name-mangle DLLs whose dependents contain insufficient internal padding to fit the mangled names and contain an overlay at the end of the binary. An exception will be raised if such a DLL is encountered. Commonly, the overlay consists of symbols that can be safely removed using the GNU strip utility, although there exist situations where the data must be present for the DLL to function properly. To remove the overlay, execute strip -s EXAMPLE.dll or use the --strip flag. To keep the overlay and skip name mangling, use the --no-mangle or --no-mangle-all flag.

• Any DLL containing an Authenticode signature will have its signature cleared if its dependencies are name-mangled or if it was built with a non-0 value for the /DEPENDENTLOADFLAG linker flag.

• delvewheel cannot repair a wheel that contains extension modules targeting more than one CPU architecture (e.g. both win32 and win_amd64). You should create a separate wheel for each CPU architecture and repair each individually.

• If your project has a delay-load DLL dependency, you must use a custom delay-load import hook when building the DLL that has the delay-load dependency. This ensures that the directory containing the vendored DLLs is included in the DLL search path when delay-loading. For convenience, we provide a suitable hook for Microsoft Visual C/C++ at delayload/delayhook.c. Add the file to your C/C++ project when building your DLL.

• An __init__.py file in a top-level package or a .py file at the root of a namespace package must be parsable by the version of Python that runs delvewheel. For instance, you cannot run delvewheel using Python 3.9 to repair a wheel containing a top-level package with an __init__.py file that uses syntax features introduced in Python 3.10. Aside from this rule, there are no other requirements regarding the relationship between the version of Python that runs delvewheel and the version(s) of Python that the wheel supports.

delvewheel's Issues

Unable to find module on Python 3.6 and 3.7 but working fine on 3.8 and 3.9

Hello, me and @kuelumbus are currently testing Windows wheel for https://github.com/kuelumbus/rdkit_platform_wheels/ . After some test we found that the wheel working just fine on Python 3.8 and 3.9, but on 3.6 and 3.7 it gives the following error.

(rdkit-win-36) C:\Users\radif\Documents\Python>python
Python 3.6.8 (tags/v3.6.8:3c6b436a57, Dec 24 2018, 00:16:47) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from rdkit import Chem
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\radif\Documents\Python\rdkit-win-36\lib\site-packages\rdkit\__init__.py", line 20, in <module>
    _delvewheel_init_patch_0_0_14()
  File "C:\Users\radif\Documents\Python\rdkit-win-36\lib\site-packages\rdkit\__init__.py", line 17, in _delvewheel_init_patch_0_0_14
    WinDLL(os.path.join(libs_dir, lib))
  File "C:\Users\radif\AppData\Local\Programs\Python\Python36\lib\ctypes\__init__.py", line 348, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: [WinError 126] The specified module could not be found

More detail on kuelumbus/rdkit-pypi#8

Any idea why this happened? Thanks.

delvewheel needed fails with PEFormatError

Running needed reports a PEFormatError:

(env) C:\Users\Scott Talbert\pycurl>delvewheel needed dist/pycurl-7.44.1-cp38-cp38-win_amd64.whl
Traceback (most recent call last):
  File "C:\Program Files\Python38\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\Python38\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\Scott Talbert\pycurl\env\Scripts\delvewheel.exe\__main__.py", line 7, in <module>
  File "c:\users\scott talbert\pycurl\env\lib\site-packages\delvewheel\__main__.py", line 72, in main
    for dll_name in patch_dll.get_direct_needed(args.file):
  File "c:\users\scott talbert\pycurl\env\lib\site-packages\delvewheel\patch_dll.py", line 78, in get_direct_needed
    with PEContext(lib_path) as pe:
  File "c:\users\scott talbert\pycurl\env\lib\site-packages\delvewheel\patch_dll.py", line 17, in __init__
    self._pe = pefile.PE(name)
  File "c:\users\scott talbert\pycurl\env\lib\site-packages\pefile.py", line 2743, in __init__
    self.__parse__(name, data, fast_load)
  File "c:\users\scott talbert\pycurl\env\lib\site-packages\pefile.py", line 2851, in __parse__
    raise PEFormatError("DOS Header magic not found.")
pefile.PEFormatError: 'DOS Header magic not found.'

Wheel:
pycurl-7.44.1-cp38-cp38-win_amd64.zip

unable to find msvcp140.dll when cross-compiling to ARM64 with cibuildwheel

I'm exploring building Windows ARM wheels for pyzmq. This uses cibuildwheel, which cross-compiles from AMD64 to ARM64, and then invokes delvewheel. The build is succeeding, but delvewheel is not finding msvcp140.dll:

  File "C:\hostedtoolcache\windows\Python\3.11.8\x64\Lib\site-packages\delvewheel\_dll_utils.py", line 403, in get_all_needed
      raise FileNotFoundError(f'Unable to find library: {dll_name}')
  FileNotFoundError: Unable to find library: msvcp140.dll
  extracting pyzmq-26.0.0a6-cp39-cp39-win_arm64.whl to C:\Users\RUNNER~1\AppData\Local\Temp\tmpy71qzmf_
  repairing C:\Users\runneradmin\AppData\Local\Temp\cibw-run-eilqynyq\cp39-win_arm64\built_wheel\pyzmq-26.0.0a6-cp39-cp39-win_arm64.whl
  finding DLL dependencies
  analyzing package-level extension module zmq\backend\cython\_zmq.cp39-win_arm64.pyd

I'm guessing I just need to specify the right --add-path to find the arm64 DLLs, but I don't really know where these things live. Any suggestion on where to look? Should I expect the default search path to be able to find these?

This is on a GitHub Actions windows-2022 runner.

Strange behaviour with Matplotlib wheels dependent on delvewheel version

Matplotlib is experiencing some strange behaviour (segmentation faults) in wheels using some but not all versions of delvewheel, and we'd appreciate some help understanding what is going on and hopefully fixing it.

The Matplotlib 3.9.1 release was yanked because although Matplotlib-only tests worked fine, using it on Windows with some downstream libraries was causing segmentation faults. I believe the original issue is actions/runner-images#10055, a change in use of C++ std::mutex by MSVC that can be worked around with a #define, but the shipping and use of MSVC DLLs (which I don't think any of us on Matplotlib fully understand) is not letting the problem go away.

The related Matplotlib issue is quite long, the most relevant comment is matplotlib/matplotlib#28551 (comment).

I think the original Matplotlib 3.9.1 used delvewheel 1.7.1. There was a period of about a week when the problem was magically solved for us which I think corresponds to delvewheel 1.7.2. Based on that we made a 3.9.1.post1 release, but unfortunately that used delvewheel 1.7.3 and we are back to experiencing some segmentation faults although only in a subset of situations that were problematic before.

The wheels we were happy with are at https://github.com/matplotlib/matplotlib/actions/runs/10227059586/job/28298087600, and wheels that are now problematic are at https://github.com/matplotlib/matplotlib/actions/runs/10272282149/job/28424163560. I have made a simple github repository to reproduce the problem at https://github.com/ianthomas23/mpl-test. The workflow to reproduce the failure is, from
https://github.com/ianthomas23/mpl-test/blob/cc04a4efbaa68a7aeda35303de6ad74792b2c6f7/.github/workflows/test.yml#L119
this:

<create new python environment>
python -m pip install -v --no-binary=contourpy contourpy
python -m pip install --only-binary=:all: --pre --upgrade --extra-index-url https://pypi.anaconda.org/scientific-python-nightly-wheels/simple matplotlib
python test_with.py

where test_with.py is

# Test with import of matplotlib

from contourpy import contour_generator
import matplotlib.pyplot as plt

print("START test_with.py")
cont_gen = contour_generator(z=[[0, 1], [2, 3]])
try:
    cont_gen.filled(2.0, 1.0)
except Exception as e:
    print("EXCEPTION HANDLER", e)
print("END test_with.py")

One can download the good and bad wheels from the above links and pip install those to confirm good/bad behaviour.

Exception: Chained function entry cannot be changed

Hi, I am trying to use delvewheel with cibuildwheel to build windows wheels for https://github.com/kuelumbus/rdkit_platform_wheels/ . Unfortunately, I get an error (https://github.com/kuelumbus/rdkit_platform_wheels/actions/runs/1195363995)

+ delvewheel repair --add-path C:\libs -w C:\Users\RUNNER~1\AppData\Local\Temp\cibuildwheelm5ku8_7m\repaired_wheel C:\Users\RUNNER~1\AppData\Local\Temp\cibuildwheelm5ku8_7m\built_wheel\rdkit_pypi-2021.3.4-cp38-cp38-win_amd64.whl
  Traceback (most recent call last):
    File "c:\cibw\python\python.3.8.10\tools\lib\runpy.py", line 194, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "c:\cibw\python\python.3.8.10\tools\lib\runpy.py", line 87, in _run_code
      exec(code, run_globals)
    File "C:\cibw\python\python.3.8.10\tools\Scripts\delvewheel.exe\__main__.py", line 7, in <module>
    File "c:\cibw\python\python.3.8.10\tools\lib\site-packages\delvewheel\__main__.py", line 70, in main
      wr.repair(args.target, no_mangles, args.no_mangle_all, args.lib_sdir)
    File "c:\cibw\python\python.3.8.10\tools\lib\site-packages\delvewheel\wheel_repair.py", line 374, in repair
      discovered, ignored = patch_dll.get_all_needed(extension_module_path, self._add_dlls, self._no_dlls, self._wheel_dirs)[:2]
    File "c:\cibw\python\python.3.8.10\tools\lib\site-packages\delvewheel\patch_dll.py", line 160, in get_all_needed
      with PEContext(lib_path) as pe:
    File "c:\cibw\python\python.3.8.10\tools\lib\site-packages\delvewheel\patch_dll.py", line 17, in __init__
      self._pe = pefile.PE(name)
    File "c:\cibw\python\python.3.8.10\tools\lib\site-packages\pefile.py", line 2742, in __init__
      self.__parse__(name, data, fast_load)
    File "c:\cibw\python\python.3.8.10\tools\lib\site-packages\pefile.py", line 3147, in __parse__
      self.full_load()
    File "c:\cibw\python\python.3.8.10\tools\lib\site-packages\pefile.py", line 3258, in full_load
      self.parse_data_directories()
    File "c:\cibw\python\python.3.8.10\tools\lib\site-packages\pefile.py", line 3553, in parse_data_directories
      value = entry[1](dir_entry.VirtualAddress, dir_entry.Size)
    File "c:\cibw\python\python.3.8.10\tools\lib\site-packages\pefile.py", line 3646, in parse_exceptions_directory
      rf.unwindinfo.set_chained_function_entry(
    File "c:\cibw\python\python.3.8.10\tools\lib\site-packages\pefile.py", line 1945, in set_chained_function_entry
      raise Exception("Chained function entry cannot be changed")
  Exception: Chained function entry cannot be changed
  repairing C:\Users\RUNNER~1\AppData\Local\Temp\cibuildwheelm5ku8_7m\built_wheel\rdkit_pypi-2021.3.4-cp38-cp38-win_amd64.whl
  finding DLL dependencies

Any idea?
Chris

"--add-dll"-specified dlls are not added to the .load-order file

This is especially an issue on Python <3.8 where the delvewheel init patch tries to preload vendored DLLs explicitly based on the .load-order file contents.

delvewheel 1.5.1 broken for Python 3.7

I noticed a breakage upon release of delvewheel 1.5.1, in psycopg 3 daily build:

• last test working with delvewheel 1.5.0
• first test broken with delvewheel 1.5.1
• test fixed again by forcing delvewheel < 1.5.1

The error is: Invalid format './wheelhouse/psycopg_c-3.2.0.dev1-cp37-cp37m-win_amd64.whl'

I understand Python 3.7 is senescent, however the metadata state that delvewheel 1.5.1 is still compatible with Python 3.7.

delvewheel/setup.cfg

Line 33 in 00acbe7

python_requires = >= 3.7

I think it would be totally reasonable to drop support for Python 3.7, as long as the package metadata are updated accordingly.

Bug with absolute_import

from future import absolute_import

Bug

There's a bug when reads the following docstring in init.py: ( Fiona 1.8.6 library )

"""
Fiona is OGR's neat, nimble, no-nonsense API.

Fiona provides a minimal, uncomplicated Python interface to the open
source GIS community's most trusted geodata access library and
integrates readily with other Python GIS packages such as pyproj, Rtree
and Shapely.

How minimal? Fiona can read features as mappings from shapefiles or
other GIS vector formats and write mappings as features to files using
the same formats. That's all. There aren't any feature or geometry
classes. Features and their geometries are just data.

A Fiona feature is a Python mapping inspired by the GeoJSON format. It
has id, 'geometry, and propertieskeys. The value ofidis a string identifier unique within the feature's parent collection. Thegeometryis another mapping withtypeandcoordinateskeys. Theproperties` of a feature is another mapping corresponding to its
attribute table. For example:

{'id': '1',
'geometry': {'type': 'Point', 'coordinates': (0.0, 0.0)},
'properties': {'label': u'Null Island'} }

is a Fiona feature with a point geometry and one property.

Features are read and written using objects returned by the
collection function. These Collection objects are a lot like
Python file objects. A Collection opened in reading mode serves
as an iterator over features. One opened in a writing mode provides
a write method.

Usage

Here's an example of reading a select few polygon features from
a shapefile and for each, picking off the first vertex of the exterior
ring of the polygon and using that as the point geometry for a new
feature writing to a "points.shp" file.

import fiona
with fiona.open('docs/data/test_uk.shp', 'r') as inp:
... output_schema = inp.schema.copy()
... output_schema['geometry'] = 'Point'
... with collection(
... "points.shp", "w",
... crs=inp.crs,
... driver="ESRI Shapefile",
... schema=output_schema
... ) as out:
... for f in inp.filter(
... bbox=(-5.0, 55.0, 0.0, 60.0)
... ):
... value = f['geometry']['coordinates'][0][0]
... f['geometry'] = {
... 'type': 'Point', 'coordinates': value}
... out.write(f)

Because Fiona collections are context managers, they are closed and (in
writing modes) flush contents to disk when their with blocks end.
"""

DELVEWHEEL file makes reproducibility difficult

The DELVEWHEEL file in dist-info contains the arguments passed to the program at repair time. Various build strategies will use temporary directories which then leak into this file, causing it to not be reproducible across runs.

It might be possible to setup a build environment and pass only relative paths for the input wheel, output directory, and lib paths. But with symlinking being tricky (impossible?) on Windows, that also becomes a difficult task.

It looks like only the Version entry in the file is used at a later time. The Arguments entry is stored but doesn't appear to be ever read again. Would you be open to omitting it, either outright or with some CLI option?

Error parsing __init__.py: docstring exists but does not start with triple quotes

First of all, thank you for creating this tool!

I'm running into an issue trying to repair a wheel:

(env) C:\Users\Scott Talbert\pycurl>delvewheel repair --add-path "C:\Users\Scott Talbert\vcpkg\installed\x64-windows\bin" dist/pycurl-7.44.1-cp38-cp38-win_amd64.whl
repairing dist/pycurl-7.44.1-cp38-cp38-win_amd64.whl
finding DLL dependencies
copying DLLs into pycurl-7.44.1.data\platlib
mangling DLL names
calculating DLL load order
patching curl\__init__.py
Traceback (most recent call last):
  File "C:\Program Files\Python38\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\Python38\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\Scott Talbert\pycurl\env\Scripts\delvewheel.exe\__main__.py", line 7, in <module>
  File "c:\users\scott talbert\pycurl\env\lib\site-packages\delvewheel\__main__.py", line 70, in main
    wr.repair(args.target, no_mangles, args.no_mangle_all, args.lib_sdir)
  File "c:\users\scott talbert\pycurl\env\lib\site-packages\delvewheel\wheel_repair.py", line 504, in repair
    self._patch_init(init_path, libs_dir_name, load_order_filename)
  File "c:\users\scott talbert\pycurl\env\lib\site-packages\delvewheel\wheel_repair.py", line 229, in _patch_init
    raise ValueError('Error parsing __init__.py: docstring exists but does not start with triple quotes')
ValueError: Error parsing __init__.py: docstring exists but does not start with triple quotes

Here is the wheel that I'm trying to repair (renamed to zip):
pycurl-7.44.1-cp38-cp38-win_amd64.zip

Suppress delay-load?

In January, I constructed some wheels with cibuildwheel and they came out fine. I did the same thing earlier this week and the resulting wheels have a .pth file that's apparently related to support for delay-load. This breaks the wheels in a virtual environment.

Is it possible to tell delvewheel not to do this?

Support pyinstaller locations for dll files

When using pyinstaller, an executable is created containing all of the required binaries/datafiles/source code. When the executable is run, the contents are extracted into a temporary directory and then executed.

This prevents the current logic in _delvewheel_init_patch_0_0_14() from working because the patch assumes that the files are in a sibling directory of the directory of the file. This is not the case when using a file generated by pyinstaller.

I propose two solutions. One is to not add the libs_dir file to the dll directory path if it doesn't exist. This would allow the dlls that are in the current directory to be loaded (probably not preferred logic).

Or...

pyinstaller sets a variable on the sys module called _MEIPASS that points to the temporary directory. If that value is defined, then patch method should use that value as the base directory for finding the dll files, and then not add os.pardir to the path.

For example: (using rdkit as an example).

def _delvewheel_init_patch_0_0_14():
    import os
    import sys
    
    if hasattr(sys, '_MEIPASS'):
        libs_dir = os.path.abspath(os.path.join(sys._MEIPASS, 'rdkit_pypi.libs'))
    else:
        libs_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir, 'rdkit_pypi.libs'))
        
    if sys.version_info[:2] >= (3, 8):
        if os.path.exists(os.path.join(sys.base_prefix, 'conda-meta')):
            os.environ['CONDA_DLL_SEARCH_MODIFICATION_ENABLE']='1'
        if os.path.exists(libs_dir):    
            os.add_dll_directory(libs_dir)
    else:
        from ctypes import WinDLL
        with open(os.path.join(libs_dir, '.load-order-rdkit_pypi-2021.3.5.1')) as file:
            load_order = file.read().split()
        for lib in load_order:
            WinDLL(os.path.join(libs_dir, lib))

And then the pyinstaller command can be run using:

pyinstaller --onefile --clean --add-binary "env\Lib\site-packages\rdkit_pypi.libs\**;rdkit_pypi.libs" test.py

I can provide an example project using rdkit and pyinstaller if necessary.

Option to patch all dlls in wheel ?

Hi !
I noticed that delvewheel will inspect only *.pyd files for dependencies while auditwheel goes through all the *.so files in the wheel.

Looking at the delvewheel code it looks like a matter of few lines to extend the search to include also dll files and I can create a proper MR that also introduces a cli switch if this makes it easier for you.

diff --git a/delvewheel/_wheel_repair.py b/delvewheel/_wheel_repair.py
index 58e4410..cdb625e 100644
--- a/delvewheel/_wheel_repair.py
+++ b/delvewheel/_wheel_repair.py
@@ -502,7 +502,7 @@ class WheelRepair:
                     item_lower = item.lower()
                     if item_lower.endswith('.py') and item_lower != '__init__.py':
                         self._patch_py_file(item_path, libs_dir, load_order_filename, depth)
-                    elif item_lower.endswith('.pyd'):
+                    elif item_lower.endswith('.pyd') or item_lower.endswith('.dll'):
                         namespace_root_ext_modules.add(item_path)
                 elif os.path.isdir(item_path) and \
                         (item not in self._root_level_module_names(package_dir) or self._get_init(item_path)):
@@ -626,7 +626,7 @@ class WheelRepair:
             if root == self._data_dir:
                 dirnames[:] = set(dirnames) & {'platlib', 'purelib'}
             for filename in filenames:
-                if filename.lower().endswith('.pyd'):
+                if filename.lower().endswith('.pyd') or filename.lower().endswith('.dll'):
                     extension_module_path = os.path.join(root, filename)
                     extension_module_paths.append(extension_module_path)
                     discovered, _, ignored, not_found = _dll_utils.get_all_needed(extension_module_path, self._no_dlls, self._wheel_dirs, 'ignore', False, False, self._verbose)
@@ -731,7 +731,7 @@ class WheelRepair:
             if root == self._data_dir:
                 dirnames[:] = set(dirnames) & {'platlib', 'purelib'}
             for filename in filenames:
-                if filename.lower().endswith('.pyd'):
+                if filename.lower().endswith('.pyd') or filename.lower().endswith('.dll'):
                     extension_module_path = os.path.join(root, filename)
                     dll_arch = _dll_utils.get_arch(extension_module_path)
                     if dll_arch != self._arch:

Do you see any issues enabling this feature ?

Usecase:
We want to bundle C++ libraries that are loaded upon runtime via python. This results in dlls in the wheel that are not linked by any pyd file and thus they are not discovered by the method that delvewheel currently employs.

Using --add-dll does not really help here because we need to add a big number of dependencies manually and we loose the mangling that delvewheel offers which we require since we are going to bundle 3rd-party libraries.

Thank you !

Don't mangle msvcr*.dll

Traceback (most recent call last):
  File "c:\program files\python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\program files\python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Program Files\Python39\Scripts\delvewheel.exe\__main__.py", line 7, in <module>
  File "c:\program files\python39\lib\site-packages\delvewheel\__main__.py", line 66, in main
    wr.repair(args.target, no_mangles, args.no_mangle_all, args.lib_sdir)
  File "c:\program files\python39\lib\site-packages\delvewheel\wheel_repair.py", line 291, in repair
    patch_dll.replace_needed(lib_path, needed, name_mangler)
  File "c:\program files\python39\lib\site-packages\delvewheel\patch_dll.py", line 163, in replace_needed
    raise RuntimeError(
RuntimeError: Unable to rename the dependencies of msvcp120.dll because this DLL has trailing data. If this DLL was created with MinGW, run the strip utility. Otherwise, use the --no-mangle flag.

msvcr120.dll should not be name-mangled by default because msvcp120.dll cannot be edited and depends on msvcr120.dll.

Error parsing __init__.py

Thanks @adang1345 for this library, it looks promising! I am trying to get it to work to build wheels for Windows for the python-igraph library, see the CI build here.

It seems that I am running into a problem with the parsing of __init__.py. For some reason, delvewheel throws

ValueError: Error parsing __init__.py: docstring exists but is not the first element of the parse tree

When trying to debug _patch_init, it seems that I am locally seeing that children[0] yields a ast.Str not a ast.Constant, which is currently being checked in

delvewheel/delvewheel/wheel_repair.py

Lines 198 to 200 in f7e1c0e

	if len(children) == 0 or not isinstance(children[0], ast.Expr) or \
	not isinstance(children[0].value, ast.Constant) or \
	children[0].value.value != docstring:

I am not too experience with ast though, so I don't know exactly what is going wrong.

libwinpthread-1.dll mangling

We recently had to add --no-mangle "libwinpthread-1.dll" in one of our private repositories because we were seeing:

RuntimeError: Unable to rename the dependencies of libgcc_s_seh-1.dll because this DLL has trailing data. If this DLL was created with MinGW, run the strip utility. Otherwise, include {'libwinpthread-1.dll'} in the --no-mangle flag. In addition, if you believe that delvewheel should avoid name-mangling a specific DLL by default, open an issue at https://github.com/adang1345/delvewheel/issues and include this error message.

I have no idea why this happened - the only change in our build was the cibuildwheel version (1.10 -> 2.3.0), which also changed the pip version (21.1 -> 21.3). Earlier today, everything was fine with cibuildwheel 1.10 / pip 21.1, but again, I don't see the connection as these bumps wouldn't change any DLL version. Both are using delvewheel 0.0.15.

I see that psycopg2 also encountered the same issue.

psycopg/psycopg@6c3cc44

Maybe you want to add libwinpthread-1.dll to the default "no-mangle" list.

Don't call add_dll_directory unless directory exists

Hi there,

Our packaging tool is running into problems (panda3d/panda3d#1492) with delvewheel-based dependencies since our tool places DLL dependencies in the root folder and does not create specific shapely.libs, scipy.libs, etc. subdirectories, which causes an error at runtime when os.add_dll_directory is called with a non-existent path.

We could masquerade as PyInstaller by setting sys._MEIPASS, but I'd rather not (I am worried about other side-effects), so we'll probably work around this by creating empty scipy.libs etc. subdirectories when a delvewheel-based wheel is detected. But it would be simpler if delvewheel simply checked whether the directory existed before calling os.add_dll_directory.

Alternatively, could you please provide guidance for how packaging tools are supposed to handle delvewheel-based wheels?

Anaconda and `os.add_dll_directory`

Hi!

I just came across your commit e4d5c91.

I'm running into similar issues with conda where it is not correctly handling the os.add_dll_directory instructions. I was planning to implement the same workaround you had, where we enable the CONDA_DLL_SEARCH_MODIFICATION_ENABLE flag (environment variable).
The fact that you're reverting this fix and the commit description both suggest that there are still a few issues with this solution. Would you mind sharing your findings with me?

For reference, this issue is where I first encountered this.

OSError with "The operation completed successfully."

We got this error message in the GitHub Actions (see log):

 tests\io\abinit\test_pseudos.py:11: in <module>
    from pymatgen.io.abinit.pseudos import Pseudo, PseudoTable
src\pymatgen\io\abinit\__init__.py:5: in <module>
    from .netcdf import (
src\pymatgen\io\abinit\netcdf.py:25: in <module>
    import netCDF4
C:\Users\runneradmin\micromamba\envs\pmg\lib\site-packages\netCDF4\__init__.py:28: in <module>
    _delvewheel_patch_1_7_0()
C:\Users\runneradmin\micromamba\envs\pmg\lib\site-packages\netCDF4\__init__.py:25: in _delvewheel_patch_1_7_0
    raise OSError('Error loading {}; {}'.format(lib, ctypes.FormatError(ctypes.get_last_error())))
E   OSError: Error loading charset-233f44715e5aacaf3a688c2faff5ddf7.dll; The operation completed successfully.

which is a piece of code in the netCDF4 package generated by delvewheel.

delvewheel/delvewheel/_wheel_repair.py

Lines 73 to 74 in e746862

	if os.path.isfile(lib_path) and not kernel32.LoadLibraryExW(ctypes.c_wchar_p(lib_path), None, 0x00000008):
	raise OSError('Error loading {{}}; {{}}'.format(lib, ctypes.FormatError(ctypes.get_last_error())))

According to the Windows documentation, this error message means no error:

ERROR_SUCCESS
0 (0x0)
The operation completed successfully.

So the error message does not make sense to me.

Unable to find library

Hi! We are trying to create wheels for windows for the first time using pypa/cibuildwheel and delvewheel. Our package uses fortran extensions compiled using f2py. When repairing the wheel, we get this error:

+ delvewheel repair --no-mangle-all -w C:\Users\RUNNER~1\AppData\Local\Temp\cibuildwheelldbcha4p\repaired_wheel --add-path C:\Users\RUNNER~1\AppData\Local\Temp\cibuildwheelldbcha4p\repaired_wheel\sisl\.libs C:\Users\RUNNER~1\AppData\Local\Temp\cibuildwheelldbcha4p\built_wheel\sisl-0.1.dev1+ga06a674-cp39-cp39-win32.whl
  Traceback (most recent call last):
    File "C:\cibw\python\pythonx86.3.9.9\tools\lib\runpy.py", line 197, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "C:\cibw\python\pythonx86.3.9.9\tools\lib\runpy.py", line 87, in _run_code
      exec(code, run_globals)
    File "C:\cibw\python\pythonx86.3.9.9\tools\Scripts\delvewheel.exe\__main__.py", line 7, in <module>
    File "C:\cibw\python\pythonx86.3.9.9\tools\lib\site-packages\delvewheel\__main__.py", line 70, in main
      wr.repair(args.target, no_mangles, args.no_mangle_all, args.lib_sdir)
    File "C:\cibw\python\pythonx86.3.9.9\tools\lib\site-packages\delvewheel\wheel_repair.py", line 392, in repair
      discovered, ignored = patch_dll.get_all_needed(extension_module_path, self._add_dlls, self._no_dlls, self._wheel_dirs)[:2]
    File "C:\cibw\python\pythonx86.3.9.9\tools\lib\site-packages\delvewheel\patch_dll.py", line 181, in get_all_needed
      raise FileNotFoundError(f'Unable to find library: {dll_name}')
  FileNotFoundError: Unable to find library: libio_m.z2qrulmp43yn7bzyxtwuv4l6mjf2c6ct.gfortran-win32.dll
  repairing C:\Users\RUNNER~1\AppData\Local\Temp\cibuildwheelldbcha4p\built_wheel\sisl-0.1.dev1+ga06a674-cp39-cp39-win32.whl
  finding DLL dependencies

Here are the CI logs: https://github.com/pfebrer/sisl/runs/4999376946?check_suite_focus=true
And here is the wheel that is built and we are trying to repair:
sisl-0.1.dev1+g1b94514-cp39-cp39-win32.zip

I should say that we are quite lost on how dll are managed by windows, so presumably we are doing things wrong :) Any pointers in the right direction would be very highly appreciated 😅

Thanks!

The __init__.py file makes our wheel unusable

After an upgrade of delvewheel around October 15 we get consistent errors when importing the wheel "repaired" with delvewheel. The wheel installs successfully and we are able to import its single module "my_module". However, if we try to access the module in any way, we get the error:

AttributeError: module 'my_module' has no attribute 'my_attribute'

The only difference before and after the upgrade is that delvewheel adds an init.py file in a subfolder. If we manually remove that file from the wheel it works perfectly again.

Is there a way to avoid this added init.py file?

Specify which dependency name(s) cannot be mangled

Traceback (most recent call last):
  File "c:\program files\python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\program files\python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Program Files\Python39\Scripts\delvewheel.exe\__main__.py", line 7, in <module>
  File "c:\program files\python39\lib\site-packages\delvewheel\__main__.py", line 66, in main
    wr.repair(args.target, no_mangles, args.no_mangle_all, args.lib_sdir)
  File "c:\program files\python39\lib\site-packages\delvewheel\wheel_repair.py", line 291, in repair
    patch_dll.replace_needed(lib_path, needed, name_mangler)
  File "c:\program files\python39\lib\site-packages\delvewheel\patch_dll.py", line 163, in replace_needed
    raise RuntimeError(
RuntimeError: Unable to rename the dependencies of msvcp120.dll because this DLL has trailing data. If this DLL was created with MinGW, run the strip utility. Otherwise, use the --no-mangle flag.

The error says that the dependencies of msvcp120.dll cannot be renamed but does not specify what the dependencies are. So I don't know which dependency(ies) to specify using the --no-mangle flag. The error message should be more informative.

Prepend rather than append to PATH with --add-path

It is more useful for path(s) specified in --add-path to take precedence over those in the PATH environment variable.

shows bundled dlls as 'needed'

I'm so pleased to see a Windows wheel repair tool!

I'm trying this out because I've just made a pyzmq release with missing DLL dependencies and I'd like to use your tool to help not make that mistake in the future. However, my existing bundling is resulting in some unexpected behavior. Running on pyzmq 21 wheel for Python 3.8, I get:

Analyzing pyzmq-21.0.0-cp38-cp38-win_amd64.whl

The following dependent DLLs will be copied into the wheel.
    vcruntime140.dll (C:\Program Files\Python38\vcruntime140.dll)
    vcruntime140_1.dll (C:\Program Files\Python38\vcruntime140_1.dll)
    libzmq.cp38-win_amd64.pyd (Error: Not Found)
    msvcp140.dll (Error: Not Found)

The following dependent DLLs will not be copied into the wheel.
    advapi32.dll
    api-ms-win-crt-convert-l1-1-0.dll
    api-ms-win-crt-heap-l1-1-0.dll
    api-ms-win-crt-runtime-l1-1-0.dll
    api-ms-win-crt-stdio-l1-1-0.dll
    api-ms-win-crt-string-l1-1-0.dll
    api-ms-win-crt-utility-l1-1-0.dll
    iphlpapi.dll
    kernel32.dll
    python38.dll
    ws2_32.dll

The problem is that msvcp140.dll and libzmq.cp38-win_amd64.pyd are already present in the top-level of the wheel due to my own incomplete steps to bundle dependencies, so should be found and not added for bundling. I would also expect msvcp140.dll to be found, since it's in the compiler's runtime-redist directory. Is this not searched by default?

It can be found with:

from setuptools import msvc
from distutils.util import get_platform

vcvars = msvc.msvc14_get_vc_env(get_platform())
vcruntime = vcvars["py_vcruntime_redist"]
redist_dir, _ = os.path.split(vcruntime)

where all the vc++ redist dlls such as msvcp140.dll, vcruntime, etc. are in redist_dir. It seems like it would make sense to add this to the default search path if it exists.

Support `SOURCE_DATE_EPOCH` environment variable for reproducibility

https://reproducible-builds.org/specs/source-date-epoch/

Library Not Found even with add-path

Thanks a ton for the library - this is awesome!

I am running into an issue where my wheel has some dependencies that aren't on system paths. If I do a delvewheel show <the_wheel> my output shows something like this:

The following DLLs will be copied into the wheel.
   ...
   arrow_python.dll (Error: Not Found)
   arrow.dll (Error: Not Found)

These arrow libraries are installed in C:\Program Files\arrow\lib

I was hoping to resolve this by running delvewheel show --add-path "C:\Program Files\arrow\lib" <the_wheel> but it doesn't seem to have any effect. FWIW running with -v` as an option doesn't increase any verbosity either

Docs: Provide additional resources / information about mangling

First off -- thanks for this great project.

We have been trying to use delvewheel on a project and noticed that when a DLL is mangled by delvewheel the e.g. SHA256 sum of the file is changed. Can the docs be expanded to include additional information about what delvewheel does to the DLLs during mangling? I think documentation here or alternatively links to external sources would be fine.

In our application, the above modification of the DLL actually made it incompatible with code running against the same (but untouched/unmangled) DLL.

Bitness detection problems?

delvewheel is detecting bitness problems when Visual Studio is loaded into the environment. Is there a way to skip over the first name-matching DLL and continue searching for a matching bitness candidate before bailing on the first name (which ends up being unsuitable)?

Full output log (unsure of how long this sticks around): https://gitlab.kitware.com/vtk/vtk/-/jobs/5743222#L3474

Relevant output:

$ delvewheel show (Get-ChildItem dist\*.whl | % FullName)
Analyzing vtk-9.0.20210427-cp36-cp36m-win_amd64.whl
Traceback (most recent call last):
  File "C:\glr\builds\vtk\vtk-ci-ext\0\.gitlab\python\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\glr\builds\vtk\vtk-ci-ext\0\.gitlab\python\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\glr\builds\vtk\vtk-ci-ext\0\build\venv\Scripts\delvewheel.exe\__main__.py", line 9, in <module>
  File "c:\glr\builds\vtk\vtk-ci-ext\0\build\venv\lib\site-packages\delvewheel\__main__.py", line 67, in main
    wr.show()
  File "c:\glr\builds\vtk\vtk-ci-ext\0\build\venv\lib\site-packages\delvewheel\wheel_repair.py", line 302, in show
    discovered, ignored, not_found = patch_dll.get_all_needed(extension_module_path, self._add_dlls, self._no_dlls, self._wheel_dirs, 'ignore')
  File "c:\glr\builds\vtk\vtk-ci-ext\0\build\venv\lib\site-packages\delvewheel\patch_dll.py", line 145, in get_all_needed
    raise OSError(f'Dependent library {lib_path} is {lib_bitness}-bit but Python interpreter is {interpreter_bitness}-bit')
OSError: Dependent library c:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\concrt140.dll is 32-bit but Python interpreter is 64-bit

Include *.lib files, if available

This is required, if the package offers also C API, like the gmpy2 does. Now we use ugly hacks to repackage wheel after the delvewheel repair: https://github.com/aleaxit/gmpy/blob/d44b2803cc7c17e093a2d06d6069902a775360be/scripts/cibw_repair_wheel_command_windows.sh

Another example is https://pypi.org/project/scipy-openblas64/

Experience with delvewheel and the PostgreSQL libpq

Hello,

I am using cibuildwheel+delvewheel to build psycopg >= 3 packages on GitLab CI runners.

The psycopg package depends on the libpq.dll (the Postgres client library) which in turn depends on a bunch of other libraries. One of them is libiconv-2.dll. It seems that the version of this library distributed with PostgreSQL cannot be managed by delvewheel, resulting in this error.

I am running other tests but it seems that using --no-mangle libiconv-2.dll produces a working package.

It also happen that the GitLab CI runner has other software installed, which include another libiconv-2.dll on the path, coming from git. delvewheel has no problem with this version of this library (but I am modifying the build pipeline to remove these unwanted files).

Just wanted to share this. I think everything is working ok and I don't think there's anything to do about the libiconv. However if you want I can try and run more tests.

Thank you for this tool!

Feature Request: Add an option to allow the user to specify which `__init__.py` gets the patch

We are currently running into the limit for namespace packages documented in the README

- `delvewheel` creates or patches `__init__.py` in each top-level package so that the DLLs are loaded properly during import. This will cause issues if you have a top-level namespace package that requires `__init__.py` to be absent to function properly.

Would it be possible to allow the user to specify the "top-level" of the package they wish to delocate? Something like

delvewheel repair ... --top-level=package/subpackage/

Alternatively, I believe that all we would need is the ability to move the patch to the subpackage's __init__.py rather than the top-level __init__.py. So maybe something like

delvewheel repair ... --patch-init=package/subpackage/__init__.py

I can make the PR if you approve of an approach.

Or is there some alternative you would suggest? On our side, we can "manually" move the patch after running develwheel, but of course an upstream fix would be ideal.

Cross link to our issue dwavesystems/dwave-preprocessing#130 if that's helpful.

Missing dependency: wheel

It seems, delvewheel assumes that this package is present. Here is a build failure with recent cibuildwheels: https://github.com/skirpichev/gmpy/actions/runs/9156931291/job/25173450737. Solved by last commit in aleaxit/gmpy#483.

DLL not found when try to import package generated.

I try to use delvewheel to embed dll into my python package, but the result package cannot be imported.

The generated folder is like this:

├── PyTAT-0.2.7.post2+gd04c114.data
│   └── platlib
│       ├── concrt140.dll
│       ├── libopenblas-506beabb.dll
│       └── msvcp140.dll
└── TAT.cp310-win_amd64.pyd

When I try to import the package, it says: Library libopenblas-506beabb.dll not found
What happened? If I simply move the libopenblas-xxx.dll to the current folder manually, it works.

Is there something like rpath set wrong?

delvewheel version: delvewheel-0.0.20

I call delvewheel by: python -m delvewheel repair --wheel-dir {dest_dir} {wheel} --add-path bin
log:

  repairing C:\Users\RUNNER~1\AppData\Local\Temp\cibuildwheelrhvnubge\built_wheel\PyTAT-0.2.7.post2+gd04c114-cp310-cp310-win_amd64.whl
  finding DLL dependencies
  copying DLLs into PyTAT-0.2.7.post2+gd04c114.data\platlib
  mangling DLL names
  calculating DLL load order
  updating PyTAT-0.2.7.post2+gd04c114.dist-info\RECORD
  repackaging wheel
  fixed wheel written to C:\Users\RUNNER~1\AppData\Local\Temp\cibuildwheelrhvnubge\repaired_wheel\PyTAT-0.2.7.post2+gd04c114-cp310-cp310-win_amd64.whl

I run the above in github action, action run url: https://github.com/hzhangxyz/TAT/runs/5691145045

__init__.py patch fails with /<package>.data/purelib/ wheel format

Firstly, thank you! This is immensely useful and it fills in important gap in Python packaging ecosystem!

I've tried this to patch ray-1.1.0 wheels - and it mostly works, aside from the fact that it fails to patch __init__.py (to add the code loading DLLs):

$ ./venv/Scripts/delvewheel repair ray-1.1.0-cp38-cp38-win_amd64.whl
repairing ray-1.1.0-cp38-cp38-win_amd64.whl
finding DLL dependencies
copying DLLs into ray.libs
mangling DLL names
calculating DLL load order
updating ray-1.1.0.dist-info\RECORD
repackaging wheel
fixed wheel written to D:\dev\ray\wheelhouse\ray-1.1.0-cp38-cp38-win_amd64.whl

This is apparently because the script skips /<package>.data/ directory:

delvewheel/delvewheel/wheel_repair.py

Line 339 in c6dafee

item != f'{distribution_name}-{version}.data' and \

Ray stores its package under \ray-1.1.0.data\purelib\ray\, which I believe is a valid way to define the wheel.

Make import symbol limit configurable

Hello, thanks for this useful tool.

I came across a scenario where delvewheel-repair does not include all necessary DLLs. This was caused by pefile returning a truncated list of imports due to a limit on the number of imports parsed. I observe in CadQuery/cadquery#1048 (comment) that increasing pefile.MAX_IMPORT_SYMBOLS at least partially resolves some issues.

My ideas for delvewheel are:

• Allow MAX_IMPORT_SYMBOLS to be specified at the command-line, e.g. delvewheel repair --max_import_symbols=1000000 leads to pefile.MAX_IMPORT_SYMBOLS = 1000000 before parsing;

• Display all pefile warnings to the user. Warning messages are accumulated on the PE object (example) and can be retrieved by PE.get_warnings().

Un-deprecate --no-mangle-all

Due to python/cpython#53584, it is not very user-friendly to have an optional argument with nargs='?' be followed by a positional. cf92927 should be reverted.

Importing installed wheel fails with "ImportError: DLL load failed while importing"

Hey,

first of all, thanks for creating this library! I've long been missing auditwheel on Windows.

I just tested it with pdftotext. delvewheel repair seems to have worked as I can see useful DLL files in pdftotext.libs. However, when I try to import it I get

PS C:\Users\bauerj> py
Python 3.9.1 (tags/v3.9.1:1e5d33e, Dec  7 2020, 17:08:21) [MSC v.1927 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pdftotext
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: DLL load failed while importing pdftotext: Das angegebene Modul wurde nicht gefunden.

Is there any way to debug this and see which module couldn't be found? I've already used https://github.com/lucasg/Dependencies on the .pyd file but it seemed to resolve all libraries just fine.

This is on Windows 10 by the way and as far as I can see no LoadLibrary is used.

If you want to look into this, the wheels can be found here: https://dev.azure.com/jhnnbr/pdftotext/_build/results?buildId=19&view=artifacts&pathAsName=false&type=publishedArtifacts

Patching __init__.py fails when module docstring is in double quotes

Two issues I found while working on my new project - first is that delvewheel requires you to use triple quotes for the module docstring. Python does not require this and my module has a single line in double quotes. I don't see why this is needed but delvewheel complains if this is the case. But before I even discovered this, a secondary issue threw me off.

Many of my function docstrings use triple quotes since they have multi-line descriptions. Considering my module docstring was in double quotes, delvewheel placed the patch template after the first multi-line function docstring, midway in the file, breaking the verification after patching.

Basically - delvewheel should not require module docstrings to be in triple quotes, and should not get distracted with function docstrings which are triple quoted.

Issues with anaconda

I've built a wheel for windows and I'm having issues when testing with using the python interpreters 3.8 and 3.9 from anaconda.
I works fine when _delvewheel_init_patch_0_0_13 is force to not use os.add_dll_directory. It also works fine with the python interpreters from python.org.

I'm not the only one, although setting the environment variable CONDA_DLL_SEARCH_MODIFICATION_ENABLE=1 does not help in my case. (using the environment variable does work actually)

vsruntime DLLs conflict after delvewheel repair

I am building wheel package for pypi, that usually is OK, but after updating some thirdparties I've got an error:

>>> from meshlib import mrmeshpy as mm
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: DLL load failed while importing mrmeshpy: The specified module could not be found.

(I am using python 3.11.9)

---

Lets say I had good whl and now I have bad whl.

Both was repaired with delvewheel repair and both have same dlls including msvcp140-hash.dll, vcruntime140-hash.dll, vcruntime140_1-hash.dll (instead of hash there was some hash described here (Name Mangling section of delvewheel documentation))

So good whl works fine and bad one does not work. After hours of investigation I found out that using one of these options makes bad one work --no-dll "msvcp140.dll;vcruntime140_1.dll;vcruntime140.dll" or --no-mangle "msvcp140.dll;vcruntime140_1.dll;vcruntime140.dll".

It seems to me that there are some conflicts with original dlls in the system that produce that error:

1. --no-dll excludes msvcp140-hash.dll, vcruntime140-hash.dll, vcruntime140_1-hash.dll from my package, so it just uses system dlls
2. --no-mangle includes msvcp140.dll, vcruntime140.dll, vcruntime140_1.dll to my package without hash that makes system leave only one dll linked, that also prevents conflict

Interesting thing that good whl does not have this problem (it has same dependencies but some of them are older versions)

---

Have anyone faced something like this and what is the best way to fix it? (Would be nice if someone could explain why this issue happened, "dll conflicts" is just a theory based on symptoms)

Thanks!

Link to SO question

Ignoring .pyd files in data/ directory

Hi,

I saw this commit changed the behavior to ignore .pyd files in the data directory outside of two specific directories. Could you elaborate on why this change was made?

Also, would you have any insight on how to export a package out of CMake that would place the .pyd files in the structure expected by delvewheel? Right now, I'm using the release prior to the above change and the .pyd files get exported relative to data/. I set the LIBRARY DESTINATION for the install() to lib/site-packages/<name>.libs so they get extracted to the same directory as the dependent DLLs that delvewheel adds into the wheel when it finds the .pyd files.

Use pefile.PE(..., fast_load=True) where possible

delvewheel currently uses pefile via PEContext, which calls pefile.PE() without enabling fast_load. This means that all types of data directories are loaded/parsed. However, I noticed that many, if not all, uses of PEContext require only a subset of data directories. This means that a lot of work is not needed.

To avoid the extra work, pefile.PE() provides the fast_load=True option, which skips parsing. For example, delvewheel.patch_dll.get_all_needed(), which needs DIRECTORY_ENTRY_IMPORT and DIRECTORY_ENTRY_DELAY_IMPORT, can do something like:

pe = pefile.PE(lib_name, fast_load=True)
pe.parse_data_directories([
    pefile.DIRECTORY_ENTRY['IMAGE_DIRECTORY_ENTRY_IMPORT'],
    pefile.DIRECTORY_ENTRY['IMAGE_DIRECTORY_ENTRY_DELAY_IMPORT'],
])

See the docstring of PE.parse_data_directories().

To provide some anecdata, on a rather large library containing thousands of symbols, parsing is sped up from around two minutes to under two seconds.

Unable to execute on Python 3.8 or earlier

>py -3.8 -m delvewheel
Traceback (most recent call last):
  File "C:\Program Files\Python38\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\Python38\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Program Files\Python38\lib\site-packages\delvewheel\__main__.py", line 4, in <module>
    from .wheel_repair import WheelRepair
  File "C:\Program Files\Python38\lib\site-packages\delvewheel\wheel_repair.py", line 13, in <module>
    from . import patch_dll
  File "C:\Program Files\Python38\lib\site-packages\delvewheel\patch_dll.py", line 76, in <module>
    on_error: str = 'raise') -> tuple[set[str], set[str], set[str]]:
TypeError: 'type' object is not subscriptable

Type hinting with built-in collection types was introduced with PEP 585 in Python 3.9, so a TypeError occurs with earlier Python versions.

Repaired Wheel Doesn't Work with py-limited-api and Future Python Versions

I was able to create a wheel with py-limited-api=38 . On other platforms, the repaired wheel is still installable on Python 3.9 / 3.10, but I've noticed on Windows that when trying to use the repaired wheel proc mon shows entries like ...\site-packages\<package_name>\python38.dll NAME NOT FOUND

Install on Windows 11

Hi,

I have used delvewheel before and it worked great! Thank you for developing and maintaining this.

Something must have changed, because now when I pip install delvewheel on Windows 11 and try to use it I get the following error:

PS C:\Users\galab\source\repos\ERGO-Code\HiGHS\build\highspy> delvewheel
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\galab\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\local-packages\Python312\Scripts\delvewheel.exe\__main__.py", line 4, in <module>
  File "C:\Users\galab\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\local-packages\Python312\site-packages\delvewheel\__main__.py", line 4, in <module>
    from ._wheel_repair import WheelRepair
  File "C:\Users\galab\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\local-packages\Python312\site-packages\delvewheel\_wheel_repair.py", line 16, in <module>
    from . import _dll_utils
  File "C:\Users\galab\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\local-packages\Python312\site-packages\delvewheel\_dll_utils.py", line 209, in <module>
    _translate_directory = _translate_directory()
                           ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\galab\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\local-packages\Python312\site-packages\delvewheel\_dll_utils.py", line 129, in _translate_directory
    interpreter_arch = get_arch(sys.executable)
                       ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\galab\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\local-packages\Python312\site-packages\delvewheel\_dll_utils.py", line 76, in get_arch
    with open(path, 'rb') as file:
         ^^^^^^^^^^^^^^^^
OSError: [Errno 22] Invalid argument: 'C:\\Users\\galab\\AppData\\Local\\Microsoft\\WindowsApps\\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\\python.exe'

I tried on a couple different machines and on a virtual machine with Windows 10 and I still get the same error.

Do you know how I can fix this one?

Thank you,
Ivet

mkl dlls and mangling

As per this suggestion

  RuntimeError: Unable to rename the dependencies of mkl_vml_def.1.dll because this DLL has trailing data. 
If this DLL was created with MinGW, run the strip utility. 
Otherwise, include {'mkl_core.1.dll'} in the --no-mangle flag. 
In addition, if you believe that delvewheel should avoid name-mangling a specific DLL by default, 
open an issue at https://github.com/adang1345/delvewheel/issues and include this error message.

Adding it to the no mangle flag does not work, presumably because another dll it depends on is renamed, and this original dll does not know about it, leading to a loading issue. Adding this dll subsequently to no mangle leads to some weird error code, which I think means it tries to lead a different dll of the same name and wrong version, hence it does not work.

Is it possible to just not change the name of this dll (because it apparently can not be done by the downstream mangler), but still patch it to load the mangled-name dll it depends on? Seems like it may fix it if I understood how these things work internally.

Improve messaging about 'strip'

Like many folks, I recently got one of these messages:

RuntimeError: Unable to rename the dependencies of libgraphblas.dll because this DLL has trailing data. If this DLL was created with MinGW, run the strip utility. Otherwise, include libgomp-1.dll;libgcc_s_seh-1.dll in the --no-mangle flag. In addition, if you believe that delvewheel should avoid name-mangling a specific DLL by default, open an issue at https://github.com/adang1345/delvewheel/issues and include this error message.

Could we improve the messaging on what "run the strip utility" means? Some of us only deal with Windows inside a GitHub Actions runner and experimentation gets tedious.

• Is strip a Windows command or do I need to install it? Is it just the GNU binutils utility?
• What strip arguments does delvewheel need? I tried strip libgraphblas.dll (with full path) and saw the same message again.
  The list of things strip can strip is long:

       strip [-F bfdname |--target=bfdname]
             [-I bfdname |--input-target=bfdname]
             [-O bfdname |--output-target=bfdname]
             [-s|--strip-all]
             [-S|-g|-d|--strip-debug]
             [--strip-dwo]
             [-K symbolname|--keep-symbol=symbolname]
             [-M|--merge-notes][--no-merge-notes]
             [-N symbolname |--strip-symbol=symbolname]
             [-w|--wildcard]
             [-x|--discard-all] [-X |--discard-locals]
             [-R sectionname |--remove-section=sectionname]
             [--keep-section=sectionpattern]
             [--remove-relocations=sectionpattern]
             [-o file] [-p|--preserve-dates]
             [-D|--enable-deterministic-archives]
             [-U|--disable-deterministic-archives]
             [--keep-section-symbols]
             [--keep-file-symbols]
             [--only-keep-debug]
             [-v |--verbose] [-V|--version]
             [--help] [--info]
             objfile...

Suggestion: follow homebrew's example for similar circumstances: emit the command you need me to run (with switches and dll name included). Even better, test for the presence of strip and just run it for me. Possibly if enabled with a --autostrip switch if you wish.