py2exe hangs in dllfinder.py

[py2exe]

py2exe hangs in dllfinder.py

[albertosottile]

Which py2exe wheel did you install? Can you paste here the output of conda list and pip freeze? Thanks.

[albertosottile]

I am going to close this since I am unable to reproduce it and I did not receive further information. Please feel free to reopen it when you are able to provide relevant data.

[nchidsey]

I ran into this too. With 0.9.3.1, on Python 3.7.

I believe it may be triggered by a DLL dependency cycle. In my case, I tracked it down to import_extension(), which adds the full paths returned by bind_image() to todo. But self._loaded_dlls only has lowercase basenames, so when it checks dll and dep_dll against it, the full paths won't be noticed and skipped.

When I added os.path.basename(...).lower() to those two checks, it fixed the infinite loop for my case.

[albertosottile]

@nchidsey Thank you for reporting this. Can you provide a minimal working example that reproduces this issue reliably?

[nchidsey]

Addendum: I also had to move the line that adds the entries to _loaded_dlls from bind_image() to import_extension(), just before bind_image() is called.

I'm afraid I don't have a nice reproduction, as the DLLs in question were a pain to come by. I used msys2 to get cairo (along with gtk3, gobject, etc), then generated MSVC .lib files from the GCC-built DLLs that msys2 makes. I will see if I can find a way to make a simple cycle, if in fact that's what is setting off the infinite loop.

[albertosottile]

@nchidsey Do you have any more insight on this bug? I am still unable to reproduce it. If not, I am afraid I have to close this.

[nchidsey]

Sorry, the way I got the DLLs for this was pretty annoying to do. I do think the bug is that one of those functions I mentioned above is using full paths to dlls, and the other is only using the filenames, thus it doesn't avoid the infinite loop it appears intended to prevent. But I don't have an easy reproduction at this time. It's okay with me if you close it; I can always come back later.

[vicsafric]

I seem to be having a similar problem where it looks to me like dllfinder.py is hanging. I'm not an expert with python or py2exe so my diagnosis might be wrong.
I have Anaconda3-2020.02 installed on my laptop. I'm trying to build an executable with py2exe. When I run py2exe in the windows cmd window, I see the message "running py2exe" but it never continues past this point. When I press ctrl C while py2exe is hanging, I see
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 232, in 'calling callback function'
File "C:\Anaconda3-2020.02\lib\site-packages\py2exe\dllfinder.py", line 133, in status_routine
@_wapi.PIMAGEHLP_STATUS_ROUTINE

The python code I'm trying to compile is very simple. It consists of 1 line: print ("Hello World!")

I copied all the inputs and outputs of the cmd window in the attached text file(py2exe hanging cmd window text.txt)

The attached file "Anaconda py2exe.txt" contains the outputs from the cmd window when I typed "pip freeze" and "conda list"

python_code.zip
py2exe hanging cmd window text.txt

Anaconda py2exe.txt

[vicsafric]

To confirm my suspicion that the problem has something to do with the interaction between py2exe and the Anaconda environment, I installed the same version of python (3.7.6) that is part of my Anaconda installation, as a separate installation. I successfully built the executable python code with py2exe without any problem. The text in the cmd window did indicate that 1 module was missing
1 missing Modules

? readline imported from cmd, code, pdb

I do not know if this means that something needs to be fixed in Anaconda or in dllfinder. Any suggestions on how this can be fixed will be very welcome. We have a team of 8 people that was planning to start using Anaconda and py2exe to build our executables.

py2exe work outside Anaconda.txt

[zarquon5]

For the record, this issue continues to be problematic as of June 2022, in the same circumstances as outlined in earlier comments. When I tried to run py2exe from within a miniconda3 py38_4.9.2 (64-bit) python3.8.5 shell, the process hung. indefinitely, consuming about 50% of the cpu, but never terminating. I tried running build_exe -v -r to see if I could get more verbose information, but all it output was:

INFO:runtime:Analyzing the code

and then hung. In order to get py2exe to work, I had to abandon using Anaconda's python environment, and do a native python 3.8.5 install; once I did this, py2exe ran as expected, so py2exe and Anaconda remain incompatible.

[sdbbs]

OK, I think I can shed a bit more light on this, unfortunately have no idea about a solution.

It seems that this occurs either with MINGW64 Python, or with Anaconda Python - for me it is MINGW64 Python, and I have started by experiencing this:

... but all it output was:

INFO:runtime:Analyzing the code

and then hung.

Also:

Can you provide a minimal working example that reproduces this issue reliably?

I hope I can:

1. Download the MSYS2 installer from https://www.msys2.org/ and install it
2. Start the program "MSYS2 MinGW 64-bit" - this will start a bash shell in its won terminal window
3. As per https://www.msys2.org/docs/updating/ , run pacman -Syu in the shell - you might be asked to close the shell and reopen it; if so, then run pacman -Syu again for a second time in the shell to get it fully updated
4. Run pacman -S mingw-w64-x86_64-python-py2exe - it should hopefully also install the required dependencies, such as Python 3
5. Create test folder and files:

$ mkdir /tmp/testpy2exe
$ cd /tmp/testpy2exe

File /tmp/testpy2exe/mymain.py:

#!/usr/bin/env python3

print("Hello, this is mymain")

File /tmp/testpy2exe/freeze.py (with the new API):

#!/usr/bin/env python3

import sys,os

import py2exe # for debug
from py2exe import freeze

py2exe_console = [{
  'script': './mymain.py',
  'dest_base': 'my-main',
  'description': "My Main",
  'product_name': 'my-main',
}]

py2exe_options = {
  'verbose': 4,
  'bundle_files': 1,
  'compressed': 1,
  'optimize': 2,
  'dist_dir': '.',
  'dll_excludes': ['w9xpopen.exe', 'crypt32.dll'],
}

print("Start py2exe freeze")
freeze(
  console=py2exe_console,
  options=py2exe_options,
  zipfile=None,
)

And finally, run freeze.py:

$ python3 freeze.py
Start py2exe freeze
INFO:runtime:Analyzing the code

... and observe the process freeze (pun intended).

---

Now, I managed to do some debugging, and the problem is an apparent recursion of DLLs within dllfinder.py, more precisely determine_dll_type. However, it is VERY tricky to get a verbose debug output.

For one, there is the undocumented 'verbose': 4, which can be used in options - although I cannot recall anymore if it did a difference here or not.

What I had to do, was directly hack in /mingw64/lib/python3.11/site-packages/py2exe/runtime.py:

# ...
    def analyze(self):
        logger.info("Analyzing the code")

        mf = self.mf = Scanner(excludes=self.options.excludes,
                               optimize=self.options.optimize,
                               dll_excludes=self.options.dll_excludes,
                               verbose=4)         # add this line
# ...

This will give a massive printout of a tree of dlls; which for me ends with (use horizontal scrollbar to scroll to the right - a LOT of indents here, that is why it looks empty):

                                                                                                                            import_hook '_heapq' Module('heapq', 'C:/msys64/mingw64/lib/python3.11/heapq.py') None 0
                                                                                                                                determine_parent Module('heapq', 'C:/msys64/mingw64/lib/python3.11/heapq.py') 0
                                                                                                                            determine_parent -> None
                                                                                                                                find_head_package None '_heapq'
                                                                                                                                    import_module '_heapq' '_heapq' None
                                                                                                                                        load_module '_heapq' 'fp' 'C:/msys64/mingw64/lib/python3.11/lib-dynload/_heapq.cp311-mingw_x86_64.pyd'

... but then, the process hangs after that.

The second hack to get a printout I had to do was in /mingw64/lib/python3.11/site-packages/py2exe/dllfinder.py:

# ...
    def determine_dll_type(self, imagename):
        """determine_dll_type must be called with a full pathname.

        For any dll in the Windows or System directory or any
        subdirectory thereof return None, except when the dll binds to
        or IS the current python dll. Additionally, return None for DLLs
        that belong to either the Visual C++ redistributable or the
        Universal C Runtime.

        Return "DLL" when the image binds to the python dll, return
        None when the image is in the windows or system directory or belongs
        to a windows framework, return "EXT" otherwise.
        """
        print("determine_dll_type imagename {}".format(imagename))   # add this line
        fnm = imagename.lower()
# ...

Now, when the second hack is added and you run python3 freeze.py, then I can see:

...
                                                                                                                                        load_module '_heapq' 'fp' 'C:/msys64/mingw64/lib/python3.11/lib-dynload/_heapq.cp311-mingw_x86_64.pyd'
determine_dll_type imagename C:\WINDOWS\SYSTEM32\KERNEL32.dll
determine_dll_type imagename C:\WINDOWS\SYSTEM32\ADVAPI32.dll
determine_dll_type imagename C:\msys64\mingw64\bin\libpython3.11.dll
determine_dll_type imagename C:\WINDOWS\SYSTEM32\VERSION.dll
determine_dll_type imagename C:\WINDOWS\SYSTEM32\WS2_32.dll
determine_dll_type imagename C:\WINDOWS\SYSTEM32\bcrypt.dll
determine_dll_type imagename C:\WINDOWS\SYSTEM32\msvcrt.dll
determine_dll_type imagename C:\msys64\mingw64\bin\libgcc_s_seh-1.dll
determine_dll_type imagename C:\WINDOWS\SYSTEM32\KERNEL32.dll
determine_dll_type imagename C:\WINDOWS\SYSTEM32\CRYPTBASE.dll
determine_dll_type imagename C:\WINDOWS\SYSTEM32\sechost.dll
determine_dll_type imagename C:\bin\texlive\2020\bin\win32\api-ms-win-core-processthreads-l1-1-1.dll
determine_dll_type imagename C:\bin\texlive\2020\bin\win32\api-ms-win-core-timezone-l1-1-0.dll
determine_dll_type imagename C:\WINDOWS\SYSTEM32\RPCRT4.dll
determine_dll_type imagename C:\WINDOWS\SYSTEM32\SECHOST.dll
determine_dll_type imagename C:\WINDOWS\SYSTEM32\CRYPTSP.dll
determine_dll_type imagename C:\WINDOWS\SYSTEM32\msvcrt.dll
determine_dll_type imagename C:\WINDOWS\SYSTEM32\ntdll.dll
...

... and this ends up in an endless loop. I could notice that some .dlls from texlive installation sneaked in there, at first I thought that was the problem - such can be removed by adding something like this in freeze.py:

# ...
import sys,os

path_entries = os.environ["PATH"].split(";")
path_entries.remove("C:\\bin\\texlive\\2020\\bin\\win32")
# path_entries.remove ... # add as many as needed to remove entries from your PATH
os.environ["PATH"] = os.pathsep.join(path_entries)
print(os.environ["PATH"])

import py2exe # for debug
# ...

... and while this will indeed remove those entries from the determine_dll_type printout - if you run python3 freeze.py again, it will again end up in an endless loop, this time like:

...
determine_dll_type imagename C:\WINDOWS\SYSTEM32\KERNEL32.dll
determine_dll_type imagename C:\WINDOWS\SYSTEM32\VERSION.dll
determine_dll_type imagename C:\WINDOWS\SYSTEM32\msvcrt.dll
determine_dll_type imagename C:\WINDOWS\SYSTEM32\WS2_32.dll
determine_dll_type imagename C:\msys64\mingw64\bin\libpython3.11.dll
determine_dll_type imagename C:\WINDOWS\SYSTEM32\bcrypt.dll
determine_dll_type imagename C:\msys64\mingw64\bin\libgcc_s_seh-1.dll
determine_dll_type imagename C:\WINDOWS\SYSTEM32\ADVAPI32.dll
determine_dll_type imagename C:\WINDOWS\SYSTEM32\NTDLL.dll
determine_dll_type imagename C:\WINDOWS\SYSTEM32\ntdll.dll
determine_dll_type imagename C:\WINDOWS\SYSTEM32\kernelbase.dll
determine_dll_type imagename C:\WINDOWS\SYSTEM32\KERNELBASE.dll
...
determine_dll_type imagename C:\WINDOWS\SYSTEM32\KERNELBASE.dll
determine_dll_type imagename C:\WINDOWS\SYSTEM32\ntdll.dll
determine_dll_type imagename C:\WINDOWS\SYSTEM32\ntdll.dll
determine_dll_type imagename C:\WINDOWS\SYSTEM32\RPCRT4.dll
determine_dll_type imagename C:\WINDOWS\SYSTEM32\KERNELBASE.dll
determine_dll_type imagename C:\WINDOWS\SYSTEM32\ntdll.dll
determine_dll_type imagename C:\WINDOWS\SYSTEM32\ntdll.dll
determine_dll_type imagename C:\WINDOWS\SYSTEM32\RPCRT4.dll
determine_dll_type imagename C:\WINDOWS\SYSTEM32\KERNELBASE.dll

Now, you will notice this starts NOT as an endless loop: e.g. C:\msys64\mingw64\bin\libpython3.11.dll is processed - and if you observe this .dll in https://github.com/lucasg/Dependencies , you'll see it depends on C:\msys64\mingw64\bin\libgcc_s_seh-1.dll - but after a short while, we end up in an endless loop, looping KERNELBASE.dll, ntdll.dll (twice), and RPCRT4.dll ...

So, yeah - how do we get rid of this endless loop, I have no idea unfortunately (note that adding 'dll_excludes': ['w9xpopen.exe', 'crypt32.dll', 'ntdll.dll', 'KERNELBASE.dll', 'RPCRT4.dll'], in options in freeze.py does NOT work, we still get recursion and endless loop hang) ...

[sdbbs]

Ok, it seems I found a fix for the above - here is the code with my changes in /mingw64/lib/python3.11/site-packages/py2exe/dllfinder.py:

        print("import_extension pyd {}".format(pyd))
        while todo:
            dll = todo.pop() # get one and check it
            print("  ie dll {} _loaded_dlls {}".format(dll, self._loaded_dlls))
            # NB 2024-03-22 _loaded_dlls is array of dict, dll is a string,
            # so `dll in self._loaded_dlls cannot work
            #if dll in self._loaded_dlls:
            #    continue
            if os.path.basename(dll).lower() in self._loaded_dlls.keys():
                 continue
            for dep_dll in self.bind_image(dll):
                print("    ie dep_dll {}".format(dep_dll))
                if dep_dll in self._loaded_dlls:
                    continue
                dll_type = self.determine_dll_type(dep_dll)
                if dll_type is None:
                    continue
                ## if dll_type == "EXT":
                ##     print("EXT", dep_dll)
                ## elif dll_type == "DLL":
                ##     print("DLL", dep_dll)
                todo.add(dep_dll)
                print("    ie todo {}".format(todo))
                self._dlls[dep_dll].add(dll)

Basically, since _loaded_dlls is a list of dicts, and dll is a string, if dll in self._loaded_dlls can never return true, and so we end up in an endless loop if there are circular dependencies between dlls. I wonder how this ever worked (probably in the past _loaded_dlls was a list of strings, in which case if dll in self._loaded_dlls would have worked).

With the above fix, printouts are something like:

  ...
  ie dll C:\msys64\tmp\testpy2exe\WS2_32.dll _loaded_dlls {'libpython3.11.dll': 'C:\\msys64\\mingw64\\bin\\libpython3.11.dll', 'kernel32.dll': 'C:\\msys64\\tmp\\testpy2exe\\KERNEL32.dll', 'msvcrt.dll': 'C:\\msys64\\tmp\\testpy2exe\\msvcrt.dll', 'libwinpthread-1.dll': 'C:\\msys64\\mingw64\\bin\\libwinpthread-1.dll', 'rpcrt4.dll': 'C:\\WINDOWS\\SYSTEM32\\RPCRT4.dll', 'ntdll.dll': 'C:\\WINDOWS\\SYSTEM32\\ntdll.dll', 'kernelbase.dll': 'C:\\WINDOWS\\SYSTEM32\\KERNELBASE.dll', 'cryptbase.dll': 'C:\\msys64\\tmp\\testpy2exe\\CRYPTBASE.dll', 'sechost.dll': 'C:\\msys64\\tmp\\testpy2exe\\SECHOST.dll', 'cryptsp.dll': 'C:\\msys64\\tmp\\testpy2exe\\CRYPTSP.dll', 'bcrypt.dll': 'C:\\msys64\\tmp\\testpy2exe\\bcrypt.dll', 'zlib1.dll': 'C:\\msys64\\mingw64\\bin\\zlib1.dll', 'libbz2-1.dll': 'C:\\msys64\\mingw64\\bin\\libbz2-1.dll'}
    ie dep_dll C:\WINDOWS\SYSTEM32\RPCRT4.dll
    ie todo {'C:\\msys64\\mingw64\\bin\\libgcc_s_seh-1.dll', 'C:\\msys64\\tmp\\testpy2exe\\msvcrt.dll', 'C:\\msys64\\tmp\\testpy2exe\\KERNEL32.dll', 'C:\\msys64\\tmp\\testpy2exe\\ADVAPI32.dll', 'C:\\WINDOWS\\SYSTEM32\\RPCRT4.dll', 'C:\\msys64\\tmp\\testpy2exe\\VERSION.dll', 'C:\\msys64\\mingw64\\bin\\libbz2-1.dll', 'C:\\msys64\\mingw64\\bin\\libpython3.11.dll', 'C:\\msys64\\tmp\\testpy2exe\\bcrypt.dll'}
    ie dep_dll C:\WINDOWS\SYSTEM32\ntdll.dll
    ie todo {'C:\\msys64\\mingw64\\bin\\libgcc_s_seh-1.dll', 'C:\\msys64\\tmp\\testpy2exe\\msvcrt.dll', 'C:\\WINDOWS\\SYSTEM32\\ntdll.dll', 'C:\\msys64\\tmp\\testpy2exe\\KERNEL32.dll', 'C:\\msys64\\tmp\\testpy2exe\\ADVAPI32.dll', 'C:\\WINDOWS\\SYSTEM32\\RPCRT4.dll', 'C:\\msys64\\tmp\\testpy2exe\\VERSION.dll', 'C:\\msys64\\mingw64\\bin\\libbz2-1.dll', 'C:\\msys64\\mingw64\\bin\\libpython3.11.dll', 'C:\\msys64\\tmp\\testpy2exe\\bcrypt.dll
'}
  ie dll ...

So, with the above fix, finally python3 freeze.py completes - with:

$ python3 freeze.py
...
\WINDOWS\\SYSTEM32\\DSROLE.dll', 'netjoin.dll': 'C:\\WINDOWS\\SYSTEM32\\NETJOIN.dll'}
INFO:runtime:Found 497 modules, 10 are missing, 0 may be missing

  10 missing Modules
  ------------------
? __main__                            imported from bdb, pdb
? _frozen_importlib                   imported from importlib, importlib.abc, zipimport
? _frozen_importlib_external          imported from importlib, importlib._bootstrap, importlib.abc, zipimport
? _winreg                             imported from platform
? asyncio.DefaultEventLoopPolicy      imported from -
? dummy.Process                       imported from multiprocessing.pool
? java.lang                           imported from platform
? org.python.core                     imported from copy, pickle
? os.path                             imported from ctypes._aix, distutils.file_util, os, pkgutil, py_compile,
 sysconfig, test.support.script_helper, tracemalloc, unittest, unittest.util
? readline                            imported from cmd, code, pdb
Building './my-main.exe'.

However, if I run the program it crashes:

$ ./my-main.exe
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
...
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00002008 (most recent call first):
  <no Python frame>

... although, that is now a different question. At least now, the hang in dllfinder.py for me is fixed!

---

EDIT: well, unfortunately I could not get the program to work; I think it boils down to this:

• 3.11 Regression, Py_Initialize; Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding · Issue #99004 · python/cpython
• The problem is that the function _PyCodecRegistry_Init in v3.11 calls PyImport_ImportModule("encodings"), whereas previously PyImport_ImportModuleNoBlock was used - and this happens during the call to Py_InitializeFromConfig
• So. PyImport_ImportModule apparently must read from the filesystem/PYTHONPATH to read in the "encodings" module - however, here the module "encodings" is in the embedded library.zip, which we can only read in thanks to the zipextimporter module
• However, zipextimporter is also a Python module (aside from its dependency _memimporter, which is however embedded in the run stub/exe)
• So if we want to load "encodings" from embedded library.zip, we must load the module zipextimporter - but loading modules can only happen after Py_InitializeFromConfig, which here as mentioned crashes because it cannot find the "encodings" library

( see also Cannot seem to embed Python 3.11 into a 64-bit Windows C++ program · Issue #115919 · python/cpython)

So, this is a bit of a chicken-and-egg problem, and unfortunately I do not know the Python C API enough to find how to load a pure Python module before Py_InitializeFromConfig() (so "encodings" from library.zip is found when PyImport_ImportModule("encodings") runs is called by Py_InitializeFromConfig()) and solve the problem - but I just wanted to document what the problem is, from what I've seen so far ...