problem with CairoSVG/CairoCffi "._generated"

[py2exe]

problem with CairoSVG/CairoCffi "._generated"

[albertosottile]

Thanks for reporting this... I fear that you will need to include a lot more than the Python packages in the bundle to make it work.

At the moment, I was not even able to start your software from sources, as I get this:

C:\cewe2pdf>python cewe2pdf.py
Traceback (most recent call last):
  File "cewe2pdf.py", line 82, in <module>
    from clpFile import ClpFile  #for clipart .CLP and .SVG files
  File "C:\cewe2pdf\clpFile.py", line 7, in <module>
    import cairosvg
  File "C:\Python\lib\site-packages\cairosvg\__init__.py", line 26, in <module>
    from . import surface  # noqa isort:skip
  File "C:\Python\lib\site-packages\cairosvg\surface.py", line 9, in <module>
    import cairocffi as cairo
  File "C:\Python\lib\site-packages\cairocffi\__init__.py", line 48, in <module>
    cairo = dlopen(
  File "C:\Python\lib\site-packages\cairocffi\__init__.py", line 45, in dlopen
    raise OSError(error_message)  # pragma: no cover
OSError: no library called "cairo" was found
no library called "libcairo-2" was found
cannot load library 'libcairo.so.2': error 0x7e
cannot load library 'libcairo.2.dylib': error 0x7e
cannot load library 'libcairo-2.dll': error 0x7e

From the documentation of CairoSVG I read:

CairoSVG and its dependencies may require additional tools during the installation: a compiler, Python headers, Cairo, and FFI headers. These tools have different names depending on the OS you are using, but:

• on Windows, you’ll have to install Visual C++ compiler for Python and Cairo;

What should I install to get CairoSVG to work? I would prefer installers that I can run from the command line, as I am running all my tests in Docker.

[BarchSteel]

Thanks for the quick reply.
Can you try install the dependencies with:

pip install -r requirements.txt
python cewe2pdf.py

[albertosottile]

I am afraid the situation is a little more complicated than just using the requirements file.

First of all, to run your code from sources, I need to have Cairo installed in my test system. Installing Cairo on Windows is quite a nightmare, especially from CLI only, so I resorted to downloading pre-built wheels from https://www.lfd.uci.edu/~gohlke/pythonlibs/ for cairocffi, which ship a copy of cairo.dll. This allowed me to run your code from sources (after installing the right VCRuntime). I suspect that for distributing your code you would need to use the same wheels and/or to add cairo.dll to your package manually.

Second, py2exe still has issues with importing implicit namespace packages (a.k.a. folders that do not include a __init__.py file). Indeed, cairocffi._generated does not contain this file and so is ignored by py2exe. You can "fix" this by creating this file in the right folder (e.g. type nul > C:\Python\Lib\site-packages\cairocffi\_generated\__init__.py or wherever your Python installation is) then rebuilding the bundle.

After this, I ran into the VERSION problems, which I am not sure how you fixed. I do not understand how and where you manually copied the two VERSION files. AFAIK, py2exe does not support anymore building executables without a "library.zip" or whatever named zip file that containes the bytecode of the required modules. All the bundle_files options just determine which extra files get embedded in the archive, but the archive is always created and includes at least all the pyc files (there was a skip_archive option but it was removed when switching to Python 3).

I tried modifying the code of py2exe to include these VERSION files in the archive but then I ran into the issue that cairocffi will never pick up this file from the archive (see the source code here), which makes cairocffi effectively incompatible with zipimport and, ultimately, with py2exe.

To summarize, I tried to tackle the issues of bundling your code with py2exe but ultimately ran into an upstream problem of one of your dependencies (cairocffi) that I do not know how to fix. Since apparently you do, I would ask you to modify the package adding the __init__.py file and try again. If you succeed, good for you :) . If not, I am afraid you would need to either contact the authors of cairocffi and ask them to make it compatible with zipimport or either open a PR yourself.

[BarchSteel]

Thank you that you tried so hard on my problem. I really appreciate this.

I'm sorry that building cairo.dll caused you so much trouble. I was using the Anaconda3 distribution where this was pre-compiled, so I did not see the negative impact for you.

Thank you for the tip, that the package needs to have an __init__.py to be included. Maybe a command line option to force the inclusion of an arbitrary package might help with similar cases, without modifying all the source packages.

Yes, I think the way cairosvg, cairocffi, etc. look for their version file is very bad for packaging. Instead of crashing, they should have a default value, since there is only a version number in that file.

[albertosottile]

Thank you for the tip, that the package needs to have an init.py to be included. Maybe a command line option to force the inclusion of an arbitrary package might help with similar cases, without modifying all the source packages.

Such an option already exists, and is the one you used in the setup.py file. The problem is that py2exe does not recognize a folder without an __init__.py file as a Python package and it incorrectly ignores it. This is of course a well-known py2exe limitation that I plan to remove, when I have some spare time.

Yes, I think the way cairosvg, cairocffi, etc. look for their version file is very bad for packaging. Instead of crashing, they should have a default value, since there is only a version number in that file.

I think they should store the version information in a Python module (a .py file) and let Python do the job of importing the file correctly, instead of doing this manually. This is what everyone else does, including py2exe...

[albertosottile]

The upcoming version 0.10.2.0 will introduce a new module finder that supports implicit namespace packages (a.k.a. packages without an __init__.py file). Preliminary wheels for this version can be found on Bintray here: https://bintray.com/py2exe/py2exe/py2exe/0.10.2.0#files

It would be great if you could install this version, repeat your tests with it, and report here what happened. However, this new version will not solve the inability to load these VERSION text files via zipimport. For that, you should still ask upstream for a patch.

[albertosottile]

I am closing this as I did not receive any further update on the matter. FYI, in case it proves impossible to make this package compatible with zipimport (and so, with py2exe) from upstream, another alternative route could be similar to what I did for matplotlib in fe559bd.