So, I'm a Python developer trying to get my company on board with the whole namespacing thing. I've got a few years of Python under my belt, and I'd like to think I'm at least reasonably competent, but some of the idiosyncrasies of Python packaging are still beyond me - particularly when it comes to the numerous distribution formats and installation methods, most of which were seemingly developed ad-hoc. Given that some PyPA issue tickets go on for pages and pages of discussion, I guess I'm at least in good company.
This sample repo has given me a big leg up in sorting through which bits work together, though it does appear that the table.md file in master is incorrect for several pkgutil tests. These ones specifically differed, working on a fresh checkout:
nox > * pkgutil(command_a=('pip', 'install', '.'), command_b=('python', 'setup.py', 'install'), interpreter='python2'): failed
nox > * pkgutil(command_a=('pip', 'install', '-e', '.'), command_b=('python', 'setup.py', 'install'), interpreter='python2'): failed
nox > * pkgutil(command_a=('python', 'setup.py', 'install'), command_b=('pip', 'install', '.'), interpreter='python2'): failed
nox > * pkgutil(command_a=('python', 'setup.py', 'install'), command_b=('python', 'setup.py', 'install'), interpreter='python2'): failed
nox > * pkgutil(command_a=('python', 'setup.py', 'develop'), command_b=('python', 'setup.py', 'install'), interpreter='python2'): failed
nox > * pkgutil(command_a=('pip', 'install', '.'), command_b=('python', 'setup.py', 'install'), interpreter='python3'): failed
nox > * pkgutil(command_a=('pip', 'install', '-e', '.'), command_b=('python', 'setup.py', 'install'), interpreter='python3'): failed
nox > * pkgutil(command_a=('python', 'setup.py', 'install'), command_b=('pip', 'install', '.'), interpreter='python3'): failed
nox > * pkgutil(command_a=('python', 'setup.py', 'install'), command_b=('python', 'setup.py', 'install'), interpreter='python3'): failed
nox > * pkgutil(command_a=('python', 'setup.py', 'develop'), command_b=('python', 'setup.py', 'install'), interpreter='python3'): failed
This is on an Ubuntu 16.04 machine using the system-level Python installations. (While I switched to Anaconda for development quite some time ago, I didn't want to have that interfere, so I removed it from my PATH for this test.) So, "python" is 2.7, "python3" is 3.5, and strangely enough pip, virtualenv, and nox have their shebangs pointing to python3.4, though that should be OK.
Looking at the virtualenv for the first of those above failures, it seemed to be because python setup.py install
created and installed a zipped bdist_egg, while pip install .
did not. I have no idea if this is due changes in setuptools or what (ie. this might of worked when you ran it before), but they're incompatible now.
swolebro@local site-packages$ cd /media/src/sample-ns/.nox/93fa2640
swolebro@local 93fa2640$ source bin/activate
(93fa2640) swolebro@local 93fa2640$ python -V; pip --version
Python 2.7.12
pip 9.0.1 from /media/src/sample-ns/.nox/93fa2640/local/lib/python2.7/site-packages (python 2.7)
(93fa2640) swolebro@local 93fa2640$ pip list
example-pkg-a (1)
example-pkg-b (1)
pip (9.0.1)
setuptools (36.6.0)
wheel (0.30.0)
(93fa2640) swolebro@local 93fa2640$ cd lib/python2.7/site-packages/
(93fa2640) swolebro@local site-packages$ tree example*
example_pkg
├── a
│ └── __init__.py
└── __init__.py
example_pkg_a-1-py2.7.egg-info
├── dependency_links.txt
├── installed-files.txt
├── PKG-INFO
├── SOURCES.txt
└── top_level.txt
example_pkg_b-1-py2.7.egg [error opening dir]
1 directory, 7 files
(93fa2640) swolebro@local site-packages$ file example_pkg_b-1-py2.7.egg
example_pkg_b-1-py2.7.egg: Zip archive data, at least v2.0 to extract
(93fa2640) swolebro@local site-packages$ deactivate
Now, there were some cases where the zipped egg from a python setup.py install
did work... but that's when using an editable or develop install to go along with it. Eg.
swolebro@local site-packages$ cd /media/src/sample-ns/.nox/5ea9ad76
swolebro@local 5ea9ad76$ source bin/activate
(5ea9ad76) swolebro@local 5ea9ad76$ python -V; pip --version
Python 2.7.12
pip 9.0.1 from /media/src/sample-ns/.nox/5ea9ad76/local/lib/python2.7/site-packages (python 2.7)
(5ea9ad76) swolebro@local 5ea9ad76$ pip list
example-pkg-a (1)
example-pkg-b (1, /media/src/sample-ns/pkgutil/pkg_b)
pip (9.0.1)
setuptools (36.6.0)
wheel (0.30.0)
(5ea9ad76) swolebro@local 5ea9ad76$ cd lib/python2.7/site-packages/
(5ea9ad76) swolebro@local site-packages$ tree example*
example_pkg_a-1-py2.7.egg [error opening dir]
example-pkg-b.egg-link [error opening dir]
0 directories, 0 files
(5ea9ad76) swolebro@local site-packages$ file example*
example_pkg_a-1-py2.7.egg: Zip archive data, at least v2.0 to extract
example-pkg-b.egg-link: ASCII text
(5ea9ad76) swolebro@local site-packages$ cat example-pkg-b.egg-link
/media/src/sample-ns/pkgutil/pkg_b
On the other hand, if you set zip_safe=False
in your setup()
call, then you only ever get unzipped .egg directories, which, according to my tests, not only worked on all of the pkgutil cases, but also on all of the py3/pep420 ones. I've never really understood the point of the directly importable zip files - I get that it theoretically loads faster (fewer dirents to read), but I've never seen stats to back that up, especially when weighed against the complexity it adds.
I'm attaching both the nox output and JSON report for the runs without and with zip_safe=False, as well as the patch for that change. Please note the nox output has color control chars, so you'll want to less -R
it. I skipped the py2/pep420 tests (guaranteed fails) and pkg_resource tests (irrelevant to me).
I encourage you to double-check my work (because I found myself manually comparing outputs and could easily have misread) and let me know what you think.
The next thing I'd like to see is an in-depth test to demonstrate the differences between MANIFEST.in, data_files, package_data, include_package_data, and the different dist types... Maybe one day I'll get around to that.
-swolebro
ns-outputs.zip
(As a zip file, since GitHub appears to apply arbitrary restrictions to the type of files it accepts. Yay, engineering!)