pypa/sample-namespace-packages

Tests against namespace packages

Python Namespace Package Examples

This repository contains samples for the various ways to create namespace packages in Python. For more details, see the documentation on namespace packages

Testing

This repo also contains testing tools to exercise installation scenarios for namespace packages.

To run the scenarios:

$ pip install -r requirements.txt
$ nox --report report.json

nox will execute all of the scenarios and report whether the namespace packages are able to be imported successfully after installation. You can use python report_to_table.py to transform the report into a markdown-friendly table.

Current status

To see the status since the last time the scenarios were run open table.md¹.

Please note:

• Mixing package types within a single namespace is not supported. While it may work in some cases, it may also break depending on the software versions used, the install commands issued, or the order of commands. It is generally advisable not to mix types.
• The pkg_resources method of namspacing is deprecated. Whenever possible, developers are encouraged to migrate away from it.
• PEP 420 was accepted as part of Python 3.3. For wider compatibility (going back to Python 2.3), use the pkgutil method.
• Zipped eggs don't play nicely with namespace packaging, and may be implicitly installed by commands like python setup.py install. To prevent this, it is recommended that you set zip_safe=False in setup.py, as we do here. Please also note that distributing packages via egg files is considered deprecated.
• The tests reported in table.md use pip with build isolation and build-backend APIs. This is triggered by the presence of a pyproject.toml file in each package source directory. If your package does not have a pyproject.toml file, pip might select a legacy (and deprecated) installation procedure, which can behave differently.

Remarks on staggered migrations

It is difficult migrate away from deprecated pkg_resources namepaces in large projects. Ideally, all packages sharing a namespace should coordinate and simultaneously drop __init__.py files to conform with PEP 420. However, developers might be interested in carrying out a staggered migration plan and temporarily mix different namespacing techniques to mitigate the migration effort and spread the work load in time.

Based on the results for the scenarios mixing pkg_resources and other namespace methods reported in table.md and legacy_table.md, we can see that (in principle) a staggered migration plan can be successful, as long as the developers are willing to accept some limitations:

• Deprecated installation methods will not be supported (e.g. python setup.py install),
• Editable installations will not be supported.

Please note, however, that these are preliminary studies. Developers should carry out an independent investigation, and check for the specific use cases they are interested in supporting.

Footnotes

1. If you would like to know about deprecated installation methods (e.g. via python setup.py install) or Python 2.7, please have a look at legacy_table.md. ↩

sample-namespace-packages's Issues

table.md in master incorrect for pkgutil; overall better compatibility with zip_safe=False.

So, I'm a Python developer trying to get my company on board with the whole namespacing thing. I've got a few years of Python under my belt, and I'd like to think I'm at least reasonably competent, but some of the idiosyncrasies of Python packaging are still beyond me - particularly when it comes to the numerous distribution formats and installation methods, most of which were seemingly developed ad-hoc. Given that some PyPA issue tickets go on for pages and pages of discussion, I guess I'm at least in good company.

This sample repo has given me a big leg up in sorting through which bits work together, though it does appear that the table.md file in master is incorrect for several pkgutil tests. These ones specifically differed, working on a fresh checkout:

nox > * pkgutil(command_a=('pip', 'install', '.'), command_b=('python', 'setup.py', 'install'), interpreter='python2'): failed
nox > * pkgutil(command_a=('pip', 'install', '-e', '.'), command_b=('python', 'setup.py', 'install'), interpreter='python2'): failed
nox > * pkgutil(command_a=('python', 'setup.py', 'install'), command_b=('pip', 'install', '.'), interpreter='python2'): failed
nox > * pkgutil(command_a=('python', 'setup.py', 'install'), command_b=('python', 'setup.py', 'install'), interpreter='python2'): failed
nox > * pkgutil(command_a=('python', 'setup.py', 'develop'), command_b=('python', 'setup.py', 'install'), interpreter='python2'): failed
nox > * pkgutil(command_a=('pip', 'install', '.'), command_b=('python', 'setup.py', 'install'), interpreter='python3'): failed
nox > * pkgutil(command_a=('pip', 'install', '-e', '.'), command_b=('python', 'setup.py', 'install'), interpreter='python3'): failed
nox > * pkgutil(command_a=('python', 'setup.py', 'install'), command_b=('pip', 'install', '.'), interpreter='python3'): failed
nox > * pkgutil(command_a=('python', 'setup.py', 'install'), command_b=('python', 'setup.py', 'install'), interpreter='python3'): failed
nox > * pkgutil(command_a=('python', 'setup.py', 'develop'), command_b=('python', 'setup.py', 'install'), interpreter='python3'): failed

This is on an Ubuntu 16.04 machine using the system-level Python installations. (While I switched to Anaconda for development quite some time ago, I didn't want to have that interfere, so I removed it from my PATH for this test.) So, "python" is 2.7, "python3" is 3.5, and strangely enough pip, virtualenv, and nox have their shebangs pointing to python3.4, though that should be OK.

Looking at the virtualenv for the first of those above failures, it seemed to be because python setup.py install created and installed a zipped bdist_egg, while pip install . did not. I have no idea if this is due changes in setuptools or what (ie. this might of worked when you ran it before), but they're incompatible now.

swolebro@local site-packages$ cd /media/src/sample-ns/.nox/93fa2640
swolebro@local 93fa2640$ source bin/activate
(93fa2640) swolebro@local 93fa2640$ python -V; pip --version
Python 2.7.12
pip 9.0.1 from /media/src/sample-ns/.nox/93fa2640/local/lib/python2.7/site-packages (python 2.7)
(93fa2640) swolebro@local 93fa2640$ pip list
example-pkg-a (1)
example-pkg-b (1)
pip (9.0.1)
setuptools (36.6.0)
wheel (0.30.0)
(93fa2640) swolebro@local 93fa2640$ cd lib/python2.7/site-packages/
(93fa2640) swolebro@local site-packages$ tree example*
example_pkg
├── a
│   └── __init__.py
└── __init__.py
example_pkg_a-1-py2.7.egg-info
├── dependency_links.txt
├── installed-files.txt
├── PKG-INFO
├── SOURCES.txt
└── top_level.txt
example_pkg_b-1-py2.7.egg [error opening dir]

1 directory, 7 files
(93fa2640) swolebro@local site-packages$ file example_pkg_b-1-py2.7.egg
example_pkg_b-1-py2.7.egg: Zip archive data, at least v2.0 to extract
(93fa2640) swolebro@local site-packages$ deactivate

Now, there were some cases where the zipped egg from a python setup.py install did work... but that's when using an editable or develop install to go along with it. Eg.

swolebro@local site-packages$ cd /media/src/sample-ns/.nox/5ea9ad76
swolebro@local 5ea9ad76$ source bin/activate
(5ea9ad76) swolebro@local 5ea9ad76$ python -V; pip --version
Python 2.7.12
pip 9.0.1 from /media/src/sample-ns/.nox/5ea9ad76/local/lib/python2.7/site-packages (python 2.7)
(5ea9ad76) swolebro@local 5ea9ad76$ pip list
example-pkg-a (1)
example-pkg-b (1, /media/src/sample-ns/pkgutil/pkg_b)
pip (9.0.1)
setuptools (36.6.0)
wheel (0.30.0)
(5ea9ad76) swolebro@local 5ea9ad76$ cd lib/python2.7/site-packages/
(5ea9ad76) swolebro@local site-packages$ tree example*
example_pkg_a-1-py2.7.egg [error opening dir]
example-pkg-b.egg-link [error opening dir]

0 directories, 0 files
(5ea9ad76) swolebro@local site-packages$ file example*
example_pkg_a-1-py2.7.egg: Zip archive data, at least v2.0 to extract
example-pkg-b.egg-link:    ASCII text
(5ea9ad76) swolebro@local site-packages$ cat example-pkg-b.egg-link
/media/src/sample-ns/pkgutil/pkg_b

On the other hand, if you set zip_safe=False in your setup() call, then you only ever get unzipped .egg directories, which, according to my tests, not only worked on all of the pkgutil cases, but also on all of the py3/pep420 ones. I've never really understood the point of the directly importable zip files - I get that it theoretically loads faster (fewer dirents to read), but I've never seen stats to back that up, especially when weighed against the complexity it adds.

I'm attaching both the nox output and JSON report for the runs without and with zip_safe=False, as well as the patch for that change. Please note the nox output has color control chars, so you'll want to less -R it. I skipped the py2/pep420 tests (guaranteed fails) and pkg_resource tests (irrelevant to me).

I encourage you to double-check my work (because I found myself manually comparing outputs and could easily have misread) and let me know what you think.

The next thing I'd like to see is an in-depth test to demonstrate the differences between MANIFEST.in, data_files, package_data, include_package_data, and the different dist types... Maybe one day I'll get around to that.

-swolebro

ns-outputs.zip
(As a zip file, since GitHub appears to apply arbitrary restrictions to the type of files it accepts. Yay, engineering!)

failing tests

Maybe my environment is bad somehow, but the results of the test run are very concerning.

$ nox --report report.json
...
$ grep "result\":" report.json   | cut -c 18- | sort | uniq -c
      1
     48 failed",
     80 skipped",
     32 success",

QUESTION: Namespace Packages

With namespace packages, you do not put the __init__.py in the root of the namespace, does setup.py create one?

Issue w/ pkgutil and setuptools > 31.0.0

Recent version of setuptools requires the namespace_packages=['xxx'] keyword in setup().
Without this, namespace packages won't work (tested with develop option).

But if you want to use pkg_resources-style namespace packages, setuptools will ask you to add :
init.py does not call declare_namespace()! Please fix it.

So your example https://github.com/pypa/sample-namespace-packages/tree/master/pkgutil don't work anymore.

Tested w/ setuptools 33.1.1 on Debian, and 36.0.1 on Arch.

Thanks

It's good as a skeleton example, but...

... is it possible to add actual py files in the packages to see how their import statements look? I'm not sure in case of namespaces if the import includes the namespaces, if it includes the package names, etc...

pkg_resource test transparently using pkgutil ?

Wouldn't this : https://github.com/jonparrott/namespace-pkg-tests/blob/master/pkg_resources_example_pkg_a/example_pkg/__init__.py#L4
mean that the package might use pkgutil while we won't be aware of it ?

Why not removing that exception handler and just let it except if pkg_resources except ?

pkgutil example breaks with pip uninstall

Uninstalling one namespace pkg uninstalls the other.

# Work out of /tmp/test
rm -rf /tmp/test;
mkdir /tmp/test;
cd /tmp/test;

# Create a virtualenv
virtualenv venv;
source venv/bin/activate;

# Clone this project
git clone https://github.com/pypa/sample-namespace-packages.git;
cd sample-namespace-packages/pkgutil/;

# Install both namespace packages
pip install pkg_a/ pkg_b/;

# This works, no errors
python -c "import example_pkg.a";
python -c "import example_pkg.b";

# Remove pkg B
yes | pip uninstall example_pkg_b;

# pkg A is now broken
python -c "import example_pkg.a";

This issue is "fixed" by modifying the setup.py of each namespace package to have packages: include only the child package.

Example showing the use of pyproject.toml file

Can namespace packages be created with pyproject.toml files only? I couldn't find any references to these type of packages.

Thank you

package data

I arrived at the example, wanting to see a solution for including packaged data, with a native namespace package.
It would be very helpful if one of both example packages has this.

using native (no pkg __init__) in py2

It seems to me that the main point PEP 420 implicit namespace package, is that every directory is considered a package, even if there is no __init__.py in there.

I noticed the result table does not reflect this, but it was for me a major hassle while developing a custom importer (relying on this for python3), and porting it to python2.

So I developed a custom importer and loader (see https://github.com/asmodehn/filefinder2) for python2.7, precisely because I needed implicit namespace packages.

I was wondering if that would relate to the features tested here, enough to be meaningful ?
If so, how should I go about integrating filefinder2 to the list of tests here ?

Keeping version information in table.md.

In #7, I added some version information to table.md, which could be useful later.

Is there any sane way to add this automatically in report_to_table.py? I pulled the versions by doing a bash loop over all the envs, source activating, calling python -V and pip list, and scanning over the output**. Putting a bunch of subprocess calls in the script is probably going to provide more headache than benefit though. May as well just do it manually by that point.

Is there any way to extract version info from a virtualenv without actually being in it? I guess we could os.walk() and get extract it from the directory names under site-packages, but that's arguably worse than subprocessing bash.

** Incidentally it looked like a handful of environments had broken pips after running all the nox sessions, but I got to look into that more later.

install namespace package

Is it possible to have a setup.py file in the native directory that installs both pkg_a and pkg_b? If so, what would that look like?

something like

native
    |-- setup.py
    |-- pkg_a
        |-- example_pkg/a
        |-- setup.py
    |-- pkg_b
        |-- example_pkg/b
        |-- setup.py

Thank you.

CI wrongly considers `noxfile.py` is designed for all sessions to pass

The sessions in noxfile.py were not originally designed to be evaluated in terms of "passing".
Instead, by analysing the README we can see that the original intent of the noxfile.py was to generate an informative table that will include situations where combining methods will pass and other where it will not pass... Making all the sessions in nox pass kind of defeat this purpose, isn't it?

Probably a clearer approach (however more costly in terms of computing resources and time), would be to run all the sessions in noxfile.py, build the new table.md using the reporting instructions described in the README.md and then fail if there is any uncommitted alteration in that file specifically.

native namespace package is missing an example of a single module portion

pep420 allows also to declare a single module namespace portion directly in/under the namespace root.

The native part of this example project is missing an example for this type of portion of a native
namespace package.

I will create a pull request to add an example demonstrating the structure of that kind of module portion.

NB: Although #13 is already mentioning/recommending something similar but is also having a sub-folder for the portion package (with an init.py module). But for a single module in the namespace root there is no sub-folder needed.