di/pip-api

An unofficial, importable pip API

pip-api's Issues

pip-api should require `six`

This package uses six in https://github.com/di/pip-api/blob/master/pip_api/_vendor/packaging/requirements.py#L12, but doesn't declare it in setup.py. Most environments install six somewhere or another, but it's possible not to, which will make importing pip_api fail.

'parse_requirements' does not consider environment markers

Trying to parse the following file

pytest>=4.6.10,<5 ; python_version < '3'
pytest>=5.4.2,<6 ; python_version >= '3'

Throws an exception

PipError: Double requirement given: pytest<6,>=5.4.2 (already in pytest<5,>=4.6.10, name='pytest')

test failure

tests fails with

========================================================================================================= short test summary info ==========================================================================================================
ERROR tests/test_installed_distributions.py::test_installed_distributions - AttributeError: module 'virtualenv' has no attribute 'create_environment'
ERROR tests/test_tests.py::test_isolation[True] - AttributeError: module 'virtualenv' has no attribute 'create_environment'
ERROR tests/test_tests.py::test_isolation[False] - AttributeError: module 'virtualenv' has no attribute 'create_environment'
ERROR tests/test_tests.py::test_all_the_right_pips - AttributeError: module 'virtualenv' has no attribute 'create_environment'

virtualenv is version 20.4.7

Identify projects which could use this library

Goals

1. Create a list of projects that currently use pip's internal API
2. Identify which use cases are not covered by pip-api
3. Open issues for each use case
4. Implement API for each use case
5. Open PR to migrate target project to pip-api

Projects successfully migrated to pip-api:

• https://github.com/peterbe/hashin/
• https://github.com/di/pytest-reqs/

Projects currently using pip's internal API

(Some likely candidates here: https://www.wheelodex.org/projects/pip/rdepends/)

• https://github.com/jazzband/pip-tools/
  • This is a big one: https://github.com/jazzband/pip-tools/blob/master/piptools/_compat/pip_compat.py

There is no LICENSE file

As per $SUBJ without license this package can't be packaged in most distributions.

Just pick a license you want add it to the repository and make sure it is shipped to pypi.

New pip release: 20.2.4

https://pypi.org/project/pip/20.2.4/

New pip release: 21.2.4

The PyPA recommended tool for installing Python packages.

https://pypi.org/project/pip/21.2.4/

Allow `installed_distributions` to be filtered for global distributions

Ref: pypa/pip-audit#7: @tetsuo-cpp pointed out that pip list and pip freeze both support the --local flag for optimistically filtering system-installed distributions.

It'd be nice to expose that as part of pip_api, so that someone could do something like this:

pip_api.installed_distributions(global=False)

...to get only the distributions installed in the local virtual environment. That, in turn, would make implementing pypa/pip-audit#7 much simpler.

For backwards compatibility, we could have global be a kwarg that defaults to True.

Thoughts?

New pip release: 20.2.3

https://pypi.org/project/pip/20.2.3/

New pip release: 21.2.3

The PyPA recommended tool for installing Python packages.

https://pypi.org/project/pip/21.2.3/

`parse_requirements` raises exception for git+ssh://… line

When using parse_requirements with a file containing an entry such as

git+ssh://[email protected]/HUB/pyfpdf.git#egg=fpdf

it fails with an exception like

pip_api.exceptions.PipError: Invalid requirement: 'git+ssh://[email protected]/HUB/pyfpdf.git#egg=fpdf'                                                                               
It looks like a path. File 'git+ssh://[email protected]/HUB/pyfpdf.git#egg=fpdf' does not exist.        

Tests are slow

The change in b484f20 was necessary but it is making the test suite quite a bit slower than it was before.

Finding a way to speed this up or otherwise scope a "bare" isolated environment creation across the entire test suite would be ideal.

_vendor/ missing from 0.0.9 on pypi

We're hitting this error:

Traceback (most recent call last):
  File "/usr/local/bin/hashin", line 6, in <module>
    from hashin import main
  File "/usr/local/lib/python2.7/site-packages/hashin.py", line 20, in <module>
    import pip_api
  File "/usr/local/lib/python2.7/site-packages/pip_api/__init__.py", line 3, in <module>
    from pip_api._vendor.packaging import version as packaging_version
ImportError: No module named _vendor.packaging

The wheel and tar.gz on pypi are both missing pip_api/_vendor/ which #31 added.

New pip release: 20.2.1

https://pypi.org/project/pip/20.2.1/

Support custom indices (and other configurations)

Per #14, pip-api should respect pip.conf or pip's environment variables for things like custom indices.

New pip release: 21.3.1

The PyPA recommended tool for installing Python packages.

https://pypi.org/project/pip/21.3.1/

New pip release: 20.2

https://pypi.org/project/pip/20.2/

Search Functionality

@di Do we have a search functionality to check if a package exists?

Long-term plan for `parse_requirements` API

The parse_requirements API is currently a bit of a wart on this project as it doesn't adhere to the general goal of using the underlying pip CLI as much as possible, and instead is mostly a reimplementation of requirements parsing in pip.

As I see it we have three options:

1. we could continue to hack on it to support new features like #120 and continue to further diverge from pip;
2. we could use some other requirements parser that includes features like #120
3. we could push for a CLI in pip that exposes this functionality
4. we could move this into pypa/packaging as a utility library that could be reused by pip.

Short term, I think we should generally do 1 to unblock downstream work, with the understanding that this isn't the long-term goal.

For 2, @pombredanne mentioned working on a standalone requirements parser, but I'm not sure what the status of that is, perhaps we could get a summary here.

I think 3 would be interesting but it's longer-term work, and I'm not sure if the pip team would be receptive to it due to what I imagine is a limited use case.

Ultimately I think 4 is the right answer here. This could involve upstreaming either the parsing logic here, in pip, in some other requirements parser, or a combination of all of them.

Make requirements and distributions case insensitive

In doing some research before writing this ticket, I learned that the django project on PyPI actually lists its project name as "Django" with a capitol "D". I've always just typed the all lowercase version into a requirements.txt or pip install command. Turns out that either is fine, per PEP 426

All comparisons of distribution names MUST be case insensitive, and MUST consider hyphens and underscores to be equivalent.

What I'm currently working towards is making an is_environment_up_to_date function, in which I need to look over the keys in the dict I got back from pip_api.parse_requirements, and see if they're all present in the dict I got back from pip_api.installed_distributions. The fact that putting "django" as a line in "requirements.txt" and having pip_api tell me that that item wasn't installed threw me for a loop for a minute.

Certainly I could solve my problem by just capitalizing the first letter of that line. However, I'd love to see this project get "better" (according to me) by more closely following relevant PEPs. I propose that both of the aforementioned functions pip_api.parse_requirements and pip_api.installed_distributions no longer return a plain dict object, but a dict-subclass that is case insensitive, like this one. Then if I have django in my requirements.txt, and Django is installed (thanks whoever made that weird, arbitrary, and unpythonic decision for Django!), then this code:

reqs = pip_api.parse_requirements('requirements.txt')
deps = pip_api.installed_distributions()
assert all(req in deps for req in reqs.keys())

will succeed.

If you agree with my proposal, I'll happily submit a PR. :)

Add a `conda` installation option for `pip-api`

Adding a conda installation option could be helpful. I have started working on this already to make pip-audit available on conda-forge channel. Once the conda-forge 💡 PR gets merged, I will push a PR in this repository to update the install-instructions in the readme.

conda install -c conda-forge pip-api

• Current status: All CI's passed. Awaiting review/merger. conda-forge/staged-recipes#17235
• Note: The PR is being used to add pip-api and pip-audit to conda-forge.

Type hints?

pip-audit is consuming pip-api, and is aiming for 100% MyPy coverage in the process.

pip-api currently doesn't have any type signatures, so we disable our MyPy checking wherever we import pip_api.

Is adding type signatures to the public APIs worth it here? If so, it's something we could dedicate engineering time to.

Pass requirements text directly to `parse_requirements`

I have a use case where it may be easier to pass a blob of text representing the contents of a requirements file, rather than a file name (because a file may not actually exist on the FS). Could we refactor this so that you can pass an argument like parse_requirements(contents=my_text_blob) and have it go around the whole process of opening and reading the file?

Alternatively, if it at least supported passing filenames or file-like objects, then I could at least pass in a StringIO object that could be opened.

New pip release: 21.1

The PyPA recommended tool for installing Python packages.

https://pypi.org/project/pip/21.1/

New pip release: 21.1.2

The PyPA recommended tool for installing Python packages.

https://pypi.org/project/pip/21.1.2/

Python 3.4 support dropped

07531fa dropped support for Python 3.4, and thus the same has happened in pytest-reqs (or maybe it was the other way around).

However there appears to be no incompatibilities with Python 3.4 in the code base. Reverting that commit here results in a green build still.

What was the reason for it? My guess is because it is EOL by Python Foundation.

This is probably a deal-breaker for coala, as it will break our minimum support for python 3.4.2, which is predicated around a travis CI requirements more so than a user requirement, and will also break our support for several LTS releases of distros which were using Python 3.4, such as Debian 8 (jessie) which has EOL June 30, 2020 -- a year away, and openSUSE 15.0, EOL November 2019.

We could play some tricks to make pytest-reqs optional so it isnt used on older versions of pytest, if that was truly necessary. Irony alert -- we would probably need di/pytest-reqs#33 merged for that to be possible.

Preserve comments and pre-evaluated markers when parsing requirements

pip-api currently (and very reasonably) strips these as part of its parse_requirements API, removing all comments and any requirement lines whose environment markers are not satisfied.

This is the correct and ideal behavior for nearly all usecases, but pypa/pip-audit#225 requires slightly different behavior from the parser: in particular, pip-audit -r ... --fix should preserve all comments in the input requirements, as well as any requirement lines (and environment markers) that weren't evaluated.

cc @tetsuo-cpp for thoughts on implementation here.

New pip release: 20.2.2

https://pypi.org/project/pip/20.2.2/

installed_distributions() gives an unhandled exception on Ubuntu 20.04 for distro-info

If installed_distributions() finds any package that it thinks has an invalid version, it throws an unhandled exception and so can't be used to look at versions of any installed packages.

Python 3.8.5 (default, Jul 28 2020, 12:59:40) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pip_api
>>> pip_api.installed_distributions()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/david/.local/lib/python3.8/site-packages/pip_api/_installed_distributions.py", line 85, in installed_distributions
    return _new_installed_distributions()
  File "/home/david/.local/lib/python3.8/site-packages/pip_api/_installed_distributions.py", line 77, in _new_installed_distributions
    ret[name] = Distribution(name, version, location)
  File "/home/david/.local/lib/python3.8/site-packages/pip_api/_installed_distributions.py", line 12, in __init__
    self.version = Version(version)
  File "/home/david/.local/lib/python3.8/site-packages/pip_api/_vendor/packaging/version.py", line 277, in __init__
    raise InvalidVersion("Invalid version: '{0}'".format(version))
pip_api._vendor.packaging.version.InvalidVersion: Invalid version: '0.23ubuntu1'
>>> 

This was first found with other packages, but has been reproduced on a fresh build of Ubuntu20.04 with just python3-pip and pip_api installed. In that test environment it fails with the distro-info package that was pre-installed in the system.

New pip release: 21.2.1

The PyPA recommended tool for installing Python packages.

https://pypi.org/project/pip/21.2.1/

New pip release: 21.1.1

The PyPA recommended tool for installing Python packages.

https://pypi.org/project/pip/21.1.1/

Feature Request: `parse_pipfile` function

With adoption increasing for Pipfile based dependency declaration, it would be great if this library could parse a Pipfile as well. Nice thing is that since a Pipfile doesn't support nested files like requirements.txt does, and the format of the file is just TOML, it should be a fairly short function that just reuses a lot of the existing functionality.

If this seems reasonable scope for this project, I'll happily submit a PR with it. :)

Expose `pip`'s cache dir

pypa/pip-audit#75 tries to re-use the pip cache whenever possible.

The pip cache's location is normally system defined (but consistent), but it can probably be overridden.

Proposed API: pip_api.cache_dir() -> Path, corresponding to the pip cache dir command.

New pip release: 21.3

The PyPA recommended tool for installing Python packages.

https://pypi.org/project/pip/21.3/

(Possibly) erroneous PipError: Double requirement given

OK, here's my setup:

base.txt

django

test.txt

-r base.txt
pytest

dev.txt

-r base.txt
-r test.txt
werkzeug

When attempting to parse dev.txt, pip_api errors with

pip_api.exceptions.PipError: Double requirement given: django (already in django, name='django')

However, I feel that this is a bug, as running pip install -r dev.txt works just fine, and what's more, it only shows the following line once in its output:

Requirement already satisfied: django in /usr/local/lib/python3.7/site-packages (from -r base.txt (line 1)) (2.2.2)

That makes me believe that pip is parsing all of these requirements files, and successfully removing duplicate requirements. So I believe that this PipError is in ... well, uh, error. It seems awfully deliberate though, so I'm not sure what problem that exception is trying to solve, when it could just silently skip over duplicate requirements, like pip seems to.

Again, happy to submit a PR, but I'm not sure what the correct approach would be to fixing this. Thanks!

test_parse_requirements_editable_file requires packaging 19.0

Recently added test_parse_requirements_editable_file requires packaging 19.0 , which wasn't updated in setup.py

See https://travis-ci.org/jayvdb/pip-api/jobs/533291247

There should be a tox factor for the minimum packaging version here and at pytest-reqs, so these problems are identified during development.

I dont believe it is acceptable for pip-api to require packaging 19.0 at this stage. It would be a very strange python env that has pip 6.0 and packaging 19.0 together.

IMO, it should be acceptable to have a version of packaging roughly equivalent to the version which was vendored by the supported pip version.

As a result, I think pip-api tests need to skip when features are not supported by the runtime version of packaging, or vendor the needed version of packaging.

New pip release: 21.1.3

The PyPA recommended tool for installing Python packages.

https://pypi.org/project/pip/21.1.3/

New pip release: 21.2.2

The PyPA recommended tool for installing Python packages.

https://pypi.org/project/pip/21.2.2/

Missing github tags

The package is at version 0.0.4 on pypi yet there is not a single tag on github.
Having tags helps with git blaming/etc, could you please add them?

Tests not distributed on pypi

Could you please update manifest.in to include tests/ so they are included on pypi distributable?
In distributions we try to run tests to validate basic python module sanity with our python stack and without it we would have to use github tarball (if there were tags, see issue #19 ).

Provide API for `pip show` info by invoking pip as a subprocess (and offer to contribute)

I recently needed to read information from a requirements file and a frozen constraints file (which I did using pip-api and works wonderfully — thanks) and then walk the tree of dependencies to identify installed packages that are directly or implicitly required by those in the requirements file.

Walking through the dependency hierarchy requires information from pip show, so I wrote code to obtain that information by running pip show in a subprocess and parsing its textual output. I think that capability would make a nice addition to the pip-api package.

Is there interest in having me contribute that functionality?

Any preferences regarding what the Python API should look like for this?

installed_distributions() always uses python 2 executable

When you execute installed_distributions() pip_api performs a python call to list the installed packages.

The used python executable is hard coded to python (if the environment variable PIPAPI_PYTHON_LOCATION is not set). It would be better to check here if the script is executed via python 2 or 3 and then use python respectively python3 depending on the used Python version.

Or alternatively simply use the current python interpreter executable (stored in the variable sys.executable)

Handle relative paths in `-r` links

This project is awesome, thank you so much!

OK, for context, here's the structure of requirements that I have:

requirements.txt
reqs/base.txt
reqs/dev.txt

requirements.txt contains the following:

-r reqs/dev.txt

reqs/dev.txt contains the following:

-r base.txt
werkzeug

reqs/base.txt contains the following:

django

This is a completely valid structure, as I can run pip install -r requirements.txt and I get both django and werkzeug installed. If I modify the first line of reqs/dev.txt to read -r reqs/base.txt, then I get the following error upon running pip install -r requirements.txt: ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'reqs/reqs/base.txt' So pip requires that references to other requirements files be expressed as a path relative to the file being parsed.

OK, so here's the problem. I call pip_api.parse_requirements('requirements.txt'), and I get the following (excerpt of a) traceback:

    reqs = pip_api.parse_requirements('requirements.txt')
  File "/usr/local/lib/python3.7/site-packages/pip_api/_parse_requirements.py", line 183, in parse_requirements
    lines = "".join(_read_file(filename)).replace("\\\n", "").splitlines()
  File "/usr/local/lib/python3.7/site-packages/pip_api/_parse_requirements.py", line 47, in _read_file
    with open(filename) as f:
FileNotFoundError: [Errno 2] No such file or directory: 'base.txt'

So it's reading requirements.txt, finding the line pointing to reqs/dev.txt, parsing it, finding a reference to base.txt, and then failing because it can't find that file in the repository root next to requirements.txt.

pip_api needs to handle file paths as being relative to the file being parsed as pip does, because setting them to be parseable by pip_api right now makes it so that my pip install command no longer works. If you agree with my conclusion, I'd be happy to submit a PR with a fix.

Ignore comments

Similar to di/pytest-reqs#35 , the default of pip-api should be like pip , which is to ignore comments.

New pip release: 20.3b1

https://pypi.org/project/pip/20.3b1/

_call.py - environment variables are being override in the subprocess.check_output method

Recently, as part of the change in commit 'Always set PIP_YES to avoid prompts' (a21d85) a dictionary was added as an env argument into the subprocess.check_output method.

When setting the env argument, the called sub-process can't access the environment variable of the base process.

The impact is that i'm getting an error (attached below) when i try to import the last version of pip-api (0.0.19)

The following code change solved the problem on my host :

i would recommend to add the PIP_YES variable upon the existing system variable.

More Information

Error message:

Subprocess module note about env variable :

Remove slogan to prevent confusion

Hi @di, could you please remove the slogan "The official unofficial pip API." to reduce the chance for creating confusion whether to base packaging tools on this?

As you know pip doesn't support an API for a good reason (maintainer sanity mostly) and it's a disservice to the community to create the impression that you personally can provide that API long-term. I get the joke of the slogan of course, and I snickered a bit, but then I thought about the possible downsides in the future. Let's not risk making things more complicated just for the sake of snark :)

Rest assured your efforts to release this 3rd-party library is not in question, in fact I really appreciate that you've done so. So thank you!

New pip release: 21.2

The PyPA recommended tool for installing Python packages.

https://pypi.org/project/pip/21.2/

Support the --path parameter when calling `pip list`

From what I can tell the --path parameter was added to pip list in 19.2

I was hoping to get pip-audit to support passing in a specific directory to audit (such as when pip install --target is used), but I suspect a change will be needed here first, possibly somewhere in:

pip-api/pip_api/_installed_distributions.py

Line 59 in b368628

def _new_installed_distributions(local: bool):

Use `pip list --format=json` when pip is sufficiently new?

It looks like pip >= 9.0.0 also supports --format=json, which is probably more reliable than parsing the column output. Thoughts?