sbidoul/pip-deepfreeze

A simple pip freeze workflow for Python application developers

pip-deepfreeze

A simple pip freeze workflow for Python application developers.

Table of contents

• About
• Intallation
• Quick Start
• How to
• FAQ
• CLI Reference
• Other tools
• Development
• Contributing

About

pip-deepfreeze aims at doing one thing and doing it well, namely managing the dependencies of a Python application in a virtual environment.

This includes:

• installing the project in editable mode, and its dependencies,
• updating the environment with new dependencies as the project evolves,
• uninstalling unused dependencies,
• refreshing dependencies,
• maintaining pinned dependencies in requirements.txt lock files,
• maintaining pinned optional dependencies in requirements-{extra}.txt lock files,
• displaying installed dependencies as a tree.

A few characteristics of this project:

• It is easy to use, with a single pip-df sync command.
• It is fast, with very little overhead on top of a regular pip install + pip freeze.
• It relies on the documented pip command line interface and its ubiquitous requirements file format.
• It assumes your project is configured using a PEP 517/660 compliant build backend but otherwise makes no assumption on the specific backend used.
• It has first class support for dependencies specified as VCS references.
• It is written in Python 3.8+, yet works in any virtual environment that has pip installed, including python 2 and python 3.6 and 3.7.
• It works with pip-less virtual environments created with python3 -m venv --without-pip.
• It is reasonably small and simple, with good test coverage and is hopefully easy to maintain.

Installation

Using pipx (recommended):

pipx install pip-deepfreeze

Using pip:

pip install --user pip-deepfreeze

Important

It is not recommended to install pip-deepfreeze in the same environment as your application, so its dependencies do not interfere with your app. By default it works with the py or python executable found in your PATH (which does what you normally expect in an activated virtualenv), but you can ask it to work within another environment using the --python option.

Quick start

Make sure your application declares its direct dependencies in pyproject.toml, or any other mechanism supported by your PEP 517/660 compliant build backend.

Create and activate a virtual environment using your favorite tool.

Tip

When you install pip-deepfreeze, it also installs pip so you don't need to install pip separately in your virtual environment (for instance, you may use python3 -m venv --without-pip to create it). However, if your virtual environment uses a Python version that is not supported by pip-deepfreeze's bundled copy of pip, you will need to install pip in the target virtual environment.

Important

When using pip older than 23.1, you may need to also install setuptools and wheel in the virtual environment for best results.

To install your project in editable mode in the active virtual environment, go to your project root directory and run:

pip-df sync

If you don't have one yet, this will generate a file named requirements.txt, containing the exact version of all your application dependencies, as they were installed.

You can then add this requirement.txt to version control, and other people collaborating on the project can install the project and its known good dependencies using pip-df sync (or pip install -r requirements.txt -e . in a fresh virtualenv).

Tip

pip-deepfreeze has experimental support for the uv pip installer. To use it, run pip-df sync --installer=uvpip.

When you add or remove dependencies of your project, run pip-df sync again to update your environment and requirements.txt.

To update one or more dependencies to the latest allowed version, run:

pip-df sync --update DEPENDENCY1,DEPENDENCY2 ...

If you need to add some dependencies from VCS references (e.g. when a library with a patch you need is not available as a release on a package index), add the dependency as usual in your project, then add the VCS reference to a file named constraints.txt like this:

DEPENDENCYNAME @ git+https://g.c/org/project@branch

Then run pip-df sync. It will update requirements.txt with a VCS reference pinned at the exact commit that was installed (you need pip version 20.1 or greater for this to work). If later you need to update to the HEAD of the same branch, simply use pip-df sync --update DEPENDENCYNAME.

When, later again, your branch is merged upstream and the project has published a release, remove the line from constraints.txt and run pip-df sync --update DEPENDENCYNAME to update to the latest released version.

How to

Creating a new project.

Follow the instructions of your favorite PEP 517/660 compliant build tool, such as hatch, setuptools, flit or others. After declaring the first dependencies, create and activate a virtualenv, then run pip-df sync in the project directory to generate pinned dependencies in requirements.txt.

Installing an existing project.

After checking out the project from source control, create and activate activate virtualenv, the run pip-df sync to install the project.

Updating to the latest version of a project.

After dependencies have been added to the project by others, update the source code from VCS, then run pip-df sync while in your activated virtualenv to bring it to the desired state: dependencies will be updated, removed or uninstalled as needed.

Adding or removing dependencies.

After you have added or removed dependencies to your build tool configuration, simply run pip-df sync to update your virtualenv. You will be prompted to uninstall unneeded dependencies.

Refreshing some pinned dependencies.

After a while you may want to refresh some or all of your dependencies to an up-to-date version. You can do so with pip-df sync --update dep1,dep2,....

Refreshing all pinned dependencies.

To update all dependencies to the latest allowed version, you can use pip-df sync --update-all. This is equivalent to removing requirements.txt then running pip-df sync. This is also roughly equivalent to reinstalling in an empty virtualenv with pip install -e . -c constraints.txt then running pip freeze > requirements.txt.

Using another package index than PyPI.

Create a file named constraints.txt in your project root, and add pip options to it, such as --index-url or --find-links. You can add any option that pip supports in requirements files.

Installing dependencies from VCS.

When one of your direct or indirect dependencies has a bug or a missing feature, it is convenient to do an upstream pull request then install from it. Assume for instance your project depends on the packaging library and you want to install a pull request you made to it. To do so, make sure packaging is declared as a regular dependency of your project. Then add the VCS reference in constraints.txt like so:

packaging @ git+https://github.com/you/packaging@your-branch

Then run pip-df sync --update packaging to install from the branch and pin the exact commit in requirements.txt for reproducibility. When upstream merges your PR and cuts a release, you can simply remove the line from constraints.txt and run pip-df sync --update packaging to refresh to the latest released version.

Working with extras.

Assuming your project configuration declares extra dependencies such as tests or docs, you can run pip-df sync --extras tests,docs to update your virtualenv with the necessary dependencies. This will also pin extra dependencies in requirements-tests.txt and requirements-docs.txt. Note that pip-deepfreeze assumes that the extras mechanism is used to specify additional dependencies to the base dependencies of the project.

FAQ

What should I put in constraints.txt? Should I add all my dependencies there?

constraints.txt is optional. The dependencies of your project must be declared primarily in pyproject.toml (or the legacy setup.py/setup.cfg). constraints.txt may contain additional constraints if needed, such as version constraints on indirect dependencies that you don't control, or VCS links for dependencies that you need to install from VCS source.

I have added a constraint in constraints.txt but pip-df sync does not honor it. What is going on?

pip-df sync always gives priority to versions pinned in requirements.txt, unless explicitly asked to do otherwise. After adding or changing constraints or VCS references for already pinned requirements, use the --update option like so:

pip-df sync --update DEPENDENCY1,DEPENDENCY2,...

pip-deepfreez erroneously complains python is not running in a virtualenv.

The most probable cause is that you used an older version of virtualenv which does not generate PEP 405 compliant virtual environments. virtualenv version 20 and later are supported, as well as the Python 3 native venv module. Should this problem be prevalent in practice, we may add support for older virtualenv versions, or add an option to ignore the virtualenv sanity check (which is only there to prevent pip-deepfreeze to corrupt the system Python packages by accident).

How can I pass options to pip?

The most reliable and repeatable way to pass options to pip is to add them in constraints.txt. The pip documentation lists options that are allowed in requirements files. Global options can also be set in the pip configuration file or passed via PIP_* environment variables (see the pip documentation for more information).

Why not using pip install and pip freeze manually?

pip-df sync combines both commands in one and ensures your environment and pinned requirements remain correct and up-to-date. Some error prone operations it facilitates include: uninstalling unneeded dependencies, updating selected dependencies, overriding dependencies with VCS references, etc.

Is there a recommended way to deploy my project in the production environment?

There are many possibilities. One approach that works well (and is recommended in the pip documentation) works with two simple steps. First you build the wheel files for your project and dependencies, using:

pip wheel --no-deps -r requirements.txt -e . --wheel-dir=./wheel-dir

Then you ship the content of the wheel-dir directory to your target environment or docker image, and run:

pip install --no-index --find-links=./wheel-dir project-name

Note the use of --no-deps when building and --no-index when installing. This will ensure that all the required dependencies are effectively pinned in requirements.txt.

CLI reference

Note

The command line interface is the only supported public interface. If you find yourself writing import pip_deepfreeze, please don't, as everything may change without notice. Or rather, get in touch to discuss your needs.

Global options

Usage: pip-df [OPTIONS] COMMAND [ARGS]...

A simple pip freeze workflow for Python application developers.

Options:
  -p, --python, --py PYTHON     The python executable to use. Determines the
                                python environment to work on. Defaults to the
                                'py' or 'python' executable found in PATH.
  -r, --project-root DIRECTORY  The project root directory.  [default: .]
  --min-version      VERSION    Minimum version of pip-deepfreeze required.
  --version                     Show the version and exit.
  -v, --verbose
  --install-completion          Install completion for the current shell.
  --show-completion             Show completion for the current shell, to copy
                                it or customize the installation.
  --help                        Show this message and exit.

Commands:
  sync  Install/update the environment to match the project requirements.
  tree  Print the installed dependencies of the project as a tree.

pip-df sync

Usage: pip-df sync [OPTIONS]

  Install/update the environment to match the project requirements, and lock new
  dependencies.

  Install/reinstall the project. Install/update dependencies to the latest
  allowed version according to pinned dependencies in requirements.txt or
  constraints in constraints.txt. On demand update of dependencies to to
  the latest version that matches constraints. Optionally uninstall unneeded
  dependencies.

Options:
  -u, --update DEP1,DEP2,...      Make sure selected dependencies are upgraded
                                  (or downgraded) to the latest allowed
                                  version. If DEP is not part of your
                                  application dependencies anymore, this
                                  option has no effect.

  --update-all                    Upgrade (or downgrade) all dependencies of
                                  your application to the latest allowed
                                  version.

  -x, --extras EXTRA1,EXTRA2,... Comma separated list of extras to install
                                  and freeze to requirements-{EXTRA}.txt.

  --post-sync-command TEXT        Command to run after the sync operation is
                                  complete. Can be specified multiple times.

  --uninstall-unneeded / --no-uninstall-unneeded
                                  Uninstall distributions that are not
                                  dependencies of the project. If not
                                  specified, ask confirmation.

  --installer [pip|uvpip]

  --help                          Show this message and exit.

pip-df tree

Usage: pip-df tree [OPTIONS]

  Print the installed dependencies of the project as a tree.

Options:
  -x, --extras EXTRA1,EXTRA2,...  Extras of project to consider when looking for
                                  dependencies.

  --help                          Show this message and exit.

Configuration

Most options can get default values from a [tool.pip-deepfreeze] section of your pyproject.toml file. For instance:

• sync.extras: default value for the --extras option of the sync command.
• sync.post_sync_commands: default value (as a list of strings) for the --post-sync-command options of the sync command.
• sync.installer
• min_version

Example:

[tool.pip-deepfreeze]
min_version = "2.0"

[tool.pip-deepfreeze.sync]
extras = "test,doc"
post_sync_commands = ["pip-preserve-requirements requirements*.txt"]
installer = "uvpip"

Other tools

Several other tools exist with a similar or overlapping scope as pip-deepfreeze.

• pip itself. pip-deepfreeze relies extensively on the pip CLI for installation and querying the database of installed distributions. In essence it is a thin wrapper around pip install and pip freeze. Some of the features here may serve as inspiration for future pip evolutions.
• pip-tools. This is the one with the most similar features. Besides the reasons explained in About above I wanted to see if it was possible to do such a thing using the pip CLI only. pip-deepfreeze is also more opinionated than pip-tools and pipdeptree, as it always does an editable install and it uses the build backend to obtain the top level dependencies.
• uv
• PDM
• Poetry
• pipenv
• pipdeptree. Works similarly as pip-df tree. It is convenient to have a tree command in pip-deepfreeze, that shares the exact same notion of top level dependencies.

Development

To run tests, use tox. You will get a test coverage report in htmlcov/index.html. An easy way to install tox is pipx install tox.

This project uses pre-commit to enforce linting (among which black for code formating, isort for sorting imports, and mypy for type checking).

To make sure linters run locally on each of your commits, install pre-commit (pipx install pre-commit is recommended), and run pre-commit install in your local clone of the pip-deepfreeze repository.

To release:

• Select the next version number of the form X.Y(.Z).
• towncrier --version X.Y(.Z).
• Inspect and commit the updated CHANGELOG.md.
• On GitHub, create a new release. Choose a tag of the form vX.Y(.Z). Click Generate release notes and copy over the content from CHANGELOG.md.

Contributing

We welcome contributions of all kinds.

Please consult the issue tracker to discover the roadmap and known bugs.

Before opening a pull request, please create an issue first to discuss the bug or feature request.

pip-deepfreeze's Issues

Pin pip, setuptools, wheel and distribute when they are dependencies

When pip, setuptools, wheel and distribute , they are never present in locked requirements.
This is because pips omits them by default in pip freeze output.

We change to call pip freeze --all but never propose to uninstall these as they may be necessary to build legacy projects without build isolation.

Generate hashes in requirements.txt

This will need to be implemented in pip freeze first.

Support environments where pkg_resources is not available

The script that inspects the content of the environment currently relies on pkg_resources, hence setuptools. In modern packaging, setuptools is not necessarily installed in the virtual environment. So we should have a mode that works with importlib[._]metadata.

Better UX when the project does not support editable installs

Switch toml parser

Looks like the packaging community is moving to tomli, which is smaller and fully toml 1.0 compliant.

Declare minimum pip-deepfreeze version in pyproject.toml

tool.pip-deepfreeze.min_version

Don't warn about `wheel` being absent when using pip >= 23.1

pip now builds in an isolated environment when wheel is absent, so no need to warn about it.

Support extras

pkg_resources.VersionConflict error when downgrading an already installed dependency

How to reproduce:

1. create project depending on packaging
2. install it
3. update project to depend on packaging<20.1

=> stack trace pkg_resources.VersionConflict

Support environments where pip is not installed

This could be done by providing pip-deepfreeze with a target pip command in addition to a target python interpreter.
That pip command could then be target-pip --python {target-python} ....

Don't show a stack trace when post sync commands fail

assertion error when running pip-df sync -e with unknown extras

When running pip-df sync -e with unknown extras, an assertion error is raised.

The desired behaviour is to warn the user but otherwise ignore the unknown extras.

Support uv

Warn if not in virtualenv?

What criteria to detect a safe situation?
What if in a virtualenv with system site packages?

Add option to update transivitive dependencies

When using --update DEP it could be useful to also update transitive dependencies of DEP.

Update not working for dependecy installed from GIT hidden ref if ref comes from a fork

Github allows you to reference a pull request as a hidden ref you can fetch. This feature is useful to keep a log of you open PR required by your project.

In order to declare a dependency to a new patched version of one of my project's dependencies, I've added into my requirements.txt.in the following line:

odoo10-addon-account-banking-mandate @ git+https://github.com/oca/bank-payment.git@refs/pull/727/head#subdirectory=setup/account_banking_mandate

and used the update command to install and freeze this new version of my dependency
pip-df sync --update odoo10-addon-account-banking-mandate

(PR from acsone/bank-payment to oca/bank-payment)

At the end of the process, the dependency is correctly installed however the updated entry into my requirements.txt file is not right. Indeed, even if the sha is the right one, the ORG into the github url is the one on which the pr has been made not the one from which the changes are submitted....

odoo10-addon-account-banking-mandate @ git+https://github.com/oca/bank-payment.git@84d5fd2cc8c9bbca6b036ad5f8d87776d1b3fdad#subdirectory=setup/account_banking_mandate

Is-it a know limitation or a bug?

Do not print a stack trace on pip errors

Platform awareness

Currently, the naming scheme of generated requirement files is hard coded (requirements.txt, requirement-{extra}.txt) and is generated for the platform/interpreter on which pip-deepfreeze runs.

It might be useful to have an optional naming scheme that includes platform/interpreter information.

Environment sanity checks

• verify pip, setuptools and wheel are installed
• check pip version, and recommend >=20.1

Pass `--no-input` to pip commands

This is important to avoid blocking silently.

Support overrides

uv supports overrides, and pip may support them at some point.

Design

• detect overrides.txt
• when installing, remove entries that are in requirements*.txt from overrides.txt
• pass the resulting overrides to the install command with -o (uv only for now)

Search for `py` before `python` when looking for the default python interpreter

--update/--update-all not working in some edge cases

When a new dependency is added to the project, and it is already installed while not being part of requirements.txt, using --update or --update-all may not update it to the latest available version.

This is a very edge case that is probably not an issue in practice.

Default extras

Developers typically need to remember to use pip-df sync -x test,dev. This is not optimal.

Two possibilities:

• sync install all extras by default
• configure default extras in pyproject.toml, in a tool.pip-deepfreeze section.

Support PEP 660 (editables)

Detect egg= requirement names

Better UX when there is no setup.py nor pyproject.toml

Add `--pre-sync-command`

Use pip inspect when available

Instead of our pip_list_json.py script. This will also resolve #54.

Rename requirements.txt.in to constraints.txt

requirements.txt.in is somewhat misnamed, as it actually contains pip constraints and options.

constraints.txt would more accurately reflect what the file is for, i.e. constraining dependencies when it is not practical or desired to put such constraints in pyproject.toml project.dependencies.

Allow non-editable install

It would appear that pip-deepfreeze tries to install the local package in editable mode:

pip install -e .

Unfortunately, I have a case where this does not work reliably. The reason being that I have multiple local packages which pip does not resolve well. The reason being that pip believes packages to be conflicting if one package is installed in editable mode, and the other is installed in non-editable mode from a path. There are also issues where pip believes there is a conflict if the paths are not identical (e.g. /src/pkg-b/../pkg-a and /src/pkg-a are the same, but pip will see a conflict).

I could not see an option with pip-deepfreeze to allow a non-editable installation. Could this be added behind a flag?

Add `lock` command when the target pip supports `pip install --dry-run --report`

Add --verbose option

To control debug level logging

Option to uninstall unneeded dependencies

Do not use pip install --constraints

Pip constraints have limitations, especially with the new resolver.

Error when adding extras to an already installed project

To reproduce, create a project with a setup.py and pip-df sync it.
Then add and extras_require with an extra named, say, dev.
Then run pip-df sync -e dev. It complains that dev is not an extra of the project.

To make it work, you need to pip uninstall the project then pip-df sync again.

Virtualenv detection may be too strict

From the FAQ: pip-deepfreeze erroneously complains python is not running in a virtualenv.

The most probable cause is that you used an older version of virtualenv which does not generate PEP 405 compliant virtual environments. virtualenv version 20 and later are supported, as well as the Python 3 native venv module. Should this problem be prevalent in practice, we may add support for older virtualenv versions, or add an option to ignore the virtualenv sanity check (which is only there to prevent pip-deepfreeze to corrupt the system Python packages by accident).