conda-incubator/conda-pypi

Better PyPI interoperability for the conda ecosystem

conda-pypi

Better PyPI interoperability for the conda ecosystem.

Important

This project is still in early stages of development. Don't use it in production (yet). We do welcome feedback on what the expected behaviour should have been if something doesn't work!

What is this?

Includes:

• conda pip: A subcommand that wraps pip to make it work with conda in a better way.
• Adds EXTERNALLY-MANAGED to your environments.

Why?

Mixing conda and PyPI is often discouraged in the conda ecosystem. There are only a handful patterns that are safe to run. This tool aims to provide a safer way of keeping your conda environments functional while mixing it with PyPI dependencies. Refer to the documentation for more details.

Contributing

Please refer to CONTRIBUTING.md.

conda-pypi's Issues

Investigate using `pip install --dry-run` + some mappings

As an alternative to grayskull, we could try with a naive pip solve first (or pip-tools), mapping things as necessary, and if that doesn't work then rely on grayskull.

Release in conda-forge

See conda-forge/staged-recipes#26650

Use parselmouth's mapping

https://github.com/prefix-dev/parselmouth

Add `uv` based solver backend

Should be similar to the current pip backend

Integrate with `- pip:` section in `environment.yml` files in `conda env`

It would be nice to have conda env handle the PyPI dependencies via our integrations for safer installs. This would be done via an opt-in plugin implemented in still non-existing hooks. Added conda/conda#13833 to discuss that in conda/conda.

Isolate externally-managed installation logic

This needs to be executed as part of the conda recipe, so it would be nice if the import path doesn't need conda and friends (importlib.resources is enough).

That said, I should also look into whether some kind of python packaging configuration allows to indicate that EXTERNALLY-MANAGED should be placed in the parent of site-packages. This is different for every Python version, and also across Unix/Windows. This way we don't need custom code being run at build time.

Migrate to conda-incubator?

Looking at the progress of the project (and in preparation for next week's PyCon US) it might make sense to commit to this project a little more by moving it to the conda-incubator.

Uncouple from `pip` a bit more

There are libraries we can use for better integration instead of relying so much on pip subprocesses. This could be part of a refactor where we isolate the components of the package a bit better.

Right now we have, roughly:

• A CLI that mimics a subset of pip's UX with a couple extra flags
• A PyPI solver backend based on: grayskull, pip; uv is planned too
• A conda/pypi mapper based on several public resources (cf-graph-countyfair, grayskull, parselmouth)
• A PyPI installer that essentially builds the command to be passed to pip. This could be reimplemented using PyPA's build and installer so conda can provide the build-system.requirements (which means we can offer compilers). This would also allow us to inject the right kind of metadata in the *.dist-info directories, in the right way.

cc @dholth

xref:

• #36 (comment)
• #35 (comment)
• conda/conda-build#5380

Amazing!

Hey, this is not an issue, I just wanted to say: thank you, what a great project!! 😍😍😍

Handle `python` upgrades

When an environment changes python versions (e.g. from 3.9 to 3.10), the conda-pip installed packages will have to be reinstalled for the new version too. Usually, conda does that automatically as part of the solve and noarch: python handling, but since right now we are not injecting that metadata, then they will be missed.

I see two solutions:

• Enhance the post-command hook so we uninstall and reinstall the relevant PyPI packages (we are not keeping track right now!)
• Convert all PyPI wheels to conda packages (maybe via conda2whl?) so this is handled by conda proper. However, this is only feasible for noarch-y packages (py_any). Python specific builds will still need to happen, which might result in solver conflicts unless we anticipate the need for new whl2conda conversions... catch 22!

Maybe the solution is a hybrid approach, but for now I think the first option is more robust.

Implement pip's editable installation method for condaverse

I wonder if, following the lack of resolution of the develop subcommand in conda-build, it would be worth implementing something similar using the capabilities of conda-pypi, basically providing a PyPI aware conda develop command.

Absorb PyPI details in `conda list`

A lot of code in conda exists to process PyPI records for conda list. This was meant to be part of the pip_interop_enabled features that were never finished. I think we can absorb those classes here and extend them as needed without having to be hindered by the backwards compatibility policies of conda. We would deprecate all of that and instruct users to install conda-pypi if they need this functionality.

This will need several items:

• Add conda list hooks to "inject" more entries in the list before is finally printed
• Deprecate code that supports pip_interop_enabled and absorb it here
• Deprecate pip_interop_enabled key. Tell users to install conda-pypi
• See how this might affect the classic solver, if at all

Install EXTERNALLY-MANAGED

Put this file in its expected place with a message that says:

• This environment is protected against naive pip installs
• To use 'conda pip' instead for extra safety, or --break-system-packages
• Where to find more information about the rationale

Add a few lines to the readme/docs explaining why folks would want to mix conda and pip/pypi.

The "Why" section only says this is something that is discouraged and that this package gives you a safe way to do it. It does explain the reasons why someone would want to mix the ecosystems. i.e. conda has strong guarantees around binary compatibility but pip/pypi has a long tail of packages that is an order of magnitude higher that the core scientific/data/ai packages that are in the conda ecosystem.

Convert to `.conda` as part of the `pip install`?

Wondering if there's a net positive in using whl2conda for this. The rough logic would be to:

1. Run the dependency analysis to get the conda and PyPI packages as we are doing now.
2. Instead of running pip install --no-deps ... directly, convert to .conda beforehand with whl2conda and then install that with conda --no-deps.
3. We would cache these converted wheels in the pkgs cache as usual.

Benefits:

• More integration in the conda environment (proper conda-meta/ files, clobbering detection, taken into account in the solves, etc).
• A mature project that has gone through the usual edge cases already so it should be more robust than this early attempt

Cons:

• One more dependency
• Delegated complexity to a 3rd party
• We could be introducing problems in the solves

Add an option to reject PyPI packages with compiled objects

By default we will only allow pure Python wheels. With a flag, we can allow people to opt-in and install things with compiled bits.

Integrate with explicit lockfiles

conda list --explicit --md5 will produce same-platform lockfiles that can be installed somewhere else via conda install --file .... These are good enough for many cases, but it doesn't handle PyPI dependencies.

I suggest:

• We add a post_command plugin for list and add extra lines with # pypi: url-to-wheel.
• Add a post-command hook to post process just the input files (not the CLI specs) and look for # pypi: url lines.

Package for PyPI

Investigate how to package this correctly (including EXTERNALLY-MANAGED in the right place, essentially) for PyPI.

Prevent clobbering by default

Analyze content of pip packages and calculate whether the installation would override things in the target environment.

Add docs about installation, usage and expected behavior

We don't have much yet :)

RFC: Rename to `conda-pypi`?

I'm not sure conda-pip is the right name for the project and package. We are implementing tool agnostic behavior with EXTERNALLY-MANAGED and while we do have the conda pip subcommand, chances are we will also have conda uv in the future (or similar). Hence I think conda-pypi might be a better name for the project, which happens to implement the pip subcommand for now.

Thoughts? This a simple change to do now but it might get a bit more complicated later when things are packaged and folks start adding this to requirements, etc.

Solve before running grayskull

We'll probably need to run a pip solve before running grayskull to get which version we can actually install, and then process that version with grayskull to obtain their deps.

Improve "conda availability" assertions with index queries

Grayskull's logic to determine whether a package is available on conda or not is based purely on the existence of the package name in anaconda.org/{channel}.

There's no concept of index, multiple channels or spec compatibility. To ensure that we try to install conda packages that indeed exists, we need to check whether the specs calculated by grayskull result in any records being found in the computed index (e.g. a recent version required by some pip package might not be available on conda-forge yet, in that case we use pip's).

Add CI for Windows

There's an installation issue on Windows that affects conda's ability load the libmamba libraries for the solver:

WARNING  conda.plugins.manager:manager.py:134 Error while loading conda entry point: conda-libmamba-solver (The specified procedure could not be found)

Found this issue: mamba-org/mamba#1728, but it's not very useful. This is normally a result of mismatched ABIs when mixing channels, but we are only using conda-forge here 🤷 I also tried downgrading to see if it was a 1.5.6 issue, but same error. I'll investigate in a subsequent PR. Disabling for now.

Originally posted by @jaimergp in #8 (comment)