pyodide/pyodide-lock

Tooling to manage the `pyodide-lock.json` file

Pyodide is a Python distribution for the browser and Node.js based on WebAssembly.

What is Pyodide?

Pyodide is a port of CPython to WebAssembly/Emscripten.

Pyodide makes it possible to install and run Python packages in the browser with micropip. Any pure Python package with a wheel available on PyPi is supported. Many packages with C extensions have also been ported for use with Pyodide. These include many general-purpose packages such as regex, PyYAML, lxml and scientific Python packages including NumPy, pandas, SciPy, Matplotlib, and scikit-learn.

Pyodide comes with a robust Javascript ⟺ Python foreign function interface so that you can freely mix these two languages in your code with minimal friction. This includes full support for error handling, async/await, and much more.

When used inside a browser, Python has full access to the Web APIs.

Try Pyodide (no installation needed)

Try Pyodide in a REPL directly in your browser. For further information, see the documentation.

Getting Started

Pyodide offers three different ways to get started depending on your needs and technical resources. These include:

• Use a hosted distribution of Pyodide: see the Getting Started documentation.
• Download a version of Pyodide from the releases page and serve it with a web server.
• Build Pyodide from source
  • Build natively with make: primarily for Linux users who want to experiment or contribute back to the project.
  • Use a Docker image: recommended for Windows and macOS users and for Linux users who prefer a Debian-based Docker image with the dependencies already installed.

History

Pyodide was created in 2018 by Michael Droettboom at Mozilla as part of the Iodide project. Iodide is an experimental web-based notebook environment for literate scientific computing and communication.

Iodide is no longer maintained. If you want to use Pyodide in an interactive client-side notebook, see Pyodide notebook environments.

Contributing

Please view the contributing guide for tips on filing issues, making changes, and submitting pull requests. Pyodide is an independent and community-driven open-source project. The decision-making process is outlined in the Project governance.

Communication

• Blog: blog.pyodide.org
• Mailing list: mail.python.org/mailman3/lists/pyodide.python.org/
• Twitter: twitter.com/pyodide
• Stack Overflow: stackoverflow.com/questions/tagged/pyodide
• Discord: Pyodide Discord

License

Pyodide uses the Mozilla Public License Version 2.0.

pyodide-lock's Issues

RFC 2: partial lockfiles

As proposed by @bollwyvl #4 (comment)

The way the in-flight jupyterlite PR works is by:

• generating partial lockfiles
  • these wouldn't be sufficient to run anything
• layering them, by name, on top of the as-loaded runtime with whatever index is hosted with it
  • this could be hoisted/normalized in pyodide's initializer with e.g. extraLockUrls: []
    • this could be non-destructive, which might be better, and potentially support "punch-out" by setting a name to a null (ick!)

The concrete things this solves there:

• auto-install-on-import
• replacing packages in the pyodide stdlib
  • IPython has an optional dependency on jedi/parso (but still downloads them)
    • we haven't been able to make jedi work well enough in jupyterlite to use, so would rather replace it with dummy shims to save a couple megabytes on the wire
  • per-PR docs builds of packages that are in the pyodide stdlib and really just want one different

Alternative proposals: #8 #10

RFC 3: using a dependency solver

Here the idea is to use some standard Python dependency solver for creating the pyodide-lock.json from multiple sources (Pyodide distribution, PyPI, third party sources etc).

For instance it could look as follows,

1. Expose a Simple JSON API for the Pyodide distribution (so that Python package managers could understand it). This should already be done, but I can't find where (cc @ryanking13 )
2. Define a list of indexes with a priority . Same idea as --extra-index-url in pip
3. Run a standard Pythopn dependency solver on a list of input requirements, specifying emscripten*_wasm32 as the platform (or whatever it is now). Here the best tool remains to be determined,
   • using solvers from pip directly as pip-tools is doing.
   • https://github.com/sarugaku/resolvelib
   • others ?
4. Reformat the results in the pyodide-lock.json format (by taking information either from the original pyodide-lock.json)

Alternative proposals in #8 #9

Make cli run under pyodide-cli?

It seems weird to have the CLI for this run with pyodide-lock not as a subcommand of pyodide, e.g. pyodide lock or pyodide lockfile? pyodide-lock has minimal dependencies (typer and rich)?

Generate JSON Schema

In pyodide/pyodide#3573 @bollwyvl requested that it would be useful to generate a JSON Schema for pyodide-lock.json.

We can do that from the current Pydantic spec, though the question is where to put it (keeping in mind that this is a Python package).

@bollwyvl could you please explain more what would work for you?

pyodide lockfile add-wheels does not append files

Hello,
I tried today to use this command to add new wheels to an existing pyodide-lock.json

pyodide lockfile add-wheels

Unfortunately it did not append any dependencies to an existing file. After some digging I figured out that the error is in this file:

https://github.com/pyodide/pyodide-lock/blob/main/pyodide_lock/cli.py#L50

The following code fixes it:

sp = PyodideLockSpec.from_json(input)
sp = add_wheels_to_spec(
        sp,
        wheels,
        base_path=base_path,
        base_url=wheel_url,
        ignore_missing_dependencies=ignore_missing_dependencies,
    )
    sp.to_json(output)

Pin to pydantic <2

Elevator Pitch

Update pyproject.toml to use pydantic >=1.10.2,<2.

Motivation

As of 0.24.1, pyodide-build pins pydantic>=1.10.2,<2 and pyodide itself still ships 1.10.x.

For the time being, this repo should likely do the same, but appears to already use some pydantic 2 features.

Design CLI API

We need to design the CLI API for this package.

In pyodide/pyodide#3573 (comment) @bollwyvl proposed,

$> pyodide-index path/to/wheels/folder
Wrote 200 packages to path/to/wheels/folder/repodata.json

and I agree this is the direction. Though given the current name of this package, it would also be more logical to call it pyodide lock IMO.

Also we need to keep in mind that the resulting lockfile would need to include information about the unvendored stdlib modules (and Python version). So it needs to have access to the original pyodide-lock.json (either via the pyodide version and looking at the CDN or by providing a path/URL to it). The difference with respect to conda index producing the repodata.json is that there,

• they can have multiple Python versions
• as far as I understand, they don't care that the included files may have some conflicts. The dependency resolution would really happen only at runtime when running conda install and combining indexes from different channels.. While in our case, by design once pyodide-lock.json is generated the dependency resolution is already done. So any package would be guaranteed to be installable with trivial dependency resolution (there is a single version for each package, and we can just ignore versions).

There are two use cases,

1. Adding/updating packages with actual files being stored on some remote CDN. In this case, extra entries in the pyodide-lock.json don't matter, since they would only load if explicitly imported, and we don't necessarily need to download all the included files locally IMO. Here I was thinking of taking something like a requirements.in as input (as in pip-tools) which would compute a consistent dependency graph merging the original pyodide-lock.json with the requirements in requirements.in and combine both (not easy).
2. Including only a subset of packages for a given application, and ship them alongside pyodide-lock.json for reproducibility. This is closer to the use case of https://github.com/pyodide/pyodide-pack: BTW I'm changing a bit the focus of that package away from rather an experimental module stripping via runtime detection to any kind of package/wheel minification. So the wheel files would be modified by that tool, but the end pyodide-lock.json would be still generated by this project.
   The challenge with this use case is that even given a list of wheels in some folder, we still need to verify that there are no missing requirements and that the dependency graph is consistent, so we need a resolver that would understand the wasm platform for finding compatible wheels.

Anyway, it's still early day, this needs more discussion. My current idea is to iterate on the implementation that would work well in practice for these use cases, while only pushing alpha releases to PyPI. Any API in this package is considered unstable and can be completely changed.

Please let me know if you have any other ideas about how this should work.

@hoodmane @ryanking13

Move test file unvendoring functionality from pyodide-build to pyodide-lock

When creating a pyodide-lock file, we separate test codes inside the wheel and pack them in a separate file to reduce the size of the wheel. This feature is highly related to the lockfile, so I think it would be nice to move it to pyodide-lock so users can use it when creating a custom lockfile using out-of-tree wheels.

On the other hand, if we move this functionality to pyodide-lock, we should support some ways to specify which packages should have their test files split and which should not. We were using unvendor_tests in the meta.yaml file, but the recipe file will not be available for out-of-tree builds.

So, probably we would need some cmdline flags such as:

--vendor-tests pkg-name pkg-name2

and also make it configurable using a config file:

[tool.pyodide.lock]
vendor-tests = ['pkg-name', 'pkg-name2']

This can also be extended to general unvendoring mechanism not only for test files as discussed in pyodide/pyodide#3092.

RFC 1: micropip.freeze() → `pyodide-lock.json`

First very simple option for creating a pyodide-lock.json mixing Pyodide packages and PyPI packages consists of,

1. install them with micropip,
2. call micropip.freeze() to get the updated `pyodide-lock.json (cf pyodide/micropip#68).
3. and then to make install on import work, parse the added wheel files and add the expected import names via #7

It can be done directly in the browser, but it's probably not optimal. The other solution is to do it in node (but this means a dependency on node).

The ideal case could be to run it native Python with micropip installed. I think it might be possible if we use the dependency resolution part but not the install part of micropip, but someone would need to try.

Alternative proposals: #9 #10

Move parse_top_level_import_name from Pyodide build

When creating the pyodide-lock.json we would need to be able to identify import names for a given pure Python wheel.
So it would be useful to move the parse_top_level_import_name function + the associated tests from Pyodide build here.

Then at a later stage use it in pyodide-build imported from this package.

0.1.0a2 not deployed to PyPI

There is a v0.1.0a2 tag but it seems like it is not deployed to PyPI.