emscripten-forge/empack

Tools to pack a conda / mamba environment into a JS & WASM bundle

empack

Pack a mamba environment with emsdk

Installation

mamba install empack -c conda-forge

or with pip:

pip install empack

Usage

You can pack the environment (located at /path/to/env) with the following command:

empack pack env --env-prefix /path/to/env

This will generate a few .tag.gz files that correspond to the packages in the environment located at /path/to/env.

You can also provide a custom config with the --config flag. A sample config is located in tests/empack_test_config.yaml

Run the following command for more information about the other CLI parameters:

empack pack env --help

empack's Issues

Add ability to relocate the root prefix of the packed env

Description

When packing an environment, the in-memory Emscripten file-system becomes the same as the real file system from which it was generated, with the same root prefix. e.g. the files generated in the CI of xeus-python will have the root-prefix /home/runner/micromamba-root/envs/xeus-python-wasm-host

It would be nice to be able to configure the root prefix (maybe with a simple search and replace in the generated files?).

Provide a more convenient way to copy the runtime files

• empack version: 3.0.0
• Operating System: Fedora 37
• Compiler: N/A

Description

When packing an environment for the web one of the required steps is to copy the runtime files (for example pyjs_runtime_browser.js) from $PREFIX/lib_js/pyjs. For example with:

cp ~/micromamba/envs/pyjs-web-env/lib_js/pyjs/pyjs_runtime_browser.* build/

Wondering if there could be a way to make that step more convenient. Maybe it could directly be done as part of the empack pack env command via a new (optional) CLI flag?

What I Did

This came up while iterating a bit on https://github.com/emscripten-forge/sample-python-repl.

Default config

Description

The empack pack command requires to provide a config, we should instead use the empack config we provide by default.

Does not work with Pydantic 2.0

Initially opened by @stephenmccalman on jupyterlite/xeus-python-demo#24

pydantic is not properly pinned, resulting in

failed to load: To define root models, use `pydantic.RootModel` rather than a field called '__root__'

This breaks xeus-python-kernel and emscripten-forge recipes

Hard pin pydantic

Pydantic has just been released and is breaking again. See jupyterlite/xeus-python-demo#31

We should not only pin the major version but also pin pydantic and pydantic-core minor versions.

move empack config from emscripten-forge/recipes repo to emscripten-forge/empack repo

It is confusing that the empack config lives in the recipes repo (ie here) rather then the empack repo (this repo).

Lets move it!

Add a bit of CI

Description

A bit of CI (Just installing and packing an environment for testing) could help

`importlib.metadata.version(<package>)` returns only NoneType

If I try to run

> import scipy
> import importlib.metadata
> type(importlib.metadata.version("scipy"))

in a JupyterLite environment built with

• jupyter_server=2.7.1
• jupyterlite-core=0.1.2
• jupyterlite-xeus-python=0.9.4

I always get the return NoneType.

This affects our attempt to run Brightway in JupyterLite, since some of our packages use a function that importlib.metadata.version("bw-processing"):

Detailed Error Message

import bw2io as bi
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[6], line 1
----> 1 import bw2io as bi

File /lib/python3.10/site-packages/bw2io/__init__.py:52
     49 from .version import version as __version__
     51 from .chemidplus import ChemIDPlus
---> 52 from .package import BW2Package
     53 from .export import (
     54     DatabaseToGEXF,
     55     DatabaseSelectionToGEXF,
   (...)
     58     lci_matrices_to_matlab,
     59 )
     60 from .backup import (
     61     backup_data_directory,
     62     backup_project_directory,
     63     restore_project_directory,
     64 )

File /lib/python3.10/site-packages/bw2io/package.py:5
      2 import warnings
      3 from time import time
----> 5 from bw2data import projects
      6 from bw2data.logs import get_logger
      7 from bw2data.serialization import JsonSanitizer, JsonWrapper

File /lib/python3.10/site-packages/bw2data/__init__.py:37
     34 from .version import version as __version__
     36 from .configuration import config
---> 37 from .project import projects
     38 from .utils import set_data_dir
     39 from .meta import (
     40     dynamic_calculation_setups,
     41     calculation_setups,
   (...)
     47     weightings,
     48 )

File /lib/python3.10/site-packages/bw2data/project.py:8
      5 from collections.abc import Iterable
      6 from pathlib import Path
----> 8 from bw_processing import safe_filename
      9 from peewee import BooleanField, DoesNotExist, Model, TextField, SQL
     10 from platformdirs import PlatformDirs

File /lib/python3.10/site-packages/bw_processing/__init__.py:48
     45 from .unique_fields import as_unique_attributes, as_unique_attributes_dataframe
     47 from .utils import get_version_tuple
---> 48 __version__ = get_version_tuple()

File /lib/python3.10/site-packages/bw_processing/utils.py:26, in get_version_tuple()
     20     except ValueError:
     21         return version_str
     23 return tuple(
     24     as_integer(v)
     25     for v in importlib.metadata.version("bw-processing")
---> 26     .strip()
     27     .split(".")
     28 )

AttributeError: 'NoneType' object has no attribute 'strip'

This works fine in the 0.23.4 version of Pyodide:

A WebAssembly-powered Python kernel backed by Pyodide
> import scipy
> import importlib.metadata
> importlib.metadata.version("scipy")
'1.9.3'

This was discussed in:

• emscripten-forge/recipes#616 (comment)

Filter out pip in packed env

Description

We should probably filter out any file under the pip site-package directory. As we won't use anything from there?

empack does not work with Python <3.10

With Python <3.10 we are getting:

TypeError: unsupported operand type(s) for |: 'ModelMetaclass' and 'ModelMetaclass'

Empack 4.0.0 packing issue with jupytercad

The lite build on JupyterCAD fails with the following. This is most likely related to the latest changes on filtering:

[LiteBuildApp] ERROR | [lite] [post_build] [jupyterlite-xeus] [ERR] Object of type PosixPath is not JSON serializable
Traceback (most recent call last):
  File "/home/runner/micromamba/envs/build-env/lib/python3.10/site-packages/doit/doit_cmd.py", line 294, in run
    return command.parse_execute(args)
  File "/home/runner/micromamba/envs/build-env/lib/python3.10/site-packages/doit/cmd_base.py", line 150, in parse_execute
    return self.execute(params, args)
  File "/home/runner/micromamba/envs/build-env/lib/python3.10/site-packages/doit/cmd_base.py", line 570, in execute
    return self._execute(**exec_params)
  File "/home/runner/micromamba/envs/build-env/lib/python3.10/site-packages/doit/cmd_run.py", line 265, in _execute
    return runner.run_all(self.control.task_dispatcher())
  File "/home/runner/micromamba/envs/build-env/lib/python3.10/site-packages/doit/runner.py", line 254, in run_all
    self.run_tasks(task_dispatcher)
  File "/home/runner/micromamba/envs/build-env/lib/python3.10/site-packages/doit/runner.py", line 213, in run_tasks
    node = task_dispatcher.generator.send(node)
  File "/home/runner/micromamba/envs/build-env/lib/python3.10/site-packages/doit/control.py", line 629, in _dispatcher_generator
    next_step = node.step()
  File "/home/runner/micromamba/envs/build-env/lib/python3.10/site-packages/doit/control.py", line 336, in step
    return next(self.generator)
  File "/home/runner/micromamba/envs/build-env/lib/python3.10/site-packages/doit/control.py", line 345, in _func
    for value in decorated(*args, **kwargs):
  File "/home/runner/micromamba/envs/build-env/lib/python3.10/site-packages/doit/control.py", line 473, in _add_task
    new_tasks = generate_tasks(to_load, task_gen, ref.__doc__)
  File "/home/runner/micromamba/envs/build-env/lib/python3.10/site-packages/doit/loader.py", line 390, in generate_tasks
    for task_dict, x_doc in flat_generator(gen_result, gen_doc):
  File "/home/runner/micromamba/envs/build-env/lib/python3.10/site-packages/doit/loader.py", line 27, in flat_generator
    for item in gen:
  File "/home/runner/micromamba/envs/build-env/lib/python3.10/site-packages/jupyterlite_core/manager.py", line 138, in _delayed_gather
    yield from _gather()
  File "/home/runner/micromamba/envs/build-env/lib/python3.10/site-packages/jupyterlite_core/manager.py", line 131, in _gather
    raise error
  File "/home/runner/micromamba/envs/build-env/lib/python3.10/site-packages/jupyterlite_core/manager.py", line 123, in _gather
    for task in getattr(addon, attr)(self):
  File "/home/runner/micromamba/envs/build-env/lib/python3.10/site-packages/jupyterlite_xeus/add_on.py", line 127, in post_build
    yield from self.copy_kernels_from_prefix()
  File "/home/runner/micromamba/envs/build-env/lib/python3.10/site-packages/jupyterlite_xeus/add_on.py", line 179, in copy_kernels_from_prefix
    yield from self.copy_kernel(kernel_dir, kernel_wasm, kernel_js)
  File "/home/runner/micromamba/envs/build-env/lib/python3.10/site-packages/jupyterlite_xeus/add_on.py", line 261, in copy_kernel
    yield from self.pack_prefix(kernel_dir=kernel_dir)
  File "/home/runner/micromamba/envs/build-env/lib/python3.10/site-packages/jupyterlite_xeus/add_on.py", line 288, in pack_prefix
    pack_env(
  File "/home/runner/micromamba/envs/build-env/lib/python3.10/site-packages/empack/pack.py", line 186, in pack_env
    pack_pkg_impl(
  File "/home/runner/micromamba/envs/build-env/lib/python3.10/site-packages/empack/pack.py", line 54, in pack_pkg_impl
    json.dump(pkg_meta, f)
  File "/home/runner/micromamba/envs/build-env/lib/python3.10/json/__init__.py", line 179, in dump
    for chunk in iterable:
  File "/home/runner/micromamba/envs/build-env/lib/python3.10/json/encoder.py", line 431, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/home/runner/micromamba/envs/build-env/lib/python3.10/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/home/runner/micromamba/envs/build-env/lib/python3.10/json/encoder.py", line 325, in _iterencode_list
    yield from chunks
  File "/home/runner/micromamba/envs/build-env/lib/python3.10/json/encoder.py", line 438, in _iterencode
    o = _default(o)
  File "/home/runner/micromamba/envs/build-env/lib/python3.10/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type PosixPath is not JSON serializable
static:jupyter-lite.json
.  pre_status:static:jupyter-lite.json
    tarball:         jupyterlite-app-0.2.3.tgz 9MB
    output:          /home/runner/work/JupyterCAD/JupyterCAD/lite/dist
    lite dir:        /home/runner/work/JupyterCAD/JupyterCAD/lite
    apps:            
    sourcemaps:      True
    unused packages: True

Cache emsdk across environments

Description

empack currently downloads emsdk and puts it under lib/python3.X/site-packages/empack/emsdk-Y/ https://github.com/emscripten-forge/empack/blob/main/empack/file_packager.py#L36.

This means that emsdk is not "cached" across environments, it's only cached for that specific empack installation.

Instead we should download emsdk under a directory outside of the prefix e.g. $HOME/.empack-emsdk/ or /.empack-emsdk/.

Can't run `pack` due to Pydantic Error: `RootModel` does not support setting `model_config['extra']`

• empack version: 3.1.0
• Operating System: ubuntu 22.04
• Compiler: emscripten

Description

Packing an environment in a GitHub action fails due to a Pydantic error from

empack/empack/file_patterns.py

Line 33 in d6ff060

class FilePattern(RootModel, extra="forbid"):

The error is:

pydantic.errors.PydanticUserError: `RootModel` does not support setting `model_config['extra']`

Traceback

Traceback (most recent call last):
  File "/opt/conda/bin/empack", line 7, in 
    from empack.cli.main import app
  File "/opt/conda/lib/python3.9/site-packages/empack/cli/main.py", line 2, in 
    from .pack import *  # noqa: F403
  File "/opt/conda/lib/python3.9/site-packages/empack/cli/pack.py", line 6, in 
    from empack.file_patterns import pkg_file_filter_from_yaml
  File "/opt/conda/lib/python3.9/site-packages/empack/file_patterns.py", line 33, in 
    class FilePattern(RootModel, extra="forbid"):
  File "/opt/conda/lib/python3.9/site-packages/pydantic/_internal/_model_construction.py", line [12](https://github.com/qutip/qutip-virtual-lab/actions/runs/5914370171/job/16039628729#step:9:13)3, in __new__
    cls: type[BaseModel] = super().__new__(mcs, cls_name, bases, namespace, **kwargs)  # type: ignore
  File "/opt/conda/lib/python3.9/abc.py", line 106, in __new__
    cls = super().__new__(mcls, name, bases, namespace, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/pydantic/root_model.py", line 50, in __init_subclass__
    raise PydanticUserError(
pydantic.errors.PydanticUserError: `RootModel` does not support setting `model_config['extra']`

What I Did

You can see the failed CI run log here and the workflow config file here, but in summary, I tried running

        empack pack env \
        --env-prefix=/opt/conda/envs/web \
        --relocate-prefix=/ \
        --no-use-cache \
        --outdir=./public/empack \
        --config=empack_config.yaml

Oddly enough, this was working just 3 days prior (without any changes to the workflow file) as you can see in this successful run. I tried pinning the empack version to 3.1.0 but that didn't help.

I don't know too much about Pydantic, but willing to submit a patch if it's an easy fix.