bytecodealliance/componentize-py

componentize-py's Issues

Use `wasi-virt`-based VFS to store Python code

Currently, componentize-py "cheats" by assuming all required Python code will be loaded during the component pre-init step, and thus CPython won't need access to any of those files at runtime. This works surprisingly well, but will certainly break for more dynamic apps.

We should probably build a VFS by default, but also offer an option to use the "cheat" if desired.

pyo3_runtime.PanicException: internal error: entered unreachable code

I ran into the following issue:

thread '<unnamed>' panicked at 'internal error: entered unreachable code', src/lib.rs:450:35
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
  File "/Users/eliabieri/Library/Caches/pypoetry/virtualenvs/widget-4qY9vehV-py3.11/bin/componentize-py", line 8, in <module>
    sys.exit(script())
             ^^^^^^^^
pyo3_runtime.PanicException: internal error: entered unreachable code

Steps to reproduce

Prerequisites: Poetry

1. Clone repo
   git clone --recurse-submodules [email protected]:eliabieri/wg_display_widget_py.git
2. Install dependencies
   poetry install
3. Generate bindings
   poetry run componentize-py --wit-path wg_display_widget_wit/wit --world widget bindings .
4. Generate component
   poetry run componentize-py --wit-path wg_display_widget_wit/wit --world widget componentize --output widget.wasm --python-path widget widget

Expand CI test coverage

We should run all the tests on Mac and Windows -- not just Linux. Also, we should test the examples to ensure they keep working.

feature: add zlib to cpython build

Some packages (in particular, aiohttp) require CPython to be built with zlib enabled. I believe this is known work but I wanted to create a tracking issue here for it. (I took a stab at this via building in wasi-wheels but bounced off the problem 😅)

Download dependencies automatically

Per yesterday's Python guest tooling meeting, we'd like componentize-py to download any dependencies found in the application's pyproject.toml (including transitive dependencies), placing them in a temporary directory for use while building the component.

Similar to micropip, componentize-py should grab any pure Python packages from PyPI, only looking to an alternative repo for packages with native extensions. We (i.e. the Bytecode Alliance) will need to maintain this repo and populate it with WASI builds of popular projects until WASI wheels are standardized and published upstream to PyPI.

To start with, componentize-py could shell out to e.g. pip --install --platform wasi-sdk-22 --abi cp311 --root <tmpdir>, but eventually we'd like to make it fully self-contained (i.e. not require Python or pip to be installed).

List errors which might be raised in function docstrings

Python does not currently have a way to represent the types of errors/exceptions which might be raised by a function in a typed function signature, so the next best thing is to list them in the docstring (e.g. following any docs derived from WIT doc comments).

Running component without WASI

Context

I have a Rust project that loads and executes wasm components using wasmtime.
The wasmtime usage can be seen here.

If I try to instantiate a component generated using componentize-py, I get an error concerning a type mismatch with a WASI type

import `wasi:clocks/wall-clock` has the wrong type

Questions

• Am I forced to use WASI when I want to run a wasm component generated by componentiue-py?
  • If yes, how would I configure wasmtime?

Apologies if these questions should be clear from the documentation. I was just not able to figure these out, even after trying for some time.

Links

• wasm component binary widget.wasm.zip
• wasm component project

Running example

I'm looking to get the example in the readme working. I have the component wasm compiled via this command:

$ componentize-py -d hello.wit -w hello componentize app -o app.wasm

Now I want to run this via wasmtime but that gives me this error:

$ wasmtime run --wasm component-model app.wasm                             
Error: failed to run main module `app.wasm`

Caused by:
    exported instance `wasi:cli/[email protected]` not present

I thought I could convert this to a core module with wasm-tools but that gives me a different error:

$ wasm-tools component new -v app.wasm -o out.wasm --adapt wasi_snapshot_preview1.reactor.wasm
error: failed to encode a component from module

Caused by:
    0: unknown table 2: exported table index out of bounds (at offset 0xb)

Ideally I'd like to take Python and covert it into wasm much like how it's done in Javy (https://github.com/bytecodealliance/javy) where the output of javy compile is code that I can execute with wasmtime <output>.wasm.

Thanks for any help or ideas! =)

re-init random seed(s) on resume

We're currently using component-init to snapshot the state of an initialized app, and the initialization process may involve getting random numbers from the host and using them to seed one or more PRNGs. That could pose a security risk if we don't force those PRNGs to be reseeded on resume, so we should find these cases in wasi-libc and/or CPython. This could also be an issue in Python code and/or native extensions, but it's not clear how we would address that; we may need to leave that up to the application programmer.

Add support for WIT resources

Implement support for WIT resources, including the canonical ABI requirements. For reference, see wit-bindgen's Rust and C implementations.

Make the types generated for WIT resources context managers

Per https://docs.python.org/3/library/stdtypes.html#typecontextmanager, the idiomatic way to represent resource types is as context managers, each with an __exit__ method that drops the resource. This will allow them to be used in with statements.

Add interface version support

Per https://github.com/WebAssembly/component-model/blob/main/design/mvp/WIT.md#package-declaration, if the WIT interface has an interface, it needs to be encoded in the component type identifier for that interface.

Remove init function and types from the final output

The pre-initialization process currently involves an exported anonymous interface and types (defined in init.wit). That should be stripped out of the final component since it's not part of the application's world.

Matrix-math example - ImportError...

Following the example of matrix-math fd5b49c I get the following error when running componentize-py -d ../../wit -w matrix-math componentize app -o matrix-math.wasm:

warning: site-packages directory not found under /home/jes/gitrepositories/wasmcloud_test/my-py-component-wasmtime16-0-0/myenv/lib
Traceback (most recent call last):
  File "/home/jes/.local/bin/componentize-py", line 8, in <module>
    sys.exit(script())
             ^^^^^^^^
AssertionError: Traceback (most recent call last):
  File "/0/app.py", line 5, in <module>
    import numpy
  File "/0/numpy/__init__.py", line 157, in <module>
    from . import random
  File "/0/numpy/random/__init__.py", line 180, in <module>
    from . import _pickle
  File "/0/numpy/random/_pickle.py", line 1, in <module>
    from .mtrand import RandomState
ImportError: dynamic module does not define module export function (PyInit_mtrand)


Caused by:
    ImportError: dynamic module does not define module export function (PyInit_mtrand)

I am using python 3.12 and have created an enviroment using virtualenv
I am more experienced in Rust than python and have been unable to figure out if it is an error on my side or with the example.

Generated component has dependency on wasi:cli/[email protected]

I'm attempting to port the CompontenizeJS example to compotentize-py.

package local:hello;

world hello {
  export hello: func(name: string) -> string;
}

componentize-py -d hello.wit -w hello bindings .

import hello

class Hello(hello.Hello):
    def hello(self, name) -> str:
        return f"Hello, {name}, from Python!"

Trying to run the component fails

componentize-py -d hello.wit -w hello componentize app -o hello.component.wasm
cargo build --release
./target/release/wasmtime-test

Error: import `wasi:cli/[email protected]` has the wrong type

Caused by:
    0: instance export `get-environment` has the wrong type
    1: expected func found nothing
make: *** [py-hello] Error 1

time.sleep sleeps indefinitely

For example, when targeting the wasi:cli command world:

from hello import exports
from time import sleep

class Run(exports.Run):
    def run(self):
        print("Hello, world!")
        sleep(0.00001)

The above prints "Hello, world!" and then seems to sleep forever.

Run MyPy on all test code and bindings

In addition to running the test suite, we should dump the app code and generated bindings in a directory and run MyPy on them, asserting that no errors are found. This will help us catch any regressions in the type annotations,

Avoid using keywords as identifiers in generated bindings

Analogous to bytecodealliance/wasmtime-py#171, we should escape any identifier names which match Python keywords by appending underscores to them.

Add support for type aliases

Currently, componentize-py generates no code for type aliases; instead, it just uses the dealiased types directly. Instead, we should generate assignment expressions for each alias and use the alias in all the same places (e.g. function parameter types, etc.) that the original WIT files used them.

Access is denied

When running componentize-py 0.4.0 on Windows for the demo I get the following error:

AssertionError: Access is denied. (os error 5)

Any ideas on where this error is coming from? It's extremely vague and I'm not sure how to determine which part of the code is generating this error

Question: Current types limitation

Hello guys,

First, this is great work, congrats!

I have been playing around and created a custom (but simple) .wit, implemented in Python a function defined in that WIT file, built the .wasm module and then I tried to call the function from a Rust application.

I was able to provide a custom record type as parameter of the function and the component built succeeded (I mean, I generated the .wasm file correctly). However, using either the wasmer or wasmtime crates, the supported set of types you can use in the call function as Value/Val is very small. Ref: https://docs.rs/wasmtime/latest/wasmtime/enum.Val.html

So my question is, am I missing something or it is actually not possible to use a custom record right now as parameter or return type of a function?

I know this is probably not related to the componentize-py tool but a more general question, but I am not sure which would be the best place to ask.

Use interface package as part of module name in generated bindings

The current Python binding generator predates the concept of WIT packages, and we need to update it. Otherwise, importing or exporting two interfaces with the same name but different packages will result in a clash.

Add contributor documentation

This should include a CONTRIBUTING.md file, plus an ARCHITECTURE.md describing the high-level architecture of the project. In addition, each source file needs doc and implementation comments.

Optionally produce component which imports libpython311.so, libc.so, etc. instead of bundling them

This would dramatically reduce binary sizes in cases where the host can provide these libraries.

Improve type mismatch detection and reporting

Currently, if Python code passes the wrong number or types of parameters to the host, or returns a result of an unexpected type, componentize-py generally reacts by panicking and trapping. That's probably unavoidable in most cases, but it should at least print an informative and actionable diagnostic prior to panicking.