kateinoigakukun/wasi-vfs

A virtual filesystem layer for WASI.

wasi-vfs

Tip

Please use bytecodealliance/WASI-Virt if you are using WASI Preview2 with Component Model

A virtual filesystem layer for WASI.

NOTICE: This project currently supports only WASI applications on the top of wasi-libc

This project provides a language and host-agnostic virtual filesystem layer for WASI.

Supported filesystems

• Embedded file system: a read only file system embedded in the .wasm binary.
• to be implemented more...

Installation

Binary

A typical installation from the release binaries might look like the following:

$ export WASI_VFS_VERSION=0.5.2

# For x86_64 Linux host machine
$ curl -LO "https://github.com/kateinoigakukun/wasi-vfs/releases/download/v${WASI_VFS_VERSION}/wasi-vfs-cli-x86_64-unknown-linux-gnu.zip"
$ unzip wasi-vfs-cli-x86_64-unknown-linux-gnu.zip

# For x86_64 macOS host machine
$ curl -LO "https://github.com/kateinoigakukun/wasi-vfs/releases/download/v${WASI_VFS_VERSION}/wasi-vfs-cli-x86_64-apple-darwin.zip"
$ unzip wasi-vfs-cli-x86_64-apple-darwin.zip

# For arm64 macOS host machine
$ curl -LO "https://github.com/kateinoigakukun/wasi-vfs/releases/download/v${WASI_VFS_VERSION}/wasi-vfs-cli-aarch64-apple-darwin.zip"
$ unzip wasi-vfs-cli-aarch64-apple-darwin.zip

# See release page for more platforms: https://github.com/kateinoigakukun/wasi-vfs/releases

$ mv wasi-vfs /usr/local/bin/wasi-vfs

Homebrew

$ brew tap kateinoigakukun/wasi-vfs https://github.com/kateinoigakukun/wasi-vfs.git
$ brew install kateinoigakukun/wasi-vfs/wasi-vfs

Building

To build the project, you need to install the wasi-sdk version 16.0 or later.

$ git clone https://github.com/kateinoigakukun/wasi-vfs.git --recurse-submodules
$ cd wasi-vfs
$ export WASI_VERSION=20
$ export WASI_VERSION_FULL=${WASI_VERSION}.0

# For x86_64 Linux host machine
$ curl -LO https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-${WASI_VERSION}/wasi-sdk-${WASI_VERSION_FULL}-linux.tar.gz
$ tar xvf wasi-sdk-${WASI_VERSION_FULL}-linux.tar.gz

# For x86_64 and arm64 macOS host machine (since wasi-sdk doesn't provide prebuilt binaries for M1, so use x64 binaries on Rosetta)
$ curl -LO https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-${WASI_VERSION}/wasi-sdk-${WASI_VERSION_FULL}-macos.tar.gz
$ tar xvf wasi-sdk-${WASI_VERSION_FULL}-macos.tar.gz

$ export WASI_SDK_PATH=$PWD/wasi-sdk-${WASI_VERSION_FULL}
$ cargo build --target wasm32-unknown-unknown

Demo

Prerequisites

Set $WASI_SDK_PATH environment variable to the path of the wasi-sdk (version 14 or later).

Build and run WASI application

$ git clone https://github.com/kateinoigakukun/wasi-vfs.git --recurse-submodules
$ cd wasi-vfs

# Build libwasi_vfs.a
$ cargo build --target wasm32-unknown-unknown

# Build a WASI app with libwasi_vfs.a
$ $WASI_SDK_PATH/bin/clang -target wasm32-unknown-wasi -o getline.wasm examples/getline.c ./target/wasm32-unknown-unknown/debug/libwasi_vfs.a

# Run the WASI app with --mapdir
$ wasmtime run --mapdir /::./examples/mnt getline.wasm -- /hello.txt
Hello

# Pack ./examples/mnt directory into a WASM binary
$ cargo run -p wasi-vfs-cli -- pack getline.wasm --mapdir /::./examples/mnt -o getline.packed.wasm

# Run the WASM binary again without --mapdir
$ wasmtime run getline.packed.wasm -- /hello.txt
Hello

Testing

Unit tests

$ CARGO_TARGET_WASM32_WASI_RUNNER=wasmtime cargo test --target wasm32-wasi

End-to-end tests

$ cargo build --target wasm32-unknown-unknown
$ LIB_WASI_VFS_A=$PWD/target/wasm32-unknown-unknown/debug/libwasi_vfs.a ./tools/run-make-test.sh

How does it work?

wasi-vfs pack command is a wrapper of wizer, which is a pre-initializer for Wasm applications. The initialization process scans the mapped directories, then copies them into in-memory virtual filesystem.

Limitations

Currently, this project only supports WASI applications on the top of wasi-libc because of the following reasons:

This project depends on wasm-ld and wasi-libc's imported symbol behavior. wasi-libc declares some external symbols to import WASI functions in C like below. When __imported_wasi_snapshot_preview1_fd_read is not defined in any input object files, wasm-ld produces a (import "wasi_snapshot_preview1" "fd_read") entry. This is how wasi-libc calls WASI functions.

int32_t __imported_wasi_snapshot_preview1_fd_read(int32_t arg0, int32_t arg1, int32_t arg2, int32_t arg3) __attribute__((
    __import_module__("wasi_snapshot_preview1"),
    __import_name__("fd_read")
));

This project exploits that external symbols to hook WASI function calls by defining them in libwasi_vfs.a. If those symbols are defined, wasm-ld doesn't produce import entries, and it links symbols normally.

Therefore, this project currently doesn't support Rust application, which calls WASI functions directly without using wasi-libc.

After module-linking and interface-types will be merged, and WASI will adopt shared-nothing architecture, this project will be able to support all WASI applications.

wasi-vfs's Issues

Possibility of adding write capabilities

In theory, is it possible to add write support to the virtual file system? Why/Why not?

Thanks

How to use `wasi-vfs` with Python?

I'd like to use wasi-vfs with Python-Wasi, as I've done with Ruby, but it seems something has to be done with the .wasm file before wasi-vfs can add the files. How can I get started? Thank you.

CC: @nickvidal @bstrie

support arm

 => ERROR [11/11] RUN wasi-vfs pack ruby.wasm --mapdir /src::./src --mapdir /usr::./head-wasm32-unknown-wasi-full/usr -o my-ruby-app.wasm                            0.2s
------
 > [11/11] RUN wasi-vfs pack ruby.wasm --mapdir /src::./src --mapdir /usr::./head-wasm32-unknown-wasi-full/usr -o my-ruby-app.wasm:
#14 0.155 qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory

fixed it by using docker build --platform linux/x86_64 but would be nice if it just worked

make mac installs easy

I stopped after the sdk clone was taking >1GB to download

any of these would be nice:

• darwin binaries
• homebrew formula
• copy-pastable clone+build instructions

Preopens & limitations

Great package! Two short questions:

1. I understand the limitations in the readme should be covered now. Is there any other known limitations?
2. Is this approach compatible with preopens from wasi (e.g. pack with vfs, then share an additional directory at wasi init)?

Duplicate symbols when building with wasi-sdk 20

I was getting a crash when initializing a wasm module that used wasi-vfs. I was using wasi-sdk 20 and thought that maybe I should build wasi-vfs against that to make sure that wasn't the issue. However, after getting it built, I get the following errors when attempting to link libwasi_vfs.a into a module. The libc included with wasi-sdk includes the rintf and rint functions already. I was wondering if you had run across this before, or if this is possibly an addition to one of wasi-vfs' dependencies?

wasm-ld: error: duplicate symbol: rintf
>>> defined in /opt/lib/libwasi_vfs.a(compiler_builtins-d324d30d7b4a99a9.compiler_builtins.50f24778-cgu.60.rcgu.o)
>>> defined in /opt/wasi-sdk-20.0+threads/bin/../share/wasi-sysroot/lib/wasm32-wasi/libc.a(math-builtins.o)

wasm-ld: error: duplicate symbol: rint
>>> defined in /opt/lib/libwasi_vfs.a(compiler_builtins-d324d30d7b4a99a9.compiler_builtins.50f24778-cgu.70.rcgu.o)
>>> defined in /opt/wasi-sdk-20.0+threads/bin/../share/wasi-sysroot/lib/wasm32-wasi/libc.a(math-builtins.o)

Plans to publish to crates.io?

I would like to experiment with this in a rust project I am working on, and I was hoping to manage it as dependency via cargo/crates.io.

Add `--exclude` option

Multiple calls to `pack`?

Is it possible to allow multiple calls to wasi-vfs pack? We'd like to add some files to a wasm module as a "standard library", but then add other directories later on with subsequent calls. It appears as though if you do that now, you only get the result of the final call.