tessi/wasmex

Execute WebAssembly from Elixir

Wasmex is a fast and secure WebAssembly and WASI runtime for Elixir. It enables lightweight WebAssembly containers to be run in your Elixir backend.

It uses wasmtime to execute Wasm binaries through a Rust NIF.

Documentation can be found at https://hexdocs.pm/wasmex.

Install

The package can be installed by adding wasmex to your list of dependencies in mix.exs:

def deps do
  [
    {:wasmex, "~> 0.8.3"}
  ]
end

Example

There is a toy Wasm program in test/wasm_test/src/lib.rs, written in Rust (but could potentially be any other language that compiles to WebAssembly). It defines many functions we use for end-to-end testing, but also serves as example code. For example:

#[no_mangle]
pub extern fn sum(x: i32, y: i32) -> i32 {
    x + y
}

Once this program compiled to WebAssembly (which we do every time when running tests), we end up with a wasmex_test.wasm binary file.

This Wasm file can be executed in Elixir:

bytes = File.read!("wasmex_test.wasm")
{:ok, pid} = Wasmex.start_link(%{bytes: bytes}) # starts a GenServer running a Wasm instance
{:ok, [42]} == Wasmex.call_function(pid, "sum", [50, -8])

What is WebAssembly?

Quoting the WebAssembly site:

WebAssembly (abbreviated Wasm) is a binary instruction format for a stack-based virtual machine. Wasm is designed as a portable target for compilation of high-level languages like C/C++/Rust, enabling deployment on the web for client and server applications.

About speed:

WebAssembly aims to execute at native speed by taking advantage of common hardware capabilities available on a wide range of platforms.

About safety:

WebAssembly describes a memory-safe, sandboxed execution environment […].

Using WebAssembly on an Elixir host is great not only, but especially for the following use cases:

• Running user-provided code safely
• Share business logic between systems written in different programming languages. E.g. a JS frontend and Elixir backend
• Run existing libraries/programs and easily interface them from Elixir

Development

To set up a development environment install the latest stable rust and rust-related tooling:

rustup component add rustfmt
rustup component add clippy
rustup target add wasm32-unknown-unknown # to compile our example Wasm files for testing
rustup target add wasm32-wasi # to compile our example Wasm/WASI files for testing

Then install the erlang/elixir dependencies:

asdf install # assuming you install elixir, erlang with asdf. if not, make sure to install them your way
mix deps.get

If you plan to change something on the Rust part of this project, set the following ENV WASMEX_BUILD=true so that your changes will be picked up.

I´m looking forward to your contributions. Please open a PR containing the motivation of your change. If it is a bigger change or refactoring, consider creating an issue first. We can discuss changes there first which might safe us time down the road :)

Any changes should be covered by tests, they can be run with mix test. In addition to tests, we expect the formatters and linters (cargo fmt, cargo clippy, mix format, mix credo) to pass.

Release

To release this package, make sure CI is green, increase the package version, and:

git tag -a v0.8.0 # change version accordingly, copy changelog into tag message
git push --tags
mix rustler_precompiled.download Wasmex.Native --all --ignore-unavailable --print

Inspect it's output carefully, but ignore NIF version 2.14 and arm-unknown-linux-gnueabihf arch errors because we don't build for them. Now inspect the checksum-Elixir.Wasmex.Native.exs file - it should include all prebuilt binaries in their checksums

Then continue with

mix hex.publish

License

The entire project is under the MIT License. Please read theLICENSE file.

Licensing

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, shall be licensed as above, without any additional terms or conditions.

wasmex's Issues

Support rustler 0.22.0-rc1

When you upgrade to OTP 24, rustler 0.21 breaks (thread panic at Erlang version 2.16 not handled, please file a a bug report.) as does wasmex due to its reliance on that version of rustler. If wasmex can upgrade to this new rc, then I can get my code to build on the newest version of Rustler (which has some pretty great syntax upgrades).

Dependabot couldn't find a mix.exs for this project

Dependabot couldn't find a mix.exs for this project.

Dependabot requires a mix.exs to evaluate your project's current Elixir dependencies. It had expected to find one at the path: /mix.exs,/native/wasmex/Cargo.toml/mix.exs.

If this isn't a Elixir project, or if it is a library, you may wish to disable updates for it from within Dependabot.

View the update logs.

Support for dynamic linking

wasmtime supports dynamic linking using —preload. Is dynamic linking supported by this library?

Example use case: https://github.com/bytecodealliance/javy#creating-and-running-a-dynamically-linked-module-on-the-cli

call_function example does not behave as expected

The function does not produces the result described in the documentation.

I expected the response described in the documentation: {:ok, [42]}. Returning the value 42 is the purpose of the example. Instead, the function returned {:ok, '*'}.

On master branch and v0.8.2 tag.
Observed on MacOS 13.1 and Ubuntu 22.04.

iex(1)> wat = "(module
...(1)> (func $helloWorld (result i32) (i32.const 42))
...(1)>  (export \"hello_world\" (func $helloWorld))
...(1)>     )"
"(module\n(func $helloWorld (result i32) (i32.const 42))\n (export \"hello_world\" (func $helloWorld))\n    )"
iex(2)> {:ok, pid} = Wasmex.start_link(%{bytes: wat})
{:ok, #PID<0.667.0>}
iex(3)> Wasmex.call_function(pid, "hello_world", [])
{:ok, '*'}

Similarly, the specifying a timeout example does not return the expected value {:ok, [42]}.

iex(13)> wat = "(module
...(13)> (func $helloWorld (result i32) (i32.const 42))
...(13)> (export \"hello_world\" (func $helloWorld))
...(13)>  )"
"(module\n(func $helloWorld (result i32) (i32.const 42))\n(export \"hello_world\" (func $helloWorld))\n )"
iex(14)> {:ok, pid} = Wasmex.start_link(%{bytes: wat})
{:ok, #PID<0.4420.0>}
iex(15)> Wasmex.call_function(pid, "hello_world", [], 10_000)
{:ok, '*'}

Require at least elixir 1.11

We just loosened our elixir requirement to ~1.10, but it seems dependencies need at least 1.11 - it more than just rustler, but here is a compile warning for rustler as an example:

==> rustler
Compiling 7 files (.ex)
Generated rustler app
warning: the dependency :rustler_precompiled requires Elixir "~> 1.11" but you are running on v1.10.4

Ability to cache compiled modules

Is it possible ( and practical ) to cache compiled modules and then load them at a later point? Especially for large wasm files?

Something like the following?

{:ok, bytes } = File.read("wasmex_test.wasm")
{:ok, module} = Wasmex.Module.compile(bytes)

### cache and load ###
{:ok} = cache_to_disk(module, path)
{:ok, module} = load_from_disk(path)

{:ok, instance } = Wasmex.start_link(%{module: module}) # starts a GenServer running this WASM instance

{:ok, [42]} == Wasmex.call_function(instance, "sum", [50, -8])

Calling WASM function should be async

The Erlang NIF documentation states that NIFs should avoid long running calls:

A native function doing lengthy work before returning degrades responsiveness of the VM, and can cause miscellaneous strange behaviors. Such strange behaviors include, but are not limited to, extreme memory usage, and bad load balancing between schedulers. Strange behaviors that can occur because of lengthy work can also vary between Erlang/OTP releases.

Calling a WASM function (Instance.call_exported_function) is exactly such a long running call. We just don't know how long the WASM function will need to execute. We have some options here (e.g. dirty schedulers) but the most promising one is to make all function calls async. But making them async means we need to figure out a way to give the function calls result back into elixir land.

In #4 we discuss a way for "Rust land" to asynchronously talk to elixir. We should probably solve this issue in a similar way to #4 to keep the communication between rust and elixir consistent.

One possible solution is to use GenServer in elixir - one elixir process per instantiated WASM instance. The rust-part of that WASM instance could use enif_send to give function call results back to the elixir process of that instance.

We still need to find a good API to instantiate instances with GenServer. Also, it would be interesting whether the rust-side should also be asynchronous and how that could be achieved.

On the elixir side, I imagine we could use the (currently empty) Wasmex module to provide a high level API around GenServer instances:

defmodule Wasmex do
  use GenServer

  @impl true
  def init(bytes) do
    {:ok, %{instance: Wasmex.Instance.from_bytes(bytes)}}
  end

  # this can be synchronous since it should be a fast call
  @impl true
  def handle_call({:exported_function_exists, name}, _from, %{instance: instance}) when is_binary(name) do
    {:reply, Wasmex.Instance.function_export_exists(instance, name), %{instance: instance}}
  end

  # this needs to start the function call and then immediately return. When the WASM part finished some form of callback/waiting/resolving needs to be done to inform the calling process of the function call result
  @impl true
  def handle_call({:call_function, name, params}, from, %{instance: instance}) do
    # ... ? this is where we need to get creative
  end

  def handle_info({:finished_function_execution, original_from, return_value}, state) do
    GenServer.reply(original_from, return_value)
    {:noreply, state}
  end
end

The :call_function handle_call could reply with a {:noreply, , %{instance: instance}} (possibly also setting the :hibernate flag according as outlined in the docs and properly setting timeouts). When the rust part sends the "i am done, here is the result" message to the elixir process, it could use reply/2 to give the result back to the original caller.

Probably, this is best expressed with an image:

We would need someone experimenting with this idea and maybe implementing a PR.

Switch to an async Rust runtime

While discussing #391 we found it's a good idea to switch Wasmex to an async Rust runtime, probably https://tokio.rs/.

An async runtime allows us to enable async wasmtime features, like

• an async config
• epoch based interruption (docs here and here)
• better handling of out-of-fuel situations - e.g. by yielding to the runtime instead of Traping

To achieve this, we must spawn tokio tasks for WASM function executions instead of spawning OS threads as we do now. We must also investigate whether we have other blocking code (looking at you, module compilation 👀 ) which must be migrated to work async to not block the event loop.

Generally, a high-level goal of this task is to be able to handle ~10k concurrent calls into WASM. A side-goal is to enable preemtive WASM scheduling (e.g. by utilizing epoch based interruptions) - we might want to implement preemtive scheduling in a follow-up issue, though, if it turns out to be too much for one change.

We might "borrow" some ideas from https://github.com/scrogson/franz (thanks @scrogson 🌻 ) and ideas from this rustler issue on how to implement async NIFs

Unable to run go code compiled to WASM

Hi!

I tried to compile this go code to wasm:

package main

import (
	"fmt"
)

func main() {
	fmt.Println("Hello from Go!")
}

Using this command:

GOOS=wasip1 GOARCH=wasm go build -o main.wasm

This produced a WASM binary that I was able to run using wasmtime:

$ wasmtime main.wasm
Hello from Go!

However, I am unable to run this using Wasmex:

iex(11)> bytes = File.read!("./native/main.wasm")
<<0, 97, 115, 109, 1, 0, 0, 0, 0, 242, 128, 128, 128, 0, 10, 103, 111, 58, 98,
  117, 105, 108, 100, 105, 100, 255, 32, 71, 111, 32, 98, 117, 105, 108, 100,
  32, 73, 68, 58, 32, 34, 102, 77, 66, 84, 53, 95, 69, 83, 122, ...>>
iex(12)> {:ok, pid} = Wasmex.start_link(%{bytes: bytes}) 
** (MatchError) no match of right hand side value: {:error, "unknown import: `wasi_snapshot_preview1::sched_yield` has not been defined"}
    (stdlib 5.2) erl_eval.erl:498: :erl_eval.expr/6
    iex:12: (file)

I tried to compile the Go code using tinygo as well. I was able to run the output using Wasmtime but got this error using Wasmex:

iex(12)> bytes = File.read!("./native/main.wasm")
<<0, 97, 115, 109, 1, 0, 0, 0, 1, 43, 8, 96, 1, 127, 0, 96, 0, 0, 96, 4, 127,
  127, 127, 127, 1, 127, 96, 2, 127, 127, 0, 96, 1, 127, 1, 127, 96, 2, 127,
  127, 1, 127, 96, 0, 1, 127, 96, 4, 127, 127, ...>>
iex(13)> {:ok, pid} = Wasmex.start_link(%{bytes: bytes}) # starts a GenServer running a Wasm instance
** (MatchError) no match of right hand side value: {:error, "unknown import: `wasi_snapshot_preview1::fd_write` has not been defined"}
    (stdlib 5.2) erl_eval.erl:498: :erl_eval.expr/6
    iex:13: (file)

My wasmex version is the latest one and I was compiling this using Go v1.21. Am I doing something wrong? :/

Arguments and returns as a string

First of all I would like to thank you for this excellent library.
I need to call a function that takes a string as an argument and that returns a string as a result.
In the examples I only saw the use of one or the other, never passing and receiving strings. It may seem trivial to expand, but I'm a little confused about the WASM API and the use of Memory to achieve this goal.
Could someone help me?

Switch to Github Actions

we use circle ci as a CI service, and they are great. However, in other projects I'm working on we're using GH actions mostly and switching over means I have to keep less things in mind.

This is very subjective and purely to reduce maintenance overhead just a tiny bit.

WebAssembly component support

Hey all, I'm really excited about the WebAssembly Components as a way to have a much higher level way to call into WebAssembly from Elixir. I'm a Rust newb (Elixir is my preferred language), but have managed to get simple examples of creating a component in JS and calling it from Rust using wasmtime to work. I'd like to investigate adding this support to wasmex, and wanted to check if there is interest in having such a feature.

Underlying Wasm runtime, wasmer -> wasmtime

Slightly related to #282

So wasmer is the underlying runtime at the moment for wasmex, would you consider switching the underlying runtime of this project to wasmtime like https://github.com/viniarck/wasmtime-ex has?

We've loved using this project and are looking to take advantage of the features that come first to wasmtime, and in a perfect world this would be done through swappable runtimes like #282. In the shorter term, is there any reservation to changing the runtime?

Thanks @tessi 🚀

Seems like rust is a prerequisite?

I was unable to use wasmex, got a pretty cryptic error about an ets table entry being invalid. I didn't save it, apologies. When I added rust to my asdf .tool-versions file I was able to startup my project. This seems to imply rust is a dependency to use wasmex, if so it would be good to add to README.

Support multiple WebAssembly engines

In the current implementation, wasmex provides an elegant Rust/NIF wrapper around the wasmer engine. One of the compelling stories of WebAssembly is that, in the right circumstances, people can choose which engine they like so that they can tune the behavior of the runtime to specific needs. For example, you might prefer interpreted over JITted when running in constrained environments, you might choose a micro-targeting runtime versus a large, robust, "cloudy" runtime.

As we've discussed in this issue in wasmCloud, it would be amazing if we had some kind of meta-package which then let wasmex consumers (such as wasmCloud) choose which of the underlying engines would be used. Ideally, this would also allow wasmCloud to release with specific optimizations for cloud, IoT, edge, etc.

Per suggestion in the original issue, I've created an issue here so that we can discuss/plan/design the work to bring this about.

Question on callbacks

Sorry about doing this through the github, I wasn't sure how else to reach you.

I'm trying to implement something where Elixir has a pointer to my Rust struct and this rust struct can make callbacks up into Elixir (e.g. after it receives a gRPC message or a broker request, etc). This callback needs to be synchronous because it needs to potentially return a value.

I see you've got this implemented with the function imports, but I don't know how much of that I need to duplicate.

I'd like to be able to do something like:

{:ok, whatevs} = MyModule.newGrpcThing(Module.For.Callbacks)

and then I have callbacks that return values to answer the gRPC or message broker requests.

Do you have a blog post or some rustler documentation you used as a reference to implement this kind of callback functionality?

Error during default engine config test and fuel consumption test

Failing tests
On master branch and v0.8.2 tag.
Observed on MacOS 13.1 and Ubuntu 22.04.

iex -S mix test

 1) test error handling handles errors occuring during Wasm execution with default engine config (WasmexTest)
     test/wasmex_test.exs:264
     Assertion with == failed
     code:  assert Wasmex.call_function(pid, :divide, [1, 0]) == {:error, String.trim(expected_reason)}
     left:  {
              :error,
              "Error during function excecution: `error while executing at wasm backtrace:\n    0: 0x394c - <unknown>!__rust_start_panic\n    1: 0x391a - <unknown>!rust_panic\n    2: 0x38ea - <unknown>!std::panicking::rust_panic_with_hook::hbafe3e603d331223\n    3: 0x2fa8 - <unknown>!std::panicking::begin_panic_handler::{{closure}}::h8ab6ee68d5b4c391\n    4: 0x2f0a - <unknown>!std::sys_common::backtrace::__rust_end_short_backtrace::h008f69666d134159\n    5: 0x3541 - <unknown>!rust_begin_unwind\n    6: 0x3c32 - <unknown>!core::panicking::panic_fmt::h1d17fc068f528130\n    7: 0x3c91 - <unknown>!core::panicking::panic::h27b5eefa3e4ff738\n    8:  0x5f4 - <unknown>!divide`."
            }
     right: {
              :error,
              "Error during function excecution: `error while executing at wasm backtrace:\n    0: 0x395d - <unknown>!__rust_start_panic\n    1: 0x3951 - <unknown>!rust_panic\n    2: 0x3921 - <unknown>!std::panicking::rust_panic_with_hook::he04cb00575f2a1e3\n    3: 0x2fc0 - <unknown>!std::panicking::begin_panic_handler::{{closure}}::hb733f0aa505760cf\n    4: 0x2f25 - <unknown>!std::sys_common::backtrace::__rust_end_short_backtrace::h6beefa0bcab220bc\n    5: 0x3575 - <unknown>!rust_begin_unwind\n    6: 0x3c47 - <unknown>!core::panicking::panic_fmt::h586d720a90aa8503\n    7: 0x3c9c - <unknown>!core::panicking::panic::h0859aaac783261b2\n    8:  0x567 - <unknown>!divide`."
            }
     stacktrace:
       test/wasmex_test.exs:288: (test)

  2) test fuel consumption &Wasmex.call_function/3 with fuel_consumption (WasmexTest)
     test/wasmex_test.exs:310
     Assertion with == failed
     code:  assert Wasmex.call_function(pid, :void, []) ==
              {:error,
               "Error during function excecution: `error while executing at wasm backtrace:\n    0:  0x3bf - <unknown>!void`."}
     left:  {:error, "Error during function excecution: `error while executing at wasm backtrace:\n    0:  0x44d - <unknown>!void`."}
     right: {:error, "Error during function excecution: `error while executing at wasm backtrace:\n    0:  0x3bf - <unknown>!void`."}
     stacktrace:
       test/wasmex_test.exs:320: (test)

Error compilation on Alpine

When I try to compile a docker image with Wasmex in Alpine I get the following error below:

error: cannot produce dylib for `wasmex v0.5.0 (/app/massa_proxy/deps/wasmex/native/wasmex)` as the target `x86_64-unknown-linux-musl` does not support these crate types

== Compilation error in file lib/wasmex/native.ex ==
** (RuntimeError) Rust NIF compile error (rustc exit code 101)
    (rustler 0.22.0) lib/rustler/compiler.ex:36: Rustler.Compiler.compile_crate/2
    lib/wasmex/native.ex:7: (module)
    (stdlib 3.15.2) erl_eval.erl:685: :erl_eval.do_apply/6
could not compile dependency :wasmex, "mix compile" failed. You can recompile this dependency with "mix deps.compile wasmex", update it with "mix deps.update wasmex" or clean it with "mix deps.clean wasmex"
The command '/bin/sh -c cd /app/massa_proxy/apps/massa_proxy     && mix deps.get     && mix release.init     && mix release' returned a non-zero code: 1
make: *** [Makefile:10: build] Error 1

I tried to install the correct target via rustup but the error persists.

RUN rustup toolchain install nightly && rustup update && rustup target add wasm32-unknown-unknown --toolchain nightly

Offer precompiled wasmex binaries

With recent work from dashbit it is now possible to have precompiled rustler binaries. I want to offer that with wasmex, so people don't have to have a rust compiler installed to use it (for most archs)

Compatibility with Lunatic.Solutions

Is possible compatibility with https://lunatic.solutions/ ?
Would it be possible to provide libraries, allowing one codebase to work on a top of Lunatic.solutions and Elixir via WASMEX? cross-linking: lunatic-solutions/lunatic#208 and https://twitter.com/GWierzowiecki/status/1658220520414822406

Interface Types Status ?

What is the status for interface types support?

Support Imported Functions

The host of a WASM instance can provide functions to the WASM instance which can be called from WASM code. This is commonly called "imported functions".

An example program (in the WAT format) is which expects the function imports.imported_func to be provided:

(module
  (func $i (import "imports" "imported_func") (param i32))
  (func (export "exported_func")
    i32.const 42
    call $i))

Wasmex does not support imported functions yet. This ticket is the place to brainstorm potential implementation ideas.

Passing Elixir/Erlang functions as callbacks

We implement WASM execution in a NIF. If our NIF gets a fun term from Erlang, we must be able to invoke that function and receive the result.

Unfortunately, Erlangs NIF API does not allow calling function terms. The only function related API seems to be to detect whether a term is a function or not.

Send a message to an erlang process

There is the NIF API call enif_send which can send a message to an erlang process. We could have a separate Erlang process running which listens to these message sends, executes the requested function and calls into the NIF again with the result. This requires some magic with threads and mutexes, though.

I found two people who implemented/proposes this:

• http://erlang.org/pipermail/erlang-questions/2018-January/094626.html

The only way to call an Erlang function from a NIF is to send a message
to a callback process that will then run the function. You can send the
fun in this message but to keep it in the NIF you will probably need to
copy it in an environment you allocated yourself (and then copy it again
in the environment that will send the message).

Calling a function from a NIF is easy. Having the NIF receive the result
of the callback is much harder.

My solution involves having a separate NIF function for receiving the
result, called by the callback process at the end of the callback execution.

This NIF function will then require a mutex lock/cond to store the
result where I will be able to read it afterwards, and to signal the
waiting thread that it can read it.

My own code runs in a separate thread from the schedulers, if you need
to wait from inside a scheduler thread things might get funny if there's
only one scheduler or if the callback takes too long.

Suffice to say that it's probably not very efficient to do all this.

• http://erlang.org/pipermail/erlang-questions/2012-August/068690.html

This seems to be a viable Option, although complex to implement and probably not very efficient.

Some WASI imported functions need to write to memory

An example of a WASI function that needs to be able to write to memory is random_get. This means one ends up in a position where one needs to pass a function as an import that can later gain access to the instance's memory. I've had to deal with this in other languages as well.

I wrote an example test and the error I see is:

15:23:54.272 [error] GenServer #PID<0.448.0> terminating
** (stop) exited in: GenServer.call(#PID<0.448.0>, {:memory, :uint8, 0}, 5000)
    ** (EXIT) process attempted to call itself 

I understand that this is a pretty strange expectation for WASI and they are aware of it, but it doesn't appear a solution will arise soon IMO. It could be useful to pass the memory into imported functions as a kind of state. What do you think?

How safe is it to run untrusted user wasm modules?

This is a general question and not a bug report

We have a need to run custom data-transform logic from users in our data platform.
We are interested in utilizing WASM due to its super quick start-up times and sandboxed nature.

However, is it practical? What are some additional security concerns we should be mindful of specifically when it comes to Elixir + WASM using wasmex

Another project we are considering is Lunatic ( https://github.com/lunatic-solutions/lunatic )
However, we would rather avoid dealing with Rust as much as we can.

Thanks!

Allow call_function to accept longer GenServer timeouts

By default, a GenServer.call assumes a timeout of 5_000. In some edge cases (for example, a wasm module that's doing a lot of I/O by way of the host runtime like wasmCloud) we might have a function that takes more than 5 seconds to finish.

We should be able to just provide an optional / default timeout parameter to the function in the following link, right? If this is already possible and I'm just not good at GenServers, please let me know :)
https://github.com/tessi/wasmex/blob/main/lib/wasmex.ex#L216

WASI support

Using the latest changes on master it should be possible to implement the WASI interface. But it would be great if we could get the default implementations for WASI functions from wasmer.

When instantiating a new module, we could pass an extra map which could be a WASI-flag. Maybe something like this:

imports = %{
  env: %{
    sum: {:fn, [:i32, :i32], [:i32], fn context, a, b -> a + b end}
  }
}

wasi = %{
  args: ["param1", "param2]
  env: %{"ENV_NAME" => "env_value"}
}

instance = start_supervised!({Wasmex, %{bytes: @import_test_bytes, imports: imports, wasi: wasi}})

Send and receive WASI std IO as Erlang messages

@josevalim came up with a great question:

Is the pipe the only way to interact with stdin/stdout/stderr? Or could it be streaming by default (i.e. send those as regular Erlang messages)?

And I love the idea, turning a WASM instance into a much more Elixir'y thing. I believe we want to send/receive raw binary's (no String, no concept of a "line").

With that, implementing a GenServer that receives stdin is fairly easy - it would take ownership of the stdin Wasmex.Pipe and write any received message into the pipe. We could optimize the pipe into an "input only" pipe - so we don't keep all input around forever as we do now, but that's an implementation detail. We probably want to configure Wasmex.Wasi.WasiOptions in a nice way for this new kind of input.

Implementing outputs (stdout, stderr) is slightly more advanced. We would need to implement a new kind of "output pipe" which sends erlang messages to a predefined pid, instead of writing the bytes to a memory buffer. We might want to implement buffering for efficiency - or maybe not, offloading that responsibility to the WASM binary (they should buffer their writes as they see fit).

---

José further wrote:

for stdin you would need buffer/queue, as you would to buffer data not yet read or block the reader until the buffer is available.

But if you can keep it as messages, then you can do reads/writes cross nodes, which is interesting too

which I didn't quite understand 😅 Why can't I just write any incoming message into the pipes memory buffer?

Anyhow, José promised to open this ticket after he had the time to look around. I went forward and already opened it, just in case. I love the idea and don't want to forget it :)
@josevalim: Feel free to still open "your" ticket if this doesn't reflect what you had in mind -- or we just adapt this ticket :)

Return better errors when a wasm function call traps

Looking as wasmer usage examples, I noticed that errors returned from a wasm function call not just have a message but also a stacktrace. No one complained yet, but I guess we want to serialize stacktraces into our error feedback.

The place to change would be within instance.rs#execute_function()

The wasmer example showcasing error handling:

    // When we call a function it can either succeed or fail. We expect it to fail.
    match result {
        Ok(_) => {
            // This should have thrown an error, return an error
            panic!("div_by_zero did not error");
        }
        Err(e) => {
            // Log the error
            println!("Error caught from `div_by_zero`: {}", e.message());

            // Errors come with a trace we can inspect to get more
            // information on the execution flow.
            let frames = e.trace();
            let frames_len = frames.len();

            for i in 0..frames_len {
                println!(
                    "  Frame #{}: {:?}::{:?}",
                    frames_len - i,
                    frames[i].module_name(),
                    frames[i].function_name().or(Some("<func>")).unwrap()
                );
            }
        }
    }

Unable to run Go code compiled for WASI

Hi @tessi !

I am fairly new to web assembly and trying to get Go code compiled for WASI to run via Elixir using Wasmex. Here is a super simple go code:

package main

//export greet
func greet() int32 {
	return 5
}

func main() {}

I can compile this using Tinygo to target WASI like this:

tinygo build -target=wasi -o tiny.wasm main.go

Now I can run this using Wasmex and call greet like this:

bytes = File.read!("./native/tiny.wasm")
{:ok, pid} = Wasmex.start_link(%{bytes: bytes, wasi: true}) 
{:ok, m} = Wasmex.module(pid)

Wasmex.call_function(pid, "greet", [])

This prints 5 as expected.

However, my next plan is to pass complex arguments to a Go program and for that I will need to write to memory. My understanding so far is that I will need to create a Wasmex.Instance using something like this:

{:ok, store} = Wasmex.Store.new_wasi(%Wasmex.Wasi.WasiOptions{})
{:ok, instance} = Wasmex.Instance.new(store, m, %{})

However, this fails with the following error:

iex(99)> {:ok, instance} = Wasmex.Instance.new(store, m, %{})
** (MatchError) no match of right hand side value: {:error, "incompatible import type for `wasi_snapshot_preview1::fd_write`"}
    (stdlib 5.2) erl_eval.erl:498: :erl_eval.expr/6
    iex:99: (file)

These are the exports and imports of my golang WASI binary:

iex(103)> Wasmex.Module.exports(m)
%{
  "_start" => {:fn, [], []},
  "asyncify_get_state" => {:fn, [], [:i32]},
  "asyncify_start_rewind" => {:fn, [:i32], []},
  "asyncify_start_unwind" => {:fn, [:i32], []},
  "asyncify_stop_rewind" => {:fn, [], []},
  "asyncify_stop_unwind" => {:fn, [], []},
  "calloc" => {:fn, [:i32, :i32], [:i32]},
  "free" => {:fn, [:i32], []},
  "greet" => {:fn, [], [:i32]},
  "malloc" => {:fn, [:i32], [:i32]},
  "memory" => {:memory, %{minimum: 2, shared: false, memory64: false}},
  "realloc" => {:fn, [:i32, :i32], [:i32]}
}
iex(104)> Wasmex.Module.imports(m)
%{
  "wasi_snapshot_preview1" => %{
    "fd_write" => {:fn, [:i32, :i32, :i32, :i32], [:i32]}
  }
}

I am not sure why it thinks the types are incompatible for fd_write. Can you please help with this? I have been trying to resolve this on my own for a few days on and off and haven't gotten anywhere. Going to do a last ditch effort to make it work with your help before giving up.

Let me know if you want me to provide any more information.

Example using WASI

This is a great project and awesome work @tessi !

Are there any examples I can take a look at that involves WASI?
Specifically, I am looking to make HTTP requests and use the filesystem from within the WASM.

Thank you!

unknown import: `wasi_snapshot_preview1::random_get` has not been defined

I was using a wasm file that built from javascript, it works fine with wasmtime run command/.

** (EXIT from #PID<0.94.0>) bad return value: {:error, "unknown import: 'wasi_snapshot_preview1::random_get' has not been defined"}
This error pops up when it run {:ok, pid} = Wasmex.start_link(%{bytes: bytes})

the soruce code of test.wasm file is actually from the demo code in https://github.com/Shopify/javy

Add rust-clippy and apply its suggestions

https://github.com/rust-lang/rust-clippy could probably help cleaning up the rust-part of this project.

To avoid merge conflicts with myself, I'd wait until #9 is done and merged. Then start a cleanup-session with clippy as the sidekiq

• add clippy as a rust dependency
• apply clippy suggestions (potentially in multiple small PRs)
• add it to the CI run (if possible) so we don't accidentally write code clippy does not approve

Calls and execution model

Hi @tessi! Great project!

My initial suggestion was around the call_exported_function. At the moment, you are passing from and returning to the caller, but I would consider accepting two arguments instead pid and a opaque term called tag. And you will reply like this:

send(pid, {tag, result})

This way, you could do this inside the GenServer:

call_exported_function(self(), {:exported_return, from})

And still match on {{:exported_return, from}, result} in handle_info. But by breaking it apart, then you could also allow someone to call directly into the instance and get the reply directly, without going to the GenServer.

In fact, you could keep the same API as today, because from is already a {pid, tag} format, and reply from the NIF as:

send(elem(from, 0), {elem(from, 1), result})

And that means you would skip sending the result to the GenServer.

---

However, as I was thinking about this, I started wondering what happens on concurrent calls in an instance. Because calls are non-blocking, today it is possible to call the same instance more than once. What happens on concurrent calls? What happens if those calls read from stdin or write to stdout? Do they use the same buffer or is it per call?

Thanks! <3

Memory offset shouldn't be influenced by the reading byte size?

I've got a situation where I need to start reading uint32's in memory starting at a byte offset like 555. However, Memory.get/4 expects the entire memory stream to be broken down into elements of the chosen element_size instead of "all bytes after the offset" which makes it impossible to read uint32's from memory unless the offset is exactly divisible by 4.

So Memory.get(memory, :uint32, 555, 0) where 555 is the byte offset is not possible.

• Did I explain that well?
• Is it alright to change the existing functions to accept a "byte offset" instead or should there be new functions?
• Do you have an idea how to achieve this in the rust code other than to always start with the <u8> view and then recalculate the view from there?

Upgrade to the wasmer rewrite

According to the wasmer team, they

are working on something very special: a new version of Wasmer with much better performance, supports for native compilation, has 4x better compilation speeds and a much more resilient API.

We got an invite to a preview of their work to be able to start adoption and could start experimenting with it.

consume_fuel function example does not work as expected

The consume_fuel function example does not behave as expected.

I expected config.consume_fuel to return true, as described. Instead, the function returned false.

iex(1)> config = %Wasmex.EngineConfig{}
%Wasmex.EngineConfig{
  consume_fuel: false,
  cranelift_opt_level: :none,
  wasm_backtrace_details: false
}
iex(2)>  |> Wasmex.EngineConfig.consume_fuel(true)
%Wasmex.EngineConfig{
  consume_fuel: true,
  cranelift_opt_level: :none,
  wasm_backtrace_details: false
}
iex(3)> config.consume_fuel
false

Similarly, wasm_backtrace_details does not return the expected value.

iex(1)> config = %Wasmex.EngineConfig{}
%Wasmex.EngineConfig{
  consume_fuel: false,
  cranelift_opt_level: :none,
  wasm_backtrace_details: false
}
iex(2)> |> Wasmex.EngineConfig.wasm_backtrace_details(true)
%Wasmex.EngineConfig{
  consume_fuel: false,
  cranelift_opt_level: :none,
  wasm_backtrace_details: true
}
iex(3)> config.wasm_backtrace_details
false