swadicalrag/wasm2lua

wasm2lua: converting WASM into Lua

🎉 wasm2lua 🎊

Have YOU ever wanted to run C++/C/Rust/javascript/banana/etc. on Lua without using native modules?

GOOD NEWS, fam: wasm2lua can compile WebAssembly modules to pure Lua (or with FFI LuaJIT for extra speed). This means that anything that can be compiled to WASM can be compiled to Lua! Long Live John Lua!

Notes

• The generated code is heavily dependent upon the LuaJIT bit library
• Some of @webassemblyjs's libraries are monkey patched at runtime
• luamin is monkey patched at runtime
• The WASI API is still under heavy development
• Emscripten is NOT supported out of the box. You will need to compile things via the WASI ecosystem.

Usage

wasm2lua <in.wasm> <out.lua>

lua-webidl <in.idl> <out.cpp> --cpp

(Run those commands with --help for other options)

See HOWTO-BINDINGS.md in this repository for reference on how to use the WebIDL binder.

Installation

npm i -g wasm2lua

Projects that use wasm2lua

• bullet3-lua (Releases, Demo), built with CMake
• brotli-lua (Releases, Demo), built with CMake
• tiny-regex-lua (Demo), built with a Makefile

BONUS: duktape via wasm2lua

int main(int argc, char *argv[]) {
	duk_context *ctx = duk_create_heap_default();

	(void) argc; (void) argv;  /* suppress warning */

	duk_push_c_function(ctx, native_print, DUK_VARARGS);
	duk_put_global_string(ctx, "print");
	duk_push_c_function(ctx, native_adder, DUK_VARARGS);
	duk_put_global_string(ctx, "adder");

	duk_eval_string(ctx, "print('Hello world!');");

	duk_eval_string(ctx, "print('2+3=' + adder(2, 3));");
	duk_pop(ctx);  /* pop eval result */

	duk_destroy_heap(ctx);

	return 0;
}

wasm2lua's Issues

Use manually compiled luajit in Travis CI

The builds are randomly failing probably because the luajit vm from the package manager is not correctly compiled.

Implement passing output buffers to c++ via webidl

namespace global {
    BrotliDecoderResult BrotliDecoderDecompress ([StringLength="encBuf",Unsigned] any encoded_size, [String="encBuf"] any encoded_buffer, [OutputBufferLengthRef="decBuf"] any decoded_size, [OutputBuffer="decBuf",Unsigned] any decoded_buffer);
};

decoded_size becomes size_t * and can be overwritten by the called code, decoded_buffer becomes unsigned char * because [Unsigned] is also present.

OutputBufferLength converts to size_t (read only).

decoded_size becomes a maxlen internally, and encoded_buffer should be a OutputBuffer (created by vm.NewOutputBuffer(maxlen).

Strings with user defined length in lua-webidl

Should I make a new extAttr for this?

interface Test {
  void Func([String="inStr"] any str, [StringLength="inStr"] any len);
};

and len is automatically filled in by the compiler as size_t + str is filled in as char * or unsigned char * if [Unsigned] also present.

Optimise br_if boolean casting

(copied from discord)

if i were to do it i'll probably add an extra field inside a phantom register called
lastBoolTransformation: string
which is set (with the code snippet before it was converted into a bool) whenever a number is cast into a bool via an operation
and then i'd add an extra parameter to getPop called shouldReturnBool and inside the function, if:

1. a phantom register is on the stack AND
2. lastBoolTransformation is set, return lastBoolTransformation
3. otherwrise, return (${phantomReg.value}) as normal

and in br_if, use that added parameter in getPop
lastBoolTransformation is guaranteed to be only written once because phantom registers should only ever be assigned to once

Failure to install using npm

When following the installation instructions in the README.md in a clean environment I ran into this error.

npm i -g wasm2lua
npm ERR! code ETARGET
npm ERR! notarget No matching version found for @webassemblyjs/[email protected].
npm ERR! notarget In most cases you or one of your dependencies are requesting
npm ERR! notarget a package version that doesn't exist.

When checking https://www.npmjs.com/package/@webassemblyjs/helper-api-error I found that there was no version 1.11.3

I speculate that this is a result of destructive changes upstream.
Likely related: #xtuc/webassemblyjs#1132

Implement arrays in lua-webidl

Failing to convert seemingly simple C program

so I converted this TTS engine to WASM (so I could wask2kt to embed in minecraft) and wondered "I wonder. can I wasm2lua this to make it a lua program"
but when I tried the wasm2lua transpiler crashed with

/usr/lib/node_modules/wasm2lua/resources/patches/decoder.js:651
        throw new _helperApiError.CompileError("Unexpected instruction: " + toHex(instructionByte));
        ^

CompileError: Unexpected instruction: 0xc0
    at parseInstructionBlock (/usr/lib/node_modules/wasm2lua/resources/patches/decoder.js:651:15)
    at parseInstructionBlock (/usr/lib/node_modules/wasm2lua/resources/patches/decoder.js:689:9)
    at parseInstructionBlock (/usr/lib/node_modules/wasm2lua/resources/patches/decoder.js:753:9)
    at parseInstructionBlock (/usr/lib/node_modules/wasm2lua/resources/patches/decoder.js:689:9)
    at parseInstructionBlock (/usr/lib/node_modules/wasm2lua/resources/patches/decoder.js:753:9)
    at parseInstructionBlock (/usr/lib/node_modules/wasm2lua/resources/patches/decoder.js:689:9)
    at parseInstructionBlock (/usr/lib/node_modules/wasm2lua/resources/patches/decoder.js:689:9)
    at parseInstructionBlock (/usr/lib/node_modules/wasm2lua/resources/patches/decoder.js:753:9)
    at parseCodeSection (/usr/lib/node_modules/wasm2lua/resources/patches/decoder.js:624:7)
    at parseSection (/usr/lib/node_modules/wasm2lua/resources/patches/decoder.js:1487:13)

Node.js v20.9.0

this appears to correlate to the i32.extend8_s opcode

Optimise `select` opcode with virtual registers

Consider using a bitvector for lua memory type tracking.

Currently memory types are tracked using a (hash)table.

Performing a table lookup for every memory operation is probably not the best idea. Type ids only take up 2 bits of space, so we should be able to store 16 cells (64 bytes) worth of type ids per bitvector cell.

Don't return from within loops

This aborts a LuaJIT trace. Perhaps put a label at the end of a function where the return is performed there?

(only affects loops with while true do wrapping it)

Optimise allocation of local variables

This way virtual registers are confined within a block and the LuaJIT's tracer heuristics can work better with inferring local variable lifespans

Graph IR

Began work on an IR today. It is locked behind a compiler flag (--useGraphIR) to facilitate easy testing + continued development alongside the main branch. It currently does not work at all.

• Convert WASM control flow to graph IR.
• Add all instructions to IR (No good dataflow yet, just for testing control flow).
• Test basic control flow. (Just link blocks together with gotos for now)
• Local dataflow: Each control block tracks which locals it uses, and the values locals are bound to when they terminate.
• Better control flow code generation.
• Proper bounding of variable lifetimes.
• Handling for very large jumps.
• Proper handling of setjmp/longjmp (setjmp should split control blocks).

Other thoughts:

• Planning to deal with block results by creating an extra pseudo-local and throwing it in block outputs.
• Track nesting to avoid the parser throwing a fit. Might be able to decompose expressions that are too large by creating a small scope with temp vars.
• Track estimate of LJ opcodes to avoid large jumps.
• Might be able to convert some loops into lua for loops.
• Maybe convert some blocks to functions? I'm not sure how the JIT feels about lambdas, will need to test.

Optimize JumpStreams

I feel like improvements are possible here, but I only have some vague ideas at the moment.

At the very least, I suspect it might be possible to flatten our loop hints so they can still be used with jumpstreams and longjmps.

Finish base WASI implementation

• __wasi_args_get()
• __wasi_args_sizes_get()
• __wasi_clock_res_get()
• __wasi_clock_time_get()
• __wasi_environ_get()
• __wasi_environ_sizes_get()
• __wasi_fd_advise()
• __wasi_fd_allocate()
• __wasi_fd_close()
• __wasi_fd_datasync()
• __wasi_fd_fdstat_get() for stdio
• __wasi_fd_fdstat_get() for file i/o
• __wasi_fd_fdstat_set_flags()
• __wasi_fd_fdstat_set_rights()
• __wasi_fd_filestat_get()
• __wasi_fd_filestat_set_size()
• __wasi_fd_filestat_set_times()
• __wasi_fd_pread()
• __wasi_fd_prestat_dir_name()
• __wasi_fd_prestat_get()
• __wasi_fd_pwrite()
• __wasi_fd_read() for stdio
• __wasi_fd_read() for file i/o
• __wasi_fd_readdir()
• __wasi_fd_renumber()
• __wasi_fd_seek()
• __wasi_fd_sync()
• __wasi_fd_tell()
• __wasi_fd_write() for stdio
• __wasi_fd_write() for file i/o
• __wasi_path_create_directory()
• __wasi_path_filestat_get()
• __wasi_path_filestat_set_times()
• __wasi_path_link()
• __wasi_path_open()
• __wasi_path_readlink()
• __wasi_path_remove_directory()
• __wasi_path_rename()
• __wasi_path_symlink()
• __wasi_path_unlink_file()
• __wasi_poll_oneoff()
• __wasi_proc_exit()
• __wasi_proc_raise()
• __wasi_random_get()
• __wasi_sched_yield()
• __wasi_sock_recv()
• __wasi_sock_send()
• __wasi_sock_shutdown()

Code generation without depending on `goto`s

It is very unfortunate world of warcraft uses Lua 5.1 which does not provide goto.

Although bit is provided, it does not provide all the functions that bitops provide which means they have to be emulated with lua apis itself.

BTW, how can I deal with wasi syscalls? should I implement those syscalls by myself?

    function __FUNCS__.func_3(reg0, reg1, reg2, reg3)
        local reg4;
        reg4 = __IMPORTS__.wasi_snapshot_preview1.fd_write(reg0,reg1,reg2,reg3);
        do return (bit_band(reg4,65535)); end;
    end
    function __FUNCS__.func_4(reg0)
        __IMPORTS__.wasi_snapshot_preview1.proc_exit(reg0);
        error('unreachable');
    end

Zero initialize locals

WebAssembly expects locals (but not parameters) in a function body to be initialized to 0 by default. wasm2lua currently initializes them as nil which causes code that relies on this invariant to break.

This minimal example is valid and should return 0, but returns nil instead.

Implement + Optimize LongInt Division

Only half of the longint division routines are implemented, only one is implemented correctly, and zero are implemented in a good way. First implement the rest correctly, then optimize. Use the same algorithm as in long.js, which has a Lua version floating around somewhere.

• Signed division
• Unsigned division

Override and inline wasm-implementations of frequently used libc functions with lua-implementations

i.e. we roll our own optimised code that plays nice with our custom memory design
e.g. (naiive memcpy example that's optimised for luajit)

    function __FUNCS__.memcpy(dest, src, len)
        local fastLen = bit.band(len,-4) -- bit.bnot(3)
        for i=0,fastLen-1,4 do
            __MEMORY_WRITE_32__(mem_0,dest+i,__MEMORY_READ_32__(mem_0,src+i))
        end
        
        for i=(len-fastLen),len-1,1 do
            __MEMORY_WRITE_8__(mem_0,dest+i,__MEMORY_READ_8__(mem_0,src+i))
        end

        return dest
    end

Before:

After:

This can be optimised further in the pure-lua memory version by allowing memset to copy over fpMap too (so that memory type hints are preserved, and speed is therefore preserved without having to convert between floats and ints)

Use `writeGoto`/`writeLabel` for longjmps

This way, jmpstreams can be used for longjmps too

cannot work on wasm32-unknown-unknown target from rust

when compiling for this target and then running wasm2lua on it you end up with

Unsupported export type: 0x2
/usr/lib/node_modules/wasm2lua/resources/patches/decoder.js:618
          throw new _helperApiError.CompileError("Unexpected valtype: " + toHex(valtypeByte));
          ^

CompileError: Unexpected valtype: 0x61
    at parseCodeSection (/usr/lib/node_modules/wasm2lua/resources/patches/decoder.js:618:17)
    at parseSection (/usr/lib/node_modules/wasm2lua/resources/patches/decoder.js:1487:13)
    at Object.decode (/usr/lib/node_modules/wasm2lua/resources/patches/decoder.js:1697:25)
    at Object.decode (/usr/lib/node_modules/wasm2lua/node_modules/@webassemblyjs/wasm-parser/lib/index.js:253:21)
    at new wasm2lua (/usr/lib/node_modules/wasm2lua/js/index.js:104:42)
    at Object.<anonymous> (/usr/lib/node_modules/wasm2lua/js/bin/wasm2lua.js:85:12)
    at Module._compile (node:internal/modules/cjs/loader:1233:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1287:10)
    at Module.load (node:internal/modules/cjs/loader:1091:32)
    at Module._load (node:internal/modules/cjs/loader:938:12)

Node.js v20.5.1

Garbage collecting objects in lua-webidl

Right now, there's no delete method exposed to lua. The plan is to expose a delete method and hook it up to __gc or something. This needs more thought. What if the object is also owned by wasm code?

Implement CPP->Lua conversion on JSImplementations

Currently, bindings are generated for JSImplementations, but values passed from wasm to Lua are not converted properly (and vice versa).

Try that and never works

Can you show us a demo on how to run

#include<stdio.h>
int main()
{
puts("Hello World\n");
}

?

Inline int64s and don't use metatables

Comparison Folding

Currently the results of comparison operations are always converted to integers, then converted back to bools when testing a conditional branch. We should store the comparison results as bools, and only convert between integers and bools if necessary. We should also try to fold the comparisons straight into conditional tests, since LuaJIT bytecode is designed to jump on comparisons.

Allow usage of `wasm2lua` without static linking to `libc` / Incorrect error message when `mallocName` is undefined

So I've been getting this project to work and I can't seem to find how to define __MALLOC__ and __FREE__.. leading to undefined is not defined (or x is not defined if using --mallocName x etc)
I've had to manually replace them with function() return 0 end, function() end after generating.

In the future I want to be able to automate this, so I wanted to put this here to see how to do so ( The examples didn't give me any clues either :v )

Also seems that someone else had the same issue as seen in the README https://github.com/EntireTwix/wasm2minecraft

This project is amazing btw, thank you for your work on it! (If I have time hopefully I can try and contribute..)
Already got it (mostly) working on windows! (examples may or may not work)