webassembly/wasi-libc

WASI libc implementation for WebAssembly

wasi-libc

wasi-libc is a libc for WebAssembly programs built on top of WASI system calls. It provides a wide array of POSIX-compatible C APIs, including support for standard I/O, file I/O, filesystem manipulation, memory management, time, string, environment variables, program startup, and many other APIs.

wasi-libc is sufficiently stable and usable for many purposes, as most of the POSIX-compatible APIs are stable, though it is continuing to evolve to better align with wasm and WASI. For example, pthread support is experimentally provided via the wasi-threads proposal.

Usage

The easiest way to get started with this is to use wasi-sdk, which includes a build of wasi-libc in its sysroot.

Building from source

To build a WASI sysroot from source, obtain a WebAssembly-supporting C compiler (currently this is only clang 10+, though we'd like to support other compilers as well), and then run:

make CC=/path/to/clang/with/wasm/support \
     AR=/path/to/llvm-ar \
     NM=/path/to/llvm-nm

This makes a directory called "sysroot" by default. See the top of the Makefile for customization options.

To use the sysroot, use the --sysroot= option:

/path/to/wasm/supporting/c/compiler --sysroot=/path/to/the/newly/built/sysroot ...

to run the compiler using the newly built sysroot.

Note that Clang packages typically don't include cross-compiled builds of compiler-rt, libcxx, or libcxxabi, for libclang_rt.builtins-wasm32.a, libc++.a, or libc++abi.a, respectively, so they may not be usable without extra setup. This is one of the things wasi-sdk simplifies, as it includes cross-compiled builds of compiler-rt, libc++.a, and libc++abi.a.

Building in pthread support

To enable pthreads support via the wasi-threads proposal, follow the above build directions with one addition: make ... THREAD_MODEL=posix. This creates additional artifacts in sysroot/lib/wasm32-wasi-threads to support --target wasm32-wasi-threads.

Arch Linux AUR package

For Arch Linux users, there's an official wasi-libc package tracking this Git repository. You might want to install other WASI related packages as well.

wasi-libc's Issues

64bit woes in Chromium

Chromium is complains about clock_t being a long long of 64bits when targeting wasm32-wasi.

TypeError: wasm function signature contains illegal type
    at clock (wasm-function[20]:0x24c5)
    at __original_main (wasm-function[11]:0x29d)
    at _start (wasm-function[10]:0x241)
    at test.html:66

Same would apply to anything that relies on other long long typedefs like

• __wasi_device_t
• __wasi_dircookie_t
• __wasi_filesize_t
• __wasi_inode_t
• __wasi_rights_t
• __wasi_timestamp_t
• __wasi_userdata_t

Upstream llvm issue? As in should the wasm32-* targets be spitting out i64 for these?

General Comments

Few comments I wanted to make in regard to a potential design:

• Each syscall should ideally have its own entry point. Contrast this with one big syscall entry point where the host implementation must switch on a syscall number.
• Relatedly, each syscall entry point should have a unique signature. For instance, emscripten currently passes a pointer to a struct that contains the arguments for each system call. This incurs overhead since the host implementation has to copy the arguments to its stack and also check if the user pointer overflow.
• I think errno should live at a higher level. Syscalls simply return -errno to signal error. Non-negative return values flag success.
• mmap() may require special consideration.

Memory management

Sharing linear memory between WASM and JavaScript without shared memory management is slightly hazardous -- one side can easily clobber what the other wrote. It would be beneficial to provide C memory management functionality in wasm libc, that can be used for both JS and WebAssembly, similar to what Rust does.

unable to compile simple httpclient wasm module

I am trying to create simple httpclient wasm module to run on wasmtime
When compiling a simple httpclient using wasi-sysroot, I see that
netdb.h is not found and socket apis 'listen' , 'bind' , 'connect' , 'accept' are missing?
Are they supported?

Trouble with the definition of O_RDWR

It seems that GNU programs do not expect O_RDONLY, O_WRONLY, and O_RDWR to share bits. This is demonstrated by this excerpt from gnulib's openat.c:

  if (flags & (O_CREAT | O_WRONLY | O_RDWR))
    {
      size_t len = strlen (filename);
      if (len > 0 && filename[len - 1] == '/')
        {
          errno = EISDIR;
          return -1;
        }
    }

This code is meant to check for a condition where write access is required but the filename ends with a slash, indicating that it should be a directory.
However, with wasi-libc, if a directory was opened with read access (flags == O_RDONLY), the value of flags & (O_CREAT | O_WRONLY | O_RDWR) is still nonzero since the definition of O_RDWR is:
https://github.com/CraneStation/wasi-libc/blob/446cb3f1aa21f9b1a1eab372f82d65d19003e924/libc-bottom-half/headers/public/__header_fcntl.h#L37

Environment variables are broken

I just encountered this issue when updating @wasmer/wasi to be compatible with wasi_snapshot_preview1).

Having this file:

use std::env;

fn main() {
    let existing = env::var_os("WASM_EXISTING");
    println!("should be set (WASM_EXISTING): {:?}", existing);
    let unexisting = env::var_os("WASM_UNEXISTING");
    println!("shouldn't be set (WASM_UNEXISTING): {:?}", unexisting);

    env::set_var("WASM_EXISTING", "NEW_VALUE");
    let existing = env::var_os("WASM_EXISTING");
    println!("Set existing var (WASM_EXISTING): {:?}", existing);
    env::set_var("WASM_UNEXISTING", "NEW_VALUE");
    let unexisting = env::var_os("WASM_UNEXISTING");
    println!("Set unexisting var (WASM_UNEXISTING): {:?}", unexisting);

    println!("All vars in env:");
    for (key, value) in env::vars() {
        println!("{}: {}", key, value);
    }
}

Compiling it with rustc env.rs --target wasm32-wasi -O

If running it without env vars:

wasmer packages/wasi/test/rs/wasi_snapshot_preview1/env.wasm

This will show the correct output:

should be set (WASM_EXISTING): None
shouldn't be set (WASM_UNEXISTING): None
Set existing var (WASM_EXISTING): Some("NEW_VALUE")
Set unexisting var (WASM_UNEXISTING): Some("NEW_VALUE")
All vars in env:
WASM_EXISTING: NEW_VALUE
WASM_UNEXISTING: NEW_VALUE

However, the following will fail (without printing anything) with error code 71:

wasmer packages/wasi/test/rs/wasi_snapshot_preview1/env.wasm --env "WASM_EXISTING=VALUE"

Note: this issue is independent of the runtime, it also happens in any other runtimes

Investigate mimalloc

It'd be nice to provide alternative malloc implementations, with different code size vs performance tradeoffs.

See the comment here about mimalloc.

Create release and/or tag for snapshots

I think it would be useful to have releases/tags for each snapshot. There is currently one release labeled v0.1-alpha from back in March. Can releases for snapshot0 and snapshot1 be added?

Review of errors and warnings, while porting/patching GNU make port

Hi there.
I wanted to practice in using LLVM backend, for WebAssembly, while compiling with Clang (without usage of emscripten, in the current pipeline).

I will have to re-write most of the code and resolve license collisions, in the future - but the general purpose of the project is pretty much obvious: it would allow to run "make" inside wasm runtime, in the browser, to perform batch execution of build scripts.

Here is my repo:

https://github.com/advancedwebdeveloper/make_wasm
.

I am currently resolving issues and patching file in https://github.com/advancedwebdeveloper/make_wasm/tree/master/src folder - but I haven't got to investigation of errors in https://github.com/advancedwebdeveloper/make_wasm/tree/master/lib directory.

It makes sense to track the summary of all libc calls, which "make" tool requires. Please advise a proper tool, for this. I would report what I have collected.

Ivan

Overuse of phony targets in makefile

Hi, I'm working on packaging this in FreeBSD Ports. Ports has separate build stages, so I need to make finish and install separately. But because most targets are .PHONY, making install starts a rebuild from scratch! I have replaced install: finish with install: as a workaround, but it would be nicer if the makefile used Make dependency tracking to avoid rebuilds.

missing pwd.h (Unix password database operations)

I took a first try at compiling libgit2 with this, but sysdir.c is looking for pwd.h:

cd /home/ctaggart/libgit2/build/src && /usr/bin/clang-8  --target=wasm32-wasi --sysroot=/home/ctaggart/wasi-sysroot/sysroot -DSHA1DC_CUSTOM_INCLUDE_SHA1_C=\"common.h\" -DSHA1DC_CUSTOM_INCLUDE_UBC_CHECK_C=\"common.h\" -DSHA1DC_NO_STANDARD_INCLUDES=1 -D_FILE_OFFSET_BITS=64 -I/home/ctaggart/libgit2/build/src -I/home/ctaggart/libgit2/src -I/home/ctaggart/libgit2/include -I/home/ctaggart/libgit2/deps/http-parser -I/home/ctaggart/libgit2/deps/zlib  -D_GNU_SOURCE  -Wall -Wextra -fvisibility=hidden -fPIC -Wdocumentation -Wno-missing-field-initializers -Wstrict-aliasing -Wstrict-prototypes -Wdeclaration-after-statement -Wshift-count-overflow -Wunused-const-variable -Wunused-function -Wformat -Wformat-security -Wint-conversion -Wno-documentation-deprecated-sync -O3 -DNDEBUG   -std=c11 -o CMakeFiles/git2internal.dir/sysdir.c.o   -c /home/ctaggart/libgit2/src/sysdir.c
/home/ctaggart/libgit2/src/sysdir.c:18:10: fatal error: 'pwd.h' file not found

Is pwd.h something that may be provided in the future? Or is this a case where I would need to add another set of defines and implement it for wasi? Sorry, a bit confused. This is all new.

Upstream llvm change adds new undefined symbol: __multi3

This seems to be the change that caused it https://reviews.llvm.org/D65143.

I guess we should somehow filter out these symbols when doing comparisons?

Migration to WebAssembly/wasi-libc

A while back we decided to go ahead with the long planned move to https://github.com/WebAssembly/wasi-libc.

What is that current status? Can you use github's "move/migrate" feature to enable old users to be seemlessly redirected?

(Is there any tracking bug for this somewhere? If so, sorry for the noise).

Update wasi libc for new threading ABI

Support for threads will require changes to the program startup code. See https://reviews.llvm.org/D65783 for details.

Change default branch name

The WASI subgroup is changing the default branch names in all repositories to main.

Please consider this a tracking issue. It is not intended for public debate.

• Change branch name
• Update CI
• Update build scripts
• Update documentation

Why fopen returns NULL?

I tried to use fopen to read an image:

#include <stdio.h>

int main()
{
    FILE *f = fopen("1.png", "rb");
    printf("file %p\n", f);
    fclose(f);
}

Build:

$ clang --target=wasm32-wasi --sysroot=/opt/wasi-sdk/share/wasi-sysroot/ test.c -o test.wasm
$ wasmtime test.wasm
$ file 0

But it returned NULL. How can I fix it?

realpath in stdlib.h but not in library

I assume we won't have realpath for WASI MVP, is that correct? It seems to included in stdlib.h.

"phread.h" not found

How to include the libc-top-half in the build process and get it in sys_root?

Line buffering issue with stdout

Hello!

I have some C code that I'm compiling with clang 8.0 and a sysroot compiled from master.

int print_random_number_of_length(int size)
{
        //setbuf(stdout, NULL);
        char* before = "before";
        host_print(before, strlen(before));

        puts("ABC\n ");
        printf("FOUND SIZE %d\n", size);
        printf("FOUND SIZE %d\n", size);
        printf("defg\n");
        while (size > 0) {
                int digit = random() % 10;
                putchar(digit + '0');
                --size;
        }

        printf("\n");
        //fflush(stdout);
        char* after = "after";
        host_print(after, strlen(after));
        return 0;
}

And here's a trace:

PRINTING FROM THE HOST: "before"
PRINTING FROM THE HOST: "after"
<call to wasi::fd_write: fd=1>
ABC
 
FOUND SIZE 2
FOUND SIZE 2
defg
75

There are a number of things going on here:

• The host call, host_print prints the string using printf with a terminating new line.
• clang changes calls to printf with a static string into calls to puts
• wasi::fd_write is only called once; this is not a buffering issue in the runtime
• writes to stdout are buffered despite new lines: notice that the host calls come before the other prints. Additionally, I've verified that it hasn't reordered the print statements in the generated Wasm.
• stdout seems to be correctly set to be line buffered: https://github.com/CraneStation/wasi-libc/blob/master/libc-top-half/musl/src/stdio/stdout.c#L11

The following things make it work correctly:

• uncommenting the fflush(stdout)
• uncommenting setbuf(stdout, NULL);
• passing in an argument large enough that about 1000 bytes are written

I spent some time looking into this, but ran into some issues and didn't want to spend too much time on this before asking for help.

I'm happy to fix this and submit a patch, but I got a bit lost in the code and print statements weren't working in all the places I'd like them to to quickly debug the issue. I'd really appreciate any pointers in the right direction you all can give!

edit:

I'm using clang source.c -Os -o wasi_example.wasm --target=wasm32-wasi -Wl,--allow-undefined -Wl,--export-all to compile in case that helps.

gethostname linkage fails, even though present in wasi-libc

I tried to compile the following code using wasi-sdk

clang hostname.c --target=wasm32-wasi --sysroot=/opt/wasi-sdk/wasi-sysroot

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int main(void) {

    char hostname[1024];
    gethostname(hostname, 1024);

    puts(hostname);

    return EXIT_SUCCESS;
}

This fails during the linking phase:

wasm-ld: error: /tmp/hostname-bd4ad1.o: undefined symbol: gethostname
clang-10: error: linker command failed with exit code 1 (use -v to see invocation)

even though gethostname is implemented by wasi-libc.

Consider CloudABI as the basis for the native API

https://github.com/NuxiNL/cloudabi

I think that for most applications, this API (or similar) is sufficient. It was originally created to run native applications obtained from untrusted sources.

errno constants are no longer usable in pre-processor macros

We had this reported by a couple of different users independently in emscripten:
emscripten-core/emscripten#9996

Our downstream fix was:
emscripten-core/emscripten#10025

Environment variables don't work if `main` isn't used

We discovered an issue in the Rust wasm32-wasi target today where a wasm binary uses std::env but doesn't end up importing the requisite wasi items to actually get env vars working.

The broken example looks like this:

#[no_mangle]
pub extern fn foo() {
    println!("{:?}", std::env::var("FOO"));
}

And when compiled:

$ rustc +nightly foo.rs --target wasm32-wasi -O --crate-type cdylib
$ wasm2wat foo.wasm | grep import
  (import "wasi_snapshot_preview1" "fd_write" (func $_ZN4wasi13lib_generated22wasi_snapshot_preview18fd_write17h8cb541fbb70a6a5bE (type 7)))

This is showing that environ_get isn't imported at all!

When compiled as a binary though it does indeed work:

fn main() {
    println!("{:?}", std::env::var("FOO"));
}

$ rustc +nightly foo.rs --target wasm32-wasi -O
$ wasm2wat foo.wasm | grep import
  (import "wasi_snapshot_preview1" "proc_exit" (func $__wasi_proc_exit (type 1)))
  (import "wasi_snapshot_preview1" "fd_write" (func $_ZN4wasi13lib_generated22wasi_snapshot_preview18fd_write17h8cb541fbb70a6a5bE (type 9)))
  (import "wasi_snapshot_preview1" "fd_prestat_get" (func $__wasi_fd_prestat_get (type 3)))
  (import "wasi_snapshot_preview1" "fd_prestat_dir_name" (func $__wasi_fd_prestat_dir_name (type 8)))
  (import "wasi_snapshot_preview1" "environ_sizes_get" (func $__wasi_environ_sizes_get (type 3)))
  (import "wasi_snapshot_preview1" "environ_get" (func $__wasi_environ_get (type 3)))

I'm told that @sunfishcode is working on a fix!

PRIxPTR format string causes "format specifies type 'unsigned int' but the argument has type 'uintptr_t' (aka 'unsigned long')" warning

The PRIxPTR macro, used to print uintptr_t values, seems to be incorrectly defined in wasi-sdk 3.0's headers.
To reproduce: compile this C file with wasi-sdk.

#include <stdio.h>
#include <stdint.h>
#include <inttypes.h>

void printsomething() {
	uintptr_t a = 0x1234;
	printf("%" PRIxPTR "\n", a);
}

Expected: no warning occurs.
Actual:

$ bin/clang --sysroot `pwd`/share/sysroot -c test.c
test.c:7:27: warning: format specifies type 'unsigned int' but the argument has type 'uintptr_t' (aka 'unsigned long') [-Wformat]
        printf("%" PRIxPTR "\n", a);
                ~~~~~~~~~~       ^
1 warning generated.

clang: error: no such file or directory: 'crs'

I'm guessing that I'm missing some build dependency. I've installed clang 8.0 and I've trying to build with

ctaggart@rustasia:~/wasi-sysroot$ make WASM_CC=/usr/bin/clang-8

But the build ends with:

clang: error: no such file or directory: 'crs'
clang: error: no such file or directory: '/home/ctaggart/wasi-sysroot/sysroot/lib/wasm32-wasi/libc.a'
Makefile:225: recipe for target '/home/ctaggart/wasi-sysroot/sysroot/lib/wasm32-wasi/libc.a' failed
make: *** [/home/ctaggart/wasi-sysroot/sysroot/lib/wasm32-wasi/libc.a] Error 1

I installed clang using instructions from http://apt.llvm.org/

sudo apt-get install clang-8 lldb-8 lld-8

Do I need something else?

Style guide

It would good to have a explicit style guide for contributors of this repo (and maybe others in Cranelift). Because there are some divergences between different files (like comment style here and here) and sometimes it hard to make code follow the style guide.

_WASI_EMULATED_MMAN: mmap does not return aligned memory

The memory returned from mmap is supposed to be page aligned.

The file offset passed in is also required to be page aligned.

Emulation for `pause`

With #162 and WASI poll_oneoff eliminating support for uninterruptable sleep, it may be useful to have WASI libc offer a pause function as an emulation feature, by having it execute a long-running poll_oneoff in an infinite loop.

Unused fields in struct stat

The WASI libc stat implementation doesn't currently assign the following fields in struct stat: st_uid, st_gid, st_rdev, st_blocks, st_blksize.

The current WASI API doesn't currently provide the information needed to fill these fields in. So for each of these fields, we should decide whether to add something to WASI, leave the field as-is with a placeholder value (they're currently set to 0), or remove the field.

trunc.c should be specialised for wasm32

Suggesting libc-top-half/musl/src/math/wasm32/trunc.c which (analogously to the other architectures) would use the ftrunc WASM instruction.

Similarly for the other available funop and fbinop intrinsics.

FE_DOWNWARD, FE_UPWARD and FE_TOWARDZERO are not defined

Hey there! I'm having quite some fun with WASI, kudos for the nice work!

Today while porting a library (QuickJS specifically) I ran into this. As the title says, those are not defined in wasi-sdk/share/sysroot/include/bits/fenv.h. I manually defined them and compilation continued though :-)

localtime is defined in the header, but has no implementation

the localtime function is present in wasi-sysroot's headers, but not in the actual libc library, so programs calling localtime fails during linking:

#include <time.h>

int main() {
	time_t a;
	localtime(&a);
}

zhuowei@zhuowei-laptop:~/wasi-sdk/bin$ ./clang-8 localtime.c 
wasm-ld: error: /tmp/localtime-230b9c.o: undefined symbol: localtime
clang-8: error: lld command failed with exit code 1 (use -v to see invocation)

Either the localtime function should be removed from the header, or an implementation should be added to the libc.

Question: how are the WASI API pointer parameters boundary checked and converted to native address

Hi, according to my understanding about the WASM spec, the two steps in the subject must be executed during the runtime serving a WASI call. I traced it down to https://github.com/CraneStation/wasmtime/blob/master/wasmtime-wasi/src/syscalls.rs and got lost. I have no knowledge about Rust programming yet, so I want to ask for any quick answer. Probably I should plan some time for learning Rust.. thanks..

Building with THREAD_MODEL = posix

Hi all,

I can build the project just fine using both the standard LLVM 8 toolchain and prebuilt wasi-sdk, however, when I set THREAD_MODEL = posix in the Makefile I get the following error:

fatal error: error in backend: Cannot select: 0x34964f8: i32 = GlobalTLSAddress<i32* @errno> 0
In function: __reallocarray
clang: error: clang frontend command failed with exit code 70 (use -v to see invocation)
clang version 8.0.1-svn369350-1~exp1~20190820121219.79 (branches/release_80)
Target: wasm32-unknown-wasi
Thread model: posix
InstalledDir: /usr/bin
clang: note: diagnostic msg: PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace, preprocessed source, and associated run script.
clang: note: diagnostic msg: 
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang: note: diagnostic msg: /tmp/reallocarray-5c3085.c
clang: note: diagnostic msg: /tmp/reallocarray-5c3085.sh
clang: note: diagnostic msg: 

********************

Before I start spending more time on it, is THREAD_MODEL = posix expected to work in the first place or is it not recommended?

I'm running on standard Ubuntu 18.04.3 LTS.

Cheers

Can we enable "squash+merge" for this project

This way we can use the github UI and still maintain a linear history.

Even better can we disable the regular (non-squash/non-rebase) form of merging in the UI?

No test suite?

I just encountered this issue #180 and I was surprised to find that there is no test suite for this WASI libc.

I think if we want to evolve WASI at a healthy pace, it will be necessary a test-suite for it, assuring that there are no unintended breaking changes moving forward.

sub-optimal math functions

Currently we are including the C implementations of several math function:

• rint
• rintf
• copysignf
• floor
• ceil
• trunc
• fabs
• copysign
• sqrt

This means that any code compiled with -fno-builtin will end up with sub-optimal implementations and because wasi-libc itself is compiled with -fno-builtin any internal references to these functions will be sub-optimal.

This was recently fixed in emscripten via: emscripten-core/emscripten#9239 emscripten-core/emscripten#9250

However, a cleaner fix might be to use arch-specific implementations like, for example:
./libc-top-half/musl/src/math/i386/rintf.s

Building wasi-sysroot on Windows?

I have Windows 10. I installed GNU make for Windows and tried to run the make WASM_CC="C:/Program Files (x86)/LLVM/bin/clang" command but got this error from make`:

File not found - *.c
File not found - *.c
File not found - *.c
File not found - *.c
File not found - *.d
File not found - *.c
File not found - *.c
File not found - *.c
Makefile:248: /dev/null: No such file or directory
make: *** No rule to make target '/dev/null'.  Stop.

I tried specifying the correct location to my clang lib folder in the Makefile, but that didn't work either.

Duplicate symbol when using no arguments version of main with lto

The linker is fussing about duplicate symbol __original_main when trying to use main with the signature int main(void)

Related #112

AT_FDCWD undefined?

I'm getting this issue building gnulib areadlink.c. The source is available here:

https://github.com/coreutils/gnulib/blob/master/lib/areadlink.c

The error is this:

wasm32-unknown-wasi-clang -DHAVE_CONFIG_H -DEXEEXT=\"\" -I. -I..  -I../intl -I../intl -DDEPENDS_ON_LIBICONV=1 -DDEPENDS_ON_LIBINTL=1    -g -O2 -c -o areadlink.o areadlink.c
areadlink.c:41:13: error: use of undeclared identifier 'AT_FDCWD'
  if (fd != AT_FDCWD)
            ^
areadlink.c:55:24: error: use of undeclared identifier 'AT_FDCWD'
  return careadlinkat (AT_FDCWD, filename, NULL, 0, NULL, careadlinkatcwd);
                       ^
2 errors generated.
make[6]: *** [Makefile:1496: areadlink.o] Error 1

AT_FDCWD should be available in wasi, correct? I can't find any direct references outside of Musl though. If it's not provided, what alternatives are available?

You can repro this with nix:

$ nix build -f channel:nixpkgs-unstable pkgsCross.wasi32.gettext --arg config '{ allowUnsupportedSystem = true; }'

/cc @Ericson2314

Question: where is the WASM API implemention?

Hi,

I want to build some basic understanding about the WASI implementation. Previously I guessed a WASI API like __wasi_fd_pread() is like a replacement of some SYS_CALL. I searched the repo source code and found it was called a few times. But I didn't see it is implemented anywhere.

Can anyone give a clue where it is implemened?

Thanks,
Kevin

`__wasilibc_init_preopen` not getting called at startup

Was encountering an assertion failure in fopen() and managed to trace it down to get_shared_map() where assert(global_map) would always fail. It seems like __wasilibc_init_preopen never initializes the global_map here causing get_shared_map (and by extension fopen) to always fail. As a quick fix, simply calling the init procedure when the global_map is NULL will work.

Rough sketch for a syscall API for argv/argc and exit status

The following is a rough sketch for a syscall API. The idea is to support a traditional C-style "main" function which in some environments is called from the wasm start function. The wasm start function has no arguments or return values, so we instead use imported functions to communicate this information.

I'm hoping this proposal will eventually help lead to something useful on its own (a standard way to invoke command-line-style wasm programs), but I'm also hoping it encourages meta-level conversations about how to design and document proposals. "main" is just the beginning ;-).

argv and argc

(func (import “@std_main/” “argv_data_len”) (result i32))
(func (import “@std_main/” “store_argv_data”) (param $data i32))
(func (import “@std_main/” “argc”) (result i32))
(func (import “@std_main/” “store_argv_pointers”)
      (param $data i32) (param $pointers i32))

Argv strings are NUL-terminated C-style strings and are concatenated into a single buffer, the argv data. argv_data_len returns the unsigned length of the argv data, including the NULs. store_argv_data stores the argv data into the default linear memory at $ptr.

argc returns the unsigned number of NUL-terminated strings are encoded in the argv data. store_argv_pointers populates an argv-style array (with length argc) of 32-bit indices at $pointers in the default linear memory, giving the starts of the NUL-terminated strings in the argv data.

store_argv_data and store_argv_offsets trap if any byte to be written would be out of bounds.

The rough startup sequence would be: call argv_data_len to obtain the argv data length, malloc a buffer of that size, call store_argv_data to write the argv data into it, call argc to obtain the number of arguments, malloc a buffer enough for that many pointers (plus one for C which needs a trailing NULL pointer), call store_argv_pointers to populate it with the argv pointers (and store NULL in the last element for C).

Strings are not required to be valid UTF-8.

Exit Status

Each wasm instance has an associated exit status variable, which is either an integer or one of a predefined set of exit messages. The value is returned to the host when the start function exits. The initial value when the start function is called is the message “success”.

(func (import “@std_main/” “set_exit_status_i32”) (param $val i32))
(func (import “@std_main/” “set_exit_status_message” (param $index i32))

set_exit_status_i32 sets the exit status variable to an integer value.

set_exit_status_message sets the exit status variable to one of a set of predefined exit messages:

index	message
0	success
1	general error
2	command-line usage error
3	memory allocation error
N in [4, 16)	error code N (not yet defined)
N in [16, 128)	exit with value N-16
N >= 128	error code N (not yet defined)

Integer values and exit messages are both mapped to the host environment in host-specific and potentially lossy ways, with the following constraints:

• an i32 value of 0 is mapped as if it were the message “success”
• An i32 value x such that 0 < x < 112 is mapped to a message “exit with value x” with “x” replaced by the value

Rationale for the magic number 112: Unix-like environments reserve exit values N >= 128 for signal termination values, and command-line shells use 127 and 126 to indicate "command not found" and "command is not executable", respectively. In theory, if shells need more values, they'll continue to count down from there. And, <sysexits.h> is a header on many systems which defines some error codes starting at 64 and counting up, currently up to 78. 112 is a somewhat arbitrary point in between these two, allowing both to grow.

Rationale for having a message API in addition to an i32 API: Some programs use exit codes to communicate specific information. For example, while 1 is commonly used to indicate an error, in grep, 1 means that no matches were found, while 2 indicates an error. The idea here is to provide an i32 API for use by portable C code that imposes minimal interpretation on the meaning, and also to provide a wasm-specific message API that programs can opt into using to evoke a specific interpretation.

Should we include lib++ and libc++abi here?

Right now we build them separately on top of this repo (at least on the wasm waterfall).

It seems that we should be including them if we want to continue to call this "sysroot". Otherwise we should rename this to "wasi-libc".. since its just one component of a sysroot.

If we do want to include them here I suggest we do so via git submodule of https://github.com/llvm-mirror/libcxx etc.. since AFAICT they don't need any modification.

_WASI_EMULATED_MMAN is doesn't write changes back to the underlying file

The munmap is supposed to do this, but also msync is completely missing.

Question: Can we use pointer of structure for the WASI API parameter?

Hi,

I just saw the API __wasi_sock_recv() is using a pointer of __wasi_iovec_t as the second parameter. I wonder if we can use data structure for the data sharing between the WASM inside and outside. There are some concerns about the memory layout and endian might be different. I also found there are some static asserts about the the structure __wasi_iovec_t, and my guess it might be related with these concerns. My question is - do we have any common guideline about using structure pointer as WASI API parameter here?

typedef struct __wasi_iovec_t {
void *buf;
size_t buf_len;
} __wasi_iovec_t;

is WASI capable of doing networking?

I don't see how these functions would work:

• socket
• accept
• bind
• listen

Building with hardening build flags

Debian builds with these build flags by default:

CFLAGS=-g -O2 -fdebug-prefix-map=/home/infinity0/var/lib/rust/wasi-sysroot=. -fstack-protector-strong -Wformat -Werror=format-security
LDFLAGS=-Wl,-z,relro

Overriding WASM_CFLAGS with these, results in the following metadata diff:

--- a/expected/wasm32-wasi/predefined-macros.txt
+++ b/expected/wasm32-wasi/predefined-macros.txt
@@ -1143,7 +1143,6 @@
 #define NAN (0.0f/0.0f)
 #define NBBY 8
 #define NCARGS 131072
-#define NDEBUG 1
 #define ND_NA_FLAG_OVERRIDE 0x00000020
 #define ND_NA_FLAG_ROUTER 0x00000080
 #define ND_NA_FLAG_SOLICITED 0x00000040
@@ -2765,6 +2764,7 @@
 #define __SIZE_MAX__ 4294967295UL
 #define __SIZE_TYPE__ long unsigned int
 #define __SIZE_WIDTH__ 32
+#define __SSP_STRONG__ 2
 #define __STDARG_H 
 #define __STDC_HOSTED__ 1
 #define __STDC_IEC_559__ 1
@@ -3173,7 +3173,7 @@
 #define and_eq &=
 #define asin(x) __tg_real_complex(asin, (x))
 #define asinh(x) __tg_real_complex(asinh, (x))
-#define assert(x) (void)0
+#define assert(x) ((void)((x) || (__assert_fail(#x, __FILE__, __LINE__, __func__),0)))
 #define atan(x) __tg_real_complex(atan, (x))
 #define atan2(x,y) __tg_real_2(atan2, (x), (y))
 #define atanh(x) __tg_real_complex(atanh, (x))
--- a/expected/wasm32-wasi/undefined-symbols.txt
+++ b/expected/wasm32-wasi/undefined-symbols.txt
@@ -13,6 +13,8 @@
 __letf2
 __lttf2
 __netf2
+__stack_chk_fail
+__stack_chk_guard
 __stack_pointer
 __subtf3
 __trunctfdf2

Of course, I don't expect hardening flags to make much sense on wasm32 right now, but perhaps they would make sense in the future. Please advise me on whether I should retain these flags, or drop them, bearing in mind the future.

Libc based on musl?

To avoid having to reimplement the entirety of libc for the reference-sysroot, I propose we take the environment-agnostic parts of musl and move them here. Musl is licensed under MIT.

Natural logarithm (log) seems broken

printf("%f ---> %f\n", 7.4225, log(7.4225)); will output
7.422500 ---> inf,
whereas
printf("%f ---> %f\n", 7.4225, log10(7.4225)); gives
7.422500 ---> 0.870550, which looks like the correct answer.

printf("%f ---> %f\n", 7.4225, log2(7.4225)); gives
7.422500 ---> -nan, which looks wrong too.

I didn't try the *f and *l versions of these.

My compiler is clang-10 from nixpkgs, which is version 10.0.0.