nuxinl/cloudlibc

CloudABI's standard C library

cloudlibc's Issues

storage size for language name too short in newlocale.c

I think, it should be 6, as with 5 the terminating nul char is not taken into account in string like "fr_FR"

cloudlibc/src/libc/locale/newlocale.c

Line 34 in 33041bf

char name[5];

which would be fine if it wasn't checked for it

cloudlibc/src/libc/locale/newlocale.c

Line 84 in 33041bf

languages[i].name[len] == '\0') {

CloudABI YAML doesn't play well with document delimiters / yamllint

YAML files are supposed to be able to begin with ---, and yamllint recommends this rule to ensure that many YAML files behave well when concatenated together. However, CloudABI's YAML parsing setup requires its YAML files to begin like %TAG ! tag:nuxi.nl,2015:cloudabi/, and then a document delimiter and capability spec document can follow.

Would it be possible to enhance the capability spec so that --- can lead the very beginning of .YML files, matching how other applications process YAML files?

segfault on freebsd 11

Hi! This is really exciting work and I look forward to trying it out.

I followed the instructions on https://nuxi.nl/cloudabi/freebsd/ but I get segfault when running helloworld..

Running ./_obj/cloudlibc-unittests also gives segfault..

FreeBSD PC1 11.0-CURRENT FreeBSD 11.0-CURRENT #0 5a410d3(HEAD): Wed Apr 6 11:14:22 PDT 2016 root@PC1:/usr/obj/usr/home/johannes/dev/freebsd/sys/GENERIC-NODEBUG amd64

Debian/Ubuntu instructions appear to be out of date

The posted installation steps for manually installing the CloudABI toolchain with the cloudabi-run utility in Debian/Ubuntu, omits some important requirements:

• yaml2argdata headers need to be installed.
• arpc needs to be installed.
• The libyaml-cpp-dev package needs to be installed.
• python3 needs to be installed.
• pypeg2 needs to be installed.
• etc.

Without these steps, cloudabi-utils fails to compile. Many of these additional steps are documented in the Dockerfile, but have not been copied to the plain vanilla Debian/Ubuntu documentation.

Could we move these commands to a standalone shell script, and have both the Dockerfile and the documentation point to that, to keep everything in sync?

Odd flushing behavior

I'm used to fprintf() intelligently flushing my output streams, e.g. when writing brief text blurbs to stderr in a Read Eval Print Loop application. However, CloudABI appears to wait until the program is terminating to flush. Could we get the flushing behavior closer to a typical GNU/libc fprintf() implicit flush?

As a workaround, I am doing this for important user feedback sections:

fprintf(console, "blahblahblah\n");

#if defined(__CloudABI__)
    fflush(console);
#endif

No warning when CloudABI entrypoint is missing

Porting applications from libc to cloudlibc is hard work, but one problem is particularly pernicious: cloudlibc toolchains do not warn the programmer when he or she forgets basic steps for correctly wiring up the new entrypoint.

• C: No warning is presented at compilation time when the entrypoint continues to use a traditional libc-style <void|int> main([int argc[, char **argv]]) signature, as opposed to the recommended void program_main(const argdata_t *ad) signature.
• Rust: No warning is presented at compilation time when the entrypoint function does not query argdata::env::argdata().

When this happens, the user observes the resulting CloudABI executable emit no output. It would be better for the runtime to present a dummy warning that the entrypoint was misconfigured. Best of all would be to detect this at compile time, and refuse to produce a binary with the wrong entrypoint.

localtime_l and mktime_l in a independent shared library

I'm curious what you think about the idea to release localtime_l and mktime_l as two functions in a independent shared library. Until today, i have not found a good library that does what these functions do, purely converting between timezones based on the IANA timezone database.

Because such a library does not exists, i integrated your localtime_l code inside my own program, but that requires (as you do now) a new release as soon as the IANA database is updated. If these functions would exist in a independent shared library, that update mechanism would be far easier, by just updating the shared library.

I also think a lot of C developers would be interested in these two functions alone without having to use the full cloudlibc library.

Inject custom clocks, RNG's

I saw the awesome talk on CloudABI and think it would be a good idea to begin offering dependency injection for the system clock and for random number generation, for even richer application tests. Could we begin designing an API towards that end?

poll test fails on linux: Errno: 52, Function not implemented

Is this a known limitation?

I first ran into this when using python:

  File "selectors.py", line 376, in select
OSError: [Errno 52] Function not implemented

Then I confirmed it using cloudlibc-unittests:

$ cloudabi-run /usr/x86_64-unknown-cloudabi/bin/cloudlibc-unittests << EOF
> %TAG ! tag:nuxi.nl,2015:cloudabi/
> ---
> tmpdir: !file
>   path: tmp-unittest
> logfile: !fd stdout
> nthreads: !!int 8
> EOF
-> argdata_serialize::seq
-> mblen::euro
-> fgetpos::eoverflow
-> fputws::success
-> wcsncpy::example1
-> mbstowcs::zero
-> malloc::zero
-> poll::file
-> twalk::empty
-> ntohl::example
-> strndup::null
Test failed
--
Statement: ASSERT_EQ(1, poll(&pfd, 1, -1))
Expected:           1 ==        0x1 == (1)
Actual:            -1 == 0xffffffff == (poll(&pfd, 1, -1))
Location:  src/libc/poll/poll_test.c:168
Errno:     52, Function not implemented

$ uname -a
Linux pav 4.7.0-cloudabi+ #1 SMP Sat Apr 8 10:56:21 CDT 2017 x86_64 x86_64 x86_64 GNU/Linux

$ (cd ~/projects/linux-cloudabi/; git log -n1)
commit 477884e20e3c68d3524a42864900a5d3bbb0e1b4
Author: Ed Schouten <[email protected]>
Date:   Sun Feb 26 22:02:46 2017 +0100

    Add missing cloudabi_vdso.lds linker script.
    
    Reported by: Dan Connolly

i686-unknown-cloudabi-cc: fatal error: 'stdio.h' file not found

Hello, I am trying to figure out if I can build 32-bit binaries with the CloudABI toolchain on macOS. I can build and run 64-bit binaries with x86_64-unknown-cloudabi-cc, but when I use the i686 variant, I get an error about a missing header file, stdio.h. I get the same behavior in Docker as well.

Do I need to install some additional LLVM components, or pass some new flags to the compiler, to get this to work?

Same error for the aarch64 and armv6 eabihf compilers.

Support win32.

That's would be great if cloundlibc would support for win32.

Unit testing binary tries to create the same directory twice?

As observed here: NuxiNL/debian-cloudabi#2

root@88e39bb79e41:/# cloudabi-run -e /usr/x86_64-unknown-cloudabi/bin/cloudlibc-unittests << EOF
%TAG ! tag:nuxi.nl,2015:cloudabi/
---
tmpdir: !file
  path: tmp-unittest
logfile: !fd stdout
EOF
WARNING: Attempting to start executable using emulation.
Keep in mind that this emulation provides no actual sandboxing.
Though this is likely no problem for development and testing
purposes, using this emulator in production is strongly
discouraged.
-> drand48::bounds
-> fdopen::bad
-> localtime_l::santiago
-> fflush::eagain
-> wcsrtombs::ascii_null_ok
-> wmemset::example
-> call_once::example
-> clock_nanosleep::monotonic_relative
-> fread::zero
-> unlinkat::examples
-> uv_ip4_addr::einval
-> wctomb::euro
-> scandirat::even_files
Test failed
--
Statement: ASSERT_EQ(0, mkdirat(state->tmpdir, test->__name))
Expected:           0 ==          0 == (0)
Actual:            -1 == 0xffffffff == (mkdirat(state->tmpdir, test->__name))
Location:  src/libc/testing/testing_execute.c:82
Errno:     20, File exists
Aborted (core dumped)

This is usually an indicator that two TEST()s are defined with the same name, causing them to race for the same unit testing data directory. A git grep doesn't seem to indicate this is the case. We should add some diagnostics to testing_execute to print which directory it was trying to create.

CMake having trouble finding sprintf / stdio.h

Hey, I'm glad to report mild success with CloudABI, building some very simple C and Rust apps! Unfortunately, when I try to use sprintf, I'm getting build errors. Am I doing something wrong?

Source

My "issue-115" branch of my tonixxx project, in the examples/fewer C project directory:

https://github.com/mcandre/tonixxx/tree/issue-115/examples/fewer

Trace (Ubuntu)

$ CC=x86_64-unknown-cloudabi-cc CXX=x86_64-unknown-cloudabi-c++ cmake .
$ cmake --build . --config Release
[ 33%] Linking C executable bin/fewer
/usr/bin/x86_64-unknown-cloudabi-ld: error: undefined symbol: sprintf
>>> referenced by fewer.c
>>>               CMakeFiles/fewer.dir/lib/fewer.c.o:(render_boi)
clang: error: linker command failed with exit code 1 (use -v to see invocation)
CMakeFiles/fewer.dir/build.make:120: recipe for target 'bin/fewer' failed
make[2]: *** [bin/fewer] Error 1
CMakeFiles/Makefile2:227: recipe for target 'CMakeFiles/fewer.dir/all' failed
make[1]: *** [CMakeFiles/fewer.dir/all] Error 2
Makefile:94: recipe for target 'all' failed
make: *** [all] Error 2

This is weird, as I am able to access other cloudlibc entities such as dprintf() and fopen() just fine in my main.c source file. Perhaps cmake is getting confused about where libc comes from when dealing with the separate file fewer.c?

Note: I am running these commands from an Ubuntu 18.04 Bionic Beaver instance in VirtualBox on macOS. When I try to build this CMake project directly on my macOS host, I can't even get the initial CMake cache to complete:

Trace (macOS)

$ CC=x86_64-unknown-cloudabi-cc CXX=x86_64-unknown-cloudabi-c++ cmake .
-- The C compiler identification is Clang 6.0.0
-- The CXX compiler identification is Clang 6.0.0
-- Check for working C compiler: /usr/local/bin/x86_64-unknown-cloudabi-cc
-- Check for working C compiler: /usr/local/bin/x86_64-unknown-cloudabi-cc -- broken
CMake Error at /usr/local/Cellar/cmake/3.13.2/share/cmake/Modules/CMakeTestCCompiler.cmake:52 (message):
  The C compiler

    "/usr/local/bin/x86_64-unknown-cloudabi-cc"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: /Users/andrew/go/src/github.com/mcandre/tonixxx/examples/fewer/CMakeFiles/CMakeTmp
    
    Run Build Command:"/usr/local/bin/gmake" "cmTC_9b5a3/fast"
    /usr/local/bin/gmake -f CMakeFiles/cmTC_9b5a3.dir/build.make CMakeFiles/cmTC_9b5a3.dir/build
    gmake[1]: Entering directory '/Users/andrew/go/src/github.com/mcandre/tonixxx/examples/fewer/CMakeFiles/CMakeTmp'
    Building C object CMakeFiles/cmTC_9b5a3.dir/testCCompiler.c.o
    /usr/local/bin/x86_64-unknown-cloudabi-cc    -o CMakeFiles/cmTC_9b5a3.dir/testCCompiler.c.o   -c /Users/andrew/go/src/github.com/mcandre/tonixxx/examples/fewer/CMakeFiles/CMakeTmp/testCCompiler.c
    Linking C executable cmTC_9b5a3
    /usr/local/Cellar/cmake/3.13.2/bin/cmake -E cmake_link_script CMakeFiles/cmTC_9b5a3.dir/link.txt --verbose=1
    /usr/local/bin/x86_64-unknown-cloudabi-cc   -Wl,-search_paths_first -Wl,-headerpad_max_install_names   CMakeFiles/cmTC_9b5a3.dir/testCCompiler.c.o  -o cmTC_9b5a3 
    /usr/local/bin/x86_64-unknown-cloudabi-ld: error: unknown argument: -search_paths_first
    clang-6.0: error: linker command failed with exit code 1 (use -v to see invocation)
    gmake[1]: *** [CMakeFiles/cmTC_9b5a3.dir/build.make:87: cmTC_9b5a3] Error 1
    gmake[1]: Leaving directory '/Users/andrew/go/src/github.com/mcandre/tonixxx/examples/fewer/CMakeFiles/CMakeTmp'
    gmake: *** [Makefile:121: cmTC_9b5a3/fast] Error 2
    

  

  CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
  CMakeLists.txt:3 (project)

Update

I spent some more time configuring my CMakeLists.txt so that the compiler options are closer to the cloudabi compiler defaults, but I am still getting this strange error. Clearly, fewer.c is #includeing the <stdio.h> file, so I am scratching my head wondering what part of this system is dropping the ball.

One thing I've noticed is that cmake tends to compile executables in multiple stages, compared to using x86_64-unknown-cloudabi-cc directly. Perhaps the compiler is dropping information about where libraries are defined, when object files are built and linked, versus building an executable all at once?

WebAssembly target

I'm entertaining the idea of using clouldabi in WebAssembly envirnment. I forked the project at https://github.com/yurydelendik/cloudlibc and it works for simple example now. So far I identified following items that will be nice to upstream:

• allow crt1.o output (current is crt0.o)
• use #define LG_PAGE 16 for wasm
• disable vDSO logic
• add/stub logic for fenv
• decide what to do with auxiliary vector

If there is an interest in accepting to above changes, I will be willing to submit pull requests.

.DEB package

Could we get .DEB packages, to automate the process of installing CloudABI toolchains, including the cloudabi-run utility? That would make it much easier for Debian/Ubuntu users to access CloudABI!

Portable thread local storage

Some microkernels (NOVA) keep segment registers fixed across context switches for the sake of performance. The problem is that is breaks compatibility with binaries that abuse FS and GS for thread local storage.

Would there be an interest in replacing the inline TLS dereferencing code that uses %fs with a function call than can be placed in the vDSO?

How to assert()?

Could cloudlibc offer a dassert() macro that accepts a file descriptor for emitting assertion errors? Is this even possible, given that assert() tends to hook deep into C++ throw?

Alternatively, NuxiNL/cloudabi-utils#11 would be fantastic!

I suspect this may be possible with dup2(), though I'm not sure how portable that function is.

Query program name programmatically

Lacking the traditional int argc, char **argv entrypoint parameters for querying command line arguments, could program.h provide another way to obtain the name of the program? This is often useful for displaying accurate help command syntax for downstream users.

Windows support

Could we get a Windows port of the cloudlibc runtime and an example cmake project illustrating how to build CloudABI binaries from Windows development hosts? So that we can grow the deployment space for CloudABI apps!

Supporting DragonflyBSD

How feasible would it be to support CloudABI on DragonflyBSD?

Are there any prerequisites to supporting CloudABI that DragonflyBSD currently lacks?

What sort of changes would need to be made to accomplish this?

Gate the system clock

cloudlibc offers engineers a tremendous gatekeeper for I/O, writing applications just once in terms of a simple API, and flexibly reconfiguring applications for many different workflows for runtime. Let's extend the same gatekeeping to calls for the current date and time. This is not so much a security measure, as a safety measure:

Gating the system clock makes it much easier to test complex applications for different clock values, such as several years in the future when important SSL certificates expire, and when 2038 breaks legacy applications. While developers are advised to program in terms of an injectable clock, most applications are not written this way. By intercepting these calls, we offer utility for integration testing this part of the total application input space.

For many systems, there will come a day when the true date breaks the application, and a convenient way to frequently reset the date to a historical value, will be the only way to resurrect these applications.

More ports!

cloudlibc has a fantastic number of ports available. Motion to spread support even wider!

• HardenedBSD (basically FreeBSD with W^X)
• OpenBSD
• Alpine (musl libc)
• Void Linux (musl libc)
• Busybox uClibC
• Haiku (missing dprintf, easy enough to shim)
• SmartOS (needs Gnulib for dprintf, openat)
• Windows (cygwin)
• Windows (MVSC), note the use of Windows file handle structures rather than ind fd’s

update tzdata to 2019 release

A new release of time zone information has been made in september 2019:
https://www.iana.org/time-zones

Provide writeable console implementation in the Rust library

The Rust tutorial for developing CloudABI programs requires the developer to write a substantial amount of unsafe code by hand in order to be able to use cloudabi::fd's with typical Rust writing functions and macros. Could we include this console write implementation in the crate by default, so that developers can use cloudabi in a more safe and convenient manner?

https://cloudabi.org/write/rust/

setvbuf::iolbf_pipe test fails on Linux

When I run the following command:

cloudabi-run /usr/x86_64-unknown-cloudabi/bin/cloudlibc-unittests << EOF
%TAG ! tag:nuxi.nl,2015:cloudabi/
---
tmpdir: !file
  path: tmp-unittest
logfile: !fd stdout
nthreads: !!int 1
EOF

one of the test cases fails with the following error:

[passing tests omitted]
-> setvbuf::iolbf_pipe
Test failed
--
Statement: ASSERT_EQ(0, ioctl(fds[0], FIONREAD, &nbytes))
Expected:           0 ==          0 == (0)
Actual:            -1 == 0xffffffff == (ioctl(fds[0], FIONREAD, &nbytes))
Location:  src/libc/stdio/setvbuf_test.c:100
Errno:     52, Function not implemented
Aborted

Extra info:

# Linux version
vagrant@vagrant:~$ uname -a
Linux vagrant 4.7.0-cloudabi+ #1 SMP Fri Jul 21 12:03:48 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

# CloudABI version
vagrant@vagrant:~$ apt-cache show x86-64-unknown-cloudabi-cloudlibc 
Package: x86-64-unknown-cloudabi-cloudlibc
Version: 0.78-1
Architecture: all
Maintainer: [email protected]
Description: cloudlibc for x86_64-unknown-cloudabi
Description-md5: 5b87b85f313f06f0c58f75cc46309a30
Homepage: https://github.com/NuxiNL/cloudlibc
Depends: x86-64-unknown-cloudabi-cloudabi, x86-64-unknown-cloudabi-compiler-rt
Filename: x86-64-unknown-cloudabi-cloudlibc_0.78-1_all.deb
Size: 1196348
SHA256: adecbabc32e77944d5c3c463fa73cf0d1b687a89fade4d3d0e91d6467a3ddca9

Package: x86-64-unknown-cloudabi-cloudlibc
Status: install ok installed
Installed-Size: 7719
Maintainer: [email protected]
Architecture: all
Version: 0.78-1
Depends: x86-64-unknown-cloudabi-cloudabi, x86-64-unknown-cloudabi-compiler-rt
Description: cloudlibc for x86_64-unknown-cloudabi
Description-md5: 5b87b85f313f06f0c58f75cc46309a30

Hello World stdout fd blackholes with some posted tutorials

Hello, I am trying to write my first CloudABI program, following along with the tutorials. I have a Clang-based CloudABI toolchain installed, along with cloudabi-run.

hello.c:

#include <stdio.h>

int main(void) {
    dprintf(1, "Hello World!\n");
}

cloudabi.yml:

%TAG ! tag:nuxi.nl,2015:cloudabi/
---
- !fd stdout

Trace:

$ x86_64-unknown-cloudabi-cc -o hello hello.c
$ cloudabi-run -e hello <cloudabi.yml
WARNING: Attempting to start executable using emulation.
Keep in mind that this emulation provides no actual sandboxing.
Though this is likely no problem for development and testing
purposes, using this emulator in production is strongly
discouraged.

So Hello World is never printed; no segmentation fault occurs, no error message about stdout access is presented. Furthermore, execution exits with a zero exit code, indicating "success".

Update

I looked at some more examples and see that file descriptor 1 appears to no longer work out of the box as stdout in CloudABI. I changed my code to:

#include <stdio.h>
#include <stdlib.h>

int m(int stdout) {
    dprintf(stdout, "Hello World!\n");
    return EXIT_SUCCESS;
}

#ifdef __CloudABI__
    #include <argdata.h>
    #include <program.h>

    void program_main(const argdata_t *ad) {
        int stdout;
        argdata_get_fd(ad, &stdout);
#else
    #include <unistd.h>

    void main() {
        int stdout = STDOUT_FILENO;
#endif
        exit(m(stdout));
    }

And am now able to build and run my lil app! This one is a polyglot, so it compiles and runs with either plain vanilla Clang or with CloudABI. Also, CloudABI appears to work with more modern Clang/LLVM/LLD versions, including v6.0. Could we update the documentation to reflect this?

Could we update the different per-OS tutorials to fix the stdout file descriptor part?

A larger question is why are guarded? I thought CloudABI was meant to protect sensitive components that can break a system. Do stdout/stderr somehow contribute to an increased attack surface?

Is cloudlibc versioned in any way?

I'm assuming that at some point calls would be added or possibly removed, and one might need preprocessor definitions to understand which version of cloudlibc is available.

I found the following in include/uv.h but that file is for event handling:

//
// Version-checking macros and functions.
//

#define UV_VERSION_MAJOR 1
#define UV_VERSION_MINOR 15
#define UV_VERSION_PATCH 0
#define UV_VERSION_IS_RELEASE 0
#define UV_VERSION_SUFFIX "cloudlibc"

#define UV_VERSION_HEX \
  (UV_VERSION_MAJOR << 16 | UV_VERSION_MINOR << 8 | UV_VERSION_PATCH)

__BEGIN_DECLS
unsigned int uv_version(void);
const char *uv_version_string(void);
__END_DECLS

Is this official versioning for the entire library?

Debian runtime packages appear to be empty?

$ dpkg --listfiles x86-64-unknown-cloudabi-c-runtime
/.
$ dpkg --listfiles x86-64-unknown-cloudabi-cxx-runtime
/.

Am I doing this right?