nuxinl / cloudabi-ports Goto Github PK

View Code? Open in Web Editor NEW

22.0 22.0 7.0 1.72 MB

Packages for CloudABI

License: Other

Python 51.55% Shell 0.31% C 4.16% C++ 0.36% Starlark 43.62%

capabilities capsicum packages sandbox unix

cloudabi-ports's People

Contributors

Stargazers

Watchers

Forkers

thedharc moreati aintnoninja dckc sgielen euclio

cloudabi-ports's Issues

how to provide directories to python programs?

The example for cloudabi for FreeBSD shows passing a directory:

tmpdir: !file
  path: tmp-unittest

I tried to follow this example but I got:

% cloudabi-run /usr/local/x86_64-unknown-cloudabi/bin/python3 < zv.yaml
Fatal Python error: can't create sys.argdata
IsADirectoryError: [Errno 31] Is a directory: 3

Current thread 0x00007fffffffcd20 (most recent call first):
Abort

This is the yaml spec I tried:

% cat zv.yaml
%TAG ! tag:nuxi.nl,2015:cloudabi/
---
stderr: !fd stderr               # Let Python write backtraces to stderr.
path: !file                      # Search path for Python modules.
    path: /usr/local/x86_64-unknown-cloudabi/lib/python3.6
cwd: !file                       # Local (in-development) python modules.
    path: .
args:                            # Arguments exposed as sys.argdata.
    stuff: junk
    cwd: !file
        path: .
command: >-                      # Inline Python script, like 'python -c'.
    from sys import argdata, stderr;
    import main;
    print(argdata['cwd'], file=stderr)

nosite and nousersite: shouldn't those be the default?

Hi @moreati,

Maybe I'm simplifying things by a bit too much, but what's the reason why we have nosite and nousersite in our argdata parsing? Wouldn't it make more sense to turn path into a sequence instead of a file descriptor and use that to list all the standard paths?

The point is, without nosite set, Python even fails to start. Ideally Python should work with only a small number of flags set.

Bad C99 features test fails with UnicodeDecodeError in m4-1.4.17/configure

I can't seem to actually build any packages.

The Bad C99 features test in src/builder.py (line 175) seems to expect ASCII:

                with open(path, 'r') as fin, open(path + '.new', 'w') as fout:
                    for l in fin.readlines():
                        # Bad C99 features test.
                        if l.startswith('#define showlist(...)'):

but there are non-ASCII characters (nonascii.py is given below):

root@8c58f1859b17:/cloudabi-ports# python3 nonascii.py /usr/obj/cloudabi-ports/build/0/m4-1.4.17/configure
25454:b'      const char input[] = "B\\217\\253\\344\\217\\251\\316er"; /* "B\xc3\xbc\xc3\x9fer" */\n'
 ==> [(63, 195), (64, 188), (65, 195), (66, 159)]
...

The result is:

root@8c58f1859b17:/cloudabi-ports# python3 -i ./build_packages.py expatINFO:src.package:BUILD m4
INFO:src.distfile:CHECKSUM /cloudabi-ports/_obj/distfiles/m4-1.4.17.tar.xz
Traceback (most recent call last):
  File "./build_packages.py", line 64, in <module>
    build_package(target_packages[(name, arch)])
...
  File "/cloudabi-ports/src/builder.py", line 76, in gnu_configure
    for l in fin.readlines():
  File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 5709: ordinal not in range(128)

nonascii.py is:

import sys

for row, line in enumerate(open(sys.argv[1], 'rb').readlines()):
    non_ascii = [(col, b) for (col, b) in enumerate(line) if b > 127]
    if non_ascii:
        print('{row}:{line}\n ==> {other}'.format(
            row=row, line=line, other=non_ascii))

Environment

I'm running on Ubuntu 16.04 in a docker container: Dockerfile gist

I used docker build -t cloudabi-toolchain . to build the docker image and then docker run --rm -ti -v /usr/local/src/cloudabi-ports:/cloudabi-ports cloudabi-toolchain bash to start the container. I have the current version of the code (e60305b) checked out in /usr/local/src/cloudabi-ports.

Python's 'path' needs to be a list instead of a single path object

Right now 'path' in the Python schema needs to be a single element. This is obviously flawed, as in practice you'd want to be able to set multiple include paths.

build_packages.py leaves behind read-only build artefacts

Probably just a --clean option or a clean.py would do, for now I run find /usr/obj/cloudabi-ports/* -exec chmod u+w {} \; && rm -rf /usr/obj/cloudabi-ports/*

Sample files

$ find /usr/obj/cloudabi-ports/* -ls
28430526    0 dr-xr-xr-x   1 alex     alex           28 Feb  9 16:42 /usr/obj/cloudabi-ports/control
28430527    4 -r--r--r--   1 alex     alex          277 Feb  9 16:42 /usr/obj/cloudabi-ports/control/control
28430528    4 -r--r--r--   1 alex     alex         1038 Feb  9 16:42 /usr/obj/cloudabi-ports/control/md5sums
28430502    0 dr-xr-xr-x   1 alex     alex            6 Feb  9 16:42 /usr/obj/cloudabi-ports/data
28430503    0 dr-xr-xr-x   1 alex     alex           46 Feb  9 16:42 /usr/obj/cloudabi-ports/data/usr
28430504    0 dr-xr-xr-x   1 alex     alex           30 Feb  9 16:42 /usr/obj/cloudabi-ports/data/usr/x86_64-unknown-cloudabi
28430505    0 dr-xr-xr-x   1 alex     alex           60 Feb  9 16:42 /usr/obj/cloudabi-ports/data/usr/x86_64-unknown-cloudabi/lib
28430506    4 -r--r--r--   1 alex     alex          890 Feb  9 16:42 /usr/obj/cloudabi-ports/data/usr/x86_64-unknown-cloudabi/lib/libspeex.la
28430507  252 -r--r--r--   1 alex     alex       256786 Feb  9 16:42 /usr/obj/cloudabi-ports/data/usr/x86_64-unknown-cloudabi/lib/libspeex.a
28430508    0 dr-xr-xr-x   1 alex     alex           16 Feb  9 16:42 /usr/obj/cloudabi-ports/data/usr/x86_64-unknown-cloudabi/lib/pkgconfig
28430509    4 -r--r--r--   1 alex     alex          320 Feb  9 16:42 /usr/obj/cloudabi-ports/data/usr/x86_64-unknown-cloudabi/lib/pkgconfig/speex.pc
28430510    0 dr-xr-xr-x   1 alex     alex           10 Feb  9 16:42 /usr/obj/cloudabi-ports/data/usr/x86_64-unknown-cloudabi/include
28430511    0 dr-xr-xr-x   1 alex     alex          194 Feb  9 16:42 /usr/obj/cloudabi-ports/data/usr/x86_64-unknown-cloudabi/include/speex
28430512    4 -r--r--r--   1 alex     alex          203 Feb  9 16:42 /usr/obj/cloudabi-ports/data/usr/x86_64-unknown-cloudabi/include/speex/speex_config_types.h
...

LLVM SafeStackColoring calculateLiveIntervals assertions

Not sure where to file this, but I'm experimenting with porting a moderately complex C++ package to cloudabi (Bitcoin Core).

In two files I've got the following crash while compiling:

Assertion failed: (!Started.test(AllocaNo)), function calculateLiveIntervals, file /wrkdirs/usr/ports/devel/llvm39/work/llvm-3.9.1.src/lib/CodeGen/SafeStackColoring.cpp, line 217.

It looks like: https://bugs.llvm.org//show_bug.cgi?id=30318

Versions:

Platform: FreeBSD 11.0-RELEASE-p2
cloudabi-toolchain-1.7         C and C++ toolchain for CloudABI
cloudabi-utils-0.18_1          Utilities for running CloudABI programs

Attached debug files:

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang-3.9: note: diagnostic msg: /tmp/server-ed8244.cpp
clang-3.9: note: diagnostic msg: /tmp/server-ed8244.sh
clang-3.9: note: diagnostic msg:

Associated debug files: server-ed8244.tar.gz

libc++abi can't find unwind symbols

Building a c++ program on FreeBSD -CURRENT with latest packages from the cloudabi pkg repo:

x86_64-unknown-cloudabi-ld: error: undefined symbol: _Unwind_RaiseException
>>> referenced by /usr/obj/cloudabi-ports/build/0/libcxxabi-5.0.0.src/src/cxa_exception.cpp
>>>               cxa_exception.cpp.o:(__cxa_throw) in archive /usr/local/x86_64-unknown-cloudabi/lib/libc++abi.a

x86_64-unknown-cloudabi-ld: error: undefined symbol: _Unwind_DeleteException
>>> referenced by /usr/obj/cloudabi-ports/build/0/libcxxabi-5.0.0.src/src/cxa_exception.cpp
>>>               cxa_exception.cpp.o:(__cxa_end_catch) in archive /usr/local/x86_64-unknown-cloudabi/lib/libc++abi.a

x86_64-unknown-cloudabi-ld: error: undefined symbol: _Unwind_RaiseException
>>> referenced by /usr/obj/cloudabi-ports/build/0/libcxxabi-5.0.0.src/src/cxa_exception.cpp
>>>               cxa_exception.cpp.o:(__cxa_rethrow) in archive /usr/local/x86_64-unknown-cloudabi/lib/libc++abi.a

x86_64-unknown-cloudabi-ld: error: undefined symbol: _Unwind_Resume
>>> referenced by /usr/obj/cloudabi-ports/build/0/libcxxabi-5.0.0.src/src/cxa_demangle.cpp
>>>               cxa_demangle.cpp.o:(__cxa_demangle) in archive /usr/local/x86_64-unknown-cloudabi/lib/libc++abi.a

x86_64-unknown-cloudabi-ld: error: undefined symbol: _Unwind_Resume
>>> referenced by /usr/obj/cloudabi-ports/build/0/libcxxabi-5.0.0.src/src/cxa_demangle.cpp
>>>               cxa_demangle.cpp.o:(__cxa_demangle) in archive /usr/local/x86_64-unknown-cloudabi/lib/libc++abi.a

They do exist in libunwind…

% objdump -t /usr/local/x86_64-unknown-cloudabi/lib/libunwind.a | sr Unwind_Resume
000000000000005a l     O .rodata.str1.1 000000000000001a .L__func__._Unwind_Resume_or_Rethrow
0000000000000000 l    d  .text._Unwind_Resume_or_Rethrow        0000000000000000 .text._Unwind_Resume_or_Rethrow
0000000000000000         *UND*  0000000000000000 _Unwind_Resume
0000000000000000 g     F .text._Unwind_Resume_or_Rethrow        0000000000000083 _Unwind_Resume_or_Rethrow
000000000000006e l     O .rodata.str1.1 000000000000000f .L__func__._Unwind_Resume
0000000000000000 l    d  .text._Unwind_Resume   0000000000000000 .text._Unwind_Resume
0000000000000000 g     F .text._Unwind_Resume   00000000000000c4 _Unwind_Resume

-fno-exceptions works, but then I don't have exceptions…

Link problem leveldb: missing leveldb::port::AcceleratedCRC32C

I get the following linker error while linking bitcoind, which links against cloudabi's leveldb:

libtool: link: x86_64-unknown-cloudabi-c++ -std=c++1z -Wstack-protector -fstack-protector-all -fPIE -g -O2 -Wall -Wextra -Wformat -Wvla -Wformat-security -Wno-unused-parameter -Wno-self-assign -Wno-unused-local-typedef -Wno-deprecated-register -Qunused-arguments -pthread -Wl,-z -Wl,relro -Wl,-z -Wl,now -pie -fno-sanitize=safe-stack -o bitcoind bitcoind-bitcoind.o  libbitcoin_server.a libbitcoin_common.a univalue/.libs/libunivalue.a libbitcoin_util.a libbitcoin_zmq.a libbitcoin_consensus.a crypto/libbitcoin_crypto.a -lleveldb /usr/local/x86_64-unknown-cloudabi/lib/libsnappy.a -lmemenv secp256k1/.libs/libsecp256k1.a -L/usr/local/x86_64-unknown-cloudabi/lib -lboost_system -lboost_filesystem -lboost_program_options -lboost_thread -lboost_chrono /usr/local/x86_64-unknown-cloudabi/lib/libssl.a /usr/local/x86_64-unknown-cloudabi/lib/libcrypto.a /usr/local/x86_64-unknown-cloudabi/lib/libevent_pthreads.a /usr/local/x86_64-unknown-cloudabi/lib/libevent.a -lzmq -pthread
/usr/local/bin/x86_64-unknown-cloudabi-ld: error: util/crc32c.cc:(function leveldb::crc32c::Extend(unsigned int, char const*, unsigned long)): undefined symbol 'leveldb::port::AcceleratedCRC32C(unsigned int, char const*, unsigned long)'
/usr/local/bin/x86_64-unknown-cloudabi-ld: error: util/crc32c.cc:(function leveldb::crc32c::Extend(unsigned int, char const*, unsigned long)): undefined symbol 'leveldb::port::AcceleratedCRC32C(unsigned int, char const*, unsigned long)'

AcceleratedCRC32C was introduced in the latest version of leveldb to accommodate architecture-specific optimizations for crc instructions, could this be something missing in the .a files or is a library missing from the link?

If this is non-straightforward I could try to create a minimal test case.

cgi.FieldStorage fails due to missing LC_TYPE

To reproduce:

% cloudabi-run /usr/local/x86_64-unknown-cloudabi/bin/python3 < bug2.yaml
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "cgi.py", line 461, in __init__
  File "locale.py", line 639, in getpreferredencoding
NameError: name 'LC_CTYPE' is not defined

where bug2.yaml looks like:

%TAG ! tag:nuxi.nl,2015:cloudabi/
---
stderr: !fd stderr               # Let Python write backtraces to stderr.
path:
    - !file                      # Search path for Python modules.
      path: /usr/local/x86_64-unknown-cloudabi/lib/python3.6
command: >
    from sys import stderr; import cgi; from io import BytesIO;
    cgi.FieldStorage(fp=BytesIO(), environ={'QUERY_STRING': 'a=b'})

for my reference:
https://github.com/dckc/ZeroVault/tree/cloudabi_wsgi 23fd185

Tracking bug: ZeroMQ

Hi @laanwj,

As I mentioned previously, I've been working towards getting ZeroMQ to work on CloudABI. I wanted to use this issue to keep track of progress. Anyway, the latest build of CloudABI Ports now includes a ZeroMQ package. To summarise:

zmq_connect() has been disabled. There is no way to connect to queues.
zmq_bind() is a bit weird: you have to use it both when letting the library create the socket, but also when passing one in. This means that we don't have a way of automatically detecting code that needs to be patched up to bind.

Anyway, I still need to test ZeroMQ a bit more extensively. At least this piece of code already builds and runs properly:

#include <program.h>
#include <stdio.h>
#include <zmq.h>

#include <argdata.hpp>
#include <cassert>
#include <cstdlib>
#include <string_view>

void program_main(const argdata_t *ad) {
  int s = -1;
  for (auto [key, value] : ad->as_map()) {
    std::string_view keystr = key->as_str();
    if (keystr == "terminal") {
      FILE *fp = fdopen(value->as_fd(), "w");
      if (fp != nullptr) {
        setvbuf(fp, nullptr, _IONBF, 0);
        fswap(fp, stderr);
        fclose(fp);
      }
    } else if (keystr == "socket") {
      s = value->as_fd();
    }
  }

  void *context = zmq_ctx_new();
  assert(context != nullptr);
  void *socket = zmq_socket(context, ZMQ_PUB);
  assert(socket != nullptr);
  int rc = zmq_setsockopt(socket, ZMQ_USE_FD, &s, sizeof(s));
  assert(rc == 0);
  rc = zmq_bind(socket, "ipc:///");
  assert(rc == 0);

  std::exit(0);
}

It can be used in combination with this config:

%TAG ! tag:nuxi.nl,2015:cloudabi/
---
terminal: !fd stdout
socket: !socket
  bind: /tmp/foo

Debian unstable complains the cloudabi-ports is insufficiently signed (weak digest)

I don't consider this urgent in any way, I'm just recording it so I have a reference.

alex@bertha:~$ sudo apt update
Hit:1 http://mirror.lchost.net/debian unstable InRelease
Hit:2 http://mirror.lchost.net/debian experimental InRelease                   
Hit:3 https://nuxi.nl/distfiles/cloudabi-ports/debian cloudabi InRelease       
Reading package lists... Done 
Building dependency tree       
Reading state information... Done
6 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: gpgv:/var/lib/apt/lists/nuxi.nl_distfiles_cloudabi-ports_debian_dists_cloudabi_InRelease: The repository is insufficiently signed by key 1EAC388A6C0D65721A2237430DA51B8531344B15 (weak digest)

For reference I'm running apt 1.2.7 and dpkg 1.18.4-88-g2adbf-dirty.

aarch64-unknown-cloudabi-freetype fails to build on Ubuntu 15.10

I'm not sure if this is relevant or noteworthy. Please feel free to e.g. close it as wontfix.

On MacbookPro11,2 running Ubuntu 15.10 the following error occurs while building freetype

checking for x86_64-pc-linux-gnu-gcc... /usr/bin/cc
checking for x86_64-pc-linux-gnu-gcc... (cached) /usr/bin/cc
checking for x86_64-pc-linux-gnu-gcc... (cached) /usr/bin/cc
checking for suffix of native executables... gcc-5.real: error trying to exec 'as': execvp: No such file or directory

It appears that cloudabi-ports/src/config.py unconditionally supplies CC_FOR_BUILD=/usr/bin/cc. On FreeBSD 10.2 this is clang 3.4, on Debian & Ubuntu it's gcc. Would you like a PR making this more like the HOST_CC and HOST_CXX platform dependent (like PERL)?

For reference I'm using the following locally, while I learn the ins and outs of cloudabi-ports

diff --git a/src/config.py b/src/config.py
index 6d0f901..709d798 100644
--- a/src/config.py
+++ b/src/config.py
@@ -25,2 +25,4 @@ FALLBACK_MIRRORS = {'https://nuxi.nl/distfiles/third_party/'}
-HOST_CC = '/usr/bin/cc'
-HOST_CXX = '/usr/bin/c++'
+HOST_CC = ('/usr/bin/clang-3.7' if platform.system() == 'Linux' else
+           '/usr/bin/cc')
+HOST_CXX = ('/usr/bin/clang++-3.7' if platform.system() == 'Linux' else
+            '/usr/bin/c++')

LLVM safestack miscompile issue

I had crashes with SafeStack in #28. After upgrading to the 4.0-devel version I'm seeing a more suble problem: some of the bitcoin tests ( key_tests/key_test1) fail. I narrowed this down to RFC6979 nonce generation in the secp256k1 library.

To be precise, this function gets compiled wrongly: https://github.com/bitcoin-core/secp256k1/blob/master/src/hash_impl.h#L235 . This results in invalid deterministic signatures.

This was a bit of a quantum bug - when adding fprintf statements, sometimes the bug went into hiding. However after disabling safe-stack with -fno-sanitize=safe-stack the problem goes away and all the tests pass!

I haven't checked the assembly to see where it exactly goes wrong (can a CloudABI program be single-stepped?) but anyhow, just wanted to report it. I think it's a bug in clang and it may very well have been solved already in SVN.

(I don't currently have enough disk space on my FreeBSD dev VM to build the ports from scratch, that's why I've been stuck with binaries)

aarch64-unknown-cloudabi gmp fails to build on Ubuntu 15.10

I'm not sure if this is relevant or noteworthy. Please feel free to e.g. close it as wontfix.

On MacbookPro11,2 running Ubuntu 15.10 the following error occurs while building gmp

checking build system type... Invalid configuration `haswell-pc-linux-gnu': machine `haswell-pc' not recognized

The cause appears to be that gmp ships a heavily customised config.sub & config.guess, which contain non-standard triplets. Hence when config.sub is replaced by cloudabi-ports/misc/config.sub the build scripts can't recognize the detected build system.

Full log/traceback

BUILD gmp
CHECKSUM /usr/cloudabi-ports/_obj/distfiles/gmp-6.1.0.tar.xz
[Errno 2] No such file or directory: '/usr/cloudabi-ports/_obj/distfiles/gmp-6.1.0.tar.xz'
FETCH http://ftp.gnu.org/gnu/gmp/gmp-6.1.0.tar.xz
CHECKSUM /usr/cloudabi-ports/_obj/distfiles/gmp-6.1.0.tar.xz
checking build system type... Invalid configuration `haswell-pc-linux-gnu': machine `haswell-pc' not recognized
configure: error: /bin/bash /usr/obj/cloudabi-ports/build/0/gmp-6.1.0/config.sub haswell-pc-linux-gnu failed
Traceback (most recent call last):
  File "./build_packages.py", line 55, in <module>
    build_package(package)
  File "./build_packages.py", line 44, in build_package
    catalog.insert(package, version, catalog.package(package, version))
  File "/usr/cloudabi-ports/src/catalog.py", line 446, in package
    package.build()
  File "/usr/cloudabi-ports/src/package.py", line 109, in build
    }, self._lib_depends)
  File "/usr/cloudabi-ports/src/package.py", line 179, in initialize_buildroot
    dep.build()
  File "/usr/cloudabi-ports/src/package.py", line 116, in build
    self._resource_directory))
  File "<string>", line 8, in build
  File "/usr/cloudabi-ports/src/builder.py", line 92, in autoconf
    builddir, os.path.join(self._path, 'configure'), args)
  File "/usr/cloudabi-ports/src/builder.py", line 352, in autoconf
    '--prefix=' + self.get_prefix()] + args)
  File "/usr/cloudabi-ports/src/builder.py", line 460, in run
    ] + command)
  File "/usr/lib/python3.4/subprocess.py", line 561, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['env', '-i', 'AR=/usr/obj/cloudabi-ports/bin/x86_64-unknown-cloudabi-ar', 'CC=/usr/obj/cloudabi-ports/bin/x86_64-unknown-cloudabi-cc', 'CC_FOR_BUILD=/usr/bin/clang-3.7', 'CFLAGS=-O2 -fstack-protector-strong -Werror=implicit-function-declaration -Werror=date-time', 'CXX=/usr/obj/cloudabi-ports/bin/x86_64-unknown-cloudabi-c++', 'CXXFLAGS=-O2 -fstack-protector-strong -Werror=implicit-function-declaration -Werror=date-time', 'CXX_FOR_BUILD=/usr/bin/clang++-3.7', 'NM=/usr/obj/cloudabi-ports/bin/x86_64-unknown-cloudabi-nm', 'OBJDUMP=/usr/obj/cloudabi-ports/bin/x86_64-unknown-cloudabi-objdump', 'PATH=/usr/obj/cloudabi-ports/bin:/usr/obj/cloudabi-ports/bin', 'PERL=/usr/bin/perl', 'PKG_CONFIG=/usr/obj/cloudabi-ports/bin/x86_64-unknown-cloudabi-pkg-config', 'RANLIB=/usr/obj/cloudabi-ports/bin/x86_64-unknown-cloudabi-ranlib', 'STRIP=/usr/obj/cloudabi-ports/bin/x86_64-unknown-cloudabi-strip', '/usr/obj/cloudabi-ports/build/0/gmp-6.1.0/configure', '--host=x86_64-unknown-cloudabi', '--prefix=/sXnLPjhMRwzJweuT', 'gmp_prog_cc_works=yes']' returned non-zero exit status 1

Debian packages for cloudabi toolchain

Currently it's necessary to manually create symlinks, and compile from source to obtain a cloudabi toolchain on Debian/Ubuntu Linux. This ticket is for tracking efforts to fix that.

clang package available
binutils packages available
clang packages in Debian repositories
binutils packages in Debian repositories
clang packages in Ubuntu repositories
binutils packages in Ubuntu repositories

How to invoke boost::tests?

The boost unittest library takes care of providing the main() function, as well as argument parsing.

I'm not sure what to do with this in cloudabi. Is there a defined way to pass in e.g. the file descriptor to print test output to?

pacman packages fail to install

Currently, pacman dies on packages from the cloudabi-ports repository.

First it successfully finishes package verification (or appears to, anyway), and then it says:

failed to commit transaction (invalid or corrupted package)

This is probably because the packages in the ports repository aren't build with makepkg but put together manually. While it's probably a bug with pacman that the error message is so unclear, there's probably a change in the package format that needs to be implemented here as well.

RFC: Mailing list(s)/forum(s) for cloudabi discussion?

Given questions/answers currently on Twitter it would be nice to have a less terse, more dedicated venue to direct people to.

Do you have a preference for

The format (e.g. a mailing list, a web forum, google groups, github issues ...)?
A particular domain (e.g. nuxi.nl, freebsd.org, cloudabi.tld ...)?

Assuming you're happy with the concept, would you like me to set something up?

Status of Python port

In the 32C3 talk it was mentioned that a port of Python to CloudABI is in progress. Is this WIP publicly available? Would you be interested in help finishing it?

Enable W ^ E in V8 for node.js build

As mentioned on twitter https://twitter.com/EdSchouten/status/993144257438502912
cloudabi executables require W^E memory

V8 runtime enabled this by default in this commit v8/v8@bf19e60
However there seems to be a conflict with WASM so the feature has been put behind a flag for now
https://bugs.chromium.org/p/v8/issues/detail?id=7272
The flag is --write-protect-code-memory

I would have thought patching this line https://github.com/nodejs/node/blob/82e475dc75751df08d80bd6e25926bd25c1794f2/deps/v8/src/flag-definitions.h#L675 to true should be enough to get this up and running but any thoughts appreciated.

Intermittent bsdtar failures running build_packages.py

I'm running ./build_packages.py on Ubuntu 16.04. Every so often a source package fails to unpack with the following traceback

...
PKG x86_64-unknown-cloudabi-uriparser-0.8.4p1.tgz
BUILD libtheora
CHECKSUM /home/alex/src/cloudabi-ports/_obj/distfiles/libtheora-1.1.1.tar.bz2

bzip2: I/O or other error, bailing out.  Possible reason follows.
bzip2: Broken pipe
    Input file = (stdin), output file = (stdout)
bsdtar: Child process exited with status 1
bsdtar: Error exit delayed from previous errors.
Traceback (most recent call last):
  File "./build_packages.py", line 63, in <module>
    build_package(package)
  File "./build_packages.py", line 52, in build_package
    catalog.insert(package, version, catalog.package(package, version))
  File "/home/alex/src/cloudabi-ports/src/catalog.py", line 933, in package
    package.build()
  File "/home/alex/src/cloudabi-ports/src/package.py", line 119, in build
    self._resource_directory))
  File "<string>", line 8, in build
  File "/home/alex/src/cloudabi-ports/src/builder.py", line 225, in extract
    ].extract(self._builder._build_directory.get_new_directory())
  File "/home/alex/src/cloudabi-ports/src/distfile.py", line 115, in extract
    target = self._extract_unpatched(target)
  File "/home/alex/src/cloudabi-ports/src/distfile.py", line 80, in _extract_unpatched
    subprocess.check_call([tar, '-xC', target, '-f', self._pathname])
  File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/obj/cloudabi-ports/bin/bsdtar', '-xC', '/usr/obj/cloudabi-ports/build/0', '-f', '/home/alex/src/cloudabi-ports/_obj/distfiles/libtheora-1.1.1.tar.bz2']' returned non-zero exit status 1

Have you seen this happen? A subsequent run will sometimes succeed, and sometimes fail. Ditto for running the bsdtar command directly. The sha256sum libtheora-1.1.1.tar.bz2 matches that in the BUILD file. The error doesn't only occur with libtheora.

In case it's relevant I'm running with the following modification to cloudabi-ports. Which I've found fixes errors when deleting the build directories

diff --git a/src/util.py b/src/util.py
index 42732c1..e1576b0 100644
--- a/src/util.py
+++ b/src/util.py
@@ -112,8 +112,13 @@ def _remove(path):
         shutil.rmtree(path)
     except FileNotFoundError:
         pass
-    except (NotADirectoryError, OSError):
+    except NotADirectoryError:
         os.unlink(path)
+    except OSError:
+        if os.path.islink(path):
+            os.unlink(path)
+        else:
+            raise


 def remove(path):