seanbruno/qemu-bsd-user

qemu bsd user mode with mips64 and arm dependent bits (bsd-user branch)

qemu-bsd-user's Issues

Capsicumized wc fails on mips64 emulation

After the recent capsicumization of wc in freebsd-current, wc fails to run as I expect there are syscalls that we just don't support/emulate here and the default operations need love on mips64.

Build a mips64 jail with poudriere, start it and enter it:

poudriere jail -c -j mips64test -m svn -v head -a mips.mips64 -x
poudriere jail -s -j mips64-test
jexec mips64-test-default

Run a bare "wc" with no input against /dev/null. It will hang until I try to get a siginfo dump from it via ctrl-t and then it will emit an error.

root@13mips64-test-default:/ # wc /dev/null
load: 0.06  cmd: qemu-mips64-static 65433 [select] 1.93r 0.01u 0.00s 0% 12104k
wc: Unable to init casper: Resource temporarily unavailable

Previously to the changes in head, wc would return something sensible:

root@13mips64-default:/ # wc /dev/null
       0       0       0 /dev/null

auditon() syscall unsupported

Noted that qemu-bsd-user doesn't have support for the auditon() syscall after exiting from an ARM jail:

root@11armv6-default:/wrkdirs/usr/ports/www/webkit2-gtk3/work/webkitgtk-2.8.5 # exit
logout
qemu: Unsupported syscall auditon()

-fstack-protector issues

Originally reported in #74 and since worked around in ports; qemu does something to irritate the stack protector. The ports workaround is effective for the old 2.x version still in ports, but 3.1 is apparently much worse about it - we hit ssp failure even before tcg is fully initialized, and even earlier sometimes if -L is used - even without the -strong variant.

We'll need to diagnose and fix it, if possible, as it consistently fails as of now.

posix_spawn no longer working on arm*

posix_spawn no longer works on arm* on head

A side effect is that packages are not fully installed

pkg-static: Cannot runscript POST-INSTALL:Operation not permitted

casper issue on mips*

There are still issues with capsicumized stuff on mips*, for instance when using strings or nm:

t2h Unsupported ancillary data: 20/65535
qemu:handle_cpu_signal received signal outside vCPU context @ pc=0x6028a2c1
uncaught target signal 6 (Abort trap) - core dumped
nm: failed to initialize fileargs: Invalid argument

Implement MAP_GUARD support

After svn r320318 MAP_GUARD support is mandatory and qemu-user doesn't support it at all. All attempts to MAP_GUARD fail.

TARGET_HW_MACHINE_ARCH is hard-coded as armv6

#define TARGET_HW_MACHINE_ARCH "armv6"
https://github.com/seanbruno/qemu-bsd-user/blob/bsd-user/bsd-user/arm/target_syscall.h#L34

We should fix that for armv7?

getrandom syscall not implemented

getrandom(2) was added in https://svnweb.freebsd.org/changeset/base/331279

At least python27 and python36 detect it in headers and will try to use it, and the build will crash if it's not implemented.

Capabilities Syscalls are unsupported.

Noted an unsupported syscall when building converters/libiconv:

checking for ranlib... ranlib
checking for awk... (cached) /usr/bin/awk
checking command to parse /usr/bin/nm output from /nxb-bin/usr/bin/cc object... qemu: Unsupported syscall cap_rights_limit()
qemu: Unsupported syscall cap_rights_limit()
qemu: Unsupported syscall cap_ioctls_limit()
qemu: Unsupported syscall cap_enter()
ok
checking for sysroot... no
checking for mt... mt
checking if mt is a manifest tool... no

__sysctlbyname not implemented

On head, there are a lot of those errors:

qemu: unsupported syscall: 570 (calling anyway)

arm-bsd-user needs ipv6 sysctl support for something or other

While testing 'ifconfig -a' I note that we're missing a handler for something or other:

root@12armv6-default:/ # ifconfig
em0: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=4019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
	ether ec:f4:bb:2f:9d:99
Qemu unsupported ioctl: cmd=0xffffffffc0286998 dir=INOUT 'i' 152 40
	media: Ethernet autoselect
	status: no carrier
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
	inet 127.0.0.1 netmask 0xff000000 
Qemu unsupported ioctl: cmd=0xffffffffc0286998 dir=INOUT 'i' 152 40
	groups: lo 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
wlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 60:67:20:05:84:2c
Qemu unsupported ioctl: cmd=0xffffffffc0286998 dir=INOUT 'i' 152 40
	groups: wlan 
	media: IEEE 802.11 Wireless Ethernet MCS mode 11ng
	status: associated
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Library path expansion in do_bsd_openat

Whilst working on adding RISC-V user mode support I noticed that it was not possible to run qemu-<arch>-static outside of a jail. I was expecting to be able to use BSD user mode in a similar fashion to Linux user mode such that the following would work:

qemu-<arch>-static -L <path-to-arch-rootfs> <arch-executable>

I tracked the problem down to the (lack of) path expansion in do_bsd_openat function in bsd-user/bsd-file.h The 'path' function has been used elsewhere to prepend the <path-to-arch-rootfs> to the file name, but in this case it has been commented out. If I replace the 'p' parameter with 'path(p)' I can run qemu-<arch>-static standalone as described above. I've tested this with arm, aarch64 and riscv64 (both standalone and in jails) and haven't encountered any issues. Without this change the program attempts to open shared libraries in, for example, /lib instead of <path-to-arch-rootfs>/lib

Linux user mode uses the 'path' function in it's call to openat (see the return value call in the do_openat function in linux-user/syscall.c).

I was wondering if there was any reason not to change 'p' to 'path(p)' in the do_bsd_openat function?

Thanks

Crashing after -CURRENT 1300021

1300021 is the only data point I have for when this started crashing upon attempting to run anything. Host system is amd64. Backtrace:

(lldb) bt
* thread #1, name = 'qemu-aarch64-static', stop reason = signal SIGABRT
  * frame #0: 0x00000000600065cf qemu-aarch64-static`tcg_reg_alloc_op at tcg.c:2613:9
    frame #1: 0x00000000600065aa qemu-aarch64-static`tcg_reg_alloc_op(s=0x0000000060755920, op=0x000000086103d000) at tcg.c:2848
    frame #2: 0x00000000600048dd qemu-aarch64-static`tcg_gen_code(s=<unavailable>, tb=0x0000000000000000) at tcg.c:3279:13
    frame #3: 0x000000006003619c qemu-aarch64-static`tb_gen_code(cpu=<unavailable>, pc=48953516, cs_base=0, flags=2147483648, cflags=0) at translate-all.c:1319:21
    frame #4: 0x0000000060034fbe qemu-aarch64-static`cpu_exec [inlined] tb_find(cpu=<unavailable>, last_tb=<unavailable>, tb_exit=<unavailable>, cf_mask=0) at cpu-exec.c:404:18
    frame #5: 0x0000000060034f58 qemu-aarch64-static`cpu_exec(cpu=0x0000000860f29380) at cpu-exec.c:731
    frame #6: 0x0000000060039944 qemu-aarch64-static`target_cpu_loop(env=0x0000000860f31598) at target_arch_cpu.h:58:18
    frame #7: 0x00000000600398b9 qemu-aarch64-static`cpu_loop(env=<unavailable>) at main.c:121:5
    frame #8: 0x000000006003a994 qemu-aarch64-static`main(argc=-5508, argv=0x0000000860f31598) at main.c:518:5
    frame #9: 0x0000000060000114 qemu-aarch64-static`_start(ap=<unavailable>, cleanup=<unavailable>) at crt1.c:76:7
(lldb)

This happens on any emulated architecture.

"unhandled CPU exception 0x10005 - aborting" when trying to build jsoncpp for aarch64

I have been attempting to build the FreeBSD port of devel/jsoncpp within poudriere for the arm64.aarch64 architecture, but I get an unhandled CPU exception. This is happening both with FreeBSD 11.0-RELEASE-p1 and 12.0-CURRENT (revision 313134). This is with the current version of the FreeBSD port of qemu-user-static, which is 2.8.50.g20170123. The exception happens near the very beginning of the build process, right as it would make the first call to c++.

Here is the exception when run in a FreeBSD 11.0-RELEASE-p1 jail:

scons: done reading SConscript files.
qemu: unhandled CPU exception 0x10005 - aborting
PC=00000040023191f0  SP=00000008606abba0
X00=0000000000000008 X01=0000004002e809e0 X02=0000000000000000 X03=0000000000000000
X04=0000000000000000 X05=0000000000000000 X06=0000000000000000 X07=0000000000000000
X08=0000004002802278 X09=0000000000007010 X10=0000000000000015 X11=0000000000007008
X12=0000000000000000 X13=0000000000000000 X14=0000004002f01fb0 X15=0000004002f01e00
X16=00000040023d5560 X17=000000400201ac4c X18=000000086a7fb1be X19=0000004002802200
X20=0000004002e15ec0 X21=0000000000000000 X22=0000000000000000 X23=0000000000000001
X24=0000000000000001 X25=0000000000000000 X26=0000000000008000 X27=00000040023dfacc
X28=ba3d7b2a63fa5f70 X29=00000008606abbc0 X30=0000004002319174 
PSTATE=80000000 N--- EL0t
qemu: uncaught target signal 6 (Abort trap) - core dumped
Abort trap

Here is the exception when run in a FreeBSD 12.0-CURRENT r313134 jail:

scons: done reading SConscript files.
qemu: unhandled CPU exception 0x10005 - aborting
PC=000000400231938c  SP=00000008606abaf0
X00=00000040028001b8 X01=0000000000000008 X02=0000000000000000 X03=0000000000000000
X04=0000004002e15ec8 X05=0000000000000000 X06=0000000000000000 X07=0000000000000001
X08=0000000000007010 X09=0000000000000008 X10=0000000000007008 X11=0000000000000060
X12=0000000000000000 X13=0000000000000000 X14=0000004002f01fb0 X15=0000004002f01e00
X16=00000040023ce6a8 X17=000000400239f080 X18=000000000000001f X19=0000000000000000
X20=0000004002e15ec0 X21=0000004002800140 X22=0000000000000001 X23=0000000000000000
X24=00000040023efe10 X25=00000040023efe54 X26=0000000000000000 X27=0000000000000000
X28=0000000000000000 X29=00000008606abb10 X30=0000004002314ac4 
PSTATE=60000000 -ZC- EL0t
qemu: uncaught target signal 6 (Abort trap) - core dumped
Abort trap

I did not try with a FreeBSD 10 jail as I have been attempting to build for use on a physical Raspberry Pi 3 which is running the same 12 revision as noted above.

aarch64 YIELD instruction triggers unhandled CU exception

[00:09:34] qemu: unhandled CPU exception 0x10004 - aborting
[00:09:34]  PC=0000000000403fcc X00=0000000000419cd8 X01=0000000000000000
[00:09:34] X02=0000000000000000 X03=0000004003000688 X04=0000000000000001
[00:09:34] X05=0000000000000000 X06=0000000000419cd8 X07=0000000000419cd8
[00:09:34] X08=0000000000000000 X09=000000000002d9d7 X10=000000000002d9d7
[00:09:34] X11=0000000000000000 X12=0000000000000000 X13=0000004002a121a8
[00:09:34] X14=0000000000000000 X15=130f5523316d0001 X16=00000040020474d4
[00:09:34] X17=00007fffdeffbc60 X18=0000000000000000 X19=0000000000419000
[00:09:34] X20=0000000000418000 X21=0000000000000000 X22=000000000041b000
[00:09:34] X23=0000000000419000 X24=0000000000000001 X25=00007fffdeffbd54
[00:09:34] X26=00000040023a4cbc X27=000000400004a430 X28=00000040020005c8
[00:09:34] X29=00007fffdeffbc70 X30=00000000004040a8  SP=00007fffdeffbc70
[00:09:34] PSTATE=60000000 -ZC- S EL0t

mips-bsd-user fails to fetch distfiles

MIPS builds are broken at the moment if distfiles haven't already been
fetched. The issue is that r308996 added AI_ADDRCONFIG to host look ups
making the silent failure in MIPS emulation a critical failure. e.g.
"ifconfig -a" never has worked, and now its required. I'm working on
fixing this but there's a lot of flipping/bopping/twisting required to
make the entire data structure big/little endian safe at the moment.

Need strace decoder for syscalls 551 and 552

When running qemu-user with -strace, syscalls 551 and 552 are undecoded:

50985 __sysctl({ 1 24 }, 2, 0x0000007fffffbe20, 0x0000007fffffbe10, 0x0000000000000000, 0x0000000000000000) = 0
50985 Unknown syscall 552
50985 Unknown syscall 552
50985 openat(-100, "/etc/libmap.conf",0x100000,0) = 3
50985 Unknown syscall 551
50985 mmap(0,102,1,2,3,0) = 0x0000004002008000
50985 close(3) = 0
50985 Unknown syscall 552
50985 Unknown syscall 552
50985 Unknown syscall 552
50985 munmap(0x4002008000,102) = 0
50985 openat(-100, "/var/run/ld-elf.so.hints",0x100000,0) = 3
50985 read(3,0x34d88,128) = 128
50985 Unknown syscall 551
50985 lseek(3,128,0,0,13,0) = 128

"checking whether printf survives out-of-memory conditions" hangs

See https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=224740

[lang/go14, arm.armv6] _umtx_op handling wrong?

I had mentioned this via e-mail to sbruno@ previously, but to formally report:

lang/go14 has an issue building, see: https://gist.github.com/kevans91/a85e3c41962db0553c5743913f3d1808

go build runtime: /wrkdirs/usr/ports/lang/go14/work/go/pkg/tool/freebsd_arm/5g: wait: interrupted system call

At the time, sbruno@ had narrowed it down to the following function: https://github.com/golang/go/blob/release-branch.go1.4/src/runtime/os_freebsd.c#L62

He had the following additional to say:

I'm trying to come up with a short test case that isn't "cross build
lang/go14" at the moment, but the issue appears to be in handling
_umtx_op() calls. I suspect this is going to be nasty.

[end of correspondence]