hermit-os/libhermit

HermitCore: A C-based, lightweight unikernel

libhermit's Issues

Missing support of "LWIP_NETCONN_SEM_PER_THREAD"

LwIP 2.0 introduces a netconn semaphore per threads. The feature could improve the performance. Currently, we didn't use it because it isn't clear where we have to initialize the semaphores. => Check if we need this feature and how we could enable it...

Using of the gs register to accelerate the access to newlib's reentrant structure

Use a custom loader to handle HermitCore binaries in a special way

The Linux kernel is extensible with custom binary formats.
This feature could be used to get rid of the create_proxy.sh script.

The HermitCore applications could be compiled with a special OS/ABI or node field in the ELF header.
This could be used by the kernel to distinguish a usual Linux application from a HermitCore one.

It's easier to implement the loader in userspace. But as shown in point 3 in my PoC, this could also be done completely in the kernel.

I prepared a proof-of-concept here: https://github.com/stv0g/snippets/tree/master/c/linux_loader_poc

MMNIF doesn't support IPv6

The branch "devel" use LwIP 2.0 with dual-stack support (IPv4 & IPv6). However, the driver mmnif, which is used in the multi-kernel version as communication interface between Linux & HermitCore, doesn't activate IPv6.

KVM entry failure

Hi!.

This command

$ HERMIT_ISLE=uhyve HERMIT_CPUS=4 HERMIT_MEM=1G HERMIT_VERBOSE=1 bin/proxy x86_64-hermit/extra/benchmarks/stream

generates three errors

KVM: receive shutdown command
proxy: 0:KVM: entry failure: hw_entry_failure_reason=0x80000021
proxy: KVM: unhandled KVM_EXIT_IO / KVM_EXIT_MMIO at port 0x0

Docker container outdated

It appears the provided Docker container does not provide many of the dependencies needed to build HermitCore as it currently is. It only comes with CMake 3.5.1 despite HermitCore requiring 3.7, and the local-cmake.sh script requires Python be installed in the container before it can be run; the hexdump utility is not installed; and there is no detectable C++ compiler.

Identical expression

/drivers/net/mmnif.c: line 586 identical
!header_phy_start_address || !header_phy_start_address

Can't link to pthread and sed.

tema3210@tema2-devhost:~/unigo$` x86_64-hermit-gccgo main.go -o main /opt/hermit/lib/gcc/x86_64-hermit/6.3.0/../../../../x86_64-hermit/bin/ld: /opt/hermit/lib/gcc/x86_64-hermit/6.3.0/../../../../x86_64-hermit/lib/libgo.a(proc.o): in function `runtime_newosproc': /work/build/x86_64-hermit/libgo/../../../libgo/runtime/proc.c:202: undefined reference to `pthread_attr_init' /opt/hermit/lib/gcc/x86_64-hermit/6.3.0/../../../../x86_64-hermit/bin/ld: /work/build/x86_64-hermit/libgo/../../../libgo/runtime/proc.c:204: undefined reference to `pthread_attr_setdetachstate' /opt/hermit/lib/gcc/x86_64-hermit/6.3.0/../../../../x86_64-hermit/bin/ld: /work/build/x86_64-hermit/libgo/../../../libgo/runtime/proc.c:218: undefined reference to `pthread_create' /opt/hermit/lib/gcc/x86_64-hermit/6.3.0/../../../../x86_64-hermit/bin/ld: /opt/hermit/lib/gcc/x86_64-hermit/6.3.0/../../../../x86_64-hermit/lib/libgo.a(thread-sema.o): in function `runtime_semacreate': /work/build/x86_64-hermit/libgo/../../../libgo/runtime/thread-sema.c:40: undefined reference to `sem_init' /opt/hermit/lib/gcc/x86_64-hermit/6.3.0/../../../../x86_64-hermit/bin/ld: /opt/hermit/lib/gcc/x86_64-hermit/6.3.0/../../../../x86_64-hermit/lib/libgo.a(thread-sema.o): in function `runtime_semasleep': /work/build/x86_64-hermit/libgo/../../../libgo/runtime/thread-sema.c:77: undefined reference to `sem_timedwait' /opt/hermit/lib/gcc/x86_64-hermit/6.3.0/../../../../x86_64-hermit/bin/ld: /work/build/x86_64-hermit/libgo/../../../libgo/runtime/thread-sema.c:107: undefined reference to `sem_wait' /opt/hermit/lib/gcc/x86_64-hermit/6.3.0/../../../../x86_64-hermit/bin/ld: /opt/hermit/lib/gcc/x86_64-hermit/6.3.0/../../../../x86_64-hermit/lib/libgo.a(thread-sema.o): in function `runtime_semawakeup': /work/build/x86_64-hermit/libgo/../../../libgo/runtime/thread-sema.c:125: undefined reference to `sem_post' /opt/hermit/lib/gcc/x86_64-hermit/6.3.0/../../../../x86_64-hermit/bin/ld: /opt/hermit/lib/gcc/x86_64-hermit/6.3.0/../../../../x86_64-hermit/lib/libgo.a(go-cgo.o): in function `_cgo_wait_runtime_init_done': /work/build/x86_64-hermit/libgo/../../../libgo/runtime/go-cgo.c:216: undefined reference to `pthread_mutex_lock' /opt/hermit/lib/gcc/x86_64-hermit/6.3.0/../../../../x86_64-hermit/bin/ld: /work/build/x86_64-hermit/libgo/../../../libgo/runtime/go-cgo.c:221: undefined reference to `pthread_cond_wait' /opt/hermit/lib/gcc/x86_64-hermit/6.3.0/../../../../x86_64-hermit/bin/ld: /work/build/x86_64-hermit/libgo/../../../libgo/runtime/go-cgo.c:225: undefined reference to `pthread_mutex_unlock' /opt/hermit/lib/gcc/x86_64-hermit/6.3.0/../../../../x86_64-hermit/bin/ld: /opt/hermit/lib/gcc/x86_64-hermit/6.3.0/../../../../x86_64-hermit/lib/libgo.a(go-cgo.o): in function `_cgo_notify_runtime_init_done': /work/build/x86_64-hermit/libgo/../../../libgo/runtime/go-cgo.c:238: undefined reference to `pthread_mutex_lock' /opt/hermit/lib/gcc/x86_64-hermit/6.3.0/../../../../x86_64-hermit/bin/ld: /work/build/x86_64-hermit/libgo/../../../libgo/runtime/go-cgo.c:242: undefined reference to `pthread_cond_broadcast' /opt/hermit/lib/gcc/x86_64-hermit/6.3.0/../../../../x86_64-hermit/bin/ld: /work/build/x86_64-hermit/libgo/../../../libgo/runtime/go-cgo.c:245: undefined reference to `pthread_mutex_unlock' /opt/hermit/lib/gcc/x86_64-hermit/6.3.0/../../../../x86_64-hermit/bin/ld: /opt/hermit/lib/gcc/x86_64-hermit/6.3.0/libgcc.a(unwind-dw2-fde.o): in function `__gthread_mutex_lock': /work/build/x86_64-hermit/libgcc/./gthr-default.h:748: undefined reference to `pthread_mutex_lock' /opt/hermit/lib/gcc/x86_64-hermit/6.3.0/../../../../x86_64-hermit/bin/ld: /work/build/x86_64-hermit/libgcc/./gthr-default.h:748: undefined reference to `pthread_mutex_lock' /opt/hermit/lib/gcc/x86_64-hermit/6.3.0/../../../../x86_64-hermit/bin/ld: /work/build/x86_64-hermit/libgcc/./gthr-default.h:748: undefined reference to `pthread_mutex_lock' /opt/hermit/lib/gcc/x86_64-hermit/6.3.0/../../../../x86_64-hermit/bin/ld: /work/build/x86_64-hermit/libgcc/./gthr-default.h:748: undefined reference to `pthread_mutex_lock' /opt/hermit/lib/gcc/x86_64-hermit/6.3.0/../../../../x86_64-hermit/bin/ld: /opt/hermit/lib/gcc/x86_64-hermit/6.3.0/libgcc.a(unwind-dw2-fde.o): in function `__gthread_mutex_unlock': /work/build/x86_64-hermit/libgcc/./gthr-default.h:778: undefined reference to `pthread_mutex_unlock' /opt/hermit/lib/gcc/x86_64-hermit/6.3.0/../../../../x86_64-hermit/bin/ld: /opt/hermit/lib/gcc/x86_64-hermit/6.3.0/libgcc.a(unwind-dw2-fde.o): in function `__gthread_mutex_lock': /work/build/x86_64-hermit/libgcc/./gthr-default.h:748: undefined reference to `pthread_mutex_lock' /opt/hermit/lib/gcc/x86_64-hermit/6.3.0/../../../../x86_64-hermit/bin/ld: /opt/hermit/lib/gcc/x86_64-hermit/6.3.0/libgcc.a(unwind-dw2-fde.o): in function `__gthread_mutex_unlock': /work/build/x86_64-hermit/libgcc/./gthr-default.h:778: undefined reference to `pthread_mutex_unlock' /opt/hermit/lib/gcc/x86_64-hermit/6.3.0/../../../../x86_64-hermit/bin/ld: /work/build/x86_64-hermit/libgcc/./gthr-default.h:778: undefined reference to `pthread_mutex_unlock' /opt/hermit/lib/gcc/x86_64-hermit/6.3.0/../../../../x86_64-hermit/bin/ld: /work/build/x86_64-hermit/libgcc/./gthr-default.h:778: undefined reference to `pthread_mutex_unlock' /opt/hermit/lib/gcc/x86_64-hermit/6.3.0/../../../../x86_64-hermit/bin/ld: /work/build/x86_64-hermit/libgcc/./gthr-default.h:778: undefined reference to `pthread_mutex_unlock' /opt/hermit/lib/gcc/x86_64-hermit/6.3.0/../../../../x86_64-hermit/bin/ld: /work/build/x86_64-hermit/libgcc/./gthr-default.h:778: undefined reference to `pthread_mutex_unlock' collect2: error: ld returned 1 exit status tema3210@tema2-devhost:~/unigo$

Such thing i've encouter by running hermits gccgo, it seems it didn't compiled these, suggest adding pthread and\or sed mirrors in include, or write manual how to do it manually, please.

xhyve: deadlock while loading Hermit app

I've been playing around with the new xhyve Hypervisor for OS X.

I managed to boot into the Kernel and mounting the new 9P share from my OS X host.
Unfortunately it crashed while loading a Hermit app.

Welcome to HermitCore (http://www.hermitcore.org/)!
/ # /hermit/tools/proxy /hermit/usr/tests/hello 1
smpboot: CPU 1 is now offline
INFO: rcu_sched detected stalls on CPUs/tasks:
    1: (0 ticks this GP) idle=191/140000000000000/0 softirq=2786/2786 fqs=0
    (detected by 0, t=5252 jiffies, g=-245, c=-246, q=0)
Task dump for CPU 1:
swapper/1       R  running task        0     0      1 0x00200000
 ffff88013aacfeb8 0000000000000000 ffff88013aacc000 ffff88013aacc000
 ffff88013aacc000 ffffffff810c78a6 0000000000000000 ffffffff81055c6a
 0000000000000286 ffffffff8106f862 ffffffff813d6198 ffffffff8123c461
Call Trace:
 [<ffffffff810c78a6>] ? 0xffffffff810c78a6
 [<ffffffff81055c6a>] ? 0xffffffff81055c6a
 [<ffffffff8106f862>] ? 0xffffffff8106f862
 [<ffffffff8123c461>] ? 0xffffffff8123c461
 [<ffffffff8102769f>] ? 0xffffffff8102769f
 [<ffffffff810614e6>] ? 0xffffffff810614e6
rcu_sched kthread starved for 5252 jiffies! g18446744073709551371 c18446744073709551370 f0x0

Syscall redirection (I/O) slow

Now that I'm using XRay for profiling and debugging I'm often transfer quite a lot of data from my application that is being profiled to the Linux system (and ultimately to my host system through p9 mount).

The reports can easily be of several Megabytes and transfering them is really slow. I'm seeing transfer speeds of 20 KiB/s right now. Is there anything we can do to speed this up? The present rate appears to be quite low anyway.

clang isn't able to build loader

If clang is the default compiler, we got following errors:

clang: warning: argument unused during compilation: '-T /home/ubuntu/HermitCore/arch/x86/loader/link.ld' /usr/bin/ld: warning: cannot find entry symbol _start; defaulting to 00000000004000f0 CMakeFiles/arch_x86_loader.dir/main.c.o: In function main: /home/ubuntu/HermitCore/arch/x86/loader/main.c:(.text+0xf): undefined reference to bss_start /home/ubuntu/HermitCore/arch/x86/loader/main.c:(.text+0x14): undefined reference to bss_end /home/ubuntu/HermitCore/arch/x86/loader/main.c:(.text+0x1f): undefined reference to bss_start /home/ubuntu/HermitCore/arch/x86/loader/main.c:(.text+0x3f): undefined reference to kernel_start /home/ubuntu/HermitCore/arch/x86/loader/main.c:(.text+0x44): undefined reference to kernel_end CMakeFiles/arch_x86_loader.dir/page.c.o:(.data+0x0): undefined reference to kernel_start libarch_x86_loader_asm.a(entry.asm.o): In function cpu_init: /home/ubuntu/HermitCore/arch/x86/loader/entry.asm:(.text+0x31): undefined reference to kernel_start /home/ubuntu/HermitCore/arch/x86/loader/entry.asm:(.text+0x36): undefined reference to kernel_end clang: error: linker command failed with exit code 1 (use -v to see invocation) CMakeFiles/arch_x86_loader.dir/build.make:277: recipe for target ldhermit.el' failed

It seems that clang ignored the flag -T

Verification of the TLB shoot down is still missing

Syscall write() broken on qemu

sys_write() returns the bytesize of the bytecounter of the written data, instead of the bytes written.
So it always returns 8, because length is a ssize_t.

https://github.com/hermitcore/libhermit/blob/423cb0e80e4b81122fb088fdbc5bb385d9232933/kernel/syscall.c#L285

Typo in vioif error message

There is a typo in HermitCore at:
https://github.com/hermitcore/libhermit/blob/423cb0e80e4b81122fb088fdbc5bb385d9232933/drivers/net/vioif.c#L368

GCov is not working with uhyve/qemu

Building/linking the examples with --coverage creates binaries that include the appropriated gcov functions, but running the binaries with uhyve or qemu does not create gcov output files.

I am not sure if this should be working, but getting code coverage would be nice for the tests. 😃

Dependabot couldn't find a Cargo.toml for this project

Dependabot couldn't find a Cargo.toml for this project.

Dependabot requires a Cargo.toml to evaluate your project's current Rust dependencies. It had expected to find one at the path: /crates/raw_cpuid/Cargo.toml.

If this isn't a Rust project, or if it is a library, you may wish to disable updates for it from within Dependabot.

You can mention @dependabot in the comments below to contact the Dependabot team.

Priority handling in the Pthread library is still missing

The implementation should be verified.

Using of a unified term for cores and CPUs

Currently we didn't differentiate...

Multicast support within mmnif

Our network driver for inter-kernel communication doesn't support multi- / broadcasts.

NPB in HermitCore

Hello

I need run the NAS parallel benchmarks (NPB) in HermitCore. Is possible this?.
NPBs are implement in Fortran, the build process require run a makefile, called make.def, this file contains compilation params.
make.def generates an executable based in classes of complexity (S, A, W). Exists three classes of benchmark: BT, SP and LU, they solves diferential ecuations systems. When I run the command "make benchmark class" with the class BT, for example, it generates an executable named BT-MZ.S.x. This executable, can it run in HermitCore?. Can it build with the build process of HermitCore?.
Is there some way for solve this problem?

Support of virtio-vsock

virtio-vsock is an interesting approach to improve the communication between HermitCore app and its proxy. An introduction is available at https://youtu.be/_bYSQ68JPwE.

Compiling application with clang and linking with x86_64-hermit-ld

Hello.

For evaluation purpose, I would like to compile an application with clang compiler, and let the hermit toolchain link the compiled files correctly.

To show you my problem, I try to compile a simple C++ hello world application :

#include <iostream>

using namespace std;

int main(int argc, char ** argv)
{
    std::cout << "Hello world !" << std::endl;
    return 0;
}

This little program compiles with the x86_64-hermit-g++ compiler, and runs fine on the proxy.

But when I try to compile this code to a .o file using clang, and then link it with x86_64-hermit-g++, I get the following error :

$ clang -c hello.cpp
$ /opt/hermit/bin/x86_64-hermit-g++ -o hello hello.o
/opt/hermit/lib/gcc/x86_64-hermit/6.3.0/../../../../x86_64-hermit/bin/ld: hello.o: Relocations in generic ELF (EM: 62)
/opt/hermit/lib/gcc/x86_64-hermit/6.3.0/../../../../x86_64-hermit/bin/ld: hello.o: Relocations in generic ELF (EM: 62)
/opt/hermit/lib/gcc/x86_64-hermit/6.3.0/../../../../x86_64-hermit/bin/ld: hello.o: Relocations in generic ELF (EM: 62)
/opt/hermit/lib/gcc/x86_64-hermit/6.3.0/../../../../x86_64-hermit/bin/ld: hello.o: Relocations in generic ELF (EM: 62)
/opt/hermit/lib/gcc/x86_64-hermit/6.3.0/../../../../x86_64-hermit/bin/ld: hello.o: error adding symbols: file in wrong format
collect2: error: ld returned 1 exit status

I'm not sure about what means "file in wrong format" here. I read on the internet that this error could be related to a .o compiled for ARM architecture, while we're trying to link it into a x86 binary executable for example. But I think it's not the case here, since I'm compiling for the same architecture. file command shows that the .o is compiled for the correct architecture (I think ?) :

$ file hello.o
hello.o: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped

Is there something that prevent us from using the hermit toolchain only for linking everything ?

NAS Parallel Benchmarks crash with a Pagefault

I tested the BT benchmark as single-kernel version with in Qemu. The application crashes with a pagefault, if I use OpenMP. The non-OpenMP version works correctly. Below I added the dump of the kernel messages...

kermit - /work/lankes/NPB3.3.1/NPB3.3-OMP> HERMIT_ISLE=qemu HERMIT_VERBOSE=1 /home/lankes/HermitCore/hermit/tools/proxy bin/bt.A.x 
qemu startup command: qemu-system-x86_64 -nographic -smp 1 -m 2G -net nic,model=rtl8139 -net user,hostfwd=tcp:127.0.0.1:18766-:18766 -chardev file,id=gnc0,path=/tmp/hermit-dRJLrp -device pci-serial,chardev=gnc0 -monitor telnet:127.0.0.1:18767,server,nowait -kernel /home/lankes/HermitCore/hermit/tools/../arch/x86/loader/ldhermit.elf -initrd bin/bt.A.x -append -freq1845 -machine accel=kvm -cpu host -net dump


 NAS Parallel Benchmarks (NPB3.3-OMP) - BT Benchmark

 No input file inputbt.data. Using compiled defaults
 Size:   64x  64x  64
 Iterations:  200       dt:   0.0008000
 Number of available threads:     1

Did HermitCore receive an exception?
qemu-system-x86_64: terminating on signal 2 from pid 19200

Dump kernel log:
================

HermitCore loader...
Loader starts at 0x100000 and ends at 0x10c009
Found mb_info at 0x9500
Map module info at 0x10d000 (ret 0)
Map first page of module 0 at 0x10e000 (ret 0)
Module 0 consists 2097
Page pool starts at 0xb3f000
Free region 0x0 - 0x9fc00
Free region 0x100000 - 0x7ffe0000
ELF file is located at 0x10e000
Map 1990 pages from physical start address 0x10f000 linear to 0x800000
Remap 1749 pages from physical start address 0x200000 linear to 0x800000
Entry point: 0x800000
[0.000][0:0][INFO] cpuid level 13
[0.000][0:0][INFO] Paging features: PSE (2/4Mb) PAE PGE PAT PSE36 NX PSE (1Gb) LM
[0.000][0:0][INFO] Physical adress-width: 40 bits
[0.000][0:0][INFO] Linear adress-width: 48 bits
[0.000][0:0][INFO] Sysenter instruction: available
[0.000][0:0][INFO] Syscall instruction: available
[0.000][0:0][INFO] Set XCR0 to 0x7
[0.000][0:0][INFO] Core 0 set per_core offset to 0x0
[0.000][0:0][INFO] Core id is set to 0
[0.000][0:0][INFO] Found and initialized FPU!
[0.000][0:0][INFO] CPU features: SSE SSE2 SSE3 SSE4.1 SSE4.2 AVX AVX2 RDRAND FMA MOVBE X2APIC FPU FXSR XSAVE OSXSAVE RDTSCP FSGSBASE CLFLUSH BMI1 BMI2 
[0.000][0:0][INFO] Ext_Save_Area_2: offset 576, size 256
[0.000][0:0][INFO] Ext_Save_Area_3: offset 0, size 0
[0.000][0:0][INFO] Ext_Save_Area_4: offset 0, size 0
[0.000][0:0][INFO] HermitCore is running on a hypervisor!
[0.000][0:0][INFO] Hypervisor Vendor Id: KVMKVMKVM
[0.000][0:0][INFO] Maximum input value for hypervisor: 0x40000001
[0.000][0:0][INFO] CR0 0x80040033, CR4 0x506a0
[0.000][0:0][INFO] size of xsave_t: 1040
[0.000][0:0][INFO] IA32_MISC_ENABLE 0x1
[0.000][0:0][INFO] IA32_PLATFORM_ID 0x0
[0.000][0:0][INFO] IA32_CR_PAT 0x7040600070406
[0.000][0:0][INFO] PAT use per default writeback.
[0.000][0:0][INFO] MTRR is enabled.
[0.000][0:0][INFO] Fixed-range MTRR is enabled.
[0.000][0:0][INFO] MTRR used per default writeback.
[0.000][0:0][INFO] Map multiboot cmdline 0x10d01b into the virtual address space
[0.000][0:0][INFO] mb_info: 0x9500
[0.000][0:0][INFO] memory_init: base 0x200000, image_size 0x40434d8, limit 0x7ffe0000
[0.000][0:0][INFO] Free region 0x0 - 0x9f000
[0.000][0:0][INFO] Free region 0x100000 - 0x7ffe0000
[0.000][0:0][INFO] Add region 0x4400000 - 0x7ffe0000
[0.000][0:0][INFO] free list starts at 0x4400000, limit 0x7ffe0000
[0.000][0:0][INFO] vma_init: reserve vma region 0x800000 - 0x4a00000
[0.000][0:0][INFO] Add region 0x100000 - 0x200000
[0.000][0:0][INFO] Found EBDA at 0x9fc0!
[0.000][0:0][INFO] Found MP config table at 0xf6ba0
[0.000][0:0][INFO] System uses Multiprocessing Specification 1.4
[0.000][0:0][INFO] MP features 1: 0
[0.000][0:0][INFO] Virtual-Wire mode implemented
[0.000][0:0][INFO] Found IOAPIC at 0xfec00000
[0.000][0:0][INFO] Map IOAPIC to 0x7fe000
[0.000][0:0][INFO] IOAPIC version: 0x11
[0.000][0:0][INFO] Max Redirection Entry: 23
[0.000][0:0][INFO] Found 1 cores
[0.000][0:0][INFO] Found APIC at 0xfee00000
[0.000][0:0][INFO] Found and enable X2APIC
[0.000][0:0][INFO] Maximum LVT Entry: 0x5
[0.000][0:0][INFO] APIC Version: 0x14
[0.000][0:0][INFO] EOI-broadcast: available
[0.000][0:0][INFO] Boot processor 0 (ID 0)
[0.010][0:0][INFO] Processor: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
[0.050][0:0][INFO] APIC calibration determined an ICR of 0x98991d
[0.050][0:0][INFO] This is Hermit 006556-dirty, build date 15384296
[0.050][0:0][INFO] Isle 0 of 1 possible isles
[0.050][0:0][INFO] Kernel starts at 0x800000 and ends at 0x48434d8
[0.050][0:0][INFO] TLS image starts at 0xed4040 and ends at 0xee5ad0 (size 0x11a90)
[0.050][0:0][INFO] BBS starts at 0xed4044 and ends at 0x48434d8
[0.050][0:0][INFO] Per core data starts at 0xecc040 and ends at 0xed4040
[0.050][0:0][INFO] Per core size 0x40
[0.050][0:0][INFO] Processor frequency: 2300 MHz
[0.050][0:0][INFO] Total memory: 2047 MiB
[0.050][0:0][INFO] Current allocated memory: 67652 KiB
[0.050][0:0][INFO] Current available memory: 1981 MiB
[0.050][0:0][INFO] Core 0 is the boot processor
[0.050][0:0][INFO] CPU 0 of isle 0 is now online (CR0 0x8004003b, CR4 0x506a0)
[0.050][0:0][INFO] Task 1 uses memory region [0x10f000 - 0x110fff] as stack
[0.050][0:0][INFO] Task 1 uses memory region [0xd1000 - 0xd2fff] as IST1
[0.050][0:0][INFO] start new task 1 on core 0 with stack address 0x10f000
[0.050][0:1][INFO] TLS of task 1 on core 0 starts at 0xd4010 (TLS)
[0.050][0:1][INFO] Initd is running
netif: netmask of interface ?? set to 255.0.0.0
netif: GW address of interface ?? set to 127.0.0.1
netif_set_ipaddr: netif address being changed
netif: IP address of interface ?? set to 127.0.0.1
netif: added interface lo IP addr 127.0.0.1 netmask 255.0.0.0 gw 127.0.0.1
[0.050][0:1][INFO] Task 2 uses memory region [0x151000 - 0x152fff] as stack
[0.050][0:1][INFO] Task 2 uses memory region [0xf8000 - 0xf9fff] as IST1
[0.050][0:1][INFO] start new task 2 on core 0 with stack address 0x151000
sys_thread_new: create_kernel_task 0, id = 2, prio = 16
[0.050][0:2][INFO] TLS of task 2 on core 0 starts at 0x192010 (TLS)
[0.050][0:2][INFO] LwIP's tcpip thread has task id 2
[0.050][0:1][INFO] TCP/IP initialized.
netif: IP address of interface ?? set to 0.0.0.0
netif: netmask of interface ?? set to 0.0.0.0
netif: GW address of interface ?? set to 0.0.0.0
[0.050][0:1][INFO] rtl8139if_init: Found RealTek RTL8139 at iobase 0xc000 (irq 11)
rtl8139if_init: MAC address 52 54 00 12 34 56 
[0.060][0:1][INFO] RTL8139: CR = 0xd, ISR = 0x0, speed = 10 mbps
netif: added interface en IP addr 0.0.0.0 netmask 0.0.0.0 gw 0.0.0.0
netif: setting default interface en
[0.060][0:1][INFO] Starting DHCPD...
netif: netmask of interface en set to 255.255.255.0
netif: GW address of interface en set to 10.0.2.2
netif_set_ipaddr: netif address being changed
netif: IP address of interface en set to 10.0.2.15
[0.070][0:1][INFO] Boot time: 70 ms
[0.070][0:1][INFO] TCP server is listening.
[0.070][0:1][INFO] Establish IP connection
netif_ip6_addr_set_state: netif address state being changed
netif: IPv6 address 0 of interface en set to FE80::5054:FF:FE12:3456/0x09
netif_ip6_addr_set: netif address being changed
netif: IPv6 address 1 of interface en set to FEC0::5054:FF:FE12:3456/0x00
netif_ip6_addr_set_state: netif address state being changed
netif: IPv6 address 1 of interface en set to FEC0::5054:FF:FE12:3456/0x08
netif_ip6_addr_set_state: netif address state being changed
netif: IPv6 address 0 of interface en set to FE80::5054:FF:FE12:3456/0x30
netif_ip6_addr_set_state: netif address state being changed
netif: IPv6 address 1 of interface en set to FEC0::5054:FF:FE12:3456/0x09
[4.570][0:1][ERROR] Page Fault Exception (14) on core 0 at cs:ip = 0x8:0xdd5bab, fs = 0xe5aa0, gs = 0, rflags 0x11202, task = 1, addr = 0xfffffffffffee990, error = 0x2 [ supervisor data write not present ]
[4.570][0:1][ERROR] rax 0x4842ee0, rbx 0xfffffffffffef5c8, rcx 0x14eb20, rdx 0x14eb08, rbp, 0x14eb18, rsp 0x14e9d0 rdi 0x14eb18, rsi 0x14eb10, r8 0, r9 0, r10 0x149ffb, r11 0x40, r12 0x40, r13 0x14eb10, r14 0x1, r15 0xfffffffffffee580
[4.570][0:1][ERROR] Heap 0x80000000 - 0x8000b000
[4.570][0:1][INF

Unik support

UniK (https://github.com/emc-advanced-dev/unik) runs and manages instances of compiled images across a variety of cloud providers. An integration of HermitCore into UniK would simplify the usage of HermitCore.

segmentation fault when running tests in multi-kernel&virtual machine mode

Hi,
I followed the instructions, trying to do some tests in multi-kernel within virtual machine mode. But no matter which test program I choose, it always generates a segmentation fault like below:

Welcome to HermitCore (http://www.hermitcore.org/)!

/ #
/ # /hermit/x86_64-hermit/extra/tests/pi
traps: pi[112] general protection ip:83ea2d sp:7ffd70178a10 error:402 in pi[800000+362000]
Segmentation fault

and objdump result shows:

000000000083ea14 <Llinux_main>:
  83ea14:       b8 04 00 00 00          mov    $0x4,%eax
  83ea19:       bb 02 00 00 00          mov    $0x2,%ebx
  83ea1e:       48 b9 06 30 9e 00 00    movabs $0x9e3006,%rcx
  83ea25:       00 00 00
  83ea28:       ba 5c 00 00 00          mov    $0x5c,%edx
  83ea2d:       cd 80                   int    $0x80
  83ea2f:       b8 01 00 00 00          mov    $0x1,%eax
  83ea34:       bb 00 00 00 00          mov    $0x0,%ebx

Is there anyway I can fix this?

fopen("file.name", "w") cannot create new file

Xray is using f = fopen(filename, "w"); to save it's reports. If the given file does not exists, it should be created, refer man fopen:

w      Truncate  file  to  zero length or create text file
       for writing.   The  stream  is  positioned  at  the
       beginning of the file.

However, this doesn't work in the sense that no file is created and fopen() returns NULL.

Page within heap area also belongs to page table

The page at virtual address 0x3FE00000 is located within the heap area. However, according to the way page tables are defined, the virtual page at this specific address belongs to the page table: it is mapped on the same physical page as the virtual page at 0x802000 (PDPT).

Thus, when an application allocate some memory with malloc, gets the page at 0x3FE00000, and tries to write to it, it overwrites the PDPT, leading to random behavior (mostly crash). Here is a program that triggers the issue:

#include <stdio.h>
#include <stdlib.h>

/* This should be sufficient to push the break past 0x3FE00000 */
#define BUFFER_SIZE	(1024 * 1024 * 1024)
#define ADDR_ISSUE	0x3FE00000

int main(void) {
	char *buffer;
	int i;

	/* Increases the break */
	buffer = malloc(BUFFER_SIZE);

	/* Actually map memory and write in it */
	for(i=0; i<BUFFER_SIZE; i++) {
		if(&(buffer[i]) == (void *)ADDR_ISSUE)
			printf("Writing in %p\n", ADDR_ISSUE);
		buffer[i] = '\0';
	}

	/* The bug makes that this should not be executed: */
	printf("Successfully allocated and mapped %llu bytes\n", BUFFER_SIZE);

	return 0;
}

With uhyve, the VM exists and KVM receives a shutdown signal. With qemu or qemu/kvm, the VM hangs. I guess the reason we do not get a page fault here is that the first entry of the PDPT at 0x802000 is mapping a large part of the address space, including the page fault handler, and this large area gets unmapped when something is written at 0x3FE00000.

One hacky solution I found is to move the start of the heap past that address, in include/hermit/stddef.h:

#define HEAP_START	(PAGE_2M_CEIL((size_t)&kernel_start + image_size + 0x3fe00000) + 4*PAGE_SIZE)

Crash upon hello_proxy execution

I'm not sure if this intended behaviour, but I suspect that something is still wrong. When I start a proxy example, it looks like something crashes and the linux kernel boots all over again. See output: https://gist.github.com/daniel-k/473aee84ff493f1a41f3

I needed to modify Qemu parameters in order to be able to start it in the first place as my machine has "only" 8GB of RAM.

-smp 8 [...] -m 4G -numa node,nodeid=0,cpus=0-3 -numa node,nodeid=1,cpus=4-7

and in linux/arch/x86/hermit/hermit.c I lowered the pool size to

static size_t pool_size = 0x10000000ULL;

although I don't know yet what this really is. However, it was needed so that HermitCore would initialize successfully and populate /sys/hermit.

HermitCore needs a logo

I want to put HermitCore onto the OSS listing at: http://fine-aachen.rwth-aachen.de/projects/

Loosing SYN packets

If HermitCore is started as classical unikernel in a VM, we lose the first SYN packet which was send from the proxy to create a connection to HermitCore. The reason is that our kernel starts not fast enough. On my test system, the boot time takes 2s.

The timeout to retry the connection is per default too high (~6s). Consequently, the first access to the system has high latency. Currently, I am not able to define my own timeout value.

The workaround (dirty hack) is currently a sleep statement to wait for the HermitCore kernel.

does hermitcore have full support for numa?

For example, allowing applications to migrate memory from one NUMA node to another

Compatibility with ARM devices / Android

I am interested to configure libhermit for Android, but I don't know how much effort can be...
Is somebody working on this? I would interested to know if they are hard dependencies to configure the toolchain for Android (with cmake)... I appreciate any response..

SMP support doesn't work in Qemu if KVM is disabled

By booting the common unikernel within Qemu (HERMIT_ISLE=qemu) and the KVM support is disabled (HERMIT_KVM=0), the kernel isn't able to activate the application processors.

Provide a docker image/deb/rpm/...

Building HermitCore from scratch takes quite long. Providing pre-built binaries would be useful for everyone who would like to just give it a try. An (seemingly) easy way to do this is https://bintray.com/ as it integrates with Travis https://docs.travis-ci.com/user/deployment/bintray

Should we define _POSIX_SOURCE per default?

To use the IP stack in a common program (C/C++/Fortran), we have to define _POSIX_SOURCE because without the definition newlib defines its own FD_SET macro. Should we define _POSIX_SOURCE per default?

Set page size to 2MB

To reduce the TLB misses, we should per default use a page size of 2MB. Currently, only the code and bss segment use a page size of 2MB.

Uhyve file read/write should not assume target buffer is contiguous in physical memory

Hello,

With Uhyve, when performing file read/write operations in the host, the target buffer is fully read or written from the guest physical memory. Such a design assumes that the buffer is contiguous in physical memory, which is not always the case. When using a buffer allocated with malloc, example of bad things that can happen are:

• Reading from a file in a buffer that is not or partially mapped to physical memory (because of on-demand mapping)
• Reading from a file in a buffer which virtual pages are not mapped to contiguous physical pages
• Writing in a file from a buffer which virtual pages are not mapped to contiguous physical pages

You can confirm these issues with these two sample programs and the corresponding Makefile:

/* read_test.c */

#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdlib.h>
#include <string.h>

#define PAGE_SIZE		4096
#define FILE_TO_READ	"fread.bin"
#define FILE_SIZE		PAGE_SIZE * 4

char verify_buf[FILE_SIZE];

int main(int argc, char **argv) {
	int fd, verify_fd, i;
	char *buf;

	printf("Read test starting\n");

	buf = malloc(FILE_SIZE);
	if(!buf) {
		fprintf(stderr, "error malloc!\n");
		return -1;
	}

	verify_fd = open(FILE_TO_READ, O_RDONLY, 0x0);
	fd = open(FILE_TO_READ, O_RDONLY, 0x0);
	if(fd == -1 || verify_fd == -1) {
		printf("error open\n");
		return -1;
	}

	if(read(verify_fd, verify_buf, FILE_SIZE) != FILE_SIZE) {
		fprintf(stderr, "read error\n");
		return -1;
	}

	if(read(fd, buf, FILE_SIZE) != FILE_SIZE) {
		fprintf(stderr, "read error\n");
		return -1;
	}
	printf("Read in malloc'd buffer done\n");

	for(i=0; i<FILE_SIZE; i++)
		if(buf[i] != verify_buf[i]) {
			fprintf(stderr, "verification failed!\n");
			fprintf(stderr, "buf[%d] == %x\n", i, buf[i]);
			fprintf(stderr, "verify_buf[%d] == %x\n", i, verify_buf[i]);
			return -1;
		}

	printf("Read verification successful!\n");

	free(buf);
	close(fd);
	return 0;
}

/* write_test.c */

#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdlib.h>
#include <string.h>

#define PAGE_SIZE		4096
#define FILE_SIZE		PAGE_SIZE * 4
#define FILE_TO_WRITE	"fwrite.bin"

char verify_buf[FILE_SIZE];

int main(int argc, char **argv) {
	char *buf;
	int fd, i;

	printf("write test starting\n");

	buf = malloc(FILE_SIZE);
	if(!buf) {
		fprintf(stderr, "error malloc\n");
		return -1;
	}

	fd = open(FILE_TO_WRITE, O_RDWR | O_CREAT | O_TRUNC, 0666);
	if(fd == -1) {
		fprintf(stderr, "error opening %s\n", FILE_TO_WRITE);
		return -1;
	}

	for(i=0; i<FILE_SIZE; i++)
		buf[i] = (i/4096) + '0';

	if(write(fd, buf, FILE_SIZE) != FILE_SIZE) {
		fprintf(stderr, "error writing in %s", FILE_TO_WRITE);
		return -1;
	}
	printf("Write from malloc'd buffer done\n");

	if(lseek(fd, 0x0, SEEK_SET) != 0) {
		fprintf(stderr, "error lseek\n");
		return -1;
	}

	if(read(fd, verify_buf, FILE_SIZE) != FILE_SIZE) {
		fprintf(stderr, "error read\n");
		return -1;
	}

	for(i=0; i<FILE_SIZE; i++)
		if(verify_buf[i] != buf[i]) {
			fprintf(stderr, "verification failed!\n");
			fprintf(stderr, "buf[%d] == %x\n", i, buf[i]);
			fprintf(stderr, "verify_buf[%d] == %x\n", i, verify_buf[i]);
			return -1;
		}

	printf("Write verification successful!\n");

	free(buf);
	close(fd);
	return 0;
}

# Makefile

HERMIT_LOCAL_INSTALL=/home/pierre/Desktop/HermitCore/prefix/
HERMIT_TOOLCHAIN_INSTALL=/opt/hermit/
READ_TEST=read_test
WRITE_TEST=write_test
READ_SAMPLE_FILE=fread.bin
WRITE_SAMPLE_FILE=fwrite.bin
READ_TEST_SRC=$(READ_TEST).c
WRITE_TEST_SRC=$(WRITE_TEST).c
CFLAGS=-Wall -Werror
LDFLAGS=-L$(HERMIT_LOCAL_INSTALL)/x86_64-hermit/lib
CC=$(HERMIT_TOOLCHAIN_INSTALL)/bin/x86_64-hermit-gcc
PROXY=$(HERMIT_LOCAL_INSTALL)/bin/proxy
VERBOSE?=0

all: $(READ_TEST) $(WRITE_TEST)

$(READ_TEST): $(READ_TEST_SRC) $(READ_SAMPLE_FILE)
	$(CC) $(CFLAGS) $(READ_TEST_SRC) -o $(READ_TEST) $(LDFLAGS)

$(WRITE_TEST): $(WRITE_TEST_SRC)
	$(CC) $(CFLAGS) $(WRITE_TEST_SRC) -o $(WRITE_TEST) $(LDFLAGS)

$(READ_SAMPLE_FILE):
	dd if=/dev/urandom of=$(READ_SAMPLE_FILE) bs=4K count=4

test: $(READ_TEST) $(WRITE_TEST)
	HERMIT_ISLE=uhyve HERMIT_VERBOSE=$(VERBOSE) $(PROXY) $(READ_TEST)
	HERMIT_ISLE=uhyve HERMIT_VERBOSE=$(VERBOSE) $(PROXY) $(WRITE_TEST)

clean:
	rm -rf $(READ_TEST) $(WRITE_TEST) *.o $(READ_SAMPLE_FILE) \
		$(WRITE_SAMPLE_FILE)

Here is my proposed solution: a patch that forces the file read and write operations to be performed on a page-by-page basis, i.e. Uhyve file read or write operation is called for each page composing the buffer. I did not extensively tested it, nor did I evaluated the performance impact (I guess it slows things down a bit).

diff --git a/kernel/syscall.c b/kernel/syscall.c
index d10691d..f7b7b00 100644
--- a/kernel/syscall.c
+++ b/kernel/syscall.c
@@ -159,6 +159,47 @@ typedef struct {
 	ssize_t ret;
 } __attribute__((packed)) uhyve_read_t;
 
+/* Pages belonging to the heap are mapped on demand, and are not always
+ * contiguous in physical memory. Thus, we need to force allocation and call
+ * the hypervisor file read function page by page.
+ */
+ssize_t uhyve_noncontiguous_read(int fd, char *buf, size_t len) {
+	ssize_t bytes_read = 0;
+	size_t cur_len = 0;
+	uhyve_read_t arg = {fd, 0x0, 0x0, -1};
+
+	while(len) {
+
+		if(!((size_t)buf % PAGE_SIZE)) {
+			cur_len = (len < PAGE_SIZE) ? len : PAGE_SIZE;
+		} else {
+			cur_len = PAGE_CEIL((size_t)buf) - (size_t)buf;
+			cur_len = (((size_t)buf + len) > PAGE_CEIL((size_t)buf))
+				? (PAGE_CEIL((size_t)buf) - (size_t)buf) : len;
+		}
+
+		/* Force mapping */
+		memset(buf, 0x0, 1);
+
+		arg.buf = (char *)virt_to_phys((size_t)buf);
+		arg.len = cur_len;
+		arg.ret = -1;
+		uhyve_send(UHYVE_PORT_READ, (unsigned)virt_to_phys((size_t)&arg));
+
+		if(arg.ret < 0)
+			return arg.ret;
+
+		bytes_read += arg.ret;
+		if(arg.ret != cur_len)
+			break;
+
+		buf += cur_len;
+		len -= cur_len;
+	}
+
+	return bytes_read;
+}
+
 ssize_t sys_read(int fd, char* buf, size_t len)
 {
 	sys_read_t sysargs = {__NR_read, fd, len};
@@ -174,11 +215,16 @@ ssize_t sys_read(int fd, char* buf, size_t len)
 	}
 
 	if (is_uhyve()) {
+
+		return uhyve_noncontiguous_read(fd, buf, len);
+
+#if 0
 		uhyve_read_t uhyve_args = {fd, (char*) virt_to_phys((size_t) buf), len, -1};
 
 		uhyve_send(UHYVE_PORT_READ, (unsigned)virt_to_phys((size_t)&uhyve_args));
 
 		return uhyve_args.ret;
+#endif
 	}
 
 	spinlock_irqsave_lock(&lwip_lock);
@@ -222,6 +268,39 @@ typedef struct {
 	size_t len;
 } __attribute__((packed)) uhyve_write_t;
 
+ssize_t uhyve_noncontiguous_write(int fd, const char *buf, size_t len) {
+	char *cur_buf = (char *)buf;
+	ssize_t bytes_written = 0;
+	size_t cur_len = 0;
+	uhyve_write_t arg = {fd, 0x0, 0x0};
+
+	while(len) {
+
+		if(!((size_t)cur_buf % PAGE_SIZE)) {
+			cur_len = (len < PAGE_SIZE) ? len : PAGE_SIZE;
+		} else {
+			cur_len = (((size_t)cur_buf + len) > PAGE_CEIL((size_t)cur_buf))
+				? (PAGE_CEIL((size_t)cur_buf) - (size_t)cur_buf) : len;
+		}
+
+		arg.buf = (char *)virt_to_phys((size_t)cur_buf);
+		arg.len = cur_len;
+		uhyve_send(UHYVE_PORT_WRITE, (unsigned)virt_to_phys((size_t)&arg));
+
+		if(arg.len == (size_t)(-1))
+			return -1;
+
+		bytes_written += arg.len;
+		if(arg.len != cur_len)
+			break;
+
+		cur_buf += cur_len;
+		len -= cur_len;
+	}
+
+	return bytes_written;
+}
+
 ssize_t sys_write(int fd, const char* buf, size_t len)
 {
 	if (BUILTIN_EXPECT(!buf, 0))
@@ -240,11 +319,16 @@ ssize_t sys_write(int fd, const char* buf, size_t len)
 	}
 
 	if (is_uhyve()) {
+
+		return uhyve_noncontiguous_write(fd, buf, len);
+
+#if 0
 		uhyve_write_t uhyve_args = {fd, (const char*) virt_to_phys((size_t) buf), len};
 
 		uhyve_send(UHYVE_PORT_WRITE, (unsigned)virt_to_phys((size_t)&uhyve_args));
 
 		return uhyve_args.len;
+#endif
 	}
 
 	spinlock_irqsave_lock(&lwip_lock);

Trigger build for RDMA migration

The CI should include a build for the RDMA migration. This can be enabled by passing ENABLE_RDMA_MIGRATION to cmake.

Fast implementation of memcmp is missing

strcpy & Co. are written in Assembly. Why do we not use Assembly for memcmp?

Why is init.asm not linked into libhermit.a?

Currently the x86 boot code is compiled into the Linux kernel and copied to hermit_trampoline during the start of a Hermit app.

I don't see a reason, why we can not directly link it into the ELF binary of a Hermit app or libhermit.a

Irrespective of this, I moved init.asm to the Linux repo, renamed it boot.asm and integrated into the kbuild system. If it's only used by the kernel, we also should build it there.

See: stv0g/linux@21e290d

Not compiling because of mm/hbmemory.o

I just checkout out latest devel and found out, that HermitCore unfortunately is not building correctly at the moment:

$ make veryclean
$ make
[...]
make[1]: Entering directory '/home/daniel/Entwicklung/HermitCore/hermit'
make[2]: Entering directory '/home/daniel/Entwicklung/HermitCore/hermit/usr'
make[2]: Nothing to be done for 'bootstrap'.
make[2]: Leaving directory '/home/daniel/Entwicklung/HermitCore/hermit/usr'
make[1]: *** No rule to make target 'mm/hbmemory.o', needed by 'libhermit.a'.  Stop.
make[1]: Leaving directory '/home/daniel/Entwicklung/HermitCore/hermit'
make: *** [Makefile:7: default] Error 2

Support OpenMP environment variables

Various existing HPC tools/scripts rely the environment variables OMP_NUM_THREAD, GOMP_CPU_AFFINITY[0] and KMP_AFFINITY[1] to work well. Supporting these would make it easier to use HermitCore.

[0] https://gcc.gnu.org/onlinedocs/libgomp/GOMP_005fCPU_005fAFFINITY.html
[1] https://software.intel.com/en-us/node/522691#KMP_AFFINITY_ENVIRONMENT_VARIABLE

proxy: KVM: receive shutdown command : No such file or directory

Hi
I am a new to hermitux, trying to understand it.

From https://github.com/ssrg-vt/hermitux, I ran all the command successfully except for the last one:
sudo HERMIT_ISLE=uhyve HERMIT_TUX=1 \ ../../../hermitux-kernel/prefix/bin/proxy \ ../../../hermitux-kernel/prefix/x86_64-hermit/extra/tests/hermitux \ ../../../musl/prefix/lib/libc.so is-dyn

that generates

PIE detected, loading application at 0x400000
proxy: KVM: receive shutdown command
: No such file or directory

Any thing I should be worry about ?
I am running Ubuntu 18
Thanks

[ solved ]

Page fault in get_frequency_from_mbinfo

On some platforms the function get_frequency_from_mbinfo in arch/x86/kernel/processors.c triggeres a page fault. => workaround: disabling of this function (see branch devel)

But the workaround increases the boot time, because the host isn't able to forward the processor frequency to the guest. => guest has to detect the frequency by itself

git clone incl. submodules only possible with GitHub account

The submodules are using the ssh clone URLs, but these are only accessible for GitHub users.

Help/error message when starting the proxy

The proxy does not show any usefull info when run without any parameter and seems to have not help message. This would really be useful for new users / forgetful people.

$ proxy
fopen: No such file or directory
fopen: No such file or directory

$ proxy --help
fopen: No such file or directory
fopen: No such file or directory

$ proxy -h
fopen: No such file or directory
fopen: No such file or directory

HermitCore shutdown hangs

Sometimes HC (at least with uhyve) does not shutdown correctly and seems to hangs indefinitely.

To reproduce just use coBench https://github.com/jbreitbart/coBench on kermit with a subset of the NPB. Feel free to use the config in my home.

Use p9 virtio to mount /hermit

We could mount the /hermit using p9. Why rebooting the VM everytime? And we would loose the genisoimage dependency.

See: http://www.linux-kvm.org/page/9p_virtio