quic/gunyah-hypervisor

Gunyah is a Type-1 hypervisor designed for strong security, performance and modularity.

Gunyah Hypervisor

Gunyah is a high performance, scalable and flexible hypervisor built for demanding battery powered, real-time, safety and security use cases.

The Gunyah Hypervisor open source project provides a reference Type-1 hypervisor configuration suitable for general purpose hosting of multiple trusted and dependent VMs.

Gunyah Origins

Gunyah is an Australian Aboriginal word. See: https://en.wiktionary.org/wiki/gunyah

The Gunyah Hypervisor was developed by Qualcomm in Sydney Australia.

Type-1 Hypervisor Concept

Gunyah is a Type-1 hypervisor, meaning that it runs independently of any high-level OS kernel such as Linux and runs in a higher CPU privilege level than VMs. It does not depend on any lower-privileged OS kernel/code for its core functionality. This increases its security and can support a much smaller trusted computing base than a Type-2 like hosted-hypervisors.

Gunyah's design principle is not dissimilar to a traditional microkernel in that it provides only a minimal set of critical services to its clients, and delegates the provision of non-critical services to non-privileged (or less-privileged) processes, wherever this is possible without an adverse impact on performance or security.

The hypervisor uses the CPU's virtualization mode and features to isolate itself from OS kernels in VMs and isolate VMs from each other. On ArM, this includes trapping and emulating registers as required, virtualizing core platform devices, Arm's GIC virtualization support, and the CPU's Stage-2 MMU to provide isolated VMs in EL1/0.

Why Gunyah

• Strong security: Mobile payments, secure user-interface, and many more security sensitive use-cases all require strong security. Gunyah's design is suited to providing strong isolation guarantees and its small size is conducive to audit.
• Performance: Mobile devices are particularly demanding. Battery powered devices demand low software overheads to get the most performance per-watt. Gunyah is designed to have high performance with minimal impact to high-level operating systems.
• Modularity: The hypervisor is designed to be modular, allowing customization and enhancement by swapping out module implementations and adding new feature via new modules.

Features

• Threads and Scheduling: The scheduler schedules virtual CPUs (VCPUs) on physical CPUs and enables time-sharing of the CPUs.
• Memory Management: Gunyah tracks memory ownership and use of all memory under its control. Memory partitioning between VMs is a fundamental security feature.
• Interrupt Virtualization: All interrupts are handled in the hypervisor and routed to the assigned VM.
• Inter-VM Communication: There are several different mechanisms provided for communicating between VMs.
• Device Virtualization: Para-virtualization of devices is supported using inter-VM communication. Low level system features and devices such as interrupt controllers are supported with emulation where required.

Platform Support

Gunyah is architected to support multiple CPU architectures, so its core design ensures architecture independence and portability in non-architecture specific areas.

Gunyah currently supports the ARM64 (ARMv8+) architecure, it uses AArch64 EL2 in VHE mode by default.

We have developed an initial port of Gunyah to the QEMU Arm System emulator. Note QEMU v7+ is recommended. Additional platforms are expected to be supported in future contributions.

Getting Started

• Terminology

• Other required Gunyah related repositories:

  • Setup Tools and Scripts (Start setup here..!!)
  • Hypervisor (Gunyah core hypervisor, this repository)
  • Resource Manager (Platform policy engine)
  • C Runtime (C runtime environment for Resource Manager)

• Setup Instructions

  • Quick Start Instructions (will take to the setup tools and scripts repository)

• Build Instructions

• Status and Changelog

Resources

• Gunyah Hypercall API

Contributions

Thank you for your interest in contributing to Gunyah!

Please read our Contributions Page for more information on contributing features or bug fixes.

Team

Gunyah was developed by Qualcomm and aims to be an open and community supported project.

Check out the AUTHORS for major contributors.

License

Gunyah is licensed on the BSD 3-clause "New" or "Revised" License. Check out the LICENSE for more details.

gunyah-hypervisor's Issues

[HYP] Unhandled trap from VM 3

Hi,

I encounter a system hang while testing the Gunyah Hypervisor with the Linux VM on the QEMU simulator.
(Follow the https://github.com/quic/gunyah-hypervisor/blob/develop/docs/test.md)

Any suggestions would be much appreciated. Thanks

[RM] Starting Resource Manager, version: 727cc85 (Tue Dec 21 03:28:12 2021 UTC)
[RM] memparcel_construct sglist:
[RM] [0]: me:fabfdee110ae6499 phys:40000000 (20000000)
[RM] memparcel_construct ret=0
[RM] create_dtbo: hyp -1 resmem -1
[RM] create_resmem_nodes: memparcel labelled 0 is the base memory at 0x40000000
[RM] init completed
[RM] UART is disabled
[HYP] sysreg_read: unhandled TID3 trap, ISS: 0x34004d. RAZ
[HYP] sysreg_read: unhandled TID3 trap, ISS: 0x380089. RAZ
[HYP] sysreg_read: unhandled TID3 trap, ISS: 0x3a0089. RAZ
[HYP] sysreg_read: unhandled TID3 trap, ISS: 0x3a0007. RAZ
[HYP] sysreg_read: unhandled TID3 trap, ISS: 0x3c0007. RAZ
[HYP] sysreg_read: unhandled TID3 trap, ISS: 0x380007. RAZ
[HYP] Unhandled trap from VM 3, ESR_EL2 = 0x62380405, ELR_EL2 = 0xffffa9bd8b497174

Help wanted!

Is it possible to work in Android?
My device chipset Qualcomm sm8350(888)
In my device have hyp partition
My kernel version 5.4
I found some data in kernel device tree

https://github.com/pjgowtham/android_kernel_oneplus_sm8350/blob/ffa3efd23c005672bff907f71a619dfd9cf82aec/arch/arm64/boot/dts/vendor/qcom/lahaina.dtsi#L402

Qemu support for gunyah hypervisor

As far as I know, Gunyah is currently working on Qemu support. I wanted to ask if you could provide compiled Qemu with Gunyah support. I have tried several times to compile the static version of Qemu using the instructions provided in the document, but without success because the compilation environment lacks some of the static version libraries.

./configure --target-list=aarch64-softmmu --enable-debug --enable-gunyah --static

https://patchew.org/QEMU/[email protected]/

There are no ".githooks" ...

There are no shared ".githooks" files:
.githooks/pre-commit
.githooks/commit-msg
mentioned in Contribution page
(same with C Runtime and Resource Manager)

An error in forward_private.c

error_t
vic_bind_hwirq_forward_private(vic_t *vic, hwirq_t *hwirq, virq_t virq)
{
error_t err = OK;

assert(hwirq->action == HWIRQ_ACTION_VIC_BASE_FORWARD_PRIVATE);

LOG(ERROR, WARN, "vic_bind_hwirq_forward_private virq {:x}", (unsigned long)virq);

// allocate for private forward
struct partition *partition = vic->header.partition;
assert(partition != NULL);

size_t size = sizeof(hwirq->vic_base_forward_private[0]) * GIC_PPI_NUM;  //here, GIC_PPI_NUM shoud be cpu count.
    //let's see how to use it later
   //		vic_forward_private_t *forward_private =
   //   	      &hwirq->vic_base_forward_private[pcpu];  
  // Am I right?


void_ptr_result_t alloc_r = partition_alloc(
	partition, size, alignof(hwirq->vic_base_forward_private[0]));
if (alloc_r.e != OK) {
	err = ERROR_NOMEM;
	goto out;
}

[Q] Run two VMs on the Gunyah

Following the steps from https://github.com/quic/gunyah-hypervisor/blob/develop/docs/test.md,

I can boot the Gunyah Hypervisor with the Linux VM on the QEMU simulator.

But it seems there is just one rootvm.

Can we run two VMs on the Gunyah Hypervisor with the Linux VM on the QEMU simulator now ?

Any suggestions would be much appreciated. Thanks

Project Architecture documentation

There is no documentation for project architecture. Need architecture-specific documentation to start using the hypervisor

Running Gunyah on Qualcomm RB5 (QRB5165 SoC)?

Hi,

would it be possible to run Gunyah on a Qualcomm RB5? Do you intend to release a configuration to support this?

Thank you,
Gergely

How to Set a Breakpoint at pvm start_kernel

1. While debugging, I found that pvm KASLR is enabled:
   [ 0.000000] KASLR enabled

2. PVM HLOS mem info
   [RM]HLOS Mem Base : 40000000
   [RM]HLOS Mem Size : 40000000
   [RM]HLOS IPA base : 40000000
   [RM]HLOS DT Ofst : f00000
   [RM]RAM FS offset : 800000

3. start_kernel addr
   root@qemuarm64:# cat /proc/kallsyms |grep start_kernel
   ffffde1763e27d0c T secondary_start_kernel
   ffffde1764fb03c0 T start_kernel
   root@qemuarm64:# cat /proc/kallsyms |more
   ffffde1763e00000 T _text

I tried to set a breakpoint, but it didn't work:
add-symbol-file ./../core-utils/build/vmlinux 0xFFFFDE1763E00000
b start_kernel

Link error while building Resource manager

Hi,

I get the following error while building the resource manager. Any pointers would be very helpful.

cheers,
Achin

[50/50] ld build/resource-manager
FAILED: build/resource-manager
/home/achgup01/work/tools/llvm/llvm-musl-install/bin/clang -target aarch64-linux-gnu -fuse-ld=lld -mcpu=cortex-a75 -static-pie -Wl,--gc-sections -static -fuse-ld=lld --rtlib=compiler-rt -Wl,-z,max-page-size=4096
-Wl,-z,separate-loadable-segments -lfdt -Wl,-Bstatic -L /home/achgup01/work/tools/sysroot/lib --sysroot=/home/achgup01/work/tools/llvm/llvm-musl-install//aarch64-linux-gnu/libc/ build/arch/armv8/obj/src/smc.c
.o build/platform/qemu/obj/src/uart_qemu.c.o build/src/obj/dt/dt_overlay.c.o build/src/obj/event/event-isr.c.o build/src/obj/exit/exit.c.o build/src/obj/guest_interface.c.o build/src/obj/hyp/memextent.c.o build/
src/obj/irq_manager/irq_manager.c.o build/src/obj/memparcel/memparcel.c.o build/src/obj/preempt/preempt.c.o build/src/obj/resource-manager.c.o build/src/obj/rpc/rm-rpc-fifo.c.o build/src/obj/rpc/rm-rpc-msgqueue.
c.o build/src/obj/rpc/rm-rpc.c.o build/src/obj/uart/uart.c.o build/src/obj/utils/address_range_allocator.c.o build/src/obj/utils/circular_buf.c.o build/src/obj/utils/dict.c.o build/src/obj/utils/vector.c.o build
/src/obj/vm_config/dtb_parser.c.o build/src/obj/vm_config/vm_config.c.o build/src/obj/vm_config/vm_config_parser.c.o build/src/obj/vm_config/vm_get_resources.c.o build/src/obj/vm_console/vm_console_simple.c.o bu
ild/src/obj/vm_creation/dto_construct.c.o build/src/obj/vm_creation/hlos_vm.c.o build/src/obj/vm_creation/second_vm.c.o build/src/obj/vm_creation/vm_creation.c.o build/src/obj/vm_mgnt/vm_mgnt.c.o -o build/resour
ce-manager
ld.lld: error: cannot preempt symbol: __init_array_start

defined in
referenced by __libc_start_main.c:13 (../src/env/__libc_start_main.c:13)
__libc_start_main.o:(.rodata+0x18) in archive /home/achgup01/work/tools/llvm/llvm-musl-install//aarch64-linux-gnu/libc//lib/libc.a

ld.lld: error: cannot preempt symbol: __init_array_end

defined in
referenced by __libc_start_main.c:13 (../src/env/__libc_start_main.c:13)
__libc_start_main.o:(.rodata+0x20) in archive /home/achgup01/work/tools/llvm/llvm-musl-install//aarch64-linux-gnu/libc//lib/libc.a

ld.lld: error: cannot preempt symbol: _DYNAMIC

defined in
referenced by __init_tls.c:80 (../src/env/__init_tls.c:80)
__init_tls.o:(.rodata+0x0) in archive /home/achgup01/work/tools/llvm/llvm-musl-install//aarch64-linux-gnu/libc//lib/libc.a

ld.lld: error: cannot preempt symbol: __fini_array_end

defined in
referenced by exit.c:15 (../src/exit/exit.c:15)
exit.o:(.rodata+0x0) in archive /home/achgup01/work/tools/llvm/llvm-musl-install//aarch64-linux-gnu/libc//lib/libc.a

ld.lld: error: cannot preempt symbol: __fini_array_start

defined in
referenced by exit.c:15 (../src/exit/exit.c:15)
exit.o:(.rodata+0x8) in archive /home/achgup01/work/tools/llvm/llvm-musl-install//aarch64-linux-gnu/libc//lib/libc.a
clang-10: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.

Query: Can gunyah-hypervisor support for any RTOS like FreeRTOS

Hello, is it possible to support to run RTOS with gunyah-hypervisor or any plan?

How to Add a test Application/Image into the VM running in gunyah

Following the Guide, I can successfully run gunyah and the VM, but I want do some more study about it,
so, I want to add A Test App( A image) into the VM, then I can run some test.
But I failed.
I tried the below two ways:

1. add the image as the qemu-system-aarch64 with "--drive " option like below:

qemu-system-aarch64 \
-machine virt,virtualization=on,gic-version=3,highmem=off \
-cpu max -m size=2G -smp cpus=8 -nographic \
-kernel ../gunyah/gunyah-hypervisor/hypvm.elf \
-device loader,file=/usr/local/src/out/Image,addr=0x41080000 \
-device loader,file=/usr/local/src/out/virt.dtb,addr=0x44200000 \
-device loader,file=/usr/local/src/out/initrd.img,addr=0x44400000 \
**-drive  format=raw,index=0,file=../gunyah/myAPP.squashfs**

The command runs successfully without error, but after login to VM OS, there nothing can be seen in any folders in the VM OS

2. Try add my APP into the initram disk image, and got error below:
   [ 36.559017] Initramfs unpacking failed: read error
   .......
   .....
   [ 55.419495] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(1,0)
   [ 55.435487] CPU: 5 PID: 1 Comm: swapper/0 Not tainted 5.14.0-rc1-00199-g45312bd762d3-dirty
   [ 55.441980] Hardware name: linux,dummy-virt (DT)
   [ 55.445334] Call trace:
   [ 55.448231] dump_backtrace+0x0/0x1d4
   [ 55.461203] show_stack+0x1c/0x28
   [ 55.467446] dump_stack_lvl+0x68/0x80
   [ 55.480107] dump_stack+0x18/0x3c
   [ 55.481206] panic+0x17c/0x36c
   [ 55.482637] mount_block_root+0x1d4/0x200
   [ 55.484015] mount_root+0x60/0x6c
   [ 55.488445] prepare_namespace+0x11c/0x198
   [ 55.489134] kernel_init_freeable+0x154/0x184
   [ 55.489837] kernel_init+0x24/0x19c
   [ 55.490759] ret_from_fork+0x10/0x18
   [ 55.494631] SMP: stopping secondary CPUs
   [ 55.514051] Kernel Offset: 0x4396b0c80000 from 0xffff800010000000
   [ 55.517683] PHYS_OFFSET: 0x40000000
   [ 55.524732] CPU features: 0x000002d1,6d300c47
   [ 55.533291] Memory Limit: none
   [ 55.552053] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(1,0) ]---

Any advice would be greatly appreciated.

Is that possible run gunyah for raspberry pi ?

Where is the definition of trigger_vectors_* function?

Hello gunyah team,

Where is the definition of the trigger_vectors_* function? For example trigger_vectors_trap_data_abort_el2_event.

Thanks

SVM loading with VIRTIO-SND

1. When launching the Secondary Virtual Machine (SVM) using Crosvm, I encounter the message "No soundcards found" despite having recompiled the Linux kernel (Image) with the CONFIG_SND_VIRTIO=y flag enabled. Does Crosvm lack support for VirtIO sound, or are additional backend configurations required in the Primary Virtual Machine (PVM) to enable VirtIO sound?

2. Instead of Crosvm, can I use Libvirt/Virt-install in the PVM to create and launch the SVM? If yes, would I need to install Libvirt/Virt-install on the PVM? Note that the PVM does not support installing packages via apt.

can I use this on rooted Galaxy Tab S9 Ultra

The CPU is Snapdragon 8 Gen 2 and the Tab S9 Ultra has 16 GB of memory. I am trying to find ways to use this device to run VMs. I would like to use Gunyah but I don't know how to install it.

Is porting to Armv8.0 underway? (If not, I'll start it)

Thank you!

A prebuilt docker instance would be really nice

I'm teaching a class at UCSD Ext and thinking of using Gunyah for students to play around.
Some are Windows users and only have WSL - Windows Subsystem for Linux.

We could not get "docker build -f gunyah/Dockerfile -t user:gunyah ." to complete because the gcc compiler or ninja ran out of memory on WSL.

Is it possible to host a prebuilt docker instance on hub.docker.com? I found a lot of Android and KVM dockers, but no Gunyah docker there.

How does this project relate to the use of hypversisor solution on official Qualcomm products?

No responding after luanching hypvm.elf in qemu

I have built the environment using provided Dockerfile on a Ubuntu 16.04 machine without errors. The image is on dockerhub: ml2558/gunyah:gunyah. However, when I follow the rest process building the hypervisor, resource manger and c runtime and pack them into single hypvm.elf binary and try to run them using qemu, it does not have any output. Is it normal?

Edit: here is what it looks like:

(gunyah-venv) root@a12fdfadd046:/home/gunyah-hypervisor# qemu-system-aarch64 -machine virt,virtualization=on,gic-version=3,highmem=off \

-cpu max -m size=2G -smp cpus=8 -nographic
-kernel /home/hypvm.elf
-device loader,file=$OUTPUT_DIR/Image,addr=$LINUX_BASE
-device loader,file=$OUTPUT_DIR/virt.dtb,addr=$DT_BASE
-device loader,file=$OUTPUT_DIR/initrd.img,addr=$INITRD_BASE

after entering the above command, the terminal just hangs (no errors, crashes, etc).

How to run secondary VM using Gunyah?

Hi, I have a question similar to #15

I succeeded in Gunyah's non-docker setup, but only the primary VM seems to be running.
I found the documentation (https://github.com/quic/gunyah-hypervisor/blob/develop/CHANGELOG.md) suggesting a Linux patch and CrosVM VMM for SVM loading.

Is there any documentation I can follow to load the secondary VM on Gunyah?
I also wonder if Gunyah can load a secondary VM without CrosVM or AVF.

Any advice would be appreciated. Thank you.