dpw/kvm-hello-world

A minimal kvm example

A minimal KVM example

kvm-hello-world is a very simple example program to demonstrate the use of the KVM API provided by the Linux kernel. It acts as a very simple VM host, and runs a trivial program in a VM. I tested it on Intel processors with the VMX hardware virtualization extensions. It might work on AMD processors with AMD-V, but that hasn't been tested.

Background

KVM is the Linux kernel subsystem that provides access to hardware virtualization features of the processor. On x86, this means Intel's VMX or AMD's AMD-V. VMX is also known as VT-x; VT-x seems to be the marketing term, whereas VMX is used in the Intel x86 manual set.

In practice, KVM is m often employed via qemu. In that case, KVM provides virtualization of the CPU and a few other key hardware components intimately associated with the CPU, such as the interrupt controller. qemu emulates all the devices making up the rest of a typical x86 system. qemu predates KVM, and can also operate independently of it, performing CPU virtualization in software instead.

But if you want to learn about the details of KVM, qemu is not a great resource. It's a big project with a lot of features and support for emulating many devices.

There's another project that is much more approachable: kvmtool. Like qemu, kvmtool does full-system emulation. unlike qemu, it is deliberately minimal, emulating just a few devices. But while kvmtool is impressive demonstration of how simple and clean a KVM-based full-system emulator can be, it's still far more than a bare-bones example.

So, as no such example seems to exist, I wrote one by studying api.txt and the kvmtool sources. (Update: When I wrote this, I had overlooked https://github.com/soulxu/kvmsample).

Notes

The code is straightforward. It:

• Opens /dev/kvm and checks the version.
• Makes a KVM_CREATE_VM call to creates a VM.
• Uses mmap to allocate some memory for the VM.
• Makes a KVM_CREATE_VCPU call to creates a VCPU within the VM, and mmaps its control area.
• Sets the FLAGS and CS:IP registers of the VCPU.
• Copies a few bytes of code into the VM memory.
• Makes a KVM_RUN call to execute the VCPU.
• Checks that the VCPU execution had the expected result.

A couple of aspects are worth noting:

Note that the Intel VMX extensions did not initially implement support for real mode. In fact, they restricted VMX guests to paged protected mode. VM hosts were expected to emulate the unsupported modes in software, only employing VMX when a guest had entered paged protected mode (KVM does not implement such emulation support; I assume it is delegated to qemu). Later VMX implementations (since Westmere aka Nehalem-C in 2010) include Unrestricted Guest Mode: support for virtualization of all x86 modes in hardware.

The code run in the VM code exits with a HLT instruction. There are many ways to cause a VM exit, so why use a HLT instruction? The most obvious way might be the VMCALL (or VMMCALL on AMD) instruction, which it specifically intended to call out to the hypervisor. But it turns out the KVM reserves VMCALL/VMMCALL for its internal hypercall mechanism, without notifying the userspace VM host program of the VM exits caused by these instructions. So we need some other way to trigger a VM exit. HLT is convenient because it is a single-byte instruction.

kvm-hello-world's Issues

License file

Thanks a lot, this is a really helpful sample!
Would it be possible to add a LICENSE file, e.g. MIT or Apache 2?

hlt instruction qemu-kvm

Hi.

I'm trying to implement that a user program in a guest OS executes an instruction(or function) and qemu receives it.
What you mentioned about vmcall and hlt is really helpful and your hello world example can do what I wanna do exactly.

I can find the ioctl(KVM_RUN) in kvm-all.c in the qemu source tree. I added a test code right after this ioctl and checked that exit_reason is same with KVM_EXIT_HLT.
First, I executed hlt instruction in a user program in a guest machine.
It caused a segfault and qemu received nothing. I thought it is because hlt is a privileged instruction.

So I added a syscall which just executes hlt into a guest kernel, and make a user program execute this syscall.
But qemu's exit reason was never the same as a KVM_EXIT_HLT.

Using perf tool, however, I can see that there are a lot of vm exit caused by the halt when I executed this syscall repeatedly. It seems that vmexit is occured, but because of somewhat reason, it is not delivered to qemu.

It might be trivial problem... but I'm really new to low level things.
What did I miss?

I will be really appreciated if you give me an advice.
Thanks in advance.

Question: where is string "Hello world!" data

0000000000000000 <_start>:
   0:   f3 0f 1e fa             endbr64 
   4:   ba 00 00 00 00          mov    $0x0,%edx
   9:   b8 48 00 00 00          mov    $0x48,%eax
   e:   e6 e9                   out    %al,$0xe9
  10:   0f b6 42 01             movzbl 0x1(%rdx),%eax
  14:   48 83 c2 01             add    $0x1,%rdx
  18:   84 c0                   test   %al,%al
  1a:   75 f2                   jne    e <_start+0xe>
  1c:   48 c7 04 25 00 04 00    movq   $0x2a,0x400
  23:   00 2a 00 00 00 
  28:   b8 2a 00 00 00          mov    $0x2a,%eax
  2d:   0f 1f 00                nopl   (%rax)
  30:   f4                      hlt    
  31:   eb fd                   jmp    30 <_start+0x30>

This is objdump result of guest64.o
It seems that no "Hello world!" string data in this code.
(I think line9 of 0x48 is ASCI code of "H")
Why can this guest output "Hello world!" when it execute on Host VMM(lvm-hello-world)?

64 bit mode broken in Ubuntu 18.04 LTS (KVM_SET_SREGS: Invalid argument)

Hi,

since Ubuntu 18.04 LTS came out 2 days ago I performed a clean install of Ubuntu 18.04 LTS 64 bit.
Unfortunately in the latest Ubuntu Version the Long Mode Version seems broken:
ioctl(vcpu->fd, KVM_SET_SREGS, &sregs) (run_long_mode() line 410 kvm-hello-world.c) fails with:

KVM_SET_SREGS: Invalid argument

All other Modes (16 bit, 32 bit, 32 bit paged) still work fine. Also reinstalling the old Version Ubuntu 17.10 64 bit makes the problem disappear, so it must be related to a change in the KVM module in the latest Ubuntu version.

Taking a closer look at the Code shows that 32 bit Paged Mode and 64 bit Mode Settings only differ in 4 Points:

-Segment Setup:
--seg.db=1 in 32 bit (=> default operand size 32 bit), =0 in 64 bit (=> 16 bit Operand Size)
--seg.l=0 in 32 bit, =1 in 64 bit (=1 enables 64 bit instruction encoding)
-Control Register Setup:
--CR4=CR4_PSE in 32 bit (4 MiB Pages), =CR4_PAE (36 bit extended physical address translation in gpt)
--EFER=0 in 32 bit, =EFER_LME in 64 bit (long mode enable)

Experimenting a bit shows that the problem is probably related to the combination of seg.l=1 and EFER=EFER_LME: when both are reset to the 32 bit settings the problem seems to disappear (to be precise it causes another problem resulting in a KVM_EXIT_SHUTDOWN but it makes the error "KVM_SET_SREGS: Invalid argument" disappear and lets the VM start).

Anyway seg.l=1 and EFER=EFER_LME are essential for entering 64 bit mode and can't be skipped of course - Maybe there was a change in Ubuntu 18.04 KVM regarding long mode in virtual machines? At least I didn't read about any such change...

Could you maybe tell me how to fix this problem?

Thanks a lot in advance for your help :)

Kind regards,
Andreas

No real-mode output?

Notice, no output after "Testing real mode".

$make
cc -Wall -Wextra -Werror -O2 -g   -c -o kvm-hello-world.o kvm-hello-world.c
as   -o guest16.o guest16.s
cc -Wall -Wextra -Werror -O2 -g -m32 -ffreestanding -fno-pic -c -o guest32.o guest.c
ld -T guest.ld -m elf_i386 guest32.o -o guest32.img
ld -b binary -r guest32.img -o guest32.img.o
cc -Wall -Wextra -Werror -O2 -g -m64 -ffreestanding -fno-pic -c -o guest64.o guest.c
ld -T guest.ld guest64.o -o guest64.img
ld -b binary -r guest64.img -o guest64.img.o
ld -T payload.ld -o payload.o
cc kvm-hello-world.o payload.o -o kvm-hello-world
./kvm-hello-world
Testing real mode
./kvm-hello-world -s
Testing protected mode
Hello, world!
./kvm-hello-world -p
Testing 32-bit paging
Hello, world!
./kvm-hello-world -l
Testing 64-bit mode
Hello, world!

How to run this proyect?

Hello, sorry for this issue but I have a problem with the kvm file.
The problem that I have is that I don't know how to run the file. When I try to run it my Kali says that
"open /dev/kvm: No such file or directory" and I don't know what to do, I followed many tutorials about how to run this kind of binaries but none of these work for me.
Can you help me please?

Undefined Reference to "_GLOBAL_OFFSET_TABLE_" when linking guest32.o

Hi,

first of all thanks a lot for all the code, I'm currently trying to get started setting up a VM with KVM api (ideally 64 bit mode later on) and these examples help me a lot in understanding how to use the KVM api 👍

Unfortunately I get an error message When trying to compile the code with "make run":

---

Original Output:
ld -T guest.ld -m elf_i386 guest32.o -o guest32.img
guest32.o: In Funktion »_start«:
guest.c:(.start+0x6): Warnung: undefinierter Verweis auf »GLOBAL_OFFSET_TABLE«
Makefile:26: die Regel für Ziel „guest32.img“ scheiterte
make: *** [guest32.img] Fehler 1

---

So ld complains about an undefined reference to "GLOBAL_OFFSET_TABLE" at offset ".start+0x6" when trying to link the guest32.o file into guest32.img using the guest.ld linker script.
(Trying to build this on an Ubuntu 17.10 64 bit OS with Intel® Core™ i5-2400 CPU inside).

Anyway 16bit and 64bit version give no errors when compiling and linking, everything works just fine here.

Did I miss anything for 32bit-mode? Unfortunately I don't have an idea how to fix this problem currently ...
Could you maybe tell me what causes this problem and how to fix it?

Thanks a lot in advance for your help :)

Kind regards,
Andreas

kvm-hello-world on AMD

I tried this code on AMD, first on the host machine and also on a virtual machine. It works as expected on the host machine, but it crashed the virtual machine (and QEMU which I used to launch the virtual machine). Do you have any suggestion on how to make it work on a virtual machine?

/usr/bin/ld: cannot use executable file 'payload.o' as input to a link

Hi,
when the makefile is executed to

 kvm-hello-world: kvm-hello-world.o payload.o
	$(CC) $^ -o $@

a error comes /usr/bin/ld: cannot use executable file 'payload.o' as input to a link

ld version is GNU ld (GNU Binutils) 2.36.1

Are there any good ideas to fix this?
thanks.

Question about -Werror in makefile

On some platforms. The compiler may check the array-bounds and see this line as an error.

cc -Wall -Wextra -Werror -O2 -m32 -ffreestanding -fno-pic -c -o guest32.o guest.c
guest.c: In function ‘_start’:
guest.c:17:9: error: array subscript 0 is outside array bounds of ‘long int[0]’ [-Werror=array-bounds=]
   17 |         *(long *) 0x400 = 42;
      |         ^~~~~~~~~~~~~~~
cc1: note: source object is likely at address zero

Removing the -Werror in makefile will fix it.

--- a/Makefile
+++ b/Makefile
@@ -1,4 +1,4 @@
-CFLAGS = -Wall -Wextra -Werror -O2
+CFLAGS = -Wall -Wextra -O2

Question on KVM

I've not been able to figure out an answer to this, but how many KVM instances can you make on a host machine, is there some limit? Is there a Kernel limit in Linux, or a hardware limit?

Obviously ram would be a limiting factor, but say I want to have 32gb of ram allocated to as many VM's as a 2MB memory limit would allow, is that possible? That would be 16k of VM instances, is that actually doable?

In fact, this is an issue, but a comment.

This code is very very helpful for the one just beginning to learn to KVM, thanks for the excellent job.
I just finished testing the MMIO emulation based on your code (changed your code to support KVM_MMIO_EXIT).

And I would like to see your code to have more and more features to show the features/functionalities of KVM.

And here is another example in github for KVM API for your reference.
https://github.com/hitmoon/kvmsample/blob/master/main.c.