fubarnetes/bhyve-multiboot

WIP: Multiboot bootloader for bhyve

fubarnetes

Scalable Kloud Komputing with bhyve

bhyve-multiboot's Issues

tests: build harness for libmultiboot.so

• implement the callbacks in userboot.h as stubs, and dlopen(3) libmultiboot.so.
• When testing using libmultiboot.so, we need to copy the library into the test directory to dlopen(3) it. I've opened freebsd/kyua#176 to find out how best to do this.
• Then we need to copy test kernels (from #2) in to check the multiboot functions

copyin cannot copy into highmem

bhyveload's copyin cannot copy into highmem, and will just wrap around when trying to:

bhyve-multiboot/tests/mock/bhyveload.c

Line 323 in 2cab4f7

to &= 0x7fffffff;

We have to make sure this doesn't happen (e.g. by ELF files that have segments in highmem). This may only be an issue using 64 bit ELFs, see #5.

tests: corpus of test kernels

We need a corpus of test kernels to test with. There's a few options we could go with here:

Non-functional kernels:

• ELF: Build a simple ELF stub and add a multiboot / multiboot2 header programmatically
• a.out kludge: Use random data with a multiboot / multiboot2 header containing address data

Functional test kernels:

• ELF without a.out kludge: test kernels in grub2. These can be found in grub-core/tests/boot/kernel-${ARCH}.S, and can be built using make multiboot.elf and make multiboot2.elf, respectively. These are licensed under the GPL though, not sure if they can simply be incorporated.
• ELF with a.out kludge: Test kernels built with rumprun.
• test kernel without a.out kludge: write one

check whether starting in 64-bit mode is possible.

see qemu/qemu@9696846

use file IO callbacks from userboot.h instead of FILE* operations

Instead of using fopen(3), fseek(3), ftell(3), fclose(3) and fileno(3), use

callbacks->open(void *arg, const char *filename, void **h_return);
callbacks->close(void *arg, void *h);
callbacks->read(void *arg, void *h, void *dst, size_t size, size_t *resid_return);
callbacks->seek(void *arg, void *h, uint64_t offset, int whence);
callbacks->stat(void *arg, void *h, int *mode_return, int *uid_return, int *gid_return, uint64_t *size_return);

The following code to get the file size will have to be rewritten using stat(2) to get the file size.

bhyve-multiboot/multiboot.c

Lines 428 to 431 in f0d394f

	/* Get the kernel file size */
	fseek(kernfile, 0L, SEEK_END);
	kernsz = ftell(kernfile);
	rewind(kernfile);

travis: fix coverage

At the moment the coverage isn't really working. This is most likely an issue with clang / llvm-cov.

4k-aligned loading

If bit 0 of the multiboot header flags indicates the kernel requests the to be loaded at a 4k-aligned offset. In this case, use allocate_aligned instead of allocate.

set up multiboot information structure

• if the os image requests it
  • get the upper and lower memory using
    

    bhyve-multiboot/userboot.h

    Lines 186 to 190 in ff5a9bc

    	/*
    	* Return guest physical memory map details
    	*/
    	void (*getmem)(void *arg, uint64_t *lowmem,
    	uint64_t *highmem);

  • set mbi->mem_lower and mbi->mem_upper and set bit 0 of mbi->flags.
  • generate a memory map buffer as described in the multiboot spec, point mbi->mmap_addr to it, set mbi->mmap_length, and set bit 6 of mbi->flags.
• make sure bit 1 of mbi->flags is cleared, as specified:

  If bit 1 in the ‘flags’ word is set, then the ‘boot_device’ field is valid, and indicates which bios disk device the boot loader loaded the OS image from. If the OS image was not loaded from a bios disk, then this field must not be present (bit 3 must be clear).

• if cmdline is given, copy it into the hypervisor, and set mbi->cmdline and set bit 2 of mbi->flags.
• if modules are given, parse their command-line specification and load them:
  • copy them into the guest, page-align them if requested by the kernel.
  • set up an array of module structures, point mbi->mods_addr to it and set mbi->mods_count
• set bit 9 of mbi->flags. copy the bootloader name into the guest and point mbi->boot_loader_name to it.
• if the kernel requests it, set up video modes - although it isn't clear what this means for headless systems.

tests: test structure offsets of multiboot header

See Section 3.1.1 of the spec and fabianfreyer@e952761 for reference

Test that load_kernel actually loads the kernel into memory

The tests load_aout_direct and load_elf_direct only check the return values of the load functions:

bhyve-multiboot/tests/test-multiboot.c

Lines 214 to 228 in 63cc3d7

	/* Load a kernel with a valid a.out kludge */
	ATF_CHECK(fread(random_buffer, 128*kiB, 1, urandom));
	mbh = (struct multiboot_header* )(random_buffer + 2*kiB);
	mbh->magic = MULTIBOOT1_MAGIC;
	mbh->flags = 0 | MULTIBOOT_AOUT_KLUDGE;
	mbh->checksum = (uint32_t) 0U-MULTIBOOT1_MAGIC-(mbh->flags);
	mbh->header_addr = 1*MiB;
	mbh->load_addr = 1*MiB;
	mbh->load_end_addr = 1*MiB + 0x100;
	mbh->bss_end_addr = 1*MiB + 0x200;
	mbh->entry_addr = 1*MiB + sizeof(struct multiboot_header);
	mb = mb_scan(random_buffer, 128*kiB);
	error = multiboot_load_aout(random_buffer, 128*kiB, mbh);
	ATF_CHECK_EQ_MSG(0, error, "multiboot_load_aout failed");

bhyve-multiboot/tests/test-multiboot.c

Lines 302 to 306 in 63cc3d7

	mb = mb_scan(kernel, kernsz);
	mbh = mb->info.mb.header;
	multiboot_load_type(kernel, kernsz, &kernelf, mbh);
	error = multiboot_load_elf(kernel, kernsz, kernelf);
	ATF_CHECK_EQ_MSG(0, error, "multiboot_load_elf failed");

Instead they should also verify that copyin actually copied something. Probably the best idea is to

• implement a function to get the lowmem_buffer and highmem_buffer pointers from mock_bhyveload.c
• then use memcmp() to verify that the appropriate parts were actually loaded

allocator: make sure all allocations are within usable memory

Currently, the allocator doesn't use any memory map and assumes all memory is usable.

Getting the memory map early would help #4.

bhyve-multiboot/userboot.h

Lines 186 to 190 in ff5a9bc

	/*
	* Return guest physical memory map details
	*/
	void (*getmem)(void *arg, uint64_t *lowmem,
	uint64_t *highmem);

For the bhyve memory map, see grub2-bhyve's implementation:

  /*
   * bhyve is going to return the following memory segments
   *
   * 0 - 640K    - usable
   * 640K- 1MB   - vga hole, BIOS, not usable.
   * 1MB - lomem - usable
   * lomem - 4G  - not usable
   * 4G - himem  - usable [optional if himem != 0]
   */
  bhyve_info.nsegs = 2;
  bhyve_info.segs = bhyve_mm;

  bhyve_mm[0].start = 0x0;
  bhyve_mm[0].end = 640*1024;		/* 640K */
  bhyve_mm[0].type = GRUB_MEMORY_AVAILABLE;

  bhyve_mm[1].start = 1024*1024;
  bhyve_mm[1].end = (memsz > lomemsz) ? lomemsz : memsz;
  bhyve_mm[1].type = GRUB_MEMORY_AVAILABLE;

  if (memsz > lomemsz) {
    bhyve_info.nsegs++;
    bhyve_mm[2].start = 4*GB;
    bhyve_mm[2].end = (memsz - lomemsz) + bhyve_mm[2].start;
    bhyve_mm[2].type = GRUB_MEMORY_AVAILABLE;
  }

An easy fix might just be:

• set MIN_ALLOC_ADDRESS to the start of the 1 MiB - Lowmem segment
• allocate the first hole from lomem to 4 GiB to reserve it
• Allocations above himem (if set, otherwise lomem) shall fail.