VM hangs at boot

[ubicloud]

VM hangs at boot

[pykello]

Code references

• Mountpoint-cache creation: linux/blob/fs/namespace.c
• smpboot: linux/blob/arch/x86/kernel/smpboot.c

Building with guest debug

General build instructions: building.md

$ cargo build --release --target=x86_64-unknown-linux-musl --all --features guest_debug
$ cargo build --target=x86_64-unknown-linux-musl --all --features guest_debug

Core dump

sudo ~/projects/cloud-hypervisor/target/x86_64-unknown-linux-musl/debug/ch-remote \
   --api-socket /tmp/ch.sock coredump file:///tmp/coredump

/opt/cloud-hypervisor/v31.0/ch-remote --api-socket /vm/vmvmhxf8/ch-api.sock pause

/opt/cloud-hypervisor/v31.0/ch-remote --api-socket /vm/vmvmhxf8/ch-api.sock coredump file:///tmp/vmvmhxf8.coredump

Crash Utility

• https://github.com/crash-utility/crash

mnt/usr/src/linux-headers-5.15.0-86/scripts/extract-vmlinux mnt/boot/vmlinuz > /tmp/vmlinux

crash /tmp/vmlinux /tmp/vmvmhxf8.coredump

...
crash: /tmp/vmlinux: no .gnu_debuglink section

Debug symbols

Installing: https://hadibrais.wordpress.com/2017/03/13/installing-ubuntu-kernel-debugging-symbols/

apt install linux-image-5.15.0-86-generic-dbgsym
crash /usr/lib/debug/boot/vmlinux-5.15.0-86-generic /tmp/vmvmhxf8.coredump

[pykello]

Linux kernel code that prints the expected message:

https://github.com/torvalds/linux/blob/d2f51b3516dade79269ff45eae2a7668ae711b25/arch/x86/kernel/smpboot.c#L1231-L1232

        pr_info("CPU0: ");
        print_cpu_info(&cpu_data(0));

https://stackoverflow.com/questions/60803129/kernel-debugging-vmlinux-gdb-py-fails-to-run-on-gdb

git clone --branch v5.15 --depth 1 https://github.com/torvalds/linux
cd linux
make oldconfig
make scripts_gdb

sudo apt install linux-image-5.15.0-83-generic-dbgsym

sudo gdb -q

(gdb) add-auto-load-safe-path ~/projects/linux/
(gdb) souce vmlinux-gdb.py
(gdb) target remote /tmp/gdb.sock
(gdb) c

https://www.kernel.org/doc/html/latest/dev-tools/gdb-kernel-debugging.html
https://wiki.ubuntu.com/DebuggingKernelWithQEMU
cloud-hypervisor/cloud-hypervisor#3213
https://mhcerri.github.io/posts/debugging-the-ubuntu-kernel-with-gdb-and-qemu/

[pykello]

sudo apt install linux-image-6.2.0-1014-azure-dbgsym

[fdr]

Ah, we forgot to close this, but it was indeed https://gitlab.com/qemu-project/qemu/-/issues/1696, and identifying that the default kernel shipped on the GitHub Action image does not have that kernel patch applied, I inspected the Azure 6.5 variant package of the kernel, found that the tick patch was applied, we tried it, and the problem went away.