qemu hangs with 100% cpu

[keystone-enclave]

qemu hangs with 100% cpu

[filamoon]

It seems spinlock_lock is hanging, as shown in the following gdb commands:

boot_loader (dtb=4128) at ../bbl/bbl.c:57
57 filter_dtb(dtb);
(gdb)
59 print_logo();
(gdb)

Breakpoint 1, boot_loader (dtb=4128) at ../bbl/bbl.c:64
64 mb();
(gdb)
65 entry_point = &_payload_start;
(gdb)
66 boot_other_hart(0);
(gdb)

Breakpoint 1, boot_loader (dtb=4128) at ../bbl/bbl.c:64
64 mb();
(gdb)
65 entry_point = &_payload_start;
(gdb)
66 boot_other_hart(0);
(gdb)
^C
Program received signal SIGINT, Interrupt.
spinlock_lock (lock=0x80021360 <sm_init_lock>) at ../machine/atomic.h:55
55 ;
(gdb)

[dayeol]

Hi,
Can you share your QEMU command line arguments?
And also, can you try git diff in the qemu source directory and share me the result?

FYI, we haven't tried upstream QEMU 4.0.0 yet.

[filamoon]

qemu-system-riscv64
-s -S -d in_asm
-D debug.log
-m 4G
-bios bootrom/bootrom.elf
-nographic
-machine virt
-kernel riscv-pk/build/bbl
-append "root=/dev/vda ro console=ttyS0"
-drive file=riscv-pk/release/busybear.bin,format=raw,id=hd0
-device virtio-blk-device,drive=hd0
-netdev type=tap,id=net0
-device virtio-net-device,netdev=net0

[filamoon]

I did not clone the qemu repo, but downloaded qemu-4.0.0.tar.xz directly from qemu.org. BTW, I failed to compile riscv-qemu since it seems conflict with the latest glibc.

[dkohlbre]

qemu 4.0.0 is not currently supported (I've been testing with the RCs). It includes the newer RISC-V CLIC interrupt handling, and Keystone doesn't yet configure it correctly.
Now that 4.0 is coming out, we'll be looking into updating to support it.

Can you elaborate on the riscv-qemu errors you got?

[edit] We should be able to take a look at 4.0 support early this week.

[dayeol]

In addition, you might need to apply the patch in patches/qemu-kernel-reloc.patch if you haven't.
(This patch is made on qemu 3.1, so I'm not sure it will work on 4.0).
This patch is for emulating the secure boot in Keystone correctly using --bios flag.

[filamoon]

@dkohlbre
Not any error logs in text at all. It just hangs with 100% CPU usage.
I used gdb to debug and found it is stuck in spinlock_lock.
It did show the logo ASCII graph twice.

[root@ip- keystone]# scripts/run.sh
bbl loader

--------- LOGO ASCII ------------

   INSTRUCTION SETS WANT TO BE FREE

initializing sm
bbl loader

--------- LOGO ASCII ------------

   INSTRUCTION SETS WANT TO BE FREE

initializing sm

[filamoon]

hmm, there is no hw/riscv/boot.c in qemu 4.0.0 ...

[filamoon]

there is no hw/riscv/boot.c in qemu 3.1.0 either...

[dkohlbre]

Ah, I meant whatever errors in building riscv-qemu 3.1 you had, since that is the one that is going to work right now.

[dayeol]

I think you should use riscv-qemu instead of upstream qemu. riscv-qemu had been the most up-to-date version for RISC-V support until very recently, and we haven't tested Keystone with upstream version.
I recommend you to use riscv-qemu 3.1 at this moment unless you have other reasons to use the other versions.

[dayeol]

• looking through the commit logs, I believe hw/riscv/boot.c has been removed from the new versions.
  We'll look into the new version.

[filamoon]

hmm, I got errors of "dereference to incomplete type ucontext_t" when compiling on Ubuntu 18.04, but it compiled ok on CentOS 7. So I will use riscv-qemu and see.

[filamoon]

It also hangs.
Compiled from latest https://github.com/keystone-enclave/riscv-qemu.
QEMU emulator version 2.12.92 (v3.0.0-rc2-83-g3b3e2ed8a6-dirty)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers

[root@ip- keystone]# scripts/run.sh
bbl loader
vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
vvvvvvvvvvvvvvvvvvvvvvvvvvvv
rrrrrrrrrrrrr vvvvvvvvvvvvvvvvvvvvvvvvvv
rrrrrrrrrrrrrrrr vvvvvvvvvvvvvvvvvvvvvvvv
rrrrrrrrrrrrrrrrrr vvvvvvvvvvvvvvvvvvvvvvvv
rrrrrrrrrrrrrrrrrr vvvvvvvvvvvvvvvvvvvvvvvv
rrrrrrrrrrrrrrrrrr vvvvvvvvvvvvvvvvvvvvvvvv
rrrrrrrrrrrrrrrr vvvvvvvvvvvvvvvvvvvvvv
rrrrrrrrrrrrr vvvvvvvvvvvvvvvvvvvvvv
rr vvvvvvvvvvvvvvvvvvvvvv
rr vvvvvvvvvvvvvvvvvvvvvvvv rr
rrrr vvvvvvvvvvvvvvvvvvvvvvvvvv rrrr
rrrrrr vvvvvvvvvvvvvvvvvvvvvv rrrrrr
rrrrrrrr vvvvvvvvvvvvvvvvvv rrrrrrrr
rrrrrrrrrr vvvvvvvvvvvvvv rrrrrrrrrr
rrrrrrrrrrrr vvvvvvvvvv rrrrrrrrrrrr
rrrrrrrrrrrrrr vvvvvv rrrrrrrrrrrrrr
rrrrrrrrrrrrrrrr vv rrrrrrrrrrrrrrrr
rrrrrrrrrrrrrrrrrr rrrrrrrrrrrrrrrrrr
rrrrrrrrrrrrrrrrrrrr rrrrrrrrrrrrrrrrrrrr
rrrrrrrrrrrrrrrrrrrrrr rrrrrrrrrrrrrrrrrrrrrr

   INSTRUCTION SETS WANT TO BE FREE

initializing sm

[dkohlbre]

That is not the version of riscv-qemu we're using in our checkouts. Please make sure you are building using the main keystone repo make buildscript. It will (assuming you are on dev branch of keystone) build everything automatically with our patches. qemu v3.0.0-rc2 is not the checkout we are using:

keystone/riscv-qemu$ ./riscv64-softmmu/qemu-system-riscv64 --version
QEMU emulator version 3.1.0
Copyright (c) 2003-2018 Fabrice Bellard and the QEMU Project developers

and

submodule status
 3cc4afdb712638ffee307eeb672047f91597386e riscv-qemu (riscv-for-master-3.1-rc2-203-g3cc4afdb71)

I've done a clean checkout-quick-setup-make on 18.04 a few times, so if you can give us the specific error you got that would help.

[filamoon]

fast-setup.sh fails. That's why I could not use the main make flow.

[root@ip- keystone]# ./fast-setup.sh
Starting...
RISCV tools are already installed
Skipping submodule 'riscv-gnu-toolchain'
error: Server does not allow request for unadvertised object d79ac1e5ee9fabccdb363c1e8acbbad0002859ca
Fetched in submodule path 'buildroot', but it did not contain d79ac1e5ee9fabccdb363c1e8acbbad0002859ca. Direct fetching of that commit failed.

[dkohlbre]

That is the root problem here, the submodules never got updated correctly on your end. (Our CI flow does the full buildroot checkout using fast-setup.sh, so its working there)

I'm not sure how to debug that one, as I just tried fetching and updating that submodule successfully.
Can you give us the output of git submodule status and git submodule update buildroot?

[dayeol]

You need to use those versions that are referred by the main top-level keystone repo, either the master branch or dev branch.

master branch docs: http://docs.keystone-enclave.org/en/latest/
dev branch docs: http://docs.keystone-enclave.org/en/dev/

I believe something about submodule remotes have been gone wrong in your end.

The buildroot hash you copied is valid in the correct remote url:

https://github.com/keystone-enclave/buildroot/tree/d79ac1e5ee9fabccdb363c1e8acbbad0002859ca

Have you cloned the latest keystone repo (either master or dev) branch?
or have you cloned them a while ago and then pulled the latest commits?

If you pulled the latest commits after you cloned the repo a while ago, you need to synchronize / update your submodules.

git submodule sync --recusrive
git submodule update --init --recursive

[filamoon]

I did a fresh checkout from https://github.com/keystone-enclave/keystone ...

git submodule status

+6795df2c5439bb95e017f4430123ea12ec5a26b1 buildroot (2017.08-679-g6795df2c54)
ed72ff181996a38c226d676e24f2dbe0ca634212 busybear-linux (v1.0-14-ged72ff1)
+56ea0b4279f42b566c6cb61ba44b41d0baa60643 linux-keystone-driver (heads/master)
1b80cbe97d2c29212398d3b74ddc54718ad32e23 riscv-gnu-toolchain ((null))
+600cf51b022ea962a8a9d17a7b259c552b4d34ac riscv-linux (v4.17-57306-g600cf51b022e)
+a280c5f19480a29ccea8b7c1c00be82e7134a749 riscv-pk (archive-1.9.1-213-ga280c5f)
+6cd6ef80287394708d34f7c62a2e78cd2a308799 riscv-qemu (riscv-for-master-3.1-rc2-3577-g6cd6ef8028)
[phone]f7e8c7c9cbab59f3c2545ad10d4f14ae sdk (heads/master)

git submodule update buildroot

error: Server does not allow request for unadvertised object d79ac1e5ee9fabccdb363c1e8acbbad0002859ca
Fetched in submodule path 'buildroot', but it did not contain d79ac1e5ee9fabccdb363c1e8acbbad0002859ca. Direct fetching of that commit failed.

[filamoon]

I switched to the dev branch and it checked out ok.
But sdk/runtime directory seems to be empty.

[filamoon]

In sdk/, I removed runtime from Makefile and went on.

After a few hours I got this error:

/data/keystone-enclave/keystone/riscv-linux/scripts/extract-cert.c:21:10: fatal error: openssl/bio.h: No such file or directory
#include <openssl/bio.h>
^~~~~~~~~~~~~~~
compilation terminated.
make[4]: *** [scripts/extract-cert] Error 1
make[3]: *** [scripts] Error 2
make[2]: *** [sub-make] Error 2
make[2]: Leaving directory /data/keystone-enclave/keystone/riscv-linux' make[1]: *** [/data/keystone-enclave/keystone/hifive-work/hifive-linux/vmlinux] Error 2 make[1]: Leaving directory /data/keystone-enclave/keystone'
make: *** [hifive] Error 2

[dkohlbre]

The sdk/runtime directory contains critical components to run any keystone applications.
It is a git submodule, and thus is empty by default.
fast-setup.sh will run the command @dayeol noted earlier:
git submodule sync --recusrive and git submodule update --init --recursive which initialize our submodules and check them out.
We extensively use git submodules to keep track of our different components. Not syncing and updating them will 100% break our builds. If you've done a fresh checkout or switched branches you must run the sync and update recursively.

The latter error indicates you don't have openssl installed.

[dkohlbre]

To be clear.
If you run a git clone on keystone, git checkout dev, fast-setup.sh, and make. Keystone will correctly setup and build. If the repository clone you have is in some bad state due to removing submodules/etc, I suggest you start over from a clean clone.

Please try a clone, checkout dev, fast-setup, and make. If those fail get back to us on what didn't work.

[filamoon]

I figured out what was wrong. I did:
clone, fast-setup, [fail], checkout dev, fast-setup, make.
Some dirs were not synced during the second fast-setup run. I removed them and git synced again and now I can finally make it.

So "checkout dev" is very important.

[dkohlbre]

Glad to hear it builds correctly.

It looks like when we updated buildroot, we broke the specific checkout master was using.
We'll update master to ensure it works.
Closing for now, we'll figure out qemu4.0 in a separate bug.