Unable to add hypervisor entitlement to qemu

[lima-vm]

Unable to add hypervisor entitlement to qemu

[christian-korneck]

@jandubois I ran into similar issues and this worked for me (macOS 11.3, Intel):

• only the relevant key in the e.xml (not both):

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>com.apple.security.hypervisor</key>
    <true/>
</dict>
</plist>

• brew install qemu
• codesign --remove-signature /usr/local/bin/qemu-system-x86_64
• codesign -s - --entitlements e.xml --force /usr/local/bin/qemu-system-x86_64

and this should not be necessary, but if you're still getting a killed: 9 error you could also try to set a gatekeeper exception:

• spctl --add /usr/local/bin/qemu-system-x86_64

[irlevesque]

TL'DR: use com.apple.security.hypervisor (and only), even on MacOS 10.15. I think the README is misleading here (I'll submit a pr shortly).

Yep, that worked for me. Thanks for digging in, @christian-korneck

[jandubois]

@christian-korneck Thanks for your comments, but they don't seem to make any difference to me.

• I already trimmed the entitlement.xml to only include the version used by Catalina.
• The qemu version installed by brew is unsigned, so --remove-signature did nothing.
• The codesign "works", but afterwards I get the "Killed: 9" response
• spctl -a already says "rejected" even for the unsigned binary, but the binary works fine.
• Running spctl --add didn't change anything; spctl -a still says "Rejected" and running the signed binary still says "Killed: 9".

This is all very mystifying for me; maybe qemu needs to be notarized in addition to being signed?

I guess it is time for me to try this on BigSur to see if that makes a difference.

[jandubois]

I guess it is time for me to try this on BigSur to see if that makes a difference.

Same problems on my Big Sur laptop. 😞

[bric3]

For the record running codesign with the entitlement with both keys didn't raise an error.

$ cat >entitlements.xml <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"\>
<plist version="1.0">
<dict>
    <!-- for OS X 10.10 - macOS 10.15 -->
    <key>com.apple.vm.hypervisor</key>
    <true/>
    <!-- for macOS 11 and later -->
    <key>com.apple.security.hypervisor</key>
    <true/>
</dict>
</plist>
EOF
$ codesign -s - --entitlements entitlements.xml --force /usr/local/bin/qemu-system-x86_64

However when starting the fedora image, I got issues to install ssh, sshfs and guest agent. Which I didn't link rught away to the qemu signing.

$ limactl start fedora.yaml      # or later with    limactl start fedora
INFO[0000] Using the existing instance "fedora"
INFO[0000] Starting QEMU
INFO[0000] SSH: 127.0.0.1:60024
INFO[0000] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0010] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0020] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0030] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0040] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0050] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0060] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0070] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0080] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0090] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0100] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0110] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0120] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0130] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0140] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0150] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0160] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0170] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0180] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0190] Waiting for the essential requirement 1 of 3: "ssh"
... same message for a while

Once I followed @christian-korneck 's #4 (comment), starting lima worked:

$ limactl start fedora
INFO[0000] Using the existing instance "fedora"
INFO[0000] Starting QEMU
INFO[0000] SSH: 127.0.0.1:60024
INFO[0000] Waiting for the essential requirement 1 of 3: "ssh"
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.x2apic [bit 21]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.tsc-deadline [bit 24]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.invpcid [bit 10]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EDX.spec-ctrl [bit 26]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.80000001H:EDX.rdtscp [bit 27]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.x2apic [bit 21]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.tsc-deadline [bit 24]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.invpcid [bit 10]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EDX.spec-ctrl [bit 26]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.80000001H:EDX.rdtscp [bit 27]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.x2apic [bit 21]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.tsc-deadline [bit 24]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.invpcid [bit 10]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EDX.spec-ctrl [bit 26]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.80000001H:EDX.rdtscp [bit 27]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.x2apic [bit 21]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.tsc-deadline [bit 24]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.invpcid [bit 10]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EDX.spec-ctrl [bit 26]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.80000001H:EDX.rdtscp [bit 27]
INFO[0010] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0020] Waiting for the essential requirement 1 of 3: "ssh"
INFO[0020] The essential requirement 1 of 3 is satisfied
INFO[0020] Waiting for the essential requirement 2 of 3: "sshfs binary to be installed"
INFO[0020] The essential requirement 2 of 3 is satisfied
INFO[0020] Waiting for the essential requirement 3 of 3: "the guest agent to be running"
INFO[0020] The essential requirement 3 of 3 is satisfied
INFO[0020] Mounting "/Users/bric3"
INFO[0021] Mounting "/tmp/lima"
INFO[0021] Waiting for the optional requirement 1 of 1: "containerd binaries to be installed"
INFO[0021] Forwarding "/run/user/501/lima-guestagent.sock" (guest) to "/Users/bric3/.lima/fedora/ga.sock" (host)
INFO[0021] Forwarding TCP port 5355
INFO[0021] Forwarding TCP port 5355
INFO[0021] The optional requirement 1 of 1 is satisfied
INFO[0021] READY. Run `lima bash` to open the shell.

$ lima bash
exit status 255
$ limactl shell fedora
bash: line 1: cd: /usr/local/opt/lima: No such file or directory
[bric3@lima-fedora bric3]$ uname -a
Linux lima-fedora 5.11.12-300.fc34.x86_64 #1 SMP Wed Apr 7 16:31:13 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

There's still some issues, that's probably due to my extract path (/usr/local/opt/lima)

[irlevesque]

I'm also having this issue. The workarounds mentioned haven't helped. I wonder if it's because SIP is enabled on my Mac?

[christian-korneck]

I wonder if it's because SIP is enabled on my Mac?

@irlevesque - the above worked for me on macOS 11.3.1 (Intel), with SIP enabled (for completeness: I have dtrace allowed csrutil enable --without dtrace, but I don't think this matters here).

I also briefly tried to get it to work on github actions (which uses MacOS 10.15 on Intel) and I ran into "killed: 9" problems. Github Actions currently has an outage, so can't test any further today.

[christian-korneck]

I also briefly tried to get it to work on github actions (which uses MacOS 10.15 on Intel) and I ran into "killed: 9" problems.

so I've now tried it twice on a vanilla MacOS 10.15.7 (Intel) VM with SIP kept enabled (aws ec2 and github actions) and this worked for me:

TL'DR: use com.apple.security.hypervisor (and only), even on MacOS 10.15. I think the README is misleading here (I'll submit a pr shortly).

assuming homebrew is already installed:

brew install qemu coreutils

cat >entitlements.xml <<EOF
> <?xml version="1.0" encoding="UTF-8"?>
> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
> <plist version="1.0">
> <dict>
>     <key>com.apple.security.hypervisor</key>
>     <true/>
> </dict>
> </plist>
> EOF


codesign -s - --entitlements entitlements.xml --force /usr/local/bin/qemu-system-x86_64
curl -L https://github.com/AkihiroSuda/lima/releases/download/v0.1.0/lima-0.1.0-Darwin-x86_64.tar.gz -o lima.tar.gz
sudo tar xf lima.tar.gz -C /usr/local/
ssh-keygen -q -t rsa -N '' -f ~/.ssh/id_rsa <<<n 2>&1 >/dev/null #create ssh keypair, needed only on vanilla machine
touch ~/.ssh/known_hosts #create ssh hosts file, needed only on vanilla machine
limactl start

[jandubois]

TL'DR: use com.apple.security.hypervisor (and only), even on MacOS 10.15. I think the README is misleading here (I'll submit a pr shortly).

Thank you @christian-korneck, this worked for me as well.

However, how can I tell if qemu is actually using the hvf acceleration, as limactl start default worked for me even when I didn't codesign the qemu binary.

I get exactly the same output on stdout/stderr after codesigning than I did before?

Given that I get qemu-system-x86_64: Error: HV_ERROR on Big Sur if I don't codesign the qemu executable, but it runs just fine on Catalina without it, I now think that -accel=hvf is maybe a no-op on Catalina, and that there might be no point in bothering with signing qemu on that macOS version.

[jandubois]

I now think that -accel=hvf is maybe a no-op on Catalina,

After reading the com.apple.vm.hypervisor description again, I think it is also possible that we simply don't need the old entitlement on Catalina because it was only required for sandboxed processes before:

The entitlement is required to use the Hypervisor APIs in a sandboxed process.

So maybe Big Sur needs the new entitlement for all processes, including non-sandboxed ones, but things work fine on Catalina without it as long as the process is not sandboxed?

At least with a quick search through the qemu repo I couldn't find any commits that would indicate that -accel=hvf would only work on Big Sur, so the theory that the entitlement is not needed on Catalina seems more plausible to me now.

[christian-korneck]

@jandubois - I think so, too. Also browsing the qemu sources I think there would be an error or at least a log message if acceleration didn't work.