Test on ARM macOS

[lloeki]

Test on ARM macOS

[toricls]

Sharing gist note written by @toricls (thank you!) for running Lima on ARM Mac

https://gist.github.com/toricls/d3dd0bec7d4c6ddbcf2d25f211e8cd7b
Screenshot: https://twitter.com/toricls/status/1432925285377142789

Just updated my installation guide to use Homebrew to install QEMU and Lima. It's dead simple now 😍
https://gist.github.com/toricls/d3dd0bec7d4c6ddbcf2d25f211e8cd7b

[cfstras]

Just reporting I've been working with this for the past two days with great success.

After creating a vm, I run lima sh -c 'sudo apt update && sudo apt install -y qemu-user-static binfmt-support' to also support amd64 containers.

[adamcstephens]

Upstream brew patched qemu, so it is no longer necessary to use the tap.

Homebrew/homebrew-core@5e8eb54#diff-5fe005c786b1e905434aaa353962cc6a58b0a6431e7f9a785076f0ad5f277a3d

─❯ qemu-system-aarch64 -accel help
Accelerators supported in QEMU binary:
hvf
tcg

[stardemo]

Just tested on an M1 Mac Mini using what's available in Homebrew:

% limactl start default
INFO[0000] Using the existing instance "default"
INFO[0000] Downloading "https://github.com/containerd/nerdctl/releases/download/v0.9.0/nerdctl-full-0.9.0-linux-arm64.tar.gz"
INFO[0000] Using cache "/Users/adam/Library/Caches/lima/download/by-url-sha256/08927199d45dbbfa40ed207c68567a6913b7b347c62c033a5c501220bd8958b2/data"
INFO[0000] [hostagent] Starting QEMU (hint: to watch the boot progress, see "/Users/adam/.lima/default/serial.log")
INFO[0000] SSH Local Port: 60022
INFO[0000] [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
INFO[0000] [hostagent] QEMU has exited
FATA[0000] exiting, status={Running:false Degraded:false Exiting:true Errors:[] SSHLocalPort:0} (hint: see "/Users/adam/.lima/default/ha.stderr.log")

Here's the latter file:

{"level":"info","msg":"Starting QEMU (hint: to watch the boot progress, see \"/Users/adam/.lima/default/serial.log\")","time":"2021-06-22T21:38:31+01:00"}
{"level":"debug","msg":"qCmd.Args: [/opt/homebrew/bin/qemu-system-aarch64 -cpu cortex-a72 -machine virt,accel=hvf,highmem=off -smp 4,sockets=1,cores=4,threads=1 -m 4096 -drive if=pflash,format=raw,readonly,file=/opt/homebrew/share/qemu/edk2-aarch64-code.fd -boot order=c,splash-time=0,menu=on -drive file=/Users/adam/.lima/default/diffdisk,if=virtio -cdrom /Users/adam/.lima/default/cidata.iso -net nic,model=virtio -net user,net=192.168.5.0/24,hostfwd=tcp:127.0.0.1:60022-:22 -device virtio-rng-pci -display none -vga none -device ramfb -device usb-ehci -device usb-kbd -device usb-mouse -parallel none -chardev socket,id=char-serial,path=/Users/adam/.lima/default/serial.sock,server,nowait,logfile=/Users/adam/.lima/default/serial.log -serial chardev:char-serial -chardev socket,id=char-qmp,path=/Users/adam/.lima/default/qmp.sock,server,nowait -qmp chardev:char-qmp -name lima-default -pidfile /Users/adam/.lima/default/qemu.pid]","time":"2021-06-22T21:38:31+01:00"}
{"level":"info","msg":"Waiting for the essential requirement 1 of 4: \"ssh\"","time":"2021-06-22T21:38:31+01:00"}
{"level":"debug","msg":"executing script \"ssh\"","time":"2021-06-22T21:38:31+01:00"}
{"level":"debug","msg":"stdout=\"\", stderr=\"ssh: connect to host 127.0.0.1 port 60022: Connection refused\\r\\n\", err=failed to execute script \"ssh\": stdout=\"\", stderr=\"ssh: connect to host 127.0.0.1 port 60022: Connection refused\\r\\n\": exit status 255","time":"2021-06-22T21:38:31+01:00"}
{"error":"exit status 1","level":"info","msg":"QEMU has exited","time":"2021-06-22T21:38:31+01:00"}
exit status 1

I got the same issue with my Mac on ARM.
Lima version: 0.6.1

I have installed lima use brew,but after installed i follow the readme.md ,uninstall the qemu 0.6.1

 brew uninstall  --ignore-dependencies qemu

reinstall qemu with patch

brew install simnalamburt/x/qemu-hvf

verify qemu version shows

$ qemu-system-aarch64 -accel help
Accelerators supported in QEMU binary:
tcg
hvf

after all these , lima works fine

[AkihiroSuda]

Thanks, testing is highly appreciated.

I'm planning to purchase an ARM Mac probably after they release the 2021 model of MacBookPro.

[AkihiroSuda]

Sharing gist note written by @toricls (thank you!) for running Lima on ARM Mac

https://gist.github.com/toricls/d3dd0bec7d4c6ddbcf2d25f211e8cd7b
Screenshot: https://twitter.com/toricls/status/1432925285377142789

[jandubois]

I was able to resolve this by setting the http_proxy and https_proxy environment variables to my local proxy.

There were various issues with environment variables in general, and proxy settings in particular in earlier lima releases, but I'm not aware of any remaining issues in v0.7.3 or later.

If the latest lima release does not automatically inherit proxy setting, then please file a separate Github issue with the details, so we can investigate.

@AkihiroSuda I'm tempted to close this bug, as it seems to just collect all manner of feedback related to M1 support instead of being specific to a single issue. I.e. I cannot tell what actions would be required to resolve it.

[lloeki]

@AkihiroSuda nixpkgs user so can't help here, although FYI there's a bit of work at nixpkgs to improve entitlements and signatures for qemu on aarch64-darwin, so it should be as easy as e.g nix-env -iA nixpkgs.qemu on that platform soon enough (it already works as expected on x86_64-darwin)

[AkihiroSuda]

@kmhagan

This is due to containerd/nerdctl#458 .
nerdctl run --rm (or nerdctl rm CONTAINERID) should work as a workaround.

[lloeki]

@AkihiroSuda Is it because of lack of hardware? I may be able to lend a hand and do some testing on mine.

[lloeki]

A quick test with a custom-built† QEMU 5.2.0 installed in /opt/qemu-5.2.0 shows the following:

$ curl -L -O https://github.com/AkihiroSuda/lima/releases/download/v0.2.0/lima-0.2.0-Darwin-arm64.tar.gz
$ mkdir lima
$ cd lima
$ tar xvzf ../lima-0.2.0-Darwin-arm64.tar.gz 
$ find . -type f | xargs xattr -d com.apple.quarantine # necessary when downloaded via Safari
$ export PATH="/opt/qemu-5.2.0/bin:$PWD/bin:$PATH"
$ limactl start # vim popped up: reviewed and accepted the defaults
INFO[0034] Downloading "https://github.com/containerd/nerdctl/releases/download/v0.8.3/nerdctl-full-0.8.3-linux-arm64.tar.gz" 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   645  100   645    0     0   2041      0 --:--:-- --:--:-- --:--:--  2041
100  152M  100  152M    0     0  9987k      0  0:00:15  0:00:15 --:--:-- 10.7M
INFO[0050] Downloaded "nerdctl-full-0.8.3-linux-arm64.tar.gz" 
INFO[0050] Attempting to download the image from "~/Downloads/hirsute-server-cloudimg-arm64.img" 
INFO[0050] Attempting to download the image from "https://cloud-images.ubuntu.com/hirsute/current/hirsute-server-cloudimg-arm64.img" 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  520M  100  520M    0     0  10.4M      0  0:00:49  0:00:49 --:--:-- 10.1M
INFO[0101] Downloaded image from "https://cloud-images.ubuntu.com/hirsute/current/hirsute-server-cloudimg-arm64.img" 
INFO[0101] Starting QEMU (hint: to watch the boot progress, see "/Users/lloeki/.lima/default/serial.log") 
INFO[0101] SSH: 127.0.0.1:60022                         
INFO[0101] Waiting for the essential requirement 1 of 3: "ssh" 
INFO[0120] The essential requirement 1 of 3 is satisfied 
INFO[0120] Waiting for the essential requirement 2 of 3: "sshfs binary to be installed" 
INFO[0129] The essential requirement 2 of 3 is satisfied 
INFO[0129] Waiting for the essential requirement 3 of 3: "the guest agent to be running" 
INFO[0129] The essential requirement 3 of 3 is satisfied 
INFO[0129] Mounting "/Users/lloeki"                     
INFO[0130] Mounting "/tmp/lima"                         
INFO[0131] Waiting for the optional requirement 1 of 1: "containerd binaries to be installed" 
INFO[0131] Forwarding "/run/user/501/lima-guestagent.sock" (guest) to "/Users/lloeki/.lima/default/ga.sock" (host) 
INFO[0134] The optional requirement 1 of 1 is satisfied 
INFO[0134] READY. Run `lima bash` to open the shell.    

As was suggested while waiting for ssh, I looked into the boot output in a second term:

$ tail -f /Users/lloeki/.lima/default/serial.log # looked good
cloud-init-local.service
         Starting Network Service...
[  OK  ] Started Network Service.
         Starting Wait for Network to be Configured...
----8<----
-----END SSH HOST KEY KEYS-----
[   38.032196] cloud-init[1297]: Cloud-init v. 21.1-19-gbad84ad4-0ubuntu3 finished at Thu, 10 Jun 2021 12:04:29 +0000. Datasource DataSourceNoCloud [seed=/dev/vdb][dsmode=net].  Up 38.02 seconds

Alright, everything seems up, let's try in a third term:

$ export PATH="/opt/qemu-5.2.0/bin:$PWD/bin:$PATH"
$ lima bash
lloeki@lima-default:/Users/lloeki/Source/sandbox/lima$ ls
bin  share
lloeki@lima-default:/Users/lloeki/Source/sandbox/lima$ echo foo > bar
bash: bar: Read-only file system
lloeki@lima-default:/Users/lloeki/Source/sandbox/lima$ mount | grep sshfs
:/Users/lloeki on /Users/lloeki type fuse.sshfs (ro,nosuid,nodev,relatime,user_id=501,group_id=1000)
:/tmp/lima on /tmp/lima type fuse.sshfs (rw,nosuid,nodev,relatime,user_id=501,group_id=1000)
lloeki@lima-default:/Users/lloeki/Source/sandbox/lima$ echo foo > /tmp/lima/bar
lloeki@lima-default:/Users/lloeki/Source/sandbox/lima$ exit
$ cat /tmp/lima/bar
foo
$ lima uname -a
Linux lima-default 5.11.0-18-generic #19-Ubuntu SMP Fri May 7 14:21:20 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

I'd say it works fine.

† followed this, only I added this patch as well

[lloeki]

qemu-system-x86_64 (5.2.0) is untested because it doesn't seem to operate properly (unrelated to lima, it goes down with Could not allocate dynamic translator buffer with FreeDOS 1.2) for some reason, although 6.0.0 works, but I did not test the latter with lima (I don't seem to be able to enable HVF).

[AkihiroSuda]

Thanks!

[verdurin]

Just tested on an M1 Mac Mini using what's available in Homebrew:

% limactl start default
INFO[0000] Using the existing instance "default"
INFO[0000] Downloading "https://github.com/containerd/nerdctl/releases/download/v0.9.0/nerdctl-full-0.9.0-linux-arm64.tar.gz"
INFO[0000] Using cache "/Users/adam/Library/Caches/lima/download/by-url-sha256/08927199d45dbbfa40ed207c68567a6913b7b347c62c033a5c501220bd8958b2/data"
INFO[0000] [hostagent] Starting QEMU (hint: to watch the boot progress, see "/Users/adam/.lima/default/serial.log")
INFO[0000] SSH Local Port: 60022
INFO[0000] [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
INFO[0000] [hostagent] QEMU has exited
FATA[0000] exiting, status={Running:false Degraded:false Exiting:true Errors:[] SSHLocalPort:0} (hint: see "/Users/adam/.lima/default/ha.stderr.log")

Here's the latter file:

{"level":"info","msg":"Starting QEMU (hint: to watch the boot progress, see \"/Users/adam/.lima/default/serial.log\")","time":"2021-06-22T21:38:31+01:00"}
{"level":"debug","msg":"qCmd.Args: [/opt/homebrew/bin/qemu-system-aarch64 -cpu cortex-a72 -machine virt,accel=hvf,highmem=off -smp 4,sockets=1,cores=4,threads=1 -m 4096 -drive if=pflash,format=raw,readonly,file=/opt/homebrew/share/qemu/edk2-aarch64-code.fd -boot order=c,splash-time=0,menu=on -drive file=/Users/adam/.lima/default/diffdisk,if=virtio -cdrom /Users/adam/.lima/default/cidata.iso -net nic,model=virtio -net user,net=192.168.5.0/24,hostfwd=tcp:127.0.0.1:60022-:22 -device virtio-rng-pci -display none -vga none -device ramfb -device usb-ehci -device usb-kbd -device usb-mouse -parallel none -chardev socket,id=char-serial,path=/Users/adam/.lima/default/serial.sock,server,nowait,logfile=/Users/adam/.lima/default/serial.log -serial chardev:char-serial -chardev socket,id=char-qmp,path=/Users/adam/.lima/default/qmp.sock,server,nowait -qmp chardev:char-qmp -name lima-default -pidfile /Users/adam/.lima/default/qemu.pid]","time":"2021-06-22T21:38:31+01:00"}
{"level":"info","msg":"Waiting for the essential requirement 1 of 4: \"ssh\"","time":"2021-06-22T21:38:31+01:00"}
{"level":"debug","msg":"executing script \"ssh\"","time":"2021-06-22T21:38:31+01:00"}
{"level":"debug","msg":"stdout=\"\", stderr=\"ssh: connect to host 127.0.0.1 port 60022: Connection refused\\r\\n\", err=failed to execute script \"ssh\": stdout=\"\", stderr=\"ssh: connect to host 127.0.0.1 port 60022: Connection refused\\r\\n\": exit status 255","time":"2021-06-22T21:38:31+01:00"}
{"error":"exit status 1","level":"info","msg":"QEMU has exited","time":"2021-06-22T21:38:31+01:00"}
exit status 1

[AkihiroSuda]

Thanks for testing, but for ARM Mac you have to apply a patch to QEMU https://gist.github.com/nrjdalal/e70249bb5d2e9d844cc203fd11f74c55

If it still does not work, please show serial.log .

[kiranchavala]

Followed the steps on m1 mac

installed the qemu patch from https://gist.github.com/nrjdalal/e70249bb5d2e9d844cc203fd11f74c55

when I try to start qemu

~ ❯ qemu-system-aarch64 \ 10:47:49 AM
-machine virt,accel=hvf,highmem=off
-cpu cortex-a72 -smp 4 -m 4G
-device virtio-gpu-pci
-device virtio-keyboard-pci
-drive "format=raw,file=edk2-aarch64-code.fd,if=pflash,readonly=on"
-drive "format=raw,file=ovmf_vars.fd,if=pflash"
-drive "format=qcow2,file=virtual-disk.qcow2"
-cdrom ubuntu-lts.iso
qemu-system-aarch64: invalid accelerator hvf

**~ ❯ limactl --debug start** 7s 10:49:31 AM
DEBU[0000] interpreting argument "default" as an instance name "default"
INFO[0000] Using the existing instance "default"
INFO[0000] Downloading "https://github.com/containerd/nerdctl/releases/download/v0.11.0/nerdctl-full-0.11.0-linux-arm64.tar.gz" (sha256:865d8648e2378e10dc2311f5a373fd0b85cbb6c03d2adafb1ca752d0e1267649)
DEBU[0000] file "/var/folders/h2/1n08pbk124d2488vvfssb13c0000gr/T/lima-download-nerdctl146249405/nerdctl-full-0.11.0-linux-arm64.tar.gz" is cached as "/Users/kiran_chavala/Library/Caches/lima/download/by-url-sha256/d6cadf5f656696cdf565db5101a16b6c6bc837cab7886b4fd0e48ac3a58dd6f4/data"
DEBU[0000] Comparing digest "sha256:865d8648e2378e10dc2311f5a373fd0b85cbb6c03d2adafb1ca752d0e1267649" with the cached digest file "/Users/kiran_chavala/Library/Caches/lima/download/by-url-sha256/d6cadf5f656696cdf565db5101a16b6c6bc837cab7886b4fd0e48ac3a58dd6f4/sha256.digest", not computing the actual digest of "/Users/kiran_chavala/Library/Caches/lima/download/by-url-sha256/d6cadf5f656696cdf565db5101a16b6c6bc837cab7886b4fd0e48ac3a58dd6f4/data"
DEBU[0000] res.ValidatedDigest=true
INFO[0000] Using cache "/Users/kiran_chavala/Library/Caches/lima/download/by-url-sha256/d6cadf5f656696cdf565db5101a16b6c6bc837cab7886b4fd0e48ac3a58dd6f4/data"
INFO[0001] [hostagent] Starting QEMU (hint: to watch the boot progress, see "/Users/kiran_chavala/.lima/default/serial.log")
DEBU[0001] [hostagent] qCmd.Args: [/opt/homebrew/bin/qemu-system-aarch64 -cpu cortex-a72 -machine virt,accel=hvf,highmem=off -smp 4,sockets=1,cores=4,threads=1 -m 4096 -drive if=pflash,format=raw,readonly,file=/opt/homebrew/share/qemu/edk2-aarch64-code.fd -boot order=c,splash-time=0,menu=on -drive file=/Users/kiran_chavala/.lima/default/diffdisk,if=virtio -cdrom /Users/kiran_chavala/.lima/default/cidata.iso -netdev user,id=net0,net=192.168.5.0/24,dhcpstart=192.168.5.15,hostfwd=tcp:127.0.0.1:60022-:22 -device virtio-net-pci,netdev=net0,mac=52:55:55:a7:10:91 -device virtio-rng-pci -display none -vga none -device ramfb -device usb-ehci -device usb-kbd -device usb-mouse -parallel none -chardev socket,id=char-serial,path=/Users/kiran_chavala/.lima/default/serial.sock,server,nowait,logfile=/Users/kiran_chavala/.lima/default/serial.log -serial chardev:char-serial -chardev socket,id=char-qmp,path=/Users/kiran_chavala/.lima/default/qmp.sock,server,nowait -qmp chardev:char-qmp -name lima-default -pidfile /Users/kiran_chavala/.lima/default/qemu.pid]
DEBU[0001] received an event event="{2021-08-23 10:49:38.526942 +0530 IST {false false false [] 60022}}"
INFO[0001] SSH Local Port: 60022
INFO[0001] [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
DEBU[0001] [hostagent] executing script "ssh"
DEBU[0001] [hostagent] qemu[stderr]: qemu-system-aarch64: -drive if=pflash,format=raw,readonly,file=/opt/homebrew/share/qemu/edk2-aarch64-code.fd: warning: short-form boolean option 'readonly' deprecated
DEBU[0001] [hostagent] qemu[stderr]: Please use readonly=on instead
DEBU[0001] [hostagent] stdout="", stderr="ssh: connect to host 127.0.0.1 port 60022: Connection refused\r\n", err=failed to execute script "ssh": stdout="", stderr="ssh: connect to host 127.0.0.1 port 60022: Connection refused\r\n": exit status 255
DEBU[0001] [hostagent] qemu[stderr]: qemu-system-aarch64: -chardev socket,id=char-serial,path=/Users/kiran_chavala/.lima/default/serial.sock,server,nowait,logfile=/Users/kiran_chavala/.lima/default/serial.log: warning: short-form boolean option 'server' deprecated
DEBU[0001] [hostagent] qemu[stderr]: Please use server=on instead
INFO[0001] [hostagent] QEMU has exited
DEBU[0001] [hostagent] qemu[stderr]: qemu-system-aarch64: -chardev socket,id=char-serial,path=/Users/kiran_chavala/.lima/default/serial.sock,server,nowait,logfile=/Users/kiran_chavala/.lima/default/serial.log: warning: short-form boolean option 'nowait' deprecated
DEBU[0001] received an event event="{2021-08-23 10:49:38.71089 +0530 IST {false false true [] 0}}"
FATA[0001] exiting, status={Running:false Degraded:false Exiting:true Errors:[] SSHLocalPort:0} (hint: see "/Users/kiran_chavala/.lima/default/ha.stderr.log")

---

**~ ❯ cat /Users/kiran_chavala/.lima/default/ha.stderr.log**

{"level":"info","msg":"Starting QEMU (hint: to watch the boot progress, see "/Users/kiran_chavala/.lima/default/serial.log")","time":"2021-08-23T10:49:38+05:30"}
{"level":"debug","msg":"qCmd.Args: [/opt/homebrew/bin/qemu-system-aarch64 -cpu cortex-a72 -machine virt,accel=hvf,highmem=off -smp 4,sockets=1,cores=4,threads=1 -m 4096 -drive if=pflash,format=raw,readonly,file=/opt/homebrew/share/qemu/edk2-aarch64-code.fd -boot order=c,splash-time=0,menu=on -drive file=/Users/kiran_chavala/.lima/default/diffdisk,if=virtio -cdrom /Users/kiran_chavala/.lima/default/cidata.iso -netdev user,id=net0,net=192.168.5.0/24,dhcpstart=192.168.5.15,hostfwd=tcp:127.0.0.1:60022-:22 -device virtio-net-pci,netdev=net0,mac=52:55:55:a7:10:91 -device virtio-rng-pci -display none -vga none -device ramfb -device usb-ehci -device usb-kbd -device usb-mouse -parallel none -chardev socket,id=char-serial,path=/Users/kiran_chavala/.lima/default/serial.sock,server,nowait,logfile=/Users/kiran_chavala/.lima/default/serial.log -serial chardev:char-serial -chardev socket,id=char-qmp,path=/Users/kiran_chavala/.lima/default/qmp.sock,server,nowait -qmp chardev:char-qmp -name lima-default -pidfile /Users/kiran_chavala/.lima/default/qemu.pid]","time":"2021-08-23T10:49:38+05:30"}
{"level":"info","msg":"Waiting for the essential requirement 1 of 4: "ssh"","time":"2021-08-23T10:49:38+05:30"}
{"level":"debug","msg":"executing script "ssh"","time":"2021-08-23T10:49:38+05:30"}
{"level":"debug","msg":"qemu[stderr]: qemu-system-aarch64: -drive if=pflash,format=raw,readonly,file=/opt/homebrew/share/qemu/edk2-aarch64-code.fd: warning: short-form boolean option 'readonly' deprecated","time":"2021-08-23T10:49:38+05:30"}
{"level":"debug","msg":"qemu[stderr]: Please use readonly=on instead","time":"2021-08-23T10:49:38+05:30"}
{"level":"debug","msg":"stdout="", stderr="ssh: connect to host 127.0.0.1 port 60022: Connection refused\r\n", err=failed to execute script "ssh": stdout="", stderr="ssh: connect to host 127.0.0.1 port 60022: Connection refused\r\n": exit status 255","time":"2021-08-23T10:49:38+05:30"}
{"level":"debug","msg":"qemu[stderr]: qemu-system-aarch64: -chardev socket,id=char-serial,path=/Users/kiran_chavala/.lima/default/serial.sock,server,nowait,logfile=/Users/kiran_chavala/.lima/default/serial.log: warning: short-form boolean option 'server' deprecated","time":"2021-08-23T10:49:38+05:30"}
{"level":"debug","msg":"qemu[stderr]: Please use server=on instead","time":"2021-08-23T10:49:38+05:30"}
{"error":"exit status 1","level":"info","msg":"QEMU has exited","time":"2021-08-23T10:49:38+05:30"}
{"level":"debug","msg":"qemu[stderr]: qemu-system-aarch64: -chardev socket,id=char-serial,path=/Users/kiran_chavala/.lima/default/serial.sock,server,nowait,logfile=/Users/kiran_chavala/.lima/default/serial.log: warning: short-form boolean option 'nowait' deprecated","time":"2021-08-23T10:49:38+05:30"}
{"level":"debug","msg":"qemu[stderr]: Please use wait=off instead","time":"2021-08-23T10:49:38+05:30"}

[kiranchavala]

Getting the same error even after

~ ❯ sudo codesign -s - --entitlements entitlements.xml --force /usr/local/bin/qemu-system-aarch64
Password:
/usr/local/bin/qemu-system-aarch64: replacing existing signature
~ ❯ cd /usr/local/bin
 /usr/local/bin ❯ qemu-system-aarch64 -accel help
Accelerators supported in QEMU binary:
tcg

[jandubois]

@kiranchavala I just followed the same steps from the gist manually, and it worked for me:

% qemu-system-aarch64 -accel help
Accelerators supported in QEMU binary:
hvf
tcg

This was on an (almost) freshly install M1 mini:

% brew ls
==> Formulae
autoconf	gdbm		glib		m4		ninja		pcre		pkg-config	readline	xz
automake	gettext		libffi		mpdecimal	[email protected]	pixman		[email protected]	sqlite

There was one comment in the gist discussion from a person having the same issue: they had python 3.8 installed. Once they moved to Python 3.9 the problem went away.

[lloeki]

Another easy route is to get UTM, which embeds up to date qemu-system-aarch64 and qemu-system-x86_64 that work right away. scratch that, it seems to not include the CLI binaries anymore.

[oo00spy00oo]

Just tested on an M1 Mac Mini using what's available in Homebrew:

% limactl start default
INFO[0000] Using the existing instance "default"
INFO[0000] Downloading "https://github.com/containerd/nerdctl/releases/download/v0.9.0/nerdctl-full-0.9.0-linux-arm64.tar.gz"
INFO[0000] Using cache "/Users/adam/Library/Caches/lima/download/by-url-sha256/08927199d45dbbfa40ed207c68567a6913b7b347c62c033a5c501220bd8958b2/data"
INFO[0000] [hostagent] Starting QEMU (hint: to watch the boot progress, see "/Users/adam/.lima/default/serial.log")
INFO[0000] SSH Local Port: 60022
INFO[0000] [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
INFO[0000] [hostagent] QEMU has exited
FATA[0000] exiting, status={Running:false Degraded:false Exiting:true Errors:[] SSHLocalPort:0} (hint: see "/Users/adam/.lima/default/ha.stderr.log")

Here's the latter file:

{"level":"info","msg":"Starting QEMU (hint: to watch the boot progress, see \"/Users/adam/.lima/default/serial.log\")","time":"2021-06-22T21:38:31+01:00"}
{"level":"debug","msg":"qCmd.Args: [/opt/homebrew/bin/qemu-system-aarch64 -cpu cortex-a72 -machine virt,accel=hvf,highmem=off -smp 4,sockets=1,cores=4,threads=1 -m 4096 -drive if=pflash,format=raw,readonly,file=/opt/homebrew/share/qemu/edk2-aarch64-code.fd -boot order=c,splash-time=0,menu=on -drive file=/Users/adam/.lima/default/diffdisk,if=virtio -cdrom /Users/adam/.lima/default/cidata.iso -net nic,model=virtio -net user,net=192.168.5.0/24,hostfwd=tcp:127.0.0.1:60022-:22 -device virtio-rng-pci -display none -vga none -device ramfb -device usb-ehci -device usb-kbd -device usb-mouse -parallel none -chardev socket,id=char-serial,path=/Users/adam/.lima/default/serial.sock,server,nowait,logfile=/Users/adam/.lima/default/serial.log -serial chardev:char-serial -chardev socket,id=char-qmp,path=/Users/adam/.lima/default/qmp.sock,server,nowait -qmp chardev:char-qmp -name lima-default -pidfile /Users/adam/.lima/default/qemu.pid]","time":"2021-06-22T21:38:31+01:00"}
{"level":"info","msg":"Waiting for the essential requirement 1 of 4: \"ssh\"","time":"2021-06-22T21:38:31+01:00"}
{"level":"debug","msg":"executing script \"ssh\"","time":"2021-06-22T21:38:31+01:00"}
{"level":"debug","msg":"stdout=\"\", stderr=\"ssh: connect to host 127.0.0.1 port 60022: Connection refused\\r\\n\", err=failed to execute script \"ssh\": stdout=\"\", stderr=\"ssh: connect to host 127.0.0.1 port 60022: Connection refused\\r\\n\": exit status 255","time":"2021-06-22T21:38:31+01:00"}
{"error":"exit status 1","level":"info","msg":"QEMU has exited","time":"2021-06-22T21:38:31+01:00"}
exit status 1

I got the same issue with my Mac on ARM.
Lima version: 0.6.1

[toricls]

Yay, thank you for sharing my note here @AkihiroSuda!

One issue I've faced during my Lima party was something related to bind mount (which is not on the gist note now).
I tried to run lima nerdctl run -d -v $(pwd):/usr/share/nginx/html -p 127.0.0.1:8080:80 nginx:alpine but it just got stuck after successfully pulled the container image, without any errors.

Either way I'll create an issue in the nerdctl repository with the repro.

[jandubois]

it just got stuck

I had the same thing happen to me (on Intel), and I thought it was because the host directories are mounted read-only into the VM, and then the app wanted to write to it from inside the container. Got distracted with something else; will test it again tomorrow.

[jandubois]

I thought it was because the host directories are mounted read-only

This doesn't seem to be causing the issue (although it would/should have caused the command to fail with an error).

I've now mounted the home directory as writable, and the container still hangs. It is supposed to copy a single file out of the container image to the mounted volume:

$ docker run -v /Users/jan/tmp/screen/bin:/mnt screen-static-coreos:latest cp -v /home/core/screen-4.8.0/screen /mnt

[wey-gu]

Hi all,
Thanks for the great work on lima!

May I know if anyone tried intel-on-arm on M1 Mac?
I followed the gist to install the patched(HVF) qemu 6.0 and the default lima instance worked like a charm.
While for x86_64, it's stuck. The log is saying on SSH but I guess it's just not booting(serial log is empty)?

Should I use x86_64 brew-ed qemu instead, or?

Thanks!

$ limactl start x
 x
? Creating an instance "x" Open an editor to override the configuration
INFO[0204] Downloading "https://github.com/containerd/nerdctl/releases/download/v0.11.1/nerdctl-full-0.11.1-linux-amd64.tar.gz" (sha256:ce7a6e119b03c3fb8ded3d46d929962fd17417bea1d5bbc07e0fce49494d8a09)
INFO[0204] Using cache "/Users/weyl/Library/Caches/lima/download/by-url-sha256/3304d173f1e1824e5be6cf84bf2f78825cf0db416c4c975dbb2458776942629e/data"
INFO[0205] Attempting to download the image from "~/Downloads/hirsute-server-cloudimg-amd64.img"
INFO[0205] Attempting to download the image from "https://cloud-images.ubuntu.com/hirsute/current/hirsute-server-cloudimg-amd64.img"
INFO[0205] Using cache "/Users/weyl/Library/Caches/lima/download/by-url-sha256/e1fed960ebd29619676c7ab7535bc83f7fb2ad71739edb6fde4e17bce0b61a47/data"
INFO[0205] [hostagent] Starting QEMU (hint: to watch the boot progress, see "/Users/weyl/.lima/x/serial.log")
INFO[0205] SSH Local Port: 62022
INFO[0205] [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
INFO[0293] [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
INFO[0380] [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
INFO[0467] [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
INFO[0553] [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
INFO[0640] [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
INFO[0727] [hostagent] Waiting for the essential requirement 1 of 4: "ssh"
FATA[0805] did not receive an event with the "running" status

$ tail -f *.log

==> ha.stderr.log <==
{"level":"debug","msg":"qemu[stderr]: qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.07H:EDX.spec-ctrl [bit 26]","time":"2021-09-06T16:27:19+08:00"}
{"level":"debug","msg":"qemu[stderr]: qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.01H:ECX.fma [bit 12]","time":"2021-09-06T16:27:19+08:00"}
{"level":"debug","msg":"qemu[stderr]: qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]","time":"2021-09-06T16:27:19+08:00"}
{"level":"debug","msg":"qemu[stderr]: qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.01H:ECX.x2apic [bit 21]","time":"2021-09-06T16:27:19+08:00"}
{"level":"debug","msg":"qemu[stderr]: qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.01H:ECX.tsc-deadline [bit 24]","time":"2021-09-06T16:27:19+08:00"}
{"level":"debug","msg":"qemu[stderr]: qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.01H:ECX.avx [bit 28]","time":"2021-09-06T16:27:19+08:00"}
{"level":"debug","msg":"qemu[stderr]: qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.01H:ECX.f16c [bit 29]","time":"2021-09-06T16:27:19+08:00"}
{"level":"debug","msg":"qemu[stderr]: qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.07H:EBX.avx2 [bit 5]","time":"2021-09-06T16:27:19+08:00"}
{"level":"debug","msg":"qemu[stderr]: qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.07H:EBX.invpcid [bit 10]","time":"2021-09-06T16:27:19+08:00"}
{"level":"debug","msg":"qemu[stderr]: qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.07H:EDX.spec-ctrl [bit 26]","time":"2021-09-06T16:27:19+08:00"}

==> ha.stdout.log <==
{"time":"2021-09-06T16:27:19.070538+08:00","status":{"sshLocalPort":62022}}

==> serial.log <==

==> ha.stderr.log <==
{"level":"debug","msg":"stdout=\"\", stderr=\"kex_exchange_identification: read: Connection reset by peer\\r\\n\", err=failed to execute script \"ssh\": stdout=\"\", stderr=\"kex_exchange_identification: read: Connection reset by peer\\r\\n\": exit status 255","time":"2021-09-06T16:28:36+08:00"}
{"level":"info","msg":"Waiting for the essential requirement 1 of 4: \"ssh\"","time":"2021-09-06T16:28:46+08:00"}
{"level":"debug","msg":"executing script \"ssh\"","time":"2021-09-06T16:28:46+08:00"}
{"level":"debug","msg":"stdout=\"\", stderr=\"kex_exchange_identification: read: Connection reset by peer\\r\\n\", err=failed to execute script \"ssh\": stdout=\"\", stderr=\"kex_exchange_identification: read: Connection reset by peer\\r\\n\": exit status 255","time":"2021-09-06T16:30:03+08:00"}
{"level":"info","msg":"Waiting for the essential requirement 1 of 4: \"ssh\"","time":"2021-09-06T16:30:13+08:00"}
{"level":"debug","msg":"executing script \"ssh\"","time":"2021-09-06T16:30:13+08:00"}
{"level":"debug","msg":"stdout=\"\", stderr=\"kex_exchange_identification: read: Connection reset by peer\\r\\n\", err=failed to execute script \"ssh\": stdout=\"\", stderr=\"kex_exchange_identification: read: Connection reset by peer\\r\\n\": exit status 255","time":"2021-09-06T16:31:30+08:00"}
{"level":"info","msg":"Waiting for the essential requirement 1 of 4: \"ssh\"","time":"2021-09-06T16:31:40+08:00"}
{"level":"debug","msg":"executing script \"ssh\"","time":"2021-09-06T16:31:40+08:00"}
{"level":"debug","msg":"stdout=\"\", stderr=\"kex_exchange_identification: read: Connection reset by peer\\r\\n\", err=failed to execute script \"ssh\": stdout=\"\", stderr=\"kex_exchange_identification: read: Connection reset by peer\\r\\n\": exit status 255","time":"2021-09-06T16:32:57+08:00"}
{"level":"info","msg":"Waiting for the essential requirement 1 of 4: \"ssh\"","time":"2021-09-06T16:33:07+08:00"}
{"level":"debug","msg":"executing script \"ssh\"","time":"2021-09-06T16:33:07+08:00"}


$ qemu-system-x86_64 -accel help
Accelerators supported in QEMU binary:
tcg

[jandubois]

While for x86_64, it's stuck. The log is saying on SSH but I guess it's just not booting(serial log is empty)?

I don't know; how long did you wait?

I once tried running the ARM version on an Intel Mac, and thought it didn't work, but I just didn't wait long enough; it took 30min to boot the VM.

Maybe I've done something wrong, but I think emulating the whole OS with a software-level emulator is just too slow. We can run just containers from a different architecture using qemu-user mode together with binfmt_misc. See also containerd/nerdctl#280

[wey-gu]

Thank you @jandubois!

I come to know software emulation of the VM level should be unusably slow and costly. I am waiting for nerdctl to support --platform :-D and before that, will give a try on bintfmt with ctr run --platform!

And it was 13 min waiting by the time I collected the log, maybe to wait longer will have the OS booted up.

Thank you so much! You rock!!!

[AkihiroSuda]

Updating installation steps for ARM (brew install simnalamburt/x/qemu-hvf)

#215

Can anyone confirm whether this works with Lima?

[wey-gu]

Updating installation steps for ARM (brew install simnalamburt/x/qemu-hvf)

#215

Can anyone confirm whether this works with Lima?

Verified it's working, replied in #215

[dkulchinsky]

Just reporting I've been working with this for the past two days with great success.

After creating a vm, I run lima sh -c 'sudo apt update && sudo apt install -y qemu-user-static binfmt-support' to also support amd64 containers.

Thanks for this tip! I've added this as a provision script in lima config:

provision:
  - mode: system
    script: |
      #!/bin/bash
      set -eux -o pipefail
      export DEBIAN_FRONTEND=noninteractive
      apt update
      apt-get install -y qemu-user-static binfmt-support

[afbjorklund]

I'm planning to purchase an ARM Mac probably after they release the 2021 model of MacBookPro.

@AkihiroSuda Can highly recommend Scaleway for testing on Apple Silicon (Mac Mini) in the cloud...

[tonykay]

Happy to contribute to testing on M1, I'm on Monterey beta 10/MBA M1 16GB. FWIW default image works (ubuntu) but fedora does not nor centos when I created a centos.yaml (pretty much the same as the fedora.yaml but using centos cloud image URLs and sha). It looks like the initial difficulty is Waiting for the essential requirement 1 of 4: "ssh" and also sshfs. I will open an issue with details later when I have more time to test.

[AkihiroSuda]

Thanks for testing, ARM edition of Fedora is now supported in the master #329

Will be included in v0.7.2 (probably next week)

[afbjorklund]

The fedora/centos issues were related to legacyBIOS: true, the ubuntu/debian should use UEFI by default and work better...

Since CentOS (CentOS Linux) is going end-of-life soon (2021-12-31), it probably needs replacing with "Stream" (CentOS Stream).

[febritecno]

i have tried on my mac M1. it seems the data in shared media is always lost when stopped and restarted on the vm container and I have problems with the network. vm can't connect to the internet like doing a clone repository action on github.com

[cedws]

Smooth sailing so far on M1 Pro, though that's not surprising because I wouldn't expect silicon-level breaking changes.
Cool project!

[kmhagan]

It's been working pretty well for me however I'm receiving an error when I stop a container and try to run it again on the same port:

$ lima nerdctl run -p 8080:80 nginxdemos/hello:plain-text
.. running and http://localhost:8080 returns results as expected ..
ctrl+c to quit
.. i see the container quit but if i go to http://localhost:8080 the connection hangs ..
$ lima nerdctl run -p 8080:80 nginxdemos/hello:plain-text
FATA[0000] failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr: time="2021-11-02T17:26:06Z" level=fatal msg="conflict with ID 1"
Failed to write to log, write /home/myname.linux/.local/share/nerdctl/1935db59/containers/default/b3af05e40ede34fd3e6eada2537704900fc221c17384ba8af5c7b9e842f02ff2/oci-hook.createRuntime.log: file already closed: unknown 

If I then try a third time the container starts but the port still hangs. If I stop and start with limactl it works again.

UPDATE: I see same behavior on intel macbook, so looks like it is not an ARM specific issue

[rfay]

Just a question: Why do we have to rely on qemu? It's a wonderful concoction, but not something that anybody would ever want to rely on. It often fails, and performance is awful. And both binaries and docker images can be built for both platforms.

[afbjorklund]

performance is awful

When using the accelerators, such as kvm/hvf/whpx, the cpu performance should be rather acceptable.

Running in emulation is slow, but it also opens up possibilities - like running "the other" architecture, etc etc

[rfay]

Docker Desktop on Mac M1 provides qemu. It makes things happen that could never happen otherwise. But the mysterious failures and performance issues make it something to take a pass on wherever possible. For "other platforms" it's also possible to build native images and native binaries. DDEV and many other tools have native binaries and docker images.

[jandubois]

Please move any further discussion of the merits of qemu to a discussion item; it is off-topic in respect to "Test on ARM macOS".

[rfay]

Sorry, moved the conversation about architecture-native binaries and images to #406

[DrPsychick]

I could offer some testing (currently Air M1, soon probably another M1 Pro). But I'm unsure what tests I can contribute.

I did some very simple cpubench tests here: https://levelup.gitconnected.com/running-amd64-linux-on-apple-m1-with-qemu-utm-64d67cccd6f8?source=friends_link&sk=35ca87f9537f73bf1e3ed74eb04e0abb

Today I tried running a docker-compose setup fully with lima. It did work quite nicely, but I also got some compose errors, especially when using -f multiple times and files basing on each other - likely nothing specific to arm.

[dlv-5ubbu]

Hi, trying to get working on an M1 Mac - Big Sur 11.4.

I've been stuck at this for a while now:

? Creating an instance "default" Proceed with the default configuration
INFO[0007] Attempting to download the image from "~/Downloads/hirsute-server-cloudimg-arm64.img"  digest=
INFO[0007] Attempting to download the image from "https://cloud-images.ubuntu.com/hirsute/current/hirsute-server-cloudimg-arm64.img"  digest=
535.75 MiB / 535.75 MiB [-----------------------------------] 100.00% 1.08 MiB/s
INFO[0505] Downloaded image from "https://cloud-images.ubuntu.com/hirsute/current/hirsute-server-cloudimg-arm64.img"
INFO[0506] Attempting to download the nerdctl archive from "https://github.com/containerd/nerdctl/releases/download/v0.13.0/nerdctl-full-0.13.0-linux-arm64.tar.gz"  digest="sha256:8a744638b01cd53c437a0d2a54c66f0ec4a918b75e93056393e7076bbf06a173"
INFO[0506] Using cache "/Users/viveksubramanian/Library/Caches/lima/download/by-url-sha256/b8e4d6c61f4294e7e6808bbb6c9d63e6515065b53f377046e0a42a4b4731e5cc/data"
INFO[0507] [hostagent] Starting QEMU (hint: to watch the boot progress, see "/Users/viveksubramanian/.lima/default/serial.log")
INFO[0507] SSH Local Port: 60022
INFO[0507] [hostagent] Waiting for the essential requirement 1 of 5: "ssh"
INFO[0517] [hostagent] Waiting for the essential requirement 1 of 5: "ssh"
INFO[0525] [hostagent] The essential requirement 1 of 5 is satisfied
INFO[0525] [hostagent] Waiting for the essential requirement 2 of 5: "user session is ready for ssh"
INFO[0525] [hostagent] The essential requirement 2 of 5 is satisfied
INFO[0525] [hostagent] Waiting for the essential requirement 3 of 5: "sshfs binary to be installed"
INFO[0565] [hostagent] Waiting for the essential requirement 3 of 5: "sshfs binary to be installed"
INFO[0605] [hostagent] Waiting for the essential requirement 3 of 5: "sshfs binary to be installed"
INFO[0645] [hostagent] Waiting for the essential requirement 3 of 5: "sshfs binary to be installed"
INFO[0685] [hostagent] Waiting for the essential requirement 3 of 5: "sshfs binary to be installed"
INFO[0725] [hostagent] Waiting for the essential requirement 3 of 5: "sshfs binary to be installed"
INFO[0766] [hostagent] Waiting for the essential requirement 3 of 5: "sshfs binary to be installed"
INFO[0806] [hostagent] Waiting for the essential requirement 3 of 5: "sshfs binary to be installed"
INFO[0846] [hostagent] Waiting for the essential requirement 3 of 5: "sshfs binary to be installed"
INFO[0886] [hostagent] Waiting for the essential requirement 3 of 5: "sshfs binary to be installed"
INFO[0926] [hostagent] Waiting for the essential requirement 3 of 5: "sshfs binary to be installed"
INFO[0966] [hostagent] Waiting for the essential requirement 3 of 5: "sshfs binary to be installed"
INFO[1007] [hostagent] Waiting for the essential requirement 3 of 5: "sshfs binary to be installed"
INFO[1047] [hostagent] Waiting for the essential requirement 3 of 5: "sshfs binary to be installed"
INFO[1087] [hostagent] Waiting for the essential requirement 3 of 5: "sshfs binary to be installed"
FATA[1106] did not receive an event with the "running" status

Is there something that I am doing wrong? I installed lima via brew install lima

[afbjorklund]

Is there something that I am doing wrong?

There might be more clues in that "serial.log" file

[zackwy]

Is there something that I am doing wrong? I installed lima via brew install lima

I'm not sure if this applies to your situation, but in my case this error was due to my local HTTP proxy configuration in macOS. After enabling video output to view the logs on init, I noticed that when installing sshfs the guest was trying to access the proxy on 127.0.0.1 rather than the address of my host. I was able to resolve this by setting the http_proxy and https_proxy environment variables to my local proxy. By default, Lima will detect this and replace localhost and 127.0.0.1 with the host address.

After this the default ubuntu image works flawlessly on the M1 for me.

[AkihiroSuda]

Sure, closing this issue.
If somebody is facing a problem, please open a new issue.