lima+wayland

[lima-vm]

lima+wayland

[bugaevc]

Hello good people of Lima 🙂

The Owl repo is now finally public, along with its dependencies like the port of Wayland itself. The announcement can be read here.

Please let me know if you succeed in building and running it.

[akihikodaki]

Hi there, I'm the author of the fork.

As you may noticed know, there is a major problem in graphics virtualization: zero-copy buffer sharing. DMA-BUF is a solution for this but specific to Linux. The problem appears all over the place.

The issue has two different parts:

1. providing a "headless" graphics device to the guest
2. integrating the guest windows to the host window system

1 is not that hard. QEMU already has egl-headless, which provides headless OpenGL for the SPICE and VNC server. It should work with ANGLE, an OpenGL ES compatibility layer which I use with my fork. However, egl-headless depends on GBM to provide zero-copy buffer sharing. The feature is optional, so you can just add #ifdefs to remove the dependency. egl-headless may require some additional code to configure ANGLE properly. As it would be a simple command line program, App Nap should not matter.

2 is the difficult part. While X was designed as a remote protocol, Wayland is designed for zero-copy buffer sharing. As such, Wayland relies on shared memory and Direct Rendering Mechanism provided by Linux kernel, which are obviously incompatible with remote communication.
A possible solution here is to create a Wayland compositor which talks a protocol for remote communication. Waypipe is a transparent proxy program specific to Wayland. WSLg used RDP with their proprietary VAIL extension to allow zero-copy buffer sharing for RAIL (remote applications integrated locally). VAIL relies on an interface of Windows and Hyper-V to share frame buffers among them.
You have to develop a Wayland compositor for macOS if you use Waypipe. If you use RDP-RAIL, you may be able to use Microsoft Remote Desktop for mac and bunch of open source software modified for WSLg. (Microsoft Remote Desktop is of course proprietary. A free alternative, FreeRDP does not support RAIL on macOS, but I think implementing RAIL on FreeRDP on macOS is much easier than implementing a full Wayland compositor.) Either way you choose, you will not have zero-copy buffer sharing. Ultimately the problem should be solved by implementing a VAIL-like protocol, which requires to hack all of the guest-side proxy, QEMU, and the host-side compositor. Today Macs have Retina displays so zero-copy buffer sharing is very important in my opinion.

[fredldotme]

Hey @akihikodaki, thanks for jumping into the discussion.

Yeah I'm aware that writing a whole Wayland compositor from scratch might be a little much, so I'm thinking of using something off-the-shelf, like Mir. I've talked to the Mir developers about a two-step PoC process:

1. Write a simple Mir shell that glReadPixel's every surface on damage and sends the pre-rendered buffer over the wire to the host compositor. It's enough for an initial PoC, though not very performant since pixels get pushed and pulled from the vGPU unnecessarily.

2. Write a Mir "platform" that receives raw, unrendered buffers (not sure when that is ready in Mir yet) and send that over the wire. It's not zero copy, but still better than approach 1.

Right now, since the RAIL/VAIL stuff seems a little complicated for my taste, I might as well implement a custom protocol for now and abstract that away as soon as other alternatives prove to become better.

[bugaevc]

Where is the repo of Owl?

On my laptop 🙂 I have not publicly released the source code yet.

Is it a library? Or an executable binary?

It is an app, so, a .app bundle. A lot like XQuartz.

[akihikodaki]

@fredldotme Apparently vsock relies on vhost, which is a Linux kernel feature. vhost-user is a userspace alternative, but it may also rely on some Linux-specific. (At least, it was not compatible enough with macOS to bring vhost-user-gpu. It provides a process isolation of graphics stack and is necessary for security and reliability.)

If vhost-user does not work, there are two alternatives: ivshmem and TCP/IP.

ivshmem stands for Inter-VM shared memory device. It can share memory and deliver interrupts among guests and the host. ivshmem-server is a server and QEMU and your application will connect to it with a library named ivshmem-client. Both of ivshmem-server and ivshmem-client are included in QEMU. It can be exposed as a PCI device to the guest but the kernel apparently does not have a driver so you will instead use uio_pci_generic. It should be trivial enough to use the device without a special driver anyway. See docs/specs/ivshmem-spec.txt to know how to use the device. Still, programming with shared memory is difficult by nature while it should provide better performance.

TCP/IP is the easiest for sure. I don't think there are much overhead when comparing with vsock. The greatest gap should exist between ivshmem and socket-based communications (including both of vsock and TCP/IP).

[fredldotme]

Thanks for the valuable info @akihikodaki! Highly appreciated!
So based on that, I'm actually thinking of implementing both.

There's something to be said of running a RPi4 connected to an iPad via USB-C and having TCP be used in that scenario, as I'm already doing regularly via SSH, as well as having a locally-optimized variant for Guest-VM-communication via shared memory, assuming it's really not doing copies somewhere.

Doing a semaphore between a guest and the host might be interesting for sure. If it turns out to be too cumbersome I will just scan out prepared buffers based on the host's vsync.

[fredldotme]

Heh, I figured "why not" and enabled IVSHMEM support (without doorbell/eventfds) on macOS:
https://github.com/fredldotme/qemu/tree/macos-ivshmem

Verified by using this tool in the guest and writing some values to the PCI device: https://github.com/Martoni/pcie_debug

Read on the host by using less on the memory-backed file it exposes.

@akihikodaki is this of interest to you? Should I open a PR and wait for a review from you?

[akihikodaki]

@fredldotme I suggest to submit the patches to the upstream. I'd appreciate if you CC me when you email them.

[bugaevc]

There has apparently been some extensive work on a macos native Wayland server. See https://mastodon.technology/@bugaevc/101603518023241841 by @bugaevc (he appears to be dev on the darling project too.)

Hi! 👋

So it sounds like you are interested in making use of Owl 🦉 (my Wayland compositor) to transparently run Linux Wayland apps?

If/when I release it, how would you expect the users to install it? Would you just tell them that they have to install Owl themselves in whatever way, or would you incorporate Owl into Lima (note: Owl is / will be GPL, so you wouldn't be able to directly link against it), or would you get Owl packaged in Homebrew and depend on it?

Wayland relies heavily on shared memory (both RAM and shared GPU memory) to pass buffer between clients and the compositor zero-copy; is that something Lima would support between Linux clients and macOS-native Owl?

[fredldotme]

@satmandu Sorry for responding so late, I wasn't notified until now about your request:

Theoretically yes, the design would be adaptable, meaning it could either go through shared memory between VM and host or some socket on the local host as would be the case for Chrome.
For macOS+QEMU I was thinking of making use of shared memory instead of a vsock, since it would be faster.
But if something like Owl is out there already I guess Lima doesn't really need it, other than to have some competition. :)

[bugaevc]

If there is any chance you might have the time or inclination to consider a release of owl soon,

Unfortunately, right now I'm very busy with fixing GNU Hurd — and that is very urgent, I have to make it in time for the upcoming Debian release.

But after that, perhaps! While I have many to-do's for it, surely I can release the source code in the not-100%-perfect state too.

some of us would be happy to beta/dogfood it.

Would anybody here also be interested in contributing to Owl?

Wayland itself presents another, larger, issue. My port of Wayland to OS X was in a very unupstreamable state, full of hacks and workarounds. Do you happen to know if someone has worked on a cleaner port since then? The Owl source code is not very useful if you can't build it.

[fredldotme]

@bugaevc Good luck with your GNU Hurd developments, I feel the pain (UBports core dev here)

Question: Is Owl expecting host-locality right now, meaning both client and server run on the same host?

[bugaevc]

By the way, while using something like RDP or VNC may indeed be nice, you better make sure your protocol of choice supports modern features like HiDPI / scaling and touchpad gestures. These are the things users rightfully expect, from both macOS and Linux desktops.

So that's one argument in favor of forwarding Wayland, which has all of this already figured out — it has to, since it's not a remote display protocol, it's the display protocol. That being said, Owl currently implements neither HiDPI nor touchpad gestures, but there's nothing stopping it from implementing features like that.

[AkihiroSuda]

Does QEMU support it? If so, yes, Lima will be able to support it.

[fredldotme]

I don't think QEMU alone is going to help here, it needs a compositor + a host-side renderer to achieve something like WSLg.
EDIT: According to the Virgl webpage it sounds like accelerated virtual GPUs are only possible with the GTK3 frontend of QEMU.

I'm not sure of the performance impact of using vsock (just read about it) but it would be interesting to have a custom Wayland compositor in the guest, where the pixel buffer of a surface is forwarded over vsock to a host-side client which displays the pre-rendered pixel buffer. I don't think DMA-BUF is possible using current means, so copying between the guest and the host seems necessary for a start.

Writing a custom guest-side Wayland compositor using something like Mir (https://mir-server.io/, there are others, but that's the one I'm most familiar with) sounds like one approach. It's supported by a company, has a pluggable platform mechanism (which could handle the pixel buffer forwarding), it's in most distro repos and would handle the Wayland client protocol.

EDIT:
I see Microsoft is using Weston + RDP on WSLg. This could also solve the problem, I'm weighting options as I'd be willing to implement something similarly using Mir. Maybe RDP is not going to cut it if one day it becomes possible to share a guest VMs GPU resources with the host system.

[fredldotme]

Is something like this of interest to the Lima developers @AkihiroSuda? I'm not sure of Lima/QEMU's limitations currently, so if there is interest I would like to use this issue to plan ahead, collecting opinions etc.

[AkihiroSuda]

I guess we can use https://github.com/knazarov/homebrew-qemu-virgl (https://gist.github.com/akihikodaki/87df4149e7ca87f18dc56807ec5a1bc5) to support this, but anyway we need to have the app nap issue to be fixed first https://gitlab.com/qemu-project/qemu/-/issues/334

[fredldotme]

Agreed. That's the QEMU fork I'm currently focusing on, yeah.
Also, the comment on how to fix App Nap was done by me. But maybe we can even get away with making an off-screen window in QEMU to allow handling OpenGL acceleration, avoiding the need to spawn an actual QEMU Cocoa window and App Nap screwing around in the first place.

What it needs though is a macOS-side app that presents the pixels pushed by the guest-side compositor to the host. I'm not familiar with GUI programming in Go, so I would probably do it in C++ using something like SDL or Qt if that's okay.

[fredldotme]

The project for my proposal is now located at: https://github.com/fredldotme/stereopticon

This repo includes sources for both the guest as well as the host side, and it's using Mir (guest compositor) and SDL (host window painter).

Both processes are able to spawn but are not exchanging data yet, that will come soon(tm).
It's now using a custom protocol (structs located in src/common/guestpixelbuffer.h) instead of RDP, good enough for most applications.
What's implemented are the ability to spawn windows, redraw them and destroy them again as the guest compositor sees fit.

I welcome contributions, but keep in mind that I haven't decided on the license yet, other than on the guest side (GPLv2 OR GPLv3)
I'm thinking the shared common code and host side might be better off with Zlib though.

[fredldotme]

I just tried to implement vsock capabilities into the compositor but it turns out that QEMU on macOS can't be built with that support enabled.
@akihikodaki do you by any chance know what's necessary to get this kickstarted?

[AkihiroSuda]

macOS seems already supporting vsock, though not supported by QEMU
https://github.com/apple/darwin-xnu/blob/xnu-7195.81.3/bsd/man/man4/vsock.4

[akihikodaki]

macOS seems already supporting vsock, though not supported by QEMU
https://github.com/apple/darwin-xnu/blob/xnu-7195.81.3/bsd/man/man4/vsock.4

The interface is for the guest, not for the host. The host implementation is up to the virtualization application, and QEMU in particular uses vhost, which is specific to Linux kernel.

[fredldotme]

I'm struggling to find ways how to word this:

My approach currently implies queue semantics. Add commands to the host-queue, read that out until empty, and proceed to do what the guest compositor wants the host to do during read.
I haven't done that sophisticated shared memory development yet, though I'm willing to try.
Having variable-size messages inside a shared memory "queue" will be hard, let alone draining it fully, without leaving traces.
Does anyone have a suggestion here on how to proceed?

EDIT: I'll go with a simple ring buffer and fake pointers (calculating offsets of messages on the fly) for a start, I'll let you know how it goes.

[fredldotme]

Arrrggh, ivshmem in QEMU requires eventfd, another Linux-only feature: https://github.com/akihikodaki/qemu/blob/master/configure#L5364

Seems the only way to properly go here is with TCP/IP.

[satmandu]

Hi all, I'm with the Chromebrew project, and we've been using google's sommelier to enable X11 on ChromeOS. Google uses this from their (slow) crostini VM in ChromeOS, whereas we're using it bare metal with the ChromeOS Exo wayland server. Google also uses that with Crostini to proxy both wayland & x11 connections (via Xwayland) to the host wayland through a virtwl device.

There has apparently been some extensive work on a macos native Wayland server. See https://mastodon.technology/@bugaevc/101603518023241841 by @bugaevc (he appears to be dev on the darling project too.)

If any of that info or work is useful....

(Also, this is off-topic, but @fredldotme Is there any chance that your https://github.com/fredldotme/stereopticon might be of use by providing an alternate wayland interface on a Linux host? We in chromebrew have to use Xwayland for some apps because the Exo Wayland server is limited and doesn't work for apps like, ironically enough, Firefox.)

[satmandu]

Hi @bugaevc ! The easiest way to have users install Owl would be to just set it as a dependency. for lima In Homebrew for instance, you would have lima having owl as a dependency, no?

[AkihiroSuda]

PR to support Owl is appreciated, but Owl should be an optional dependency

[AkihiroSuda]

Where is the repo of Owl? Is it a library? Or an executable binary?

[satmandu]

One other reason for lima+wayland (through owl or whatever) is the possibility of virtualizing ChromeOS, which would be great for debugging releases on our Chromebrew project. (I already have a script up for building docker containers for ChromeOS which we use in our project: https://gist.github.com/satmandu/6829f229d18363ddeaf52d4b43a80623 )

@bugaevc If there is any chance you might have the time or inclination to consider a release of owl soon, some of us would be happy to beta/dogfood it.

[bugaevc]

Owl is "just" a Wayland compositor, much like all the other compositors. It's not some virtualization or containerization solution z and it's not aware of them in any way.

Compositors (Owl included) expect clients to connect and talk to them over a Unix domain socket; that's enough to get basic Wayland communication going. So you'll have to somehow proxy the data though the VM boundary.

Then to actually show anything on the screen, the client has to pass some rendered content to the compositor; in Wayland this is done out-of-line in buffers. Owl supports two kinds of buffers:

• Shared memory buffers (as defined in the core Wayland protocol). For this to work, the client has to create a (perhaps anonymous) file, map it into memory, render some pixels into it, and pass the file descriptor to the compositor. This is easy to do if both the client and the compositor run on the same host, and will be non-trivial for you.
• Shared IOSurface buffers (via the zowl-iosurface protocol extension) for zero-copy hardware-accelerated OpenGL rendering. zowl-iosurface is of course Owl-specific; (most) other compositors support something analogous using low-level Linux GPU primitives (dmabufs). In addition to somehow proxying the IOSurfaces themselves, you'll also have to do something with the Mach ports that represent them, as that is how Owl expects the clients to send it the IOSurfaces.

There's the waypipe project that has already figured out many details of proxying Wayland over network, including shared buffers; you will probably want to either use it directly or use their code somehow.

[fredldotme]

Thanks for the quick response @bugaevc, I'm asking because of design decisions in the bigger picture and compatibility questions. Your design is interesting for sure, so I'd like to have something to compare to.

To not rely on hacks and risk compatibility with existing Wayland clients, no matter the implementation, with Stereopticon I decided to go the route of pre-rendering the individual frames on the guest-side (since you can't easily share a PRIME fd'ed GBM buffer across VM boundaries), transferring them over shared memory to the host and presenting them there.
In this world the guest knows best on how to deal with Wayland, it's extensions, how to pass buffers between client and compositor, etc. I just take the result of a frame draw buffer swap over shared memory and read it into a host-side presenter .app.
As long as Virgl is used in the VM it should be "fast enough".

The way I see it the design of Stereopticon is in between Waypipe and Owl.

[bugaevc]

I don't think I'm making this clear enough 🙂

Owl is not concerned with isolation and virtualization, or with forwarding anything. There's no "guest" and "host" as far as Owl is concerned, just the one machine Owl runs on, and its clients. There's no specific design in that, every Wayland compositor works that way.

Your Stereopticon, and Waypipe are concerned with forwarding, hosts and guests, etc.

To get a working Wayland forwarding from some guest to a host running macOS, you'll want to run a compositor (Owl) on the host, and some forwarding solution like Waypipe between the guest and the host. The compositor is not involved in the forwarding, and doesn't even know any forwarding takes place.

Alternatively you could do something like you're describing: let the clients do all the Wayland things entirely on the guest, and then come up with another protocol (that is not Wayland) between the host and the guest, and display the bitmaps on the host somehow. In this case, you don't need a Wayland compositor on the host at all.

[satmandu]

@bugaevc Thanks. I'm well aware that a compositor doesn't deal with forwarding or virtualization. But a wayland compositor is essential for proxying wayland apps to.

But currently, there is no meaningful wayland compositor on macos. And we'll need a working compositor to get wayland apps working at all on macos if we ever want to proxy wayland apps from a VM out to a display on macos.

[satmandu]

And @bugaevc I think you make the same arguments that Google made when they decided to use wayland as the basis for the ChromeOS display server. :)

[gautam-dev-maker]

@bugaevc I am new to this, how do I use Owl rep to actually see the GUI. Could someone pls guide me.

Thank You.