GFX-HAL as a graphics backend

[skyne98]

GFX-HAL as a graphics backend

[Drakulix]

First of all, thanks for reaching out!

We definitely want integration with GFX. The development of gfx-hal and its kinda difficult low-level api surface and also much more pressing problems were delaying it. So anyone, who wants to start working on it, is highly welcome.

In theory, it should be possible to use gfx-hal for headless drawing, mapping the resulting image to the framebuffer via DRM or EGL, right?

In theory yes. In practice however headless is sometimes not really headless, but still depends on some kind of graphical system. And the much bigger issue at hand is, that this might be really slow. Because without integration with the native buffer types we would likely need to copy the rendered image out of the GPU to the CPU and back to the framebuffer of the GPU. I do not know, if gfx-hal's api does give us the possibility of directly mapping a rendered buffer from the headless instance. It does not look that way and I am not in favor of this approach anyway.

Also, maybe there is an option to extend the gfx backends themselves to allow creating surfaces/instances from, for example, EGL streams?

This is the right way forward. gfx-hal's backends seem to be very focused on certain windows systems though and this has to be lifted and become trait-based.

E.g. the gfx-backend-gl (something we could use today, because we have a gl-loader on top of out DrmDevices) hardly depends on glutin for surface creation: https://docs.rs/gfx-backend-gl/0.1.0/gfx_backend_gl/struct.Surface.html

The gfx-backend-vulkan (which we want to be able to support in the future) on the other hand is supporting a selection of backends: https://docs.rs/gfx-backend-vulkan/0.1.0/gfx_backend_vulkan/struct.Instance.html

But not the one we actually want to have, which is the VK_KHR_display extension. Although untested, vulkano has support for it: https://docs.rs/vulkano/0.11.1/vulkano/swapchain/index.html#creating-a-surface-from-a-monitor

We could do this by implementing our own backend, but that is a huge deal, completely unnecessary and would likely duplicate a lot of code.

But we might get away with implementing our own backend, re-using all types of gfx-backend-gl, but swap the surface type with our own, that might support creation from a device. That is however highly speculative and could break on new versions of gfx-hal, if it even works.

What I would actually like to see is a change of the gfx-backend-apis. Creation of those types should not depend on specific types of e.g. glutin, but use traits instead - we can then implement - and provide implementations for glutin out of the box.

The way forward would likely be:

• Monkey-patch gfx_backend_gl so it works with smithay
• Try to reduce this patch to a minimal diff
• Open an Issue/PR over at gfx. Discuss our approach and use-case and figure the rest out with them.
• Actually merge and advertise support in smithay
• Later do the same with gfx_backend_vulkan once we have vulkan support

[skyne98]

First of all, thanks for such a detailed response!

In my opinion, hacking together support for gfx_backend_gl can be very helpful to start experimenting with different already existing renderers, as well as trying to implement new ones. This will also allow to theoretically compare performance of Smithay GFX rendering vs Wayland/X11 GFX rendering.

Also, I would have loved to spend some time on it during this weekend, but I feel like I would be able to get much further with some mentoring on the matter.

[Drakulix]

but I feel like I would be able to get much further with some mentoring on the matter

Sure thing.

So for the smithay side, what you want to support is objects implementing the GLGraphicsBackend trait (most notably EglDevice).

You will most likely need all of those six methods at some point in gfx-backend-gl to get this working.

You will need to provide an alternative to the glutin.rs window implementation for smithay using those GLGraphicsBackends, which represent your surfaces, instead of the window.
You can see in the glutin.rs file, that the get_proc_address function for example is used to initialize PhysicalDevice.

swap_buffers is the second really important function and it seems to be called by Queue::present in gfx-hal. And the implementation of gfx_backend_gl seems to just call the window.swap_buffers function: https://docs.rs/crate/gfx-backend-gl/0.1.0/source/src/queue.rs (see at the bottom).
So there another feature flag for smithay is required.

I have no idea, where gfx-hal gets its dimensions from, but somewhere it will be read from the window, which is where you will need to use get_framebuffer_dimensions.

get_pixel_format is needed for compatibility reasons, you might get rendering errors, if the wrong format is used for rendering. I have also no real idea, where that happens. But since smithay is usually default to very common formats, you might get around that at first just to hack something together.

Last functions are is_current and make_current. I have no idea, how gfx handles making gl contexts current. glium checks it before rendering and smithays api kinda relies on that fact.
It is very easy to get multiple active contexts in a compositor environment (e.g. by using multiple gpus or screens) and that needs to be handled somehow. Since gfx-hal does not seem to explicitly support multiple contexts, it might just be okay to call make_current on surface creation once and leave it that way for testing. This works as long as you only initialize one surface.

[skyne98]

@Drakulix, ok, thanks for the info, I will notify you as soon as I get to it!

Also, a couple of questions for now:

1. So, the steps should be:
   1.1. Implement a backend for gfx-hal, that will allow it to run on windowless OpenGL
   1.2. Make smithay rely on gfx-hal abstraction, instead of actual GL?
2. What is the most efficient way to rapidly test the changes?

[elinorbgr]

IIUC, it would actually be the other way around : make gfx-hal use smithay as its GL provider. This would make it possible to use projects like rendy to draw the framebuffer contents in a context managed by smithay.

[skyne98]

Yeah, actually rendy and gfx-rs webrender fork are the things that inspired me to think about it as they both have great potential in creating UI systems.

[elinorbgr]

@skyne98 wrt to gfx-rs support, in a first step focusing on openGL.

The gfx-backend-gl crate requires you to provide a glutin::Window to setup the context. This is not very nice for smithay integration, as glutin is deeply tied to winit.

Ideally, gfx-backend-gl would abstract its needs from glutin behind a trait, which would let smithay provide an implementation of this trait, like for glium::backend::Backend. And then, they could have a glutin cargo feature that provides an implementation of this trait for glutin::Window, allowing us to opt-out of glutin support to avoid depending on it.

[skyne98]

@vberger but why should we worry about this? Why not provide a whole-new backend for gfx-rs which uses glium (gfx-backend-smithay-gl)? I think making a duplicate of gfx-backend-gl and rewriting it to use another OpenGL library should be pretty straightforward?

[elinorbgr]

Oh yes, writing such a backend would work, if you're willing to put the effort of maintaining a fork of gfx-backend-gl with smithay swapped for glutin.

I suggested working with gfx-rs to make gfx-backend-gl generic over gl providers because it seems to be much less effort in the long run.

[skyne98]

Have anyone talked to them in regards to this? Or should I feel free to start a discussion there?

[elinorbgr]

I think you can start the discussion. 👍