uwerat/vnc-eglfs

VNC server for Qt/Quick on EGLFS

VNC for Qt/Quick on EGLFS

This project implements a VNC server for Qt/Quick windows.

The basic idea of this server is to grab each frame from the GPU and to forward it to the RFB protocol.
This implementation is not affected by limitions of the software renderer and allows having native OpenGL code in application code ( custom scene graph nodes ).

As the Qt/Quick technology is for "fluid and dynamic user interfaces" the VNC server has to be able to forward full updates ( f.e. fade in/out ) with an acceptable rate to the viewer, what makes image compression more or less mandatory. Fortunately modern GPUs usually offer encoding for JPEG and H.264. Encoding on the GPU also avoids the expensive calls of glReadPixels for each frame.

Obvious limitations:

• no partial updates

  A VNC server always transfers complete frames without trying to optimize for partial updates. Despite the fact that there is no ( at least I do not know one ) unexpensive way to identify the update regions it would not help much for situations like swiping, fading in/out etc. Of course this leads to more traffic than necessary for other situations like pressing a button. Using a compression like H.264, that is using differences between frames, might be the best solution.

• windows instead of screens

  Usually VNC is used to mirror a complete screen/desktop with several windows from different applications. However the "recommended plugin for modern Embedded Linux devices" EGLFS only supports top level windows in fullscreen mode - what kind of equals windows and screens. For other platforms - like xcb or wayland - good solutions for mirroring a desktop are available.

The code might work with all Qt versions >= 5.6.

Project Status

The current status of the implementation was tested with remote connections to an application running on EGLFS and XCB.

Numbers depend on the capabilities of the devices and the size/content of the window, but on my test system ( Intel i7-8559U ) I was able to display a window with 800x800 pixels with >= 20fps remotely, when JPEG compression is enabled. The number was found by running xtigervncviewer -Log '*:stderr:100' as viewer.

These features are implemented:

• mandatory parts of the RFB V3.3 protocol ( including VNC authentication )

  This similar to what is supported by the Qt VNC plugin ( + mouse wheel, additional key codes )

• Tight/JPEG

  Using the encoder from Qt's image I/O system, usually a wrapper for: libjpeg-turbo

The following important parts are missing:

• Authentication ( > V3.3 )

• H.264

  Looks like support for H.264 has been added recently to the TigerVNC viewer.

• hardware video acceleration: VA_API

  Without compressing the frames early on the GPU the performance of the pipeline suffers from expensive glReadPixels calls and what is needed to compress the image on the CPU.
  If you are familiar with libva and want to help: let me know.

How to use

There are 2 way how to enable VNC support for an applation:

• C++ API

  The C++ API allows to configure, start and stop a VNC servers individually.

• VNC platform integration proxy

  The VNC platform proxy allows to start VNC servers for applications that can't be modified or recompiled.

Both solutions are affected by the following environment variables:

• QVNC_GL_PORT

  The first unused port >= $QVNC_GL_PORT will be used when starting a server

• QVNC_GLTIMER_INTERVAL

  each server is periodically checking if a new frame is available and the viewer is ready to accept it. Increasing the interval might decrease the number of updates being displayed in the viewer.

• QVNC_GL_PASSWORD

  A string of max. 8 characters. Setting a non empty password enables VNC authentication.

Application code

The most simple way to enable VNC support is to add the following line somewhere:

#include <VncNamespace.h>

VNC::setup();

If you want to get rid of the local windows you have several options:

• using the gbm platform plugin
• using the undocumented "offscreen" platform, that comes with Qt ( X11 only )
• reconfiguring a headless mode ( EGLFS only )

VNC platform integration proxy

If you do not want ( or can't ) touch application code you can load the VNC platform plugin proxy by using one of these: keys. The proxy simply does the initialization above before loading the real plugin following the "vnc" prefix.

Assuming library and plugin are installed in "/usr/local/vnceglfs":

# export QT_DEBUG_PLUGINS=1
export QT_QPA_PLATFORM_PLUGIN_PATH="/usr/local/vnceglfs/plugins/platforms"
export LD_LIBRARY_PATH="/usr/local/vnceglfs/lib"

export QT_QPA_PLATFORM=vnceglfs # vncxcb, vncwayland, vncoffscreen, vncgbm

vnc-eglfs's Issues

Image composition should happen on dedicated threads

At the moment the framebuffer is grabbed on the scene graph thread, but converting it into the format being requested by the client happens in the GUI thread.

Probably the best way to solve this is to move each instance of VncClient in its own thread. This also allows to use blocking reads, when waiting for client messages.

Text input

There are problems with special keys like "Shift", that end up as an extra character in the text input field

Open H.264 Encoding

Considering that the image composition happens on the GPU and many modern GPUs offer H.264 encoding this sounds like an interesting approach to solve low bandwidth problems, when always sending fullscreen updates.

Not sure if there is a VNC viewer on te market that supports H.264 and we might need to implement our own client - at least for testing.

Avoid extra flipping of the framebuffer

Instead of flipping the grabbed framebuffer we could invert the order of the lines when iterating in inverted order when sending the lines to the wire. ( minor performance improvement )

Authentication/Security

We need to have some sort of support for the Authentication/Security being defined in the RFB specification.
makes sense to support V3.3, V3.7 and V3.8 then as the RFB protocol differs for this functionality.

Cursor with alpha

Support for the pseudo encoding "Cursor with Alpha" ( -314 ) should be low hanging fruits

Cleaning up the dirty region on window resizes

When resizing a window we need to remove all rectangles from the dirtyRegion.

cmake

Using cmake instead of qmake

Platform plugin

We could offer an optional platform plugin, that creates a headless environment ( using gbm ) and starts VNC support.

We would have the same limitations as with the Qt VNC platform plugin ( local/remote access are mutually exclusive ), but we would have VNC support for applications without any adjustments to the source code.

Maybe we even find another type of plugin that is loaded by every Qt/Quick applicaton, that could be used.

libjpeg-turbo

Let's see if libjpeg-turbo ( https://libjpeg-turbo.org ) is that faster

Video Acceleration (VA) API

Let's see how to make better use of the GPU: https://intel.github.io/libva/index.html

• JPEG encoding
• H.264 encoding
• ???

Better Cursor support

How to deal with cursors being updated by shaders ?

zlib encoding

zlib encoding should be an easy to have compression

Support non true color clients

Currently clients, that do not have true color support are rejected. Not sure how important such feature is in 2022 - but anyway.

Linker does not find the vncgl library

/usr/bin/ld: cannot find -lvncgl

Minimum required Qt 5 version is probably 5.13

Got this error:

../../git/src/VncServer.cpp:235:17: error: 'class QImage' has no member named 'convertTo'

According to the doc, this method exists starting from Qt 5.13. I don't know if there are other methods which require even superior releases, but this for sure requires something newer than 5.12.