merovius/srvfb

Stream a framebuffer device over HTTP

srvfb - Stream framebuffer content over HTTP

This repository contains a small webserver that can serve the contents of a linux framebuffer device as video over HTTP. The video is encoded as a series of PNGs, which are served in a multipart/x-mixed-replace stream. The primary use case is to stream a reMarkable screen to a computer and share it from there via video-conferencing or capturing it. For that reason, there is also a proxy-mode, which streams the frames as raw, uncompressed data from the remarkable and can then do the png-encoding on a more powerful machine. Whithout that, the framerate is one or two frames per second, which might not be acceptable (it might be, though).

This should be considered a tech demo in the current state. The code is not particularly clean, it's not in any way secured, probably not very efficient and it's taylored specifically to the reMarkable (e.g. it can only stream 16-bit grayscale images). Feel free to use it and report any bugs you find, but I don't make any promises in regards to support or stability and any issues not directly related to my usecase will likely be closed.

You can see a short video demonstrating this in this tweet

Note: This project is not compatible with the reMarkable 2 and I don't intend to change that. In general, you should consider this project in maintenance mode at best - I don't regularly use it anymore, so I'm not really incentivized to improve it.

Installation and usage

You need a working Go installation and ssh-access to your reMarkable. You can then obtain, install and run the code via

go get -d -u github.com/Merovius/srvfb
GOARCH=arm GOARM=7 GOOS=linux go build github.com/Merovius/srvfb
scp srvfb [email protected]:
ssh [email protected] ./srvfb -device /dev/fb0 -listen :1234

If you then open http://10.11.99.1:1234/ in your browser (only Chrome is tested) you should see the stream from your reMarkable. To use proxy-mode, run (in a separate terminal)

go build github.com/Merovius/srvfb
./srvfb -listen localhost:1234 -proxy 10.11.99.1:1234

and open http://localhost:1234/ in your browser.

Once you can see the reMarkable screen in your browser (via proxy or not), clicking on the image should rotate it by 90°.

This repository also contains systemd unit files to run srvfb automatically (using socket activation). For security reasons, it only listens on the USB network, though. To use it, run

cd $(go env GOPATH)/src/github.com/Merovius/srvfb
GOARCH=arm GOOS=linux go build github.com/Merovius/srvfb
scp srvfb [email protected]:/usr/bin
scp contrib/srvfb.service contrib/srvfb.socket [email protected]:/etc/systemd/system
ssh [email protected] systemctl enable --now srvfb.socket

License

Apart where otherwise noted, this code is published under the Apache License, Version 2.0:

Copyright 2018 Axel Wagner

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

srvfb's Issues

adapting to reMarkable 2

reMarkable 2 uses 32bit graphics ... could this be implemented? I'm happy to help with testing.
I guess having a command line option (-16 / -32) could be a good way to switch.
And thanks for this very useful piece of software!

Make framebuffer-handling more robust

We currently assume 16-bit grayscale and read files from /sys to discover resolution and the like. Not only is that not very robust against changes or portable to other devices - it also currently provides wrong values (the virtual height of the framebuffer seems to be not fully used).

There are ioctls we can use to get framebuffer-details, maybe they provide better values. They also provide us with actual color-depth.

Unable to execute

I build this on a Windows 10 using the setting you gave.
Managed to scp to the root folder but when I tried to run it, it gives me a "Permission denied".
Anything I did wrong?

Increasing framerate

Here's an idea.
In theory, using javascript and a canvas element one could render the raw data stream in the browser.
This would make the proxy mode redundant and give flexibility: for example one could try to send only updated parts with coordinates and the js would only redraw the interested region of the canvas.
Device events could be used to identify the affected region (i.e. the one around the pen events) and for good measure a whole frame can be sent at regular intervals.

I have no idea if this attacks the real bottlenecks.

Google cast support

I don't know if this is feasible, but we might consider streaming directly via Google cast. The advantage is that we save a round-trip in latency when screen-sharing - the device can directly stream into a hangout or to a cast-capable device, without bouncing via the browser.

However, this would probably require doing actual video-encoding, it's not clear the reMarkable can handle that in software.

Autoupdates

We should optionally support auto-updates. This isn't entirely trivial and needs some design work, to make it work reliably and securely.

/cc @warpfork because he'd never speak to me again if I screw this up ^^

Nice idea, and I believe remarkable should update their own tools internally as well

I have tested it, but there is a big delay. Like 2-3 seconds. Which is very annoying.
I understand you have created this in your free time for your own use case. But since you are using a remarkable how do you experience what I am experiencing. Is it fast for you?

Feedback

Thanks a lot for srvfb. It seems to work great so far.
I noticed the following points:

1. It is right that one has to run srvfb on the remarkable and on the machine to create the proxy? The readme is a bit vague here, as it points out "run in another terminal"
   If I do not start srvfb on the rM, I get a connection refused message. Maybe something more specific like "srvfb on the rM is not running" would enhance the UX.

2. Running in proxy mode (by starting srvfb on the rM and on the host), I noticed that the very last change will not be updated (Firefox, Arch Linux).

3. I read about the UI idea and was missing a feature to temporary stop the transmission. That would allow me to prepare something e.g. a small sketch and send it on screen when it is ready and in one go, useful e.g. in a class settings.

4. I can report that the internal screenshot function of firefox works well.

Thanks again, I really like this project.

RGB streaming

First of all, thanks for the efforts. Very interesting for my usecase (using the remarkable instead of the blackboard for teaching). I would like to have an RGB streaming mode which is needed when annotating pdf slides. I would implement it myself but I am not proficient in go.

Marker pen doesn't render in correct colors

Love the poc!!!! i would love to use it for training. any plan to increase fps in the near future?

no file descriptor passed

When I am executing the binary it says:

./srvfb -device /dev/fb0                                                                               
2018/12/03 17:04:25 no file descriptor passed by service manager and no -listen set

Some documentation on the PNG implementation

Hi @Merovius. Really great tool you built here ❤️. I've been trying to understand the PNG implementation you wrote there. My hot take was your implementation of PNG. It's really cool that it works but am lost on how it does.

Can you add some comments on functions or add some links to how understand it.

General feedback

Hi,

first of all, very nice little tool! I've been playing around with it and it seems to do one thing and do it reasonably well. I still have a few remarks:

• It works very well in firefox, feel free to update the readme accordingly :)
• there seems to be some kind of go-routine leak. When I first start srvfb, the CPU utilization is 0. This jumps to around 50% once an http client connects. When the screen isn't updating, the utilization drops down to around 20% and stays there, even if the client closes the connection. Opening a new connection has the CPU utilization spike to 60% and then drop down to between 30 and 40% after the client closes etc. Only after a certain timeout, the cpu utilization drops back down to 0. (After the log reports writes to broken pipes.)
• some frames are still sent redundant. Here is how I know: In firefox, when accessing /video directly, I can watch individual frames being rendered. The last frame following some screen change is always rendered and transmitted twice.

Authentication

We should have some way to do HTTP-authentication. This would enable us to listen on the wifi-interface too, meaning we wouldn't have to be connected via USB. The main question is how to configure credentials.

• TLS is hard to do, because we don't have a publicly available domain name. So the user would have to provide certificates for us (bad UX) or we'd have to rely on self-signed certs (insecure).
• Basic Auth requires the user to set a password - again, bad UX. We could authenticate against /etc/shadow, but without TLS that's a bad idea too.
• I think a somewhat user-friendly way would be a trust-on-first-use U2F auth - the user connects to the device, enrolls a security key and can then - using that key - authenticate as privileged. But not everyone has U2F keys, sadly.
• We could make this configurable via the web-UI itself. The user connects over USB, sets up an authentication mechanism (e.g. a basic password) and enables listening over wifi. However, the usb-only restriction is currently implemented in the service-file, changing that programmatically is icky…

Needs some thinking.

Don't send images if they didn't update

We can save a bit of bandwidth and CPU by only encoding/sending PNG's when the screen content actually changes.

Shut down server when idle

With socket activation, we should be able to completely shut down the server when we aren't serving anything (maybe after a ~1m timeout or so). That should help a little bit preserving battery life

Nicer UI

The Web-UI is extremely minimal (or… non-existent). We should add some swagger, have hoverable buttons for rotating and maybe animate the rotation and somesuch.

Missing license

Hey, could you add a license file so other developers can know what they can, and cannot do with your code?

the other way?

Would it be possible to use the remarkable as a second screen for a computer?

Add the `remarkable-tablet` topic to the repo

In an effort to make projects for the reMarkable tablet easier to find on GitHub, a common topic named remarkable-tablet has been established (See reHackable/awesome-reMarkable#1).

It would be awesome if you could tag this project with the remarkable-tablet topic.

There's more info on Classifying your repository with topics available in the GitHub help.

A list of already tagged projects can be found here: Projects in Topic remarkable-tablet.

(copied from rien/reStream#20!)

Download PNG

Because we use x-mixed-replace, it's not possible to save the streamed images (chrome ends up in an endless download). We should provide a button to download a screenshot.