I am a happy owner of a reMarkable 2 tablet. The device is easy to use out-of-the-box. The only thing I am missing is a proper way to stream the content on my laptop to broadcast it while in visio.
Different hack exists to do so, but I wanted something easy to deploy with very few dependencies and configurations. On top of that, I am always looking for projects to code and learn new stuff. Coding a tool to fulfill my need is a perfect way to achieve both goals.
This article explains how the goMarkableStream tool works.
In this post, you will find:
- some material about the
/proc/filesystem on Linux;
- gRPC client and server generation from a protobuf definition;
- A pair of embedded certificates for mutual authentication.
Getting a picture from the tablet
The first thing to figure out is how to get a picture from the reMarkable.
The remarkable is an armv7 based device running a Linux OS. SSH access is provided, so it is pretty easy to log as
root on the tablet.
The casual way to grab a picture is by querying the framebuffer. The Linux kernel exposes a framebuffer device addressable via a device node (typically
/dev/fb0). This device aims to provide an abstraction, so the software doesn’t need to know anything about the low-level (hardware register) stuff.
My first attempt failed: querying the device
/dev/fb0 does not work on the reMarkable 2. Brilliant people made some reverse engineering and provided a good explanation on this website. In essence:
The rm2 does not use the embedded epdc (Electronic Paper Display Controller) of the imx7. Instead the e-Ink display is connected directly to the LCD controller. This means all stuff that the epdc would normally do is now done in software…
This means that the framebuffer is not exposed in
/dev/fb0 by the kernel but by software.
To get an image, we need to get the portion of RAM’s address containing the bitmap of the tablet’s image, and we know it is not referenced by the kernel.
The address of the framebuffer
To get the global framebuffer’s address in the RAM, we will query a process that knows it already. The main application of the remarkable handling the GUI is called
xochitl. It is a closed source software; therefore, there is no way to find what we are looking for by modifying the code.
Note: This is not entirely accurate. It is possible to hack the process, but this goes far beyond my skills. See the remarkable2-framebuffer for more info.
The Linux kernel traces the memory mapping per process and exposes it in the
proc/[pid]/maps pseudo-file (see man 5 procfs).
By analyzing the maps, it appears that the
xochitl process is virtually mapping the address of the framebuffer to the pseudo-device.
The global framebuffer is therefore located at
0x74044000 in the RAM. The RAM of the process
xochitl is accessible through a call to
/proc/[pid]/mem (once again see man 5 procfs).
Now, how many bytes should we extract?
The resolution of the reMarkable 2 is 1404x1872. Therefore, let’s grab 2628288 bytes:
Our first screenshot
Let’s fetch the
image.raw and convert it to a readable format with imagemagick:
Then, we can display the image that may look like this:
Building an application
Now that we are able to grab a picture, let’s build an application to grab a flow in real-time.
Overall architecture and principle
The application is working in client/server mode. The server is getting the raw pictures in an infinite loop and serving them on the network. It is then the responsibility of the client to fetch the raw pictures from the wire and to encode it into a video stream.
A trivial implementation would be to open a network connection on level 4 and use the TCP protocol as a support to the byte stream. Nevertheless, this would induce some work to set up some delimiters between each frame and handle the bad messages.
Therefore, it is a good idea to embed each picture into a message and to rely on the capabilities of a framework to do proper encoding decoding.
So far, the widest option is to use protocol buffers as it will use a decent typing mechanism while remaining compact and easy to use.
The message represents an image, and is define like this:
Treating the flow of the messages to handle a picture one by one is part of a level 7 protocol. Instead of writing our own, let’s keep on working with protobuf use gRPC. gRPC is a high-performance, open-source universal RPC framework that runs on top of HTTP/2. The network overhead is therefore low, and the communication between the client and the server remains efficient.
Our streaming service will expose a
GetImage function that will grab the picture from memory and send it on the wire:
The implementation of both the client and the server is made in Go.
protoc tool generates the skeleton of the streaming service:
Amongst some utility to handle the serialization and deserialization of the protobuf message (see the doc Image for more info), the gRPC framework exposes some
StreamServer is an interface. It is now our responsibility to
create a structure that fulfills the interface, and that is actually implementing the
GetImage mechanism (getting the image from the memory as exposed before)
Our server is a basic structure handling a couple of elements:
r field is a pointer to the
/proc/[pid]/mem file from where we will read the data.
pointerAddr is the location of the framebuffer in this file (0x74044000) in our example, and
runnable is a channel that is used to handle the requests and avoid burning the CPU of the reMarkable (TL;DR: two consecutive calls to
GetImage will have to wait to be able to consume
runnable and a goroutine is putting one event every x millisecond in the runnable queue).
Basically the implementation of the
GetImage is trivial:
The magic is simply to read the bytes, put it in an image and return it to the caller. Exposing the service is simply instanciating the objects and using the tools build by the gRPC framework:
The client simply dial the server and calls the
GetImage remote procedure in an endless loop:
Then it encodes the
response into a JPEG file and adds it to an MJPEG stream.
The creation and exposition of the MJPEG stream is not detailed in this post as it is slightly out of context. Please see the code if you want more info.
Even if HTTP/2 does not require any encryption (see here), a lot of implementation only supports the protocol if used over an encrypted connexion.
The Go implementation of gRPC requires by default an encryption channel (that can be bypassed with the use of an
Insecure method, but we all know that is not a good way to Go ;)).
It is, therefore, a good practice to implement this security mechanism that will avoid sniffing of the pictures from the wifi if you use the tool on an untrusted network.
As I do not want anything difficult to maintain, I am generating a self-signed certificate that I am embedding on both the client and the server with the new
embed command of the Go language.
I also implement a mutual authentication mechanism. Therefore, only a known client can connect to the server.
The certificate is generated per build (via a set of
go:generate commands). Therefore, if you want to enhance security, it is your responsibility to generate new binaries, and to store them in a safe place, somewhere on your computer (as they contain the certificate).
I agree that it’s not the most secure option, but it is good enough for most use cases.
Generating the certificate
The certificate is generated in pure go code:
- An internal package is in charge of the certificate sorcery (see the certificate doc).
- A simple CLI generates the file (see the code).
certpackage (see doc here exposes a single function
GetCertificateWrapper()returning a ready-to-use configuration based on the embeded certificate (
Wiring the TLS into the gRPC server is straightforward:
- For the server:
- For the client:
That’s all folks!
The tool seems to work as expected for most users. At least it is good enough for me. I do not plan to add any fancy features. Do not hesitate to give it a try if you own a tablet:
The repo also contains a
goreleaser file if you want to build you own release with your own certificates.
Here is a video of the final product: