From its inception, an important use case for xrdesktop was that applications that are already running on the 2D desktop should be seamlessly available in VR at the press of a button. Although it is possible to implement something like this in a desktop agnostic fashion with pure xlib/xcb proof of concept: x3d, for performance and robustness reasons we opted for code integration into window managers. Typically, window managers have an internal abstraction of the windows they manage that is much easier to reliably consume than the raw X11 protocol.

The first two window manager integrations we started out with were a patchset for gnome-shell and a kwin plugin. The window manager integration approach unfortunately suffers several drawbacks.

Most importantly however, not everyone is running kwin or gnome-shell, and adding integration into more and more window managers would result in an unmaintainable mess.

Each window manager is shipped in a different version on different distributions too. The gnome-shell patchset needs to be adjusted for every major version change. The kwin plugin needs to be compiled for a specific kwin version, meaning an Ubuntu PPA with an updated KDE version would break a packaged version of the plugin.

While the ability to mirror already running 2D applications without good performance has been a unique feature of xrdesktop, which is enjoyed by our users and will still be maintained, we have also heard many voices who desire a more native XR solution, especially looking at standalone devices.

Jump to a section: Introducing wxrd | Demo | VR keyboard input and wlroots | Backends and renderers | "headless" wlroots | Wayland buffers and Vulkan | System requirements | Running headless on SBCs

Introducing wxrd

As an alternative to the window manager integration, we developed a standalone client based on wlroots and the wxrc compositor which had been dormant until recently. wxrd follows a similar approach as Simula VR (haskell, godot) or Stardust (stereokit), where the XR desktop is a standalone application that will not mirror existing windows from a 2D desktop, but only show windows in XR that have been explicitly started on this XR desktop. Other noteworthy pioneers are motorcar, a kwayland based VR Wayland compositor, and safespaces, an entire custom Wayland and X11 implementation. What sets xrdesktop apart from these alternatives is that xrdesktop's tech stack is focused on a small footprint. Instead of depending on large frameworks and libraries, xrdesktop with wxrd focuses on minimal dependencies and a manageable scope.

The wxrd standalone client is implemented as a Wayland compositor, but that doesn't mean you have to run your desktop on Wayland to use it. Wlroots supports various backends, for example its X11 backend allows running wlroots based compositors in a X11 desktop window.

Thanks to wlroots' xwayland implementation, not only can wxrd display Wayland applications in VR, but also X11 applications.

There are major advantages of running a standalone compositor over a 2D window manager integration:

• When run in a Wayland or X11 session, wlroots opens a window that provides an easy way to directly grab keyboard and mouse input and forward it to a specific VR window. Future work will include keyboard and mouse based UX in addition to VR controller input.
• Windows are not restricted by the limitations of a 2D desktop anymore and they can be arbitrarily large (though currently hardware limited to the maximum size of a Vulkan texture).
• Windows rendered in wxrd run at the native refresh rate of the HMD, whereas the windows mirrored with the gnome-shell and kwin integration run at the refresh rate of the original monitor they are displayed on.
• Fonts can be rendered at a pixel density adjusted for VR and not for the currently connected monitor.
• In environments where no desktop monitor but only a HMD is connected, running a 2D window manager is not desirable.

Some drawbacks:

• Only applications that are specifically launched to display on this standalone compositor are shown in VR (e.g. WAYLAND_DISPLAY=wayland-0), the windows already running on your 2D desktop won't be mirrored;
• Graphics vendor Wayland support might be limited on Vulkan implementations like the one from NVIDIA. Even though it does support gbm and wlroots in general, at the time of writing VK_EXT_image_drm_format_modifier is not supported, which is required to share buffers from Wayland with Vulkan.

Demo

Part 1 shows how xwayland seamlessly allows X11 applications to run. glxgears is running at 144 frames per second, the native refresh rate of the Valve Index used in the recording of this video.

Part 2 shows xrgears (another OpenXR application) being launched from wxrd.

Part 3 demonstrates how both a physical keyboard and the VR keyboard can be used to input text into applications. Even special unicode characters like Emojis work.

Part 4 demonstrates how the window focus for keyboard input can be changed with just the physical keyboard without the use of VR controllers. This lays the technical groundwork for UX with keyboard and mouse in addition to VR controllers.

VR keyboard input and wlroots

The Wayland core protocol specifies keyboard input via keymaps and keycodes. A keymap describes the layout of a physical keyboard, including which characters are available on that keyboard. The keycode then describes specifically for this layout which key is pressed.

Advanced text input methods are being added to Wayland with a text-input and input-method protocol. These input methods support sending unicode characters like emojis and strings directly to applications, and allow native implementations of rich predictive text input. Input methods unfortunately have the drawback that Wayland clients need to support the protocol to be able to receive text.

Furthermore, a virtual keyboard protocol is actively being developed with an interest on being able to provide input with a virtual keyboard for all applications. Unfortunately this work is not finished yet.

Therefore wxrd starts with a basic implementation of emulating keyboard input with keymaps created by the xkbcommon library and keycodes. Gamescope and Phosh's squeekboard implement this mode too.

Because keymaps aren't large enough to hold all possible unicode characters, every time the compositor wants to send text to a client, it dynamically creates a new keymap that contains exactly the characters contained in the string, sets this keymap as the active keymap, then sends the corresponding key codes.

With this technique, wxrd can process all unicode characters including emojis from xrdesktop's VR keyboard.

In the future we hope to use the advanced input method protocols for experimenting with new text input methods that can only be achieved in VR.

Backends and renderers

In wlroots the main purposes of a renderer are informing applications about available/supported formats, and when applications submit a rendered frame in either a pixel buffer in system ram or a file descriptor pointing at GPU memory, the renderer imports this memory so the compositor can present them in some composited way.

In the wlr_renderer interface header the functions a renderer has to implement can be seen. The most important ones are:

• get_drm_fd: The implementation must return a file descriptor of the render node used for this renderer. wlroots' own drm backend acquires drm master and uses /dev/dri/card0 for this. The headless/noop backend use /dev/dri/renderD128 that allows unprivileged access but forbids various ioctls that control displays etc. - custom renderers might want to use this rendernode too. If there are multiple GPUs, wxrd will need to be able to match the VkPhysicalDevice acquired from the OpenXR runtime with the rendernode that matches that device.
• get_dmabuf_texture_formats: Tells Wayland clients rendering to dmabufs what texture formats they can use. This includes drm format modifiers.
• texture_from_buffer: Wayland clients submit their rendered frames as buffers. Wayland clients can submit either an array of pixel data in system ram or a reference (dma-buf) to GPU memory.

wlroots 0.14 creates an internal gles2 based renderer with wlr_backend_autocreate(). This was changed in wlroots master (0.15) and a custom renderer can completely replace the internal default renderer.

wxrd implements a custom renderer called wxrd_renderer based on xrdesktop's gulkan library. Another project that implements a custom Vulkan renderer for wlroots is gamescope.

A native Vulkan renderer in wlroots has been contributed in wlroots master (0.15) but it is not yet ready for some of the usage required by wxrd: The wlroots Vulkan renderer will internally initialize a VkInstance and VkDevice, whereas xrdesktop internally must use the VkInstance and VkDevice that has been handed down from the OpenXR runtime. Future functionality like "Add a function to get a VkImage from a wlr_texture" from issue 3267 will help with sharing Vulkan textures between the Vulkan instances from wlroots and xrdesktop. Eventually we will be able to drop most of our own wxrd_renderer implementation in favor of wlroots' Vulkan native renderer.

"headless" wlroots

The X11 and Wayland backends provided by wlroots work fine for users who already run a 2D desktop session on X11 or Wayland. On standalone devices it is desirable to not run such a desktop session at all.

Unfortunately when we try to run wxrd without an X server or Wayland compositor "on drm", we run into the problem that in this setup we also want to run a VR compositor on drm. Both the wlroots drm backend and a VR compositor require to be drm master to run.

"drm master" is a permission that only one process at a time can acquire, which enables this process to issue ioctls that for example control the display resolution and refresh rate. Usually this process is the X.org server or a Wayland compositor. Many embedded devices that only need to display a single application don't bother running such a "heavy" graphical session and implement rendering their application directly on drm, requiring the drm master permission. Unfortunately the wlroots drm backend and the monado compositor are both such applications.

wlroots has a solution to this, the headless backend. wlroots 0.14 also provided a noop backends that compositors with custom renderers like gamescope preferred because the noop backend was more lightweight, for example it was the only backend not creating a renderer by default. Now in wlroots master (0.15) the headless backend became so similar to the noop backend, that the noop backend was dropped.

wxrd uses the headless backend when it detects being run on drm or when the environment variable WXRD_HEADLESS=1 is used on X11 or Wayland. When wxrd is started in its default mode in X11 or Wayland, it creates an empty window that is used to grab physical keyboard and mouse input. In headless mode wxrd does not create a "window" at all.

This means that currently headless mode only processes input from the VR controller and VR keyboard. In the future, wxrd can solve this by dynamically opening a window like gamescope to grab physical mouse and keyboard input on X11 and Wayland, and on drm wxrd can attach a "libinput" backend provided by wlroots. Prior art for these implementations exists in gamescope too.

Wayland buffers and Vulkan

An unexpected obstacle in the implementation of wxrd was the state of importing dma-buf based Wayland buffers into Vulkan. Importing a block of GPU memory from a dma-buf file descriptor into Vulkan is easy in theory and something xrdesktop's gulkan library was already capable of doing - as long as the layout of the GPU memory is "linear" and "not tiled".

Modern GPUs and drivers typically store texture data in hardware specific ways, including hardware specific compression and tiling, which helps with bandwidth and GPU memory usage in general. For typical users of graphics APIs this is entirely transparent but when sharing a reference to a certain block of GPU memory between two different APIs or application processes, the receiving end typically has to be told with what settings this block of memory is expected to be used. Thus drm format modifiers and an accompanying Vulkan extension VK_EXT_image_drm_format_modifier were created.

"A drm format modifier is one 64-bit number that has to be passed from the exporter to the importer" - Sounds simple, right? Not quite, as some drm format modifiers specify a storage format that uses multiple memory planes. A buffer could consist of multiple memory planes for two reasons:

1. It is "disjoint". This means it uses a format like YCBCR that is physically stored as multiple "disjoint" parts in memory. These formats aren't very common for desktop applications and are currently not importable in xrdesktop;
2. The image is stored in a simple format like RGBA in a compressed format in the first plane and a second plane stores opaque metadata about this compression.

One property of the second case is that while each plane is represented by a separate dma-buf, both of these dma-bufs will point to the same memory location and the second plane will be signified by an offset. The Vulkan specification currently does not document well how this case must be handled.

It is clear is that when first creating the VkImage that will be used to import the buffer, a VkSubresourceLayout struct must be chained to VkImageCreateInfo::pNext, containing one VkSubresourceLayout struct with metadata like size and memory offset per each memory plane.

The explicit mechanism for binding multiple memory planes to one VkImage is by calling vkBindImageMemory2 with an array of multiple VkBindImagePlaneMemoryInfo. However doing this is only allowed for disjoint formats. One suggestion I have heard was that this should be allowed if the VkImage was created with the tiling format VK_IMAGE_TILING_DRM_FORMAT_MODIFIER_EXT, though this is not specified and doesn't seem to work in practice either. It is unclear anyway why VkBindImagePlaneMemoryInfo duplicates the memory offset info already passed earlier in the VkSubresourceLayout struct. If the imported image is using different offsets than the offsets the VkImage was created with, what would happen?

The de-facto way currently used in wlroots' Vulkan renderer and gamescope is:

1. Check if the dma-bufs point to the same memory with an ioctl and error out if not;
2. Import only the first plane. Metadata like the offset of the second plane has been passed earlier as a VkSubresourceLayout and the driver will sort out access to the second plane internally.

This behavior is not well documented and feels not very natural to the API. Hopefully this API can be improved in the future.

xrdesktop's gulkan library contains an example demonstrating this behavior by allocating GPU memory for a format with a drm modifier with 2 memory planes using gbm, exporting each of the planes to an fd and importing those fds into a GulkanTexture.

System requirements

Interop between Wayland and Vulkan with proper handling of GPU memory is a relatively new topic and the respective Vulkan extensions are only implemented in recent versions of Mesa. Additionally to compile wxrd, a recent version of the Vulkan headers has to be provided by the distribution.

Required:

• VK_EXT_image_drm_format_modifier is absolutely required for wlroots. It is implemented in Mesa 21.1 and newer. For Ubuntu this means 21.04 or newer.

Optional:

• wlroots' vulkan renderer requires VK_EXT_physical_device_drm. It is implemented in Mesa 21.2, For Ubuntu this means 21.10 and newer. Additionally the Vulkan headers on Ubuntu 20.04 are too old to build wlroots master. Because the native Vulkan renderer from wlroots is not used by wxrd yet, it can be safely disabled when compiling wlroots.

For Building and running wxrd, see the readme.

Running headless on SBCs

As a proof of concept we were able to run wxrd on Monado without a 2D window manager running, directly on DRM. By setting the environment variable XRT_COMPOSITOR_FORCE_VK_DISPLAY=X where X is the number of the display to use you can tell Monado to run on DRM. Support for this and the screen idendifiers can be determined with the vkdisplayinfo tool.

Desktop PCs have become incredibly small. This Ryzen Embedded box can run radv and a fully featured modern Linux graphics stack using a stock x86_64 Arch Linux. It could be hooked up to a battery for a fully portable use case.

Since the Rasberry Pi 4 can run Vulkan through Mesa's v3dv implementation, we gave it a shot to run xrdestkop on it, using Arch Linux ARM for AArch64. Due to an issue where presenting the swapchain to the HMD display fails with the error code VK_ERROR_SURFACE_LOST_KHR we were not able to render to a HMD at the time of writing, though using a regular monitor in direct mode is possible. We also were only able to run Monado with the xrdesktop samples, since wxrd requires the VK_EXT_drm_format_modifier extension, which is currently unavailable for the RPi4.

Thanks to Collabora R&D for funding this work. Get wxrd here: https://gitlab.freedesktop.org/xrdesktop/wxrd

Join the discussion at #xrdesktop on IRC and on Discord to get involved. Follow @xrdesktop on Twitter for updates.

Related Posts

An xrdesktop summer of code

xrdesktop 0.14 with OpenXR support is here!

Moving the Linux desktop to another reality

Related Posts

An xrdesktop summer of code

xrdesktop 0.14 with OpenXR support is here!

Moving the Linux desktop to another reality