Simplifying AGL's existing Wayland-based graphical stack and avoiding the use of modules that aren't maintained upstream has lead to the creation of a new compositor based on libweston, bringing more reliable and fine-grained system control.

Automotive Grade Linux (AGL) is a collaborative project of the Linux Foundation about improving open source components for automotive use cases. Using a common reference platform for test and integration purposes, it follows an upstream-first policy in publishing the improvements.

On the graphical side of things, the AGL project is using Wayland, a protocol for a compositor to talk to its clients as well as a C library implementation of that protocol. With it, it delivers a rich user experience (usability and user satisfaction) and supports customization of design and behaviour as to meet different needs of vehicle type.

Initial state of affairs

When the project began, weston was chosen as the compositor, which is the reference implementation of a Wayland compositor, while for window management functionality it relied on ivi-shell (In-Vehicle Infotainment) together with an extension, called wayland-ivi-exension. A demo platform image of AGL comes with a handful of demo applications, done with the Qt, which abstracts the protocol communication between the client and the compositor. Additional functionality was in place under the form of library, to control and signal back to the compositor when applications were started, among other things.

Management of applications, starting, running and stopping them is done in AGL with AFM Application Framework Management, which is an umbrella name to denote the suite of tools and daemons that handle all of that. It is integrated with systemd and with the current security model, SMACK (Simplified Mandatory Access Control Kernel), a Linux kernel security module. Applications can use AFM to hang off data, and to pass it down to other services. Together with AFM, and with the help of a library, applications could tell the compositor which application to activate or to switch to.

Simplifying the graphical stack

Trimming down these abstractions, simplifying the way clients interact with the compositor, and avoid using modules that aren't really maintained upstream were the reasons behind looking at alternatives to ivi-shell. On the desktop, xdg-shell is currently de-facto protocol for handling all window management related functionality.

Wayland protocol has a window-like interface embedded into its protocol (called wl_shell), but xdg-shell has long time ago deprecated it and instead of adding it in the Wayland protocol namespace, it was integrated together with some other useful protocols into wayland-protocols project. The whole purpose of wayland-protocols is to enhance the Wayland protocol with new functionality and bring new extensions entirely. Compositors are free to implement, modify, enhance, and add new extensions to wayland-protocols but they need to do so in consensus.

Besides the core Wayland protocol and extended functionality from wayland-protocols, a compositor can provide and implement additional protocol extensions (custom to that compositor). By using such private extensions we align with the AGL project and its requirements, without compromising specific functionality and allows to add or improve the current ones.

With that in mind, the approach was to create a new compositor, called agl-compositor and implement dedicated private extensions, rather than trying to modify weston itself, which the AGL project would have been required to keep and maintain for itself, as a fork.

A compositor based on libweston

The compositor used currently in AGL, just like weston, is built on top of libweston and libweston-desktop. The latter, among other things, is required as it provides the server side implementation of the xdg-shell protocol which underlying toolkits (like Qt/Chromium project) makes use of to deliver desktop-like functionality. The former is used to provide back-ends and rendering support, effectively managing the HW, besides implementing the Wayland protocol.

The high-level goal of libweston is to decouple the compositor from the shell implementation.

A further rationale for creating a custom compositor was to serve as a base for customisation, and use it to gather and to bring in useful features back into libweston. And with that, to expose the potential of developing a framework which could be easily customizable, keeping the internals sufficiently hidden but powerful enough to achieve custom functionality if one would so desire.

In (traditional) desktop environments, the compositor and shell UI are one and the same. This is not necessarily true within AGL, where the compositor is separated from the shell's user-interface components, allowing shell UI to be started independently by some other mechanisms, and, in the same time, allows the possibility of being de-coupled from other components that can perform specific or general I/O tasks.

Normally, when the compositor and shell UI live in different processes, such as in Weston's reference desktop-shell implementation, the compositor starts the UI client, however with AGL's current application-framework implementation, the process of starting the shell UI is handled by the application-framework directly. Aside from that, one can also modify the configuration file, add the shell UI path, and the compositor will attempt to start it.

Private extensions

Compositors can define and implement custom extensions to further control application behaviour. For AGL, we have two private extensions defined. One targeted at defining surface roles commonly found in desktop environments (like panels, and backgrounds), which a shell client would bind to, and one targeted at regular application(s) that might require additional functionality: being able to display/activate its own surface or other's application surface, implement some kind of split screen management of windows, or dialog/pop-ups that exhibit always-on-top property even if the active surface has been changed.

Clients can make use of these private extensions to define other kind of roles for instance dialog/pop-ups or full-screen roles, and split windows vertically or horizontally. It includes the ability to activate other applications, assuming that the surfaces have been created, and the capability of delaying presentation for the client shell. Doing so, all the information is displayed at once, rather than waiting for the toolkit to map/show the surface.

An application identification mechanism was required to be able to activate other clients windows/surfaces. A string-based identifier name was chosen which can be used by the client to set an application-based identifier using the xdg-shell protocol. This identification process is needed and works in tandem with the application-framework to provide a life-cycle mechanism for managing applications. Specifically, some applications want to be able to display, or hide, programmatically, their own window (Alexa application), or other's application window (launcher) and they can't explicitly access what it is implicitly managed by the compositor, so this provides the means to do just that.

Available toolkits, application conversions and available eco-systems

Users and OEM vendors alike have the possibility, depending on their use-cases, to either use some kind of a toolkit (Qt/GTK) for writing application, or use plain C and the Wayland library to communicate with the compositor directly, without any toolkit abstraction.

Currently, the demo applications in AGL use the Qt platform with Chromium being at this phase, a second-class citizen, being currently in the works of achieving the same level of integration as QtWayland (the underlying library that abstracts the Wayland interaction) has at this moment. The Qt platform has long been favoured in embedded systems so it feels natural why AGL project chose it over other alternatives. In the same time, as web applications are for quite some time now permeating the application development scene, it also felt natural to add support for a runtime that gives that option, which in AGL was achieved with the help of the Chromium project.

For normal applications, not needing the ability to activate or displaying other's application surface, would basically mean that it would use what the toolkit has to offer, simplifying the application handling even more. Under Qt, the client shell can use QPA (Qt Platform Abstraction) to gain access to Wayland primitives, and implicitly is being able use the private extensions.

On the Chromium side of things, that happens indirectly, as Chromium doesn't expose the Wayland primitives. Not only that, but on the Chromium platform, there's another mid-layer component, called WAM (WebApplicationManager) with the purpose of handling web applications life-cycle.

So, controlling and passing information from a web application, that resembles that of a shell client, has to travel more than a few levels in the software stack, until it reaches the lower layers in Chromium where the Wayland communication and interaction takes place. WAM and Chromium development is largely handled by Igalia, with Collabora adding support for the private extension for the Ozone interface abstraction, which Chromium projects uses now to handle the display/graphical interaction with the lower stack levels.

Streaming buffers and receiving events to and from remote outputs

Quite a common feature, in the infotainment market, is the ability to stream out buffers/data to remote outputs. For instance, super-imposing the navigation application, between the speedometer and tachometer, in the IC (Instrument Cluster) of a car is such a scenario. Just like weston, the AGL compositor is capable of loading up libweston modules and make use of them. And just like weston, the AGL compositor loads up the remoting-plugin to achieve the same thing.

Further more, to cope with situations where the output is just a panel/display, without some kind of compositor driving it, the necessity of handling input events is an important feature to have, giving the user to possibility to manipulate the application/environment as he or she seems fit. The compositor loads a plug-in that streams out the buffers to an output remotely, with another plug-in handling the input events. The events, which are sent back from the display to the compositor, are generated with the help of Wayland-eque protocol that works over the network, called Waltham.

Together, they provide proper means to achieve a seamless integration with other display devices in the car cabin.

Policies and Role Base Arbitration

The compositor contains an API useful for implementing user-defined policies. It contains a policy engine, and installs by default an allow-all kind of policy. The policy engine controls if the client using the private extensions is permitted to perform those calls. Not only that, but with some policies, it won't allow the client to bind to the interfaces in the first place. That happens with the deny-all policy, which is able to retrieve the client's SMACK label and compares it with the ones statically defined. In the works, there's a new policy model, called Role Based Arbitration.

While the other two policies are embedded into the compositor, the RBA policy model is an off the-shell policy. Obviously, vendors and users can hook up their own policies, just like RBA did. These all work towards satisfying the driver distraction mitigation requirement for the AGL project, as to avoid overwhelming the driver with too much information.

Reactive rules

The policy engine is stateful, and allows the ability to inject back events, such that it allows the user to add custom rules into a policy and, depending on the event received by the policy engine, to execute a rule match for that event. Further more, the framework allows adding new states and events and the default implementation has code for handling events like showing or hiding the application specified in the policy rule. The most common example to exemplify this feature is the ability to show a custom application, like displaying the rear view camera application, when the automobile has been put in reverse.

Further enhancements, other users and uses

Expanding and improving the compositor

The agl-compositor only adds basic functionality, and depending on other use-cases, one might want to further enhance or tailor the compositor to fit their needs. It contains the bare minimum, the necessary bits to bring up the screen and makes use of private extensions to better integrate with the existing AGL display/graphics infrastructure. Being so lightweight, it encourage users to contribute and to expand it further.

A future avenue to pursue is to simplify the way in which users can contribute and, most likely, customize the compositor to better fit their use-cases. Developers not familiar with libweston/libweston-desktop might find it easier to use an abstraction layer, with the potential to shorten the time to market.

Make use of weston's testing framework

Another potential improvement would be to re-use the existing upstream testing framework from weston, and use it to test out the compositor internals, and to test out private extensions in various scenarios.

As the testing framework provides access to various compositor internals making it useful to add additional debugging mechanism, which were previous available to ivi-shell users from a command line, to control and manipulate, for instance, the Z-order of surfaces or activate interactively applications' surfaces as to name a few examples.

Other toolkits and the Apertis project

With recent updates to the compositor, support for GTK/GDK toolkit has been added, besides the Qt and Chromium platform. While doing that, we also gained support for the Apertis platform for running the agl-compositor as its Wayland compositor. Apertis is a infrastructure tailored to the automotive needs and fit for a wide variety of electronic devices, but more information about compositor changes and about Apertis project will be handled in a future blog post.

Related Posts

WirePlumber, the PipeWire session manager

Testing Weston DRM/KMS backends with virtme and VKMS

Developing Wayland Color Management and High Dynamic Range

Related Posts

WirePlumber, the PipeWire session manager

Testing Weston DRM/KMS backends with virtme and VKMS

Developing Wayland Color Management and High Dynamic Range