As mentioned in a previous blog post, we have been busy working on improving the kernel integration landscape on multiple fronts, one of which includes improving the quality of the tests available for all.

Despite most of the functionality of each platform coming from drivers, which will expose interfaces to allow userspace to interact with the underlying hardware, we still didn't have an established way to detect regressions in any of the platform's drivers setup. The majority of the subsystems in the kernel share this common driver framework, so it's time we made use of that to identify issues lower in the stack more easily and generically.

A standalone test that checks for device probes does already exist, called bootrr, and we have been running it on KernelCI for a while. However, we have noticed some important issues with it. The biggest problem is that it relies on unstable ABIs from the kernel (specifically, device and driver names), which would occasionally change from one kernel version to the next, resulting in false regressions being reported by the test, and requiring it to be updated to work differently for each kernel version. The other problem is that it requires, for each new platform, a list of all of its devices to be crafted (See here for an example). This results in a lot of effort for both enabling and maintaining tests for each platform.

In order to improve the situation, we have worked with the community and landed a new kselftest upstream. The key benefit of this test is that it relies on the Devicetree, which is already maintained for each platform, to provide the list of devices present. Therefore no additional effort is needed to enable it on a new platform or to keep it compatible with changes in the kernel.

The test works by iterating over the nodes in the currently loaded Devicetree and verifying whether there's a corresponding device and driver bound to it through sysfs. Not all Devicetree nodes are expected to register a device and have it bound to a driver though, so the test is able to ignore those based on the list of Devicetree compatibles that it extracts from all drivers in the kernel source.

We have recently added this new test in KernelCI. For the MT8195-based Tomato Chromebook, for example, you can see the results for the Linux-next tree below (available here).

As can be seen, there are 181 passing tests - devices that have been checked to have successfully bound to drivers; 34 skipped tests - devices that don't have a matching driver in the kernel and therefore aren't checked; and 11 failing tests of which:

• 5 are devices that compose the display pipeline and are indeed failing to probe but will be fixed as soon as this patch lands
• 3 from the audio pipeline due to an issue with DMA memory setup in the DSP
• 1 due to an incomplete description of a PMIC in the Devicetree 
• 2 from missing configs for ramoops and coreboot support.

In short, all of them correctly indicate missing support for the platform, and as soon as they are fixed upstream, they will enable us to detect regressions reliably.

Naturally the new DT kselftest only covers devices described in the Devicetree. For ACPI-based platforms, we have a series adding a similar kselftest currently on the mailing list. And for discoverable buses such as USB and PCI, we are currently discussing the best approach on the mailing list. We'll cover them in more detail in future articles.

As we work towards making tests for the kernel that are as robust and require as little maintenance as possible, we will strive to make them first-class citizens in the kernel development process. Through this technique, we can increase confidence in every step - from patch submission to product integration.