Pekka Paalanen
November 19, 2020
Share this post:
Reading time:
Wayland (the protocol and architecture) is still lacking proper consideration for color management. Wayland also lacks support for high dynamic range (HDR) imagery which has been around in movie and broadcasting industry for a while now (e.g. Netflix HDR UI).
While there are well established tools and workflows for how to do color management on X11, even X11 has not gained support for HDR. There were plans for it (Alex Goins, DeepColor Visuals), but as far as I know nothing really materialized from them. Right now, the only way to watch HDR content on a HDR monitor in Linux is to use the DRM KMS API directly, in other words, not use any window system, which means not using any desktop environment. Kodi is one of the very few applications that can do this at all.
This is a story about starting the efforts to fix the situation on Wayland.
Color management for Wayland has been talked about on and off for many years by dozens of people. To me it was obvious from the start that color management architecture on Wayland must be fundamentally different from X11. I thought the display server must be part of the color management stack instead of an untrusted, unknown entity that must be bypassed and overridden by applications that fight each other for who gets to configure the display. This opinion was wildly controversial and it took a long time to get my point across, but over the years some color management experts started to open up to new ideas and other people joined in the opinion as well. Whether these new ideas are actually better than the ways of old remains to be seen, though. I think the promise of getting everything and more to work better is far too great to not try it out.
The discussions started several times over the years, but they always dried out mostly without any tangible progress. Color management is a wide, deep and difficult topic, and the required skills, knowledge, interest, and available time did not come together until fairly recently. People did write draft protocol extensions, but I would claim that it was not really until Sebastian Wick started building on top of them that things started moving forward. But one person cannot push such a huge effort alone even for the simple reason that there must be at least one reviewer before anything can be merged upstream. I was very lucky that since summer 2020 I have been able to work on Wayland color management and HDR for improving ChromeOS, letting me support Sebastian's efforts on a daily basis. Vitaly Prosyak joined the effort this year as well, researching how to combine the two seemingly different worlds of ICC and HDR, and how tone-mapping could be implemented.
I must also note the past efforts of Harish Krupo, who submitted a major Weston merge request, but unfortunately at the time reviewers in Weston upstream were not much available. Even before that, there were experiments by Ville Syrjälä. All these are now mostly superseded by the on-going work.
Currently the active people around the topic are me (Collabora), Vitaly Prosyak (AMD), and Naveen Kumar (Intel). Sebastian Wick (unaffilated) is still around as well. None of us is a color management or HDR expert by trade, so we are all learning things as we go.
The foundation for the color management protocol are ICC profile files for describing both output and content color spaces. The aim is for ICCv4, also allowing ICCv2, as these are known and supported well in general. Adding iccMAX support or anything else will be possible any time in the future.
As color management is all about color spaces and gamuts, and high dynamic range (HDR) is also very much about color spaces and gamuts plus extended luminance range, Sebastian and I decided that Wayland color management extension should cater for both from the beginning. Combining traditional color management and HDR is a fairly new thing as far as I know, and I'm not sure we have much prior art to base upon, so this is an interesting research journey as well. There is a lot of prior art on HDR and color management separately, but they tend to have fundamental differences that makes the combination not obvious.
To help us keep focused and explain to the community about what we actually intend with Wayland color management and HDR support, I wrote the section "Wayland Color Management and HDR Design Goals" in color.rst (draft). I very much recommend you to read it so that you get a picture what we (or I, at least) want to aim for.
Elle Stone explains in their article how color management should work on X11. As I wanted to avoid repeating the massive email threads that were had on the wayland-devel mailing list, I wrote the section "Color Pipeline Overview" in color.rst (draft) more or less as a response to her article, trying to explain in what ways Wayland will be different from X11. I think that understanding that section is paramount before anyone makes any comment on our efforts with the Wayland protocol extension.
HDR brings even more reasons to put color space conversions in the display server than just the idea that all applications should be color managed if not explicitly then implicitly. Most of the desktop applications (well, literally all right now) are using Standard Dynamic Range (SDR). SDR is a fuzzy concept referring to all traditional, non-HDR image content. Therefore, your desktop is usually 100% SDR. You run your fancy new HDR monitor in SDR mode, which means it looks just like any old monitor with nothing fancy. What if you want to watch a HDR video? The monitor won't display HDR in SDR mode. If you simply switch the monitor to HDR mode, you will be blinded by all the over-bright SDR applications. Switching monitor modes may also cause flicker and take a bit of time. That would be a pretty bad user experience, right?
A solution is to run your monitor in HDR mode all the time, and have the window system compositor convert all SDR application windows appropriately to the HDR luminance, so that they look normal in spite of the HDR mode. There will always be applications that will never support HDR at all, so the compositor doing the conversion is practically the only way.
For the protocol, we are currently exploring the use of relative luminance. The reason is that people look at monitors in wildly varying viewing environments, under standard office lighting for example. The environment and personal preferences affect what monitor brightness you want. Also monitors themselves can be wildly different in their capabilities. Most prior art on HDR uses absolute luminance, but absolute luminance has the problem that it assumes a specific viewing environment, usually a dark room, similar to a movie theatre. If a display server would show a movie with the absolute luminance it was mastered for, in most cases it would be far too dark to see. Whether using relative luminance at the protocol level turns out to be a good idea or not, we shall see.
The Wayland color management and HDR protocol extension proposal is known as wayland/wayland-protocols!14 (MR14). Because it is a very long running merge request (the bar for landing a new protocol into wayland-protocols is high) and there are several people working on it, we started using sub-merge-requests to modify the proposal. You can find the sub-MRs in Sebastian's fork. If you have a change to propose, that is how to do it.
Obviously using sub-MRs also splits the review discussions into multiple places, but in this case I think it is a good thing, because the discussion threads in Gitlab are already massive.
There are several big and small open questions we haven't had the time to tackle yet even among the active group; questions that I feel we should have some tentative answers before asking for wider community comments. There is also no set schedule, so don't hold your breath. This work is likely to take months still before there is a complete tentative protocol, and probably years until these features are available in your favourite Wayland desktop environments.
If you are an expert on the topics of color management or HDR displays and content, or a developer interested in contributing to the project, you are warmly welcome to join the development.
19/12/2024
In the world of deep learning optimization, two powerful tools stand out: torch.compile, PyTorch’s just-in-time (JIT) compiler, and NVIDIA’s…
08/10/2024
Having multiple developers work on pre-merge testing distributes the process and ensures that every contribution is rigorously tested before…
15/08/2024
After rigorous debugging, a new unit testing framework was added to the backend compiler for NVK. This is a walkthrough of the steps taken…
01/08/2024
We're reflecting on the steps taken as we continually seek to improve Linux kernel integration. This will include more detail about the…
27/06/2024
With each board running a mainline-first Linux software stack and tested in a CI loop with the LAVA test framework, the Farm showcased Collabora's…
26/06/2024
WirePlumber 0.5 arrived recently with many new and essential features including the Smart Filter Policy, enabling audio filters to automatically…
Whether writing a line of code or shaping a longer-term strategic software development plan, we'll help you navigate the ever-evolving world of Open Source.
한국어 버전의 Collabora.com 보기
Acesse Collabora.com em Português
Comments (17)
Erwin Burema:
Nov 20, 2020 at 05:30 AM
Hi,
This is really good news and great to hear, for personal reasons I had to drop out of this but great to hear you people have picked this up. For me this is the reason holding me back switching to Wayland. I've had hoped to be able to contribute again somewhere this year but sadly haven't been able to yet.
Regards
Erwin
Reply to this comment
Reply to this comment
A:
Nov 20, 2020 at 12:13 PM
> For the protocol, we are currently exploring the use of relative luminance.
Would that result in like, a setup that could potentially react based on laptop/phone sensors? or would it be relative to some other aspect?
Reply to this comment
Reply to this comment
Pekka Paalanen:
Nov 20, 2020 at 03:35 PM
Hi Erwin, you're always welcome back!
Hi A, I haven't thought about it, but why not, I suppose, if the sensors control the display brightness in general. The protocol documentation has a draft about "EDR", that should give a better idea about the relative luminance we are thinking of at the moment.
Reply to this comment
Reply to this comment
awenumick:
Nov 23, 2020 at 10:48 AM
ICCs shouldn’t be used for HDR as they are literally domain bound. It would require ICCMax extensions to do it properly. curv and para tags are explicitly domain bound in ICC, and scaling or other hacks do not seem prudent.
> Adding iccMAX support or anything else will be possible any time in the future.
Wouldn't implementing iccmax from the get go would make more sense, rather than it being an afterthought.
Reply to this comment
Reply to this comment
Vitaly Prosyak:
Nov 23, 2020 at 03:16 PM
Hi awenumick,
There is plugin to LMCS for HDR conversions:
https://gitlab.freedesktop.org/swick/weston/-/merge_requests/1
Reply to this comment
Reply to this comment
Pekka Paalanen:
Nov 23, 2020 at 03:31 PM
Yes, we have our doubts about whether ICCv4 is enough for or compatible with HDR, and do not yet know what other parameters are needed. However, ICCv2 and ICCv4 are the foundation of color management in the wild currently, and they have mature software available, also in forms and with licenses we can use. I'm not sure that's the case with iccMAX yet, is it?
The protocol draft also has provisions for HDR without ICC profiles.
Reply to this comment
Reply to this comment
awenumick:
Nov 23, 2020 at 07:14 PM
>The protocol draft also has provisions for HDR without ICC profiles.
Haven't read the draft; honestly I don't have much knowledge on color management. But what is your plan here?
Reply to this comment
Reply to this comment
Pekka Paalanen:
Nov 24, 2020 at 08:44 AM
Please, let's have technical discussions in Gitlab after you have read through the current draft. Thanks!
Reply to this comment
Reply to this comment
Steffen:
Nov 24, 2020 at 11:50 AM
> This work is likely to take months still before there is a complete tentative protocol, and probably years until these features are available in your favourite Wayland desktop environments.
The ICCv2 is already considered legacy for a long time and if it takes another couple of years until this feature becomes available in the Wayland desktop environments, isn't it likely to be definitely deprecated at this time then. How much resources would you save if you dropped it?
Reply to this comment
Reply to this comment
spider-mario:
Nov 25, 2020 at 02:27 PM
Nice, thanks.
> The output color space here refers to the color space and OETF used for the
final framebuffer content, or more precisely, the electrical (digital) signal
to be transmitted through the wire to a monitor.
I think “inverse EOTF” is meant here, not OETF. OETF goes from scene-referred to encoded and doesn’t involve the display at all, as described in pages 9-11 of BT.2390-8: https://www.itu.int/dms_pub/itu-r/opb/rep/R-REP-BT.2390-8-2020-PDF-E.pdf
> Absolute Luminance Content
> TBD
> […]
> How to map an absolute luminance range to monitor range?
As far as PQ is concerned, this is usually handled by the display itself, again as mentioned in BT-2390-8 (section 5.4). 5.4.1 contains an example of mapping that can be used for that purpose.
Reply to this comment
Reply to this comment
Vitaly Prosyak:
Nov 25, 2020 at 05:36 PM
Thanks for your valuable comment spider-mario!
You are correct, but i found detailed description on page 47 about display-referred mapping vs scene-referred.
>Regarding BT-2390-8 section 5.4.
I will appreciate if you can review the following implementation of calculating the EETF using ICC profiles:
https://gitlab.freedesktop.org/swick/weston/-/merge_requests/1
My understanding that it can be used for luminance compression in following cases:
HDR(PQ) to HDR(PQ) and HDR(PQ) to SDR.
For conversion between HDR as HLG the relative luminance could be used.
Please, review the following comment about all HDRSDR use cases:
https://gitlab.freedesktop.org/wayland/wayland-protocols/-/merge_requests/14#note_634310
Reply to this comment
Reply to this comment
Erwin Burema:
Nov 25, 2020 at 09:19 PM
For your second point there will be/are screens that support an option to bypass the internal tone mapping of the display (e.g. Freesync2 https://gpuopen.com/learn/using-amd-freesync-2-hdr-color-spaces/), this is useful for applications that, want accuracy (since those can go directly from internal scene referred to output instead of having to go trough (rec2020.)PQ first and then through the monitor black box), latency (since those can skip the extra tone map step of the display) or both. So in general it would be possible to leave this to the display but in certain cases there would be a need to do this internally as well (especially if the application isn't full-screen and needs to be composite with SDR content, or different HDR content that is in rec2020.pq/hlg).
Reply to this comment
Reply to this comment
Vitaly Prosyak:
Nov 25, 2020 at 10:20 PM
Awesome comment Erwin!
I read your colour management discussions on wayland-devel.
Freesync2 or Native mode.
This what we agreed to drive first and the following MR was created:
https://gitlab.freedesktop.org/swick/wayland-protocols/-/merge_requests/4
RFC Wayland compositor choose display colour mode.
Please see color.rst for details.
However based on my understanding all HDR conversions should be implemented.
Reply to this comment
Reply to this comment
Erwin Burema:
Nov 26, 2020 at 09:51 AM
Of course since not all displays will have a native/freesync2 mode, some (probably cheaper displays) will only support HDR in the format of rec2020.PQ or rec2020.HLG (or even something else in the future) so those should be supported! And though I think those screens are probably not the best for productivity (e.g. photo/video editing) or gaming (potential latency issue), those screen should still be pretty descent for the average consumer and thus should be supported.
Reply to this comment
Reply to this comment
Dan:
Nov 25, 2020 at 04:18 PM
"solution is to run your monitor in HDR mode all the time"
I'm not sure current monitors will be consistent enough in this mode, as they use trics to displsy more DR.
Reply to this comment
Reply to this comment
tdennist:
Nov 27, 2020 at 04:00 AM
in color.rst (draft) as of commit 5637d2d3, you mention Redshift as one of the the things that 'play with gamma settings, and ... change colors'.
The way I read this makes me think that you will be explicitly trying to make it such that things like Redshift can not work. In the case of photo/video workflows I fully agree that I don't want to unknowingly have something messing with gamma and colors, but Redshift exists because folks found a need for it. I recall that in some situations military folks do a 'rig for red' to preserve night vision and to reduce the ability of adversaries to see light going out of portholes, Redshift makes it easy to shift everything on the screen 'cheaply'.
Am I reading your plans wrong or are you planning some other way for whole display surface (all monitors) to be able to do something like Redshift does?
Thanks for taking on the work and trying to explain it to the rest of us.
Reply to this comment
Reply to this comment
Pekka Paalanen:
Nov 27, 2020 at 01:26 PM
Just to clarify, if you want to get in touch with the developers, please log in to https://gitlab.freedesktop.org/ (you can do it with any existing GItlab.com, Github, Google or Twitter accounts).
Then post your comments, there are several ways: as a new discussion in https://gitlab.freedesktop.org/wayland/wayland-protocols/-/merge_requests/14 or as an issue in https://gitlab.freedesktop.org/swick/wayland-protocols/-/issues or if you want to propose a change, as a merge request in https://gitlab.freedesktop.org/swick/wayland-protocols/-/merge_requests .
Yes, the new issues or MRs should indeed be posted in Sebastian's personal fork of wayland-protocols. That is the official place where we cooperate before MR14 is merged, and it's linked from MR14.
Personally I do not intend to participate in discussions in any other forums. This topic attracts a lot of attention and I can't possibly discuss it everywhere while also trying to make progress. That means also the comments on this blog post.
Thank you for understanding.
Reply to this comment
Reply to this comment
Add a Comment