Times are changing: LLVM has become more than a spare to GCC, such that glibc - the last big GCC bastion, is now working towards supporting LLVM as a first-class citizen.

This post is a sequel to last year's A tale of two toolchains and glibc. Unfamiliar readers are recommended to read the previous post for a better understanding of the progress being discussed here. It has been more than a year, so it's interesting to see what has changed, but first let's take a step back and ask why.

Why work on this?

Common wisdom argues against putting all eggs in one basket, and this type of diversification is also important in software. Just like how programming languages ideally have multiple independent compiler implementations, a software project written in a specific language should ideally have multiple independent toolchains capable of building the project.

Considering how many C language implementations and toolchains have been developed in the past 45+ years (yes, it's been that long!), it is surprising that such a critical, central, and long-lived component of our modern systems, such as glibc, is still only buildable with a single toolchain, the venerable old GNU/GCC, despite LLVM becoming the clear dominant alternative.

There are more technical and non-technical reasons for choosing GNU or LLVM, and it is up to each project and developer to decide. To be able to decide, however, a choice must be possible in the first place. This is what is changing and we believe that having this choice is worthwhile.

Acknowledgements

First, we extend a big heartfelt thank you to glibc upstream maintainer Adhemerval Zanella, whose upstream glibc development efforts and knowledge made last year's progress possible. We would also like to acknowledge and thank all those who contribute in different ways to make glibc + LLVM a reality, including but not limited to: Manoj Gupta, Fangrui Song, Nick Desaulniers, Maxim Kuvyrkov, and many others.

Complexity and challenges

A few challenges have been described at a high level in our previous post and Fangrui Song wrote a further excellent blog post detailing some of the known problems. More "unknown problems" should be expected, even breakage due to how LLVM and GNU differently handle a semantic ambiguity was encountered.

Another significant challenge is the number of hardware architectures officially supported by glibc. Supporting them all would require a large development and testing effort and is not feasible (how many people still develop on Motorola 68k?) even though LLVM might have back-ends to generate code for many of them (m68k itself was added recently).

Thus, a decision has been made to focus on 32-bit ARM, AArch64, x86, and x86_64. Others in the FOSS community are welcome to step in and support more architectures. Even the subset of our four directly targeted architectures are a rather big effort and have variants and corner cases of their own; for example ARM "thumb" instruction set builds are currently broken, but -marm works.

Bootstrapping

We further explore this particularly interesting challenge because it is also relevant to the glibc + LLVM users or distributions: boostrapping glibc with an LLVM toolchain.

Bootstrapping is a well-known chicken-and-egg computer science problem. The C library is considered part of the toolchain and normal glibc + GNU/GCC bootstrapping is well understood and used by distributions.

The main problem when bootstrapping glibc + LLVM is avoiding the "normal" build & runtime dependencies on GNU/GCC projects. Fortunately, upstream glibc already supports builds with the LLD linker and the binutils tools can be replaced by their LLVM equivalents, which have improved greatly over the years.

Replacing the libgcc runtime dependency is harder because it is required by the Linux Standard Base. Fortunately again, earlier this year, LLVM got the ability to create a substitute libgcc by linking necessary functions from its compiler-rt and libunwind projects.

Two minor complication still need accounting for:

1. compiler-rt is written in C++ so we need to link it against libcxx instead of libstdcxx.
2. glibc deprecated its libcrypt, but compiler-rt requires at least a crypt.h header, now provided by the libxcrypt project.

Once all these considerations are out of the way, we only need to break the multiple dependency loops between all these components. Here are some dependency loop examples:

glibc -> compiler-rt -> glibc
glibc -> llvm-libunwind -> glibc
compiler-rt -> libxcrypt -> glibc -> compiler-rt
compiler-rt -> libcxx -> llvm-libunwind -> glibc -> compiler-rt

The above dependencies can be plotted in a directed cyclic graph (note the llvm-libunwind node is duplicated to make the graph a bit cleaner):

One breaks these loops by building only the minimum required parts of each component. They can unlock building the bigger parts of the next components in the dependency chain until we arrive at a fully working C library and compiler. This process is called multi-stage bootstrapping.

For example, here is a brief high level description of the steps necessary to bootstrap glibc + LLVM for an aarch64 cross-compiler on an x86_64 host:

1. Install headers for libxcrypt, glibc, and linux kernel
2. Build compiler-rt without sanitizers for required builtins
3. Build glibc with -unwindlib=none
4. Build llvm-libunwind
5. Build full glibc against libunwind
6. Build libcxx
7. Build full compiler-rt against glibc and libcxx

The bootstrap patches for ChromiumOS (more on ChromiumOS's usage is in the next section) are in this Gerrit topic branch and are expected to land soon. More chicken-and-egg style problems exist; for example, the crossdev tool cannot be extended to support proper LLVM + glibc until glibc actually supports it, so for now we use a downstream patch for it as well to prove the concept.

Development & testing setup

Quickly iterating is a very important requirement in toolchain construction in order to quickly rebuild and test a complete functioning system with various toolchain modifications. The Gentoo Linux build system portage is an excellent tool for this task because it was designed with cross-compilation toolchains in mind. By adding a llvm USE flag to the glibc ebuild, we can quickly switch, rebuild, and compare GNU vs LLVM results, as well as rebuild any other packages in the software stack as necessary.

We also chose to use the Gentoo-based ChromiumOS because it has a comprehensive automated integration testing suite named Tast. It can be run on all targeted hardware architectures and makes testing and identifying problems significantly faster, using real-world use-cases on real hardware or virtual machines.

Gentoo Linux also has excellent glibc package maintainership, with frequent uprevs, bugfixes, and backports. At the time of our development, glibc v2.35 was used, on top of which Gentoo added around 150 backports & fixes, ChromiumOS around 10, on top of which a further 163 patches and backports were added to enable the LLVM builds.

Results and future plans

We have reached a point where the ChromiumOS test suites pass, but the LLVM builds are intentionally not enabled by default (the glibc llvm USE flag is disabled), because some compromises have been made like the previously-mentioned disabling of thumb mode for ARM builds. More work is required.

The next steps are to upstream more patches to the glibc project and to continue testing/fixing more corner-cases. This upstream effort will be led by Adhemerval, and thanks to his awesome work, some patches have already landed for the in-development v2.37. Future releases will see more LLVM support improvements until hopefully one day no downstream patches will be required. Once support is upstream, it needs to be maintained and tested, so it will not regress.

We end this blog post with a "call to arms" for all people and projects interested in building glibc with LLVM: it is still in early development, but it is finally happening. Distro bootstrapping scripts will need to be modified and more widespread testing is required. Maybe some will even find the courage to build an LLVM-only glibc system before we do. :) This new year is the time to contribute if you find it interesting and worthwhile: it's the beginning of a brave new toolchain world.