Setting up QEMU-KVM for kernel development

Setting up QEMU-KVM for kernel development

Frédéric Dalleau
January 16, 2017

Share this post:

Reading time:

Introduction

Before doing linux kernel development, I started by typing make in a kernel tree. After booting, I always had some non working peripherals. So my second step was to use a distribution specific build procedure. For example, the Ubuntu kernel build instructions can be found at https://wiki.ubuntu.com/KernelTeam/GitKernelBuild. It works, and one can easily build a kernel and install it, with all peripheral working. But this method will quickly reach its limitations to write new kernel code. On a decent computer (i7 5600U), the build/test cycle lasts about 30 minutes. It is possible to build only the needed module and insmod/rmmod, but in case of a crash followed by a rebooting, the developer loses its work environment.

The next step is to run the kernel inside a virtual machine.

Booting kernel in a virtual machine

VirtualBox is well known, very user friendly and supports a large amount of different OSes. Installation of a VirtualBox will be under the hour mark. But rebuilding the Ubuntu kernel is still a 30 minute cycle. Additionnaly exchanging files between Virtualbox and the host will involve some kind of networking or file sharing that have to be setup. For kernel development, Virtualbox Guest additions have to be rebuilt often when the kernel is updated.

Qemu is another virtual machine. A complete distro can be installed into it. But it has a very interesting option: -kernel. With that option, QEMU will boot the kernel binary provided as argument. Ubuntu users can try :

$ sudo qemu-system-x86_64 -kernel /boot/vmlinuz-`uname -r`

This will boot your kernel within QEMU, but an error occurs immedialy: There is no filesystem to boot. Also, since /boot is readable only by root, sudo permission is required. This is not needed with a user built kernel.

Adding a rootfs

debootstrap allows to install a debian distribution in a directory. Before going too fast, if your file system is mounted with the nodev option, it won’t be possible to create device nodes. Instead, we will mount a qemu image file on a directory and use debootstrap in the mount point we created as in the following:

IMG=qemu-image.img
DIR=mount-point.dir
qemu-img create $IMG 1g
mkfs.ext2 $IMG
mkdir $DIR
sudo mount -o loop $IMG $DIR
sudo debootstrap --arch amd64 jessie $DIR
sudo umount $DIR
rmdir $DIR

The target rootfs is a matter of taste. For learning purposes, using busybox would be very interesting too. But for development purposes, having all the debian development tools in the rootfs is very useful.

Boot as follow:

$ sudo qemu-system-x86_64 -kernel /boot/vmlinuz-`uname -r`\
                          -hda qemu-image.img\
                          -append "root=/dev/sda single"

To silence the warning about raw format, replace “-hda qemu-image.img” with “-drive file=qemu-image.img,index=0,media=disk,format=raw”

Boot as a single user to change root password and create a user.

Building and booting a kernel

Now it is time to build your own kernel. There exist a make KVM configuration target that tunes an existing configuration and makes it usable from QEMU. However, it will not create a .config file from scratch. So we will start from generic config file and kvmify it. It is still possible to build a dedicated config file that will allow for shorter build time, but that would require several iterations.

git clone --depth=1 git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
cd linux
make x86_64_defconfig
# if building an older kernel use 'make kvmconfig" instead if below command fails
make kvm_guest.config
make -j 8

Use the resulting file in the command line below. We can drop sudo.

$ qemu-system-x86_64 -kernel arch/x86/boot/bzImage
                     -hda qemu-image.img
                     -append "root=/dev/sda"

Reduce the boot time with kvm

KVM accelerates x86 virtualization in QEMU. It will only accelerate x86 platforms. It is as simple as adding a command line option.

$ qemu-system-x86_64 -kernel bzImage
                     -hda qemu-image.img
                     -append "root=/dev/sda"
                     --enable-kvm

Now your debootstrap image boots in less than two seconds. This can be checked in dmesg. Before –enable-kvm, systemd is started 5.9 seconds after boot. After enabling, systemd is started after 1.7 seconds. A more than 3 times shorter boot time. And it just can’t be compared to ubuntu. Note that we don’t have a full user interface up, so we cannot compare apple with peaches.

Connecting into QEMU

Initially, QEMU displays its own screen in a dedicated window. For a terminal use case, this is not really pratical as it gets into the ALT tab list, the keyboard and mouse capture are not suitable for this use either. Copying and pasting also aren’t very practical. It is much more convenient to remove the graphic interface and instruct the kernel to write to ttyS0 that qemu redirect to the terminal in –nographic mode.

$ qemu-system-x86_64 -kernel bzImage
                     -append "root=/dev/sda console=ttyS0"
                     -hda qemu-image.img
                     --enable-kvm
                     --nographic

Time to leave

Typing halt in qemu will stop the kernel, but the qemu process would continue running on the host and would have to be killed. The proper command to terminate the virtual machine is :

$ shutdown -h now

A second part to this post is planned, stay tuned!

Original post

Kernel debugging with QEMU: An overview of tools available

Quick hack: Building ChromiumOS for QEMU

Kernelci.org automated bisection

Kernel debugging with QEMU: An overview of tools available

Quick hack: Building ChromiumOS for QEMU

Kernelci.org automated bisection

Comments (27)

Johan:
Aug 13, 2017 at 12:56 PM

Thanks for a great post!

I noticed two things while trying this out. It's not possible to use debootstrap with a qcow2 disk image. It will fail when it tries to create the device nodes, raw works flawlessly though. Secondly, I was not able to boot into single user mode using Debian Stretch. The root account was locked. But this can be worked around by using chroot and setting a root password.

Reply to this comment

Reply to this comment
1. Frédéric Dalleau:
  Aug 21, 2017 at 08:09 AM
  
  Hello Johan,
  Thanks for feedback, you're right, on stretch chroot must be used after the debootstrap step:
  
  sudo debootstrap --arch amd64 stretch $DIR
  
  sudo chroot $DIR
  
  passwd
  
  The raw image can be converted to qcow2 and the resulting file can be used in the command lines instead:
  
  qemu-img convert -O qcow2 qemu-img.img qemu-img.qcow2
  
  Best regards,
  Frédéric
  
  Reply to this comment
  
  Reply to this comment
Dongliang Mu:
Sep 22, 2017 at 12:37 AM

I use the shell script you mention at first to generate qemu-stretch.img(stretch) successfully.
And then I use the following command to boot the kernel 4.12.0-1.
qemu-system-x86_64 -kernel /boot/vmlinuz-4.12.0-1-amd64 -drive file=qemu-stretch.img,index=0,media=disk,format=raw -append "root=/dev/sda1 single"

But the kernel panic and shows : "VFS - unable to mount root fs"

Reply to this comment

Reply to this comment
1. Frédéric Dalleau:
  Oct 09, 2017 at 12:30 PM
  
  Hello Dongliang,
  Assuming you specified the appropriate kernel and disk image location, you could check the parameters passed to kernel command line. The following worked for me (changed /dev/sda1 to /dev/sda).
  
  qemu-system-x86_64 -kernel arch/x86/boot/bzImage -drive file=../qemu-image.img,index=0,media=disk,format=raw -append "root=/dev/sda single"
  
  Please note in the previous comment about the changes to add a password to a stretch image.
  Best regards,
  Frédéric
  
  Reply to this comment
  
  Reply to this comment
  1. Dongliang:
    Oct 10, 2017 at 02:29 PM
    
    I redo the experiment with the script you provide in the blog, and I build one kernel in my laptop. I try the following command:
    qemu-system-x86_64 -kernel ~/Repos/linux/arch/x86/boot/bzImage -drive file=qemu-image.img,index=0,media=disk,format=raw -append "root=/dev/sda single"
    
    The error message is :
    VFS: unable to mount root fs on unknown-block(0,0)
    
    I don't modify anything in the script.
    
    Reply to this comment
    
    Reply to this comment
    1. Frédéric Dalleau:
      Oct 11, 2017 at 06:28 AM
      
      Hello Dongliang,
      The error "VFS - unable to mount root fs" occurs if the image exists but the filesystem could not be mounted.
      Many possibilities exists: the image do not contain a filesystem, or the kernel does not support the selected filesystem.
      According to your blog link, you are using 'sudo mkfs.ext2' and this should not needed. I suggest you investigate in this direction.
      Frédéric
      
      Reply to this comment
      
      Reply to this comment
      1. Dongliang Mu:
        Oct 11, 2017 at 05:46 PM
        
        Thanks for your tip. I try to redo the experiment in the Ubuntu 14.04 LTS. It is successful. Amazing!
        In my previous Debian System, "/sbin" is not in the PATH by default. So I directly add "sudo" to change it to higher privilege.
        However, when I use "/sbin/mkfs.ext2" in the script and redo the experiment in my Debian Buster system, it still does not work.
        That's really really wired.
        
        Reply to this comment
        
        Reply to this comment
        
        Frédéric Dalleau:
        Oct 12, 2017 at 06:47 AM
        
        Hello,
        
        It's a great progress!
        Did you try to copy the working image and kernel on the other machine (with scp) and run the same command line?
        
        Regards,
        Frédéric
        
        Reply to this comment
        
        Reply to this comment
        
        Dongliang Mu:
        Oct 12, 2017 at 02:49 PM
        
        I tried to copy image generated by my Debian Buster system to Ubuntu 14.04 LTS, the image could work. And the system could boot the secure shell with jessie system.
        But when I copied the image generate by Ubuntu 14.04 LTS to Debian Buster, the image that boots successfully in Ubuntu cannot be booted in Debian Buster.
        I checked the version of two systems:
        Debian Buster : qemu 1:2.10.0+dfsg-1 amd64
        Ubuntu 14.04 : qemu 2.0.0+dfsg-2ubuntu1.36 amd64
        
        Several reasons for those failure:
        - Qemu changes some behaviors from 2.0.0 to 2.10.0;
        - Ubuntu compiles in different way from Debian, (small possibility);
        
        I will try to install one Ubuntu 16.04 and Debian Stretch and test image there. And at same time send one email to Debian Qemu QA Mailing List.
        They may have some great understanding about this problem.
        
        Reply to this comment
        
        Reply to this comment
        
        introom:
        Nov 17, 2017 at 01:14 PM
        
        Hi, dongliang
        
        The exact cause of this VFS error depends on your environment.
        
        One reason is the misconfiguration of the kernel. Normally "make defconfig" followed by a "make kvmconfig" should do.
        Another reason is the virtualization platform, if you are using nested virtualization, especially KVM run inside VMWARE , notice that qemu machine type must be no later than 2.6 if you wanna use virtio driver. Be warned, this is a bug.
        
        Reply to this comment
        
        Reply to this comment
Dongliang Mu:
Oct 10, 2017 at 03:12 PM

https://mudongliang.github.io/2017/09/21/set-up-qemu-for-kernel-development.html

I have created one post to show how I redo the experiment.

Reply to this comment

Reply to this comment
Vakul Garg:
Dec 27, 2017 at 06:06 AM

Thanks for this wonderful post.

The tips provided just jump started me with the kernel version for which I want to develop my kernel module. Earlier, I was struggling to build my kernel development env using user-mode-linux.

Now, I want to have a shared folder between host and virtual machine. How can I do so?
It seems that the root file system is mounted read only by linux in virtual machine.
How can I make it read/write? I may need to install/update packages into root file system.

Also some addition to this blog to enable networking in virtual machine would be a great value add.

Reply to this comment

Reply to this comment
1. Frédéric Dalleau:
  Jan 26, 2018 at 06:14 PM
  
  Hi Vakul,
  Thanks for your comment, from the top of my head, I'd say you can remount / read write using that command :
```
sudo mount -o rw,remount /
```
  For networking, check out this article: https://www.collabora.com/news-and-blog/blog/2017/03/13/kernel-debugging-with-qemu-overview-tools-available/
  
  Regards,
  Frédéric
  
  Reply to this comment
  
  Reply to this comment
elmazzun:
Dec 29, 2017 at 02:05 PM

Nice tutorial!
I followed all the instructions and I managed to boot the last kernel image, but I can't enter single user mode and reset the password.
When you say "Boot as a single user to change root password and create a user", I can't enter GRUB or any other particular mode and I'm stuck in the login screen, without being able to proceed.
How can I boot as a single user?

Reply to this comment

Reply to this comment
1. Frédéric Dalleau:
  Jan 26, 2018 at 06:05 PM
  
  Hi Elmazzun,
  You might want to check those steps:
  https://www.collabora.com/news-and-blog/blog/2017/01/16/setting-up-qemu-kvm-for-kernel-development/#qcom182
  Regards,
  Frédéric
  
  Reply to this comment
  
  Reply to this comment
Sreyas:
Feb 28, 2018 at 07:40 PM

Hi Frederic,

Thanks for the great post.
I followed the procedure that you've posted here with debootstrap with works fine.

I am trying to make changes to the ext4 filesystem of the linux kernel. I have configured the ext4 to load as a module with "make menuconfig" but I am still unable to find /lib/modules after booting my kernel with qemu.

I am not able to find any modules.

Do let me know if you can help.

Reply to this comment

Reply to this comment
Ciro Santilli:
May 28, 2018 at 11:03 AM

I have created a fully automated QEMU + Buildroot setup that gets you up on a minimal system in a single command: https://github.com/cirosantilli/linux-kernel-module-cheat

Reply to this comment

Reply to this comment
David:
Sep 23, 2018 at 02:00 PM

works nice, however,
1) I'm getting errors that my file system is read only. any idea how to change that? (when I'm in console inside the qemu).

2) Is there an easy way to do this with ubuntu sever flavor?

Reply to this comment

Reply to this comment
1. Ciro Santilli:
  Sep 25, 2018 at 06:35 AM
  
  Hi David,
  
  1) I can't reproduce, please post a detailed bug report at: https://github.com/cirosantilli/linux-kernel-module-cheat/issues
  
  2) it will happen in the next few weeks, subscribe to: https://github.com/cirosantilli/linux-kernel-module-cheat/issues/35
  
  Reply to this comment
  
  Reply to this comment
  1. Ciro Santilli:
    Sep 25, 2018 at 08:07 AM
    
    Ah, I thought that was a reply to me, nevermind last post.
    
    Reply to this comment
    
    Reply to this comment
2. Ciro Santilli:
  Oct 05, 2018 at 01:31 PM
  
  I managed to get both network and Ubuntu with the following bootstrap setup: https://askubuntu.com/questions/281763/is-there-any-prebuilt-qemu-ubuntu-image32bit-online/1081171#1081171
  
  Also achieved a similar Debian one with network: https://unix.stackexchange.com/questions/275429/creating-bootable-debian-image-with-debootstrap/473256#473256
  
  Reply to this comment
  
  Reply to this comment
irfan ullah:
Jul 15, 2019 at 09:43 AM

Thank you very much for very useful post. I am facing the problem with the log in.
After booting the kernel with the commands 1, and 2 (given at the end), I have encountered with the problem that I am not able to logged in. For each user name, and password I am getting the same error "log in incorrect".
I am running Ubuntu 16.04.6 LTS(xenial), and I have tried "sudo chroot $DIR", and "passwd" mentioed at (3), but it didn't work for me.
Thank you :).

1). "sudo debootstrap --arch amd64 xenial $DIR"
2). "qemu-system-x86_64 -kernel /boot/vmlinuz-5.2.0 -append "root=/dev/sda console=ttyS0" -hda qemu-image.img --enable-kvm --nographic",
3). https://www.collabora.com/news-and-blog/blog/2017/01/16/setting-up-qemu-kvm-for-kernel-development/#qcom182

Reply to this comment

Reply to this comment
anon nospam:
Oct 01, 2020 at 06:02 PM

WARNING: 'make kvmconfig' will be removed after Linux 5.10
Please use 'make kvm_guest.config' instead.

Reply to this comment

Reply to this comment
Me:
May 13, 2021 at 02:02 PM

Please why do I get this error (I read the documentation but I can't decipher it):

end Kernel panic - not syncing: No working init found. Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance.

Reply to this comment

Reply to this comment
1. Ezequiel Garcia:
  May 14, 2021 at 01:33 PM
  
  Hello Me,
  
  Some of us have moved to using virtme, which is a tool wrapping qemu, and solving its complex setup.
  Maybe this post can help you get started:
  
  https://www.collabora.com/news-and-blog/blog/2018/09/18/virtme-the-kernel-developers-best-friend/
  
  You can use virtme's github repository issue tracker for other questions, in case you need more help.
  
  Good luck,
  Ezequiel
  
  Reply to this comment
  
  Reply to this comment
2. Sebastian Reichel:
  May 14, 2021 at 01:53 PM
  
  Hi Me,
  
  The error means, that the kernel does not find /sbin/init, which is the first userspace process started by the kernel. That process is expected to start any other required user space process. If it does not exist there is no user space, which is obviously a fatal error. The kernel also suggest, that you can override the path to the binary by adding a kernel parameter, e.g. "init=/usr/local/bin/my-own-init". That's not something you need at this point, though. My guess is, that something went wrong during debootstrap and systemd-sysv has not been installed into your image (which provides /sbin/init in a default Debian system).
  
  Have fun,
  
  -- Sebastian
  
  Reply to this comment
  
  Reply to this comment
MLK:
Aug 12, 2021 at 02:34 PM

If you ever meet this error: while setting root password restart the VM by adding "rw" to the " -append" parameter (just before "single"):
qemu-system-x86_64 -kernel /boot/vmlinuz-`uname -r` -drive "file=qemu-image.img,index=0,media=disk,format=raw" -append "root=/dev/sda rw single"

Reply to this comment

Reply to this comment

Add a Comment

Search the newsroom

Latest Blog Posts

Faster inference: torch.compile vs TensorRT

19/12/2024

In the world of deep learning optimization, two powerful tools stand out: torch.compile, PyTorch’s just-in-time (JIT) compiler, and NVIDIA’s…

Mesa CI and the power of pre-merge testing

08/10/2024

Having multiple developers work on pre-merge testing distributes the process and ensures that every contribution is rigorously tested before…

A shifty tale about unit testing with Maxwell, NVK's backend compiler

15/08/2024

After rigorous debugging, a new unit testing framework was added to the backend compiler for NVK. This is a walkthrough of the steps taken…

A journey towards reliable testing in the Linux Kernel

01/08/2024

We're reflecting on the steps taken as we continually seek to improve Linux kernel integration. This will include more detail about the…

Building a Board Farm for Embedded World

27/06/2024

With each board running a mainline-first Linux software stack and tested in a CI loop with the LAVA test framework, the Farm showcased Collabora's…

Smart audio filters with WirePlumber 0.5

26/06/2024

WirePlumber 0.5 arrived recently with many new and essential features including the Smart Filter Policy, enabling audio filters to automatically…

About Collabora

Whether writing a line of code or shaping a longer-term strategic software development plan, we'll help you navigate the ever-evolving world of Open Source.

한국의 국기 한국어 버전의 Collabora.com 보기