Integrating libcamera into PipeWire

Integrating libcamera into PipeWire

Raghavendra Rao
September 11, 2020

Share this post:

Reading time:

Earlier this year, my colleagues Julian Bouzas and George Kiagiadakis wrote about PipeWire and Wireplumber, the PipeWire session manager. Since then, PipeWire has continued to evolve with the recent integration of libcamera, a library to support complex cameras. In this blog post, I'll explain why libcamera exists, what it does, and how we integrated it in PipeWire.

libcamera

Cameras are complex devices which require heavy hardware image processing. The complexity in the userspace camera application development originates with the evolution of complex hardware. libcamera addresses this problem and offers ease of camera application development. libcamera can be described as a full camera stack library in the userspace to provide the feature that do not belong in the kernel.

The core of libcamera framework exposes kernel driver APIs to userspace while the libcamera application layer abstracts the developer from complex hardware usage. libcamera supports multiple video streams from a single device. In real time use cases such as video conferencing, we may need to preview streams with different resolutions than what we tend to stream over the network. It is natural that we would like to display the live streaming simaultaneously while we capture the video with different resolutions. libcamera offers a perfect solution to fulfill these requirements.

Having said that, what if we'd like to stream a camera device data using different applications simaultaneously? Well, PipeWire answers this question. Besides acting as an interface between hardware devices and applications, PipeWire eases the sharing of hardware devices between applications. Moreover, integration of libcamera into PipeWire brings all the abilities offered by libcamera into PipeWire. With this, PipeWire can become a first class user of modern cameras.

PipeWire-libcamera interworking

As camera application development through v4l becomes more and more complex, Collabora decided to tackle this by integrating libcamera into PipeWire to offer all the evolutions coming from both the frameworks.

The integration work has been merged into PipeWire master.

Give it a try!

Please follow the steps below to try libcamera in PipeWire.

Make sure that the linux kernel version is 5.0.0 or above.
Make sure that the system user is added to video group to access /dev/media*. You can run the command sudo adduser `whoami` video and restart the session.
Make sure that meson build package is installed in your system.
Make sure that sdl2 package is installed in your system.
Make sure that libdrm package is installed in your system.
Follow Getting libcamera to obtain the source and install it. You can also use the commands below to build and install.
```
meson --prefix <path/to/install dir> --buildtype=plain build
cd build
ninja -C . install
```
Follow Getting PipeWire to obtain the source. Configure with meson and install it. Use the following commands:
```
meson --prefix <path/to/install dir> -Dlibcamera=true build
cd build
ninja -C . install
```
Now you can test libcamera spa plugin. Run the command "./build/spa/examples/local-libcamera" from the PipeWire directory. You should be able to see the camera captured data being rendered.

We will be blogging more about this integration work and improvements to be made in the future, explaining the motivation and further progress we make. Stay tuned!

WirePlumber 0.3 released, now ready for the desktop

PipeWire, the media service transforming the Linux multimedia landscape

WirePlumber, the PipeWire session manager

WirePlumber 0.3 released, now ready for the desktop

PipeWire, the media service transforming the Linux multimedia landscape

WirePlumber, the PipeWire session manager

Comments (16)

Rogan Dawes:
Sep 14, 2020 at 05:04 PM

Is this something that could easily be exported as a remote desktop using VNC? Given the availability of cheap HDMI input dongles that appear as a UVC camera, it would be neat to be able to expose them using a regular "remote desktop" protocol. I guess an input filter would be required to receive the keystrokes and mouse movements, of course. Those could then be emitted using a HID Gadget and ConfigFS or similar.
I did look at doing this with Xvncserver, but the rawfb support was a bit tricky to figure out, and I never got that working.

Reply to this comment

Reply to this comment
1. Raghavendra:
  Sep 15, 2020 at 06:57 PM
  
  Hello Rogan Dawes,
  
  Thank you for your response.
  
  The idea behind the integration of libcamera was to ease the Camera application development with modern cameras and bring all the capabilities of libcamera to PipeWire.
  
  It is interesting to note that PipeWire already supports remote desktop sharing with wayland, gnome-remote-desktop and browsers such as firefox, chrome etc. PipeWire acts as interface between the Application and hardware to share the media via "dmabuf" buffers. Applications such as VNC/RDP can connect to PipeWire creating sink nodes to consume the Video frames captured by source nodes such as libcamera node, in this case.
  
  gnome-remote-desktop daemon enables GNOME to offer remote desktop sharing using VNC with PipeWire. This daemon supports GNOME on X as well as GNOME on Wayland. If you have got ubuntu installed then you may want to recompile mutter with remote desktop option enabled to make it work on ubuntu.
  
  Thanks,
  Raghavendra
  
  Reply to this comment
  
  Reply to this comment
Andrew Zhukov:
Jan 26, 2021 at 12:57 PM

Can't get it work with latest libcamera and pipewire. Seems like pipewire uses old libcamera version...

Reply to this comment

Reply to this comment
1. Raghavendra Rao:
  Feb 03, 2021 at 11:48 AM
  
  Hello Andrew Zhukov,
  
  Thanks for your message.
  
  The issue has been fixed and the fix has been merged into pipewire master. You can now check with pipewire master branch,
  
  Thanks,
  Raghavendra Rao
  
  Reply to this comment
  
  Reply to this comment
Rogan Dawes:
Jun 15, 2022 at 08:40 AM

Thanks for your previous response. Unfortunately, I never managed to get it working. :-(

Any thoughts of implementing PipeWire support in uvc-gadget? That would allow a complete video processing pipeline to be exposed to a host computer as a simple webcam. While there are existing projects that expose e.g. a Raspberry Pi camera as a webcam (e.g. https://github.com/showmewebcam/showmewebcam), there are more and more AI-enabled cameras that could benefit from a standardised interface to implement their processing of the stream before it gets to the host. I'm thinking that PipeWire is a pretty good candidate to be this standardised interface?

Reply to this comment

Reply to this comment
1. Raghavendra Rao:
  Jun 16, 2022 at 11:38 AM
  
  Hello Rogan Dawes,
  
  Thanks for your message.
  
  This is an interesting idea, it seems. We would certainly look into it and get back on this.
  
  Thanks,
  Raghavendra Rao
  
  Reply to this comment
  
  Reply to this comment
  1. Rogan Dawes:
    Mar 07, 2024 at 02:41 PM
    
    Hi Raghavendra,
    
    Glad to see you are still following up on this blog post. Were you ever able to make any progress on integrating uvc-gadget into pipewire?
    
    Regards,
    
    Rogan
    
    Reply to this comment
    
    Reply to this comment
    1. Raghavendra:
      Mar 08, 2024 at 12:45 PM
      
      Hello Rogan,
      
      Thanks for your message.
      
      Unfortunately, I could not work on this so far. I will post you an update on this.
      
      Regards,
      Raghavendra
      
      Reply to this comment
      
      Reply to this comment
      1. Rogan Dawes:
        Mar 08, 2024 at 02:12 PM
        
        Thank you. I think you would find a number of users for it if you did (if that is an important metric for you :-) )
        
        Reply to this comment
        
        Reply to this comment
      2. Rogan Dawes:
        Mar 08, 2024 at 02:21 PM
        
        Actually, a quick googling turned up work that Pengutronix did last year, integrating uvcsink into the GStreamer framework. https://indico.freedesktop.org/event/5/contributions/257/attachments/119/177/20230926-gstreamer-uvcsink-mgrzeschik.pdf
        
        I wonder how much (if any) of it could be reused in pipewire?
        
        Reply to this comment
        
        Reply to this comment
        
        Raghavendra:
        Mar 12, 2024 at 03:23 AM
        
        Hello Rogan,
        
        Thanks for your inputs.
        
        This is interesting. We will look into this and check the feasibility. We will update you shortly.
        
        Regards,
        Raghavendra
        
        Reply to this comment
        
        Reply to this comment
    2. Raghavendra:
      Mar 08, 2024 at 12:45 PM
      
      Hello Rogan,
      
      Thanks for your message.
      
      Unfortunately, I could not work on this so far. I will post you an update on this.
      
      Regards,
      Raghavendra
      
      Reply to this comment
      
      Reply to this comment
  2. Rogan Dawes:
    Mar 07, 2024 at 02:42 PM
    
    Hi Raghavendra,
    
    Glad to see you are still following up on this blog post. Were you ever able to make any progress on integrating uvc-gadget into pipewire?
    
    Regards,
    
    Rogan
    
    Reply to this comment
    
    Reply to this comment
Joel Winarske:
Mar 06, 2024 at 05:00 PM

Hi there,

Thanks for your work on this.

I'm hitting issue with Fedora 39 trying to run the example:

$ ./local-libcamera
libdecor-gtk-WARNING: Failed to initialize GTK
Failed to load plugin 'libdecor-gtk.so': failed to init
[48:46:15.784065619] [1687584] INFO Camera camera_manager.cpp:284 libcamera v0.2.0+50-1bf2d707-dirty (2024-03-06T08:44:06-08:00)
[E] spa.libcamera [libcamera-source.cpp:1029 impl_init()] unknown camera id
can't make factory instance: -2
can't create libcamera-source: -2
can't make nodes: -2

When it does work are `all` the libcamera controls + parameters available from pw-cli?

Reply to this comment

Reply to this comment
1. Raghavendra:
  Mar 07, 2024 at 04:47 AM
  
  Hello Joel Winarske,
  
  Thank you for your response. I will check this issue and get back to you.
  Also, I will get back to you on your question about "pw-cli".
  
  Thanks & Regards,
  Raghavendra
  
  Reply to this comment
  
  Reply to this comment
2. George Kiagiadakis:
  Mar 08, 2024 at 03:34 PM
  
  Hi Joel,
  
  The local-libcamera executable is merely a development test tool and I wouldn't recommend trying to get it to work, as it does not really integrate with PipeWire.
  
  On the latest WirePlumber releases, the libcamera plugin is loaded by default, so you should be able to list and use the cameras without running any additional code, provided that the plugin is installed. First of all, ensure that you have the `pipewire-plugin-libcamera` package installed. Then, restart wireplumber and check `wpctl status -n`, under the Video -> Sources section. You should be able to see sources starting their name with "libcamera_input."
  
  You can then run `pw-dump libcamera_input...(that full name)...` on any of the sources appearing there to see the available controls (under the params -> PropInfo section).
  
  In order to use the source, try a gstreamer pipeline with `pipewiresrc`, like this: `gst-launch-1.0 pipewiresrc target-object="libcamera_input...." ! autovideosink`
  
  Controls can be changed with `pw-cli` indeed. Something like: `pw-cli set-param Props '{ contrast = 1.1, brightness = 0.1 }'`
  
  I hope this is helpful.
  
  Best regards,
  George
  
  Reply to this comment
  
  Reply to this comment

Add a Comment

Search the newsroom

Latest Blog Posts

Faster inference: torch.compile vs TensorRT

19/12/2024

In the world of deep learning optimization, two powerful tools stand out: torch.compile, PyTorch’s just-in-time (JIT) compiler, and NVIDIA’s…

Mesa CI and the power of pre-merge testing

08/10/2024

Having multiple developers work on pre-merge testing distributes the process and ensures that every contribution is rigorously tested before…

A shifty tale about unit testing with Maxwell, NVK's backend compiler

15/08/2024

After rigorous debugging, a new unit testing framework was added to the backend compiler for NVK. This is a walkthrough of the steps taken…

A journey towards reliable testing in the Linux Kernel

01/08/2024

We're reflecting on the steps taken as we continually seek to improve Linux kernel integration. This will include more detail about the…

Building a Board Farm for Embedded World

27/06/2024

With each board running a mainline-first Linux software stack and tested in a CI loop with the LAVA test framework, the Farm showcased Collabora's…

Smart audio filters with WirePlumber 0.5

26/06/2024

WirePlumber 0.5 arrived recently with many new and essential features including the Smart Filter Policy, enabling audio filters to automatically…

About Collabora

Whether writing a line of code or shaping a longer-term strategic software development plan, we'll help you navigate the ever-evolving world of Open Source.

한국의 국기 한국어 버전의 Collabora.com 보기