Here at Collabora we are pretty interested at improving QA tools in GStreamer. Thibault is for example doing a great job on gst-validate ensuring that a lot of code paths are regularly tested using real life scenarios. Last year I added Valgrind support to gst-validate allowing us to automatically detect memory leaks in test scenarios. My goal was to integrate this as part of GStreamer's automatic QA to prevent memory leak regressions. While this can sometimes be a good approach to track leaks it has a few downsides:
- Valgrind can be very CPU and/or memory consuming which can be a problem with longer scenarios or on limited hardware such as embedded devices.
- As a result running the full tests suite with valgrind can take ages.
- Valgrind checks for any potential memory leak which can lead to a lot of false positives or leaks in low level system libraries on which we have few control. We usually work around this problem using suppression files but they are generally very fragile and depend a lot on the system/distribution which has been used for testing.
I tried to solve these issues by trying a new approach using GstTracer. Tracers are a new mechanism introduced in GStreamer 1.8 allowing tools to hook into GStreamer internals and collect data. So I started by adding tracer hooks when GstObject and GstMiniObject are created and destroyed. Then I implemented a new tracer tracking the lifetime of (mini)objects and listing those which are still alive when the application is exiting. This worked pretty well but I needed a way to discard objects which are intentionally leaked (false positives). To do so I introduced a new (mini)object flag allowing us to mark such objects.
I'm pretty happy with the result, while proof testing this tool I found and fixed dozens of leaks into Gstreamer (core, plugins and tests). Some of those fixes have already reached the 1.8.2 release. It's also very easy to use and doesn't require any external tool unlike Valgrind (which can be tricky to integrate on some platforms).
To use it you just have to load the leaks tracer with your application and enable tracer logs:
GST_TRACERS="leaks" GST_DEBUG="GST_TRACER:7" gst-launch-1.0 videotestsrc num-buffers=10 ! fakesink
You can also filter out the types of GstObject or GstMiniObject tracked to reduce memory consumption:
GST_TRACERS="leaks(GstEvent,GstMessage)" GST_DEBUG="GST_TRACER:7" gst-launch-1.0 videotestsrc num-buffers=10 ! fakesink
This tracer has recently be merged into GStreamer core and will be part of the 1.9.1 release.
As future enhancements I implemented live tracking and checkpointing support using signals like I already did in gobject-list a while ago. I'd also like to be able to display the creation stack trace of leaked objects to easily spot the leaked instances. Finally, I opened a bug to discuss the integration of the tracer with the QA system.
Comments (2)
vandana:
Apr 27, 2021 at 10:03 AM
Hello, my question is on the very large size of the trace logs. Is it possible to limit that, since I only need the stats overview:
export GST_DEBUG=GST_TRACER:7,GST_BUFFER*:0,GST_EVENT:0,GST_MESSAGE:0 ; export GST_TRACERS="stats;rusage"; export GST_DEBUG_FILE=/gstlog.txt
Overall Statistics:
Number of Threads: 4
Number of Elements: 19
Number of Bins: 2
Number of Pads: 43
Number of GhostPads: 3
Number of Buffers passed: 188902
Number of Events sent: 584
Number of Message sent: 932
Number of Queries sent: 20647
Time: 0:01:06.139425106
Reply to this comment
Reply to this comment
Olivier Crête:
Apr 27, 2021 at 03:13 PM
In summary, there is no way right now to do the stats without doing a giant log. To do that, one would need to write a new tracer with this feature.
Reply to this comment
Reply to this comment
Add a Comment