kirushyk/gst-instruments

Easy-to-use profiler for GStreamer

GStreamer Instruments

Set of performance profiling and data flow inspection tools for GStreamer pipelines.

gst-instruments

gst-instruments displays the trace file.

Opening the file from the command-line is supported:

$ gst-instruments-1.0 pipeline0.gsttrace

gst-top

Inspired by top and perf-top, this utility displays performance report for the particular command, analyzing GStreamer ABI calls.

$ gst-top-1.0 gst-launch-1.0 audiotestsrc num-buffers=1000 ! vorbisenc ! vorbisdec ! fakesink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Redistribute latency...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Got EOS from element "pipeline0".
Execution ended after 0:00:00.225688000
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...
ELEMENT        %CPU   %TIME   TIME
vorbisenc0      73.8   74.3    172 ms
vorbisdec0      13.5   13.6   31.4 ms
audiotestsrc0    8.1    8.2   18.9 ms
fakesink0        4.0    4.0   9.21 ms
pipeline0        0.0    0.0      0 ns
$

Generating the trace file

Intercepts GStreamer ABI calls and records communication between the app and GStreamer into the trace file.

macOS example

$ DYLD_INSERT_LIBRARIES=/usr/local/lib/libgstintercept.dylib \
    DYLD_FORCE_FLAT_NAMESPACE= \
    GST_DEBUG_DUMP_TRACE_DIR=. \
    gst-launch-1.0 audiotestsrc num-buffers=1000 ! vorbisenc ! vorbisdec ! fakesink
Setting pipeline to PAUSED ...
[...]
Freeing pipeline ...
$ ls *.gsttrace
pipeline0.gsttrace

Linux example

$ LD_PRELOAD=/usr/lib/libgstintercept.so \
    GST_DEBUG_DUMP_TRACE_DIR=. \
    gst-launch-1.0 audiotestsrc num-buffers=1000 ! vorbisenc ! vorbisdec ! fakesink
Setting pipeline to PAUSED ...
[...]
Freeing pipeline ...
$ ls *.gsttrace
pipeline0.gsttrace

gst-report

Generates performance report for input trace file.

$ gst-report-1.0 playbin.gsttrace
ELEMENT               %CPU   %TIME   TIME
mad0                   33.4   26.5    806 ms
source                 10.5    8.3    253 ms
aconv                   9.8    7.8    237 ms
streamsynchronizer0     8.4    6.6    202 ms
mpegaudioparse0         8.2    6.5    197 ms
typefind                7.1    5.7    172 ms
inputselector0          7.1    5.6    171 ms
id3demux0               7.0    5.5    168 ms
volume                  6.9    5.5    166 ms
resample                6.7    5.3    161 ms
playsink                5.9    4.7    142 ms
aqueue                  5.6    4.5    136 ms
fakesink0               5.3    4.2    127 ms
abin                    4.2    3.4    102 ms
uridecodebin0           0.0    0.0    161 us
conv                    0.0    0.0   20.0 us
audiotee                0.0    0.0      0 ns
identity                0.0    0.0      0 ns
decodebin0              0.0    0.0      0 ns
playbin                 0.0    0.0      0 ns
$

You can generate performance graph in DOT format:

gst-report-1.0 --dot playbin.gsttrace | dot -Tsvg > perf.svg

gst-instruments's Issues

Linkage error

Getting below error while executing "gst-top-1.0 gst-launch-1.0 videotestsrc ! glimagesink"
CANNOT LINK EXECUTABLE "gst-launch-1.0": library "/home/xxx/xxx/xxx/lib/libgstintercept.so.0" not found

** (gst-top-1.0:11500): WARNING **: gst-launch-1.0 exited with code 256

Also I don't find file with "libgstintercept.so.0" anyware .
Please help.

make install does not install shared files

I started packaging gst-instruments for arch, but when trying to generate an svg file, i see that the shared files are missing:

gst-report-1.0 --dot gst-top.gsttrace | dot -Tsvg > perf.svg
Warning: No such file or directory while opening /usr/share/gst-instruments/push.svg
Warning: No or improper image="/usr/share/gst-instruments/push.svg" for node "en0x55e0a632a0d0_pad_0x55e0a6332030"
Warning: No such file or directory while opening /usr/share/gst-instruments/push.svg
...

Taking a look at the package content lists the following files (the package is generated with ./autogen --prefix=/usr and then make install):

gst-instruments-git /usr/
gst-instruments-git /usr/bin/
gst-instruments-git /usr/bin/gst-instruments-1.0
gst-instruments-git /usr/bin/gst-report-1.0
gst-instruments-git /usr/bin/gst-top-1.0
gst-instruments-git /usr/lib/
gst-instruments-git /usr/lib/gstreamer-1.0/
gst-instruments-git /usr/lib/gstreamer-1.0/libgstinstruments.so
gst-instruments-git /usr/lib/gstreamer-1.0/libgstinstruments.so.0
gst-instruments-git /usr/lib/gstreamer-1.0/libgstinstruments.so.0.0.0
gst-instruments-git /usr/lib/libgstintercept.so
gst-instruments-git /usr/lib/libgstintercept.so.0
gst-instruments-git /usr/lib/libgstintercept.so.0.0.0

Am i missing something ?

For reference, here is my WIP PKGBUILD:

# Maintainer: Florent Thiéry [email protected]
pkgname=gst-instruments-git
pkgver=0.1
pkgrel=1
pkgdesc="Set of performance analyzing tools for time profiling and data flow inspection in GStreamer apps."
arch=('i686' 'x86_64')
url="https://github.com/kirushyk/gst-instruments"
license=('GPL3')
depends=('gstreamer' 'gtk3' 'vala' 'xdot')
source=('git+https://github.com/kirushyk/gst-instruments.git')
sha512sums=('SKIP')

build() {
    cd gst-instruments
    ./autogen.sh --prefix=/usr
    make
}

package() {
    cd gst-instruments
    make DESTDIR="$pkgdir/" install
}

how to install this

I'm trying to install this application but I don't know how to install this on a linux machine.

I tried with automake and autoconf but nothing worked correctly. I always get many errors on this like here:

are supported and installed on your system.

perl: warning: Falling back to the standard locale ("C").
configure.ac:5: error: required file './compile' not found
configure.ac:5: 'automake --add-missing' can install 'compile'
configure.ac:6: error: required file './config.guess' not found
configure.ac:6: 'automake --add-missing' can install 'config.guess'
configure.ac:6: error: required file './config.sub' not found
configure.ac:6: 'automake --add-missing' can install 'config.sub'
configure.ac:14: error: required file './install-sh' not found
configure.ac:14: 'automake --add-missing' can install 'install-sh'
configure.ac:6: error: required file './ltmain.sh' not found
configure.ac:40: error: required file './missing' not found
configure.ac:40: 'automake --add-missing' can install 'missing'

Can you please update the readme so people can follow the installation instructions?

Add a --method command line option for gst-top to choose between ABI interception and GStreamer tracing ways

At this moment this is hardcoded.

GUI app should open file given in command line arguments

In case some trace filename is passed in command line arguments, application should open it.

Error: <stdin>: syntax error in line 1 near '('

This is my pipeline:

 PIPELINE="GST_DEBUG_FILE=output.log GST_DEBUG_NO_COLOR=1 \
     LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libgstintercept.so GST_DEBUG_DUMP_TRACE_DIR=. \
     gst-launch-1.0 --gst-debug-level=3 \
     filesrc location=detection.mp4 ! decodebin ! videoconvert ! \
     xvimagesink sync=true"
 
 eval $PIPELINE

I'm using Ubuntu18.04 and GStreamer 1.14

pipeline0.gsttrace.log

This is the gsttrace file (added log suffix because Github doesn't allow it otherwise).

Am I missing something?

How to use intercept lib ?

Your example is:

DYLD_INSERT_LIBRARIES=/usr/local/lib/libgstintercept.dylib \
	DYLD_FORCE_FLAT_NAMESPACE= \
	GST_DEBUG_DUMP_TRACE_DIR=. \
	gst-play-1.0 ~/Music/Snowman.mp3 --audiosink=fakesink

I'm trying to generate a trace file, but nothing is generated; where is the libgstintercept.dylib supposed to be ? I tried with libgstintercept.so but no joy

$ DYLD_INSERT_LIBRARIES=/usr/lib/libgstintercept.so DYLD_FORCE_FLAT_NAMESPACE= GST_DEBUG_DUMP_TRACE_DIR=. gst-launch-1.0 audiotestsrc num-buffers=1000 ! vorbisenc ! vorbisdec ! fakesink
$ ls *.gsttrace
ls: cannot access '*.gsttrace': No such file or directory

My arch package contains:

gst-instruments-git /usr/
gst-instruments-git /usr/bin/
gst-instruments-git /usr/bin/gst-instruments-1.0
gst-instruments-git /usr/bin/gst-report-1.0
gst-instruments-git /usr/bin/gst-top-1.0
gst-instruments-git /usr/lib/
gst-instruments-git /usr/lib/gstreamer-1.0/
gst-instruments-git /usr/lib/gstreamer-1.0/libgstinstruments.so
gst-instruments-git /usr/lib/gstreamer-1.0/libgstinstruments.so.0
gst-instruments-git /usr/lib/gstreamer-1.0/libgstinstruments.so.0.0.0
gst-instruments-git /usr/lib/libgstintercept.so
gst-instruments-git /usr/lib/libgstintercept.so.0
gst-instruments-git /usr/lib/libgstintercept.so.0.0.0
gst-instruments-git /usr/share/
gst-instruments-git /usr/share/gst-instruments/
gst-instruments-git /usr/share/gst-instruments/pull.svg
gst-instruments-git /usr/share/gst-instruments/push.svg

Refactoring: tracing library

We need to delegate parsing, tracing & logging into separate tracing library, think about its name.

Move to gst-devtools

I think big parts of that tool would have their place in gst-devtools, and the tracer itself might be moved to -core (not sure, I have not check the code well enough yet).

What do you think about that? I can handle the move if you agree about it.

unable to find gst-instrument-1.0 binary

Hi,
I am trying to run gst-instrument-1.0 on Linux 20.04, x86_64 system but I get a response back saying
gst-instrument-1.0: command not found

I am able to create trace file, view them using cmd,
gst-report-1.0 --dot pipeline0.gsttrace | dot -Tsvg > perf.svg
but not able to run using
gst-instruments-1.0 pipeline0.gsttrace

I didn't find any binary for gst-instrument-1.0 except the lib libgstinstruments.so

couldn't find task error

I've recorded some trace data but I cannot visualize the data file, all I get is:

gst-report-1.0 pipeline0.gsttrace 
couldn't find task 0x78696d01e12048

(gst-report-1.0:71207): GLib-CRITICAL **: 19:03:11.589: g_array_free: assertion 'array' failed
could not read trace file: pipeline0.gsttrace

Here's the recorded trace file (compressed) that triggers this issue:

pipeline0.gsttrace.gz

Trace files suffix should be put in configuration-time constant

.gsttrace occurrences should be replaced with substitutions.

Live mode support

Hi, I dont see support live pipeline, where periodically we have tracer info dumped. Can someone confirm this?

Also, is there a way to get tracer information through some callback function/gstBus instead of dumping into a file?

Refactoring: make tracing library be dynamic

Make tracing library be a dynamic .so, used by libgstintercept and tools.

Integrate with new tracing subsystem

It would be nice to use events logged by GStreamer tracing stuff

Implement pushing / pulling overhead calculation

Is gst-top utilization normalized?

Is the %CPU printout representing all cores in the system or just a single core?

That is if I have a 4 core system, and gst-top has a printout of 2.4% for a specific element, does that imply that for the duration of the trace, only 2.4% of the system 4 cores are used?

Error: <stdin>: syntax error in line 1 near '(' when ran gst-instruments-1.0 with trace file

I ran LD_PRELOAD=/usr/local/lib/aarch64-linux-gnu/libgstintercept.so GST_DEBUG_DUMP_TRACE_DIR=. ./app to generate a pipeline.gsttrace file.

gst-report-1.0 pipeline.gsttrace works but gst-instruments-1.0 pipeline.gsttrace give the following error:
Error: : syntax error in line 1 near '('

In my pipeline, I use Ctrl+C to stop the pipeline, not sure if this effects the tracer. Please advice.

Implicit declaration of function 'trace_add_entry'

Hi

I wanted to give a try to branch wip/trace but I have found the following issue:

While building:

/home/ricardo/curro/qt5022/build-qt5022/build/tmp/work/bobcat_64-poky-linux/gst-instruments/git-r0/git/libs/gst/intercept/gstintercept.c: In function 'dump_hierarchy_info_if_needed':
/home/ricardo/curro/qt5022/build-qt5022/build/tmp/work/bobcat_64-poky-linux/gst-instruments/git-r0/git/libs/gst/intercept/gstintercept.c:156:5: warning: implicit declaration of function 'trace_add_entry' [-Wimplicit-function-declaration]
     trace_add_entry (pipeline, g_strdup_printf ("element-discovered %p %s %s 0", pipeline, LGI_ELEMENT_NAME (pipeline), LGI_OBJECT_TYPE_NAME (pipeline)));
     ^
/home/ricardo/curro/qt5022/build-qt5022/build/tmp/work/bobcat_64-poky-linux/gst-instruments/git-r0/git/libs/gst/intercept/gstintercept.c: In function 'gst_element_set_state':
/home/ricardo/curro/qt5022/build-qt5022/build/tmp/work/bobcat_64-poky-linux/gst-instruments/git-r0/git/libs/gst/intercept/gstintercept.c:459:7: warning: implicit declaration of function 'gst_element_dump_to_file' [-Wimplicit-function-declaration]
       gst_element_dump_to_file (element, filename);
       ^

While running

root@qt5022:~/kk# gst-top-1.0 gst-launch-1.0 qteccamerasrc name=src src.vfsrc ! fakesink
Setting pipeline to PAUSED ...
gst-launch-1.0: symbol lookup error: /usr/lib/libgstintercept.so.0: undefined symbol: trace_add_entry

** (gst-top-1.0:3679): WARNING **: gst-launch-1.0 exited with code 32512

I have taken a quick look to the code and it seems to be a big refactoring on the way, therefore I did not dare to change the code.

I guess you are already aware of this...

Best regards and thanks again for gst-instruments

Want to get more information with gst-top.

gst-top provides cpu, time usage and data flow informations in each elements.
I want cpu cycles and instructions informations additionally.
Can I upload pull requests for this?

On OS X move main menu to the menu bar

Now main menu is inside the app's window, that's wrong.

Total CPU usage does not match with the summed up CPU usage reported by gst-report

Hello,

We tried the Linux command LD_PRELOAD=/usr/lib/libgstintercept.so.... to determine the individual plugin's CPU usage for a pipeline on a 8 core CPU. The summed up CPU usage does not match with the CPU usage reported by top command, which too was noted during run.

Any suggestion in this regard would help.

Thanks,
Subbarao

"Please install libtool" happens when run ./autogen.sh

"Please install libtool" happens when run ./autogen.sh

Distro

invain@mymate:/work/taos/gst-instruments$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.4 LTS"
invain@mymate:/work/taos/gst-instruments$ uname -a
Linux mymate 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

How to reproduce

s$ ./autogen.sh
Checking for autoreconf...
/usr/bin/autoreconf
Checking for pkg-config...
/usr/bin/pkg-config
Checking for aclocal...
/usr/bin/aclocal
Checking for libtool...
Please install libtool.

Unable to run gst-top-1.0

Hi,
I'm not able to run the core command gst-top-1.0 for any pipeline. Any time I run it the result is always the same:

gst-play-1.0: symbol lookup error: /usr/local/lib/libgstintercept.so.0: undefined symbol: for_each_buffer
** (gst-top-1.0:23818): WARNING **: gst-play-1.0 exited with code 32512

Could you please help me how to solve this?
Thx;-)

Any ETA on the next (0.2.4) release?

Pick license

Should I use LGPL v.2 or LGPL v.2.1 or LGPL v.3?

Use binary format for storing trace

This will help to decrease overhead, speed up parsing & ease refactoring.

Android how to measure cpu usage?

Hello,
In android, how to measure cpu usage? this project supports Android Well ?
I use libgstandroid_gstreamer.so, how can I generate Trace files?

How to load this in a custom pipeline?

We're trying to load this into our Python pipeline, however using LD_PRELOAD doesn't seem to have an effect. I've tried to load the plugin through the GstRegistry, but this has no effect?

Thanks for the help

gst-top-1.0: In case command failed don't show prev. results

Now gst-top-1.0 makes gst-report-1.0 parse trace in any case: this may result in showing old results left by last successful call.

gst-top freeze after "Freeing pipeline"

I'm trying to use gst-top on an Nvidia Xavier (Jetson platform). I was able to build it without problem, but when I execute gst-top-1.0 on my pipeline and then hit Ctrl-C to exit I get:

Interrupt: Stopping pipeline ...
Execution ended after 0:00:08.821753480
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

And it freezes at this point. I have to press Ctrl-C again to interrupt.

Any ideas about what might cause such behavior? I have tried with various pipelines and get the same...

Thanks!

Refactoring: get rid of data model in tool's folder

Get rid of extra code in instruments/ folder: headstones, data model stuff, rename to tools/.

Make OS X version be build-able with .pkg version of OS X

Now it can be only built with brew.

playbin3 not work ?

playbin2 work well,

gst-top-1.0 gst-launch-1.0 playbin uri=http://10.9.44.116/vod/ts/1080p/1080p.ts
gst-instruments-1.0 gst-top.gsttrace

playbin3 not work

gst-top-1.0 gst-launch-1.0 playbin3 uri=http://10.9.44.116/vod/ts/1080p/1080p.ts
gst-instruments-1.0 gst-top.gsttrace

Error: : syntax error in line 1 near '('
Error: : syntax error in line 1 near '('
Error: : syntax error in line 1 near '('
Error: : syntax error in line 1 near '('
Error: : syntax error in line 1 near '('
Error: : syntax error in line 1 near '('
Error: : syntax error in line 1 near '('

Breaks with playbin3

It breaks setting the environment variable USE_PLAYBIN3=1.

This breaks:

 USE_PLAYBIN3=1 gst-top-1.0 gst-play-1.0 <file>

This does not break:

 USE_PLAYBIN3=0 gst-top-1.0 gst-play-1.0 <file>

Error in terminal:

** (gst-top-1.0:15288): WARNING **: gst-play-1.0 exited with code 139

Make ui option be auto by default

Refactoring: make UI work directly with data model

Now it just calls tools and works as selection / preview app.