Comments (17)
Hi, somebody pinged me about this problem for help. The easiest way to check what your app exactly uses is to generate a pipeline dot file to some directory GST_DEBUG_DUMP_DOT_DIR=/some/dir
.
AFAIU from quick glance at your code, you use GTK media stream to play video in your app and set it as Shell background, right? GTK media stream is a very simple and basic example for testing video playback within GTK4, but in order to maximize performance and reduce CPU usage as much as possible, what you really should do is use custom GStreamer pipeline instead. This way you can for example not software decode audio at all instead just muting it, etc.
from gnome-ext-hanabi.
About the Clapper player API that you mentioned, will that be a library like libvlc or libmpv, but for Gtk4?
Something like GstPlay
API that GStreamer offers but easier to use for people not familiar with GStreamer and hopefully not as buggy. Right now I am using modified version of their older API in Clapper but I want to write and switch into using my own that is easier to understand, has more cool features and works better with (but not limited to) GTK4. Feel free to ask any Clapper related stuff on Clapper matrix channel π
from gnome-ext-hanabi.
Hi @VarLad, thanks for reporting. I'm not an expert to this actually, so I might be wrong.
It depends on what kind of GPU you have. The backend used here is gstreamer.
First, try to get some info with gst-inspect
. If you are using NVidia card, this should be expected output:
β― gst-inspect-1.0 nvcodec
Plugin Details:
Name nvcodec
Description GStreamer NVCODEC plugin
Filename /usr/lib64/gstreamer-1.0/libgstnvcodec.so
Version 1.20.3
License LGPL
Source module gst-plugins-bad
Source release date 2022-06-15
Binary package Fedora GStreamer-plugins-bad package
Origin URL http://download.fedoraproject.org
cudadownload: CUDA downloader
cudaupload: CUDA uploader
nvh264dec: NVDEC h264 Video Decoder
nvh264enc: NVENC H.264 Video Encoder
nvh264sldec: NVDEC H.264 Stateless Decoder
nvh265dec: NVDEC h265 Video Decoder
nvh265enc: NVENC HEVC Video Encoder
nvh265sldec: NVDEC H.265 Stateless Decoder
nvjpegdec: NVDEC jpeg Video Decoder
nvmpeg2videodec: NVDEC mpeg2video Video Decoder
nvmpeg4videodec: NVDEC mpeg4video Video Decoder
nvmpegvideodec: NVDEC mpegvideo Video Decoder
nvvp8dec: NVDEC vp8 Video Decoder
nvvp8sldec: NVDEC VP8 Stateless Decoder
nvvp9dec: NVDEC vp9 Video Decoder
nvvp9sldec: NVDEC VP9 Stateless Decoder
16 features:
+-- 16 elements
If you are using Intel/AMD, I guess this should be the expected one:
β― gst-inspect-1.0 vaapi
Plugin Details:
Name vaapi
Description VA-API based elements
Filename /usr/lib64/gstreamer-1.0/libgstvaapi.so
Version 1.20.3
License LGPL
Source module gstreamer-vaapi
Source release date 2022-06-15
Binary package gstreamer-vaapi
Origin URL Unknown package origin
vaapidecodebin: VA-API Decode Bin
vaapih263dec: VA-API H263 decoder
vaapih264dec: VA-API H264 decoder
vaapimpeg2dec: VA-API MPEG2 decoder
vaapimpeg4dec: VA-API MPEG4 decoder
vaapipostproc: VA-API video postprocessing
vaapisink: VA-API sink
vaapivc1dec: VA-API VC1 decoder
8 features:
+-- 8 elements
From gstreamer documentation, it says
Sneak peek: if properly setup, you do not need to do anything special to activate hardware acceleration; GStreamer automatically takes advantage of it.
So, if the process has a high CPU usage, probably there is something wrong with your setup. Or the video you are trying to play is not HW acceleration supported by gstreamer.
If your GPU use VA-API (Intel/AMD), then you should also check vainfo
and see if there is any entrypoint.
Some articles that might be helpful:
https://wiki.archlinux.org/title/Hardware_video_acceleration
https://lifestyletransfer.com/how-to-install-gstreamer-vaapi-plugins-on-ubuntu/
from gnome-ext-hanabi.
That being said, it seems that I also couldn't get the HW acceleration working on my side. (High CPU usage when compare to other video player, for example Clapper which is also gstreamer-based AFAIK)...
Need some investigation. Any advice is welcome!
from gnome-ext-hanabi.
@jeffshee I think for gstreamer plugin for vaapi we have gstreamer1-vaapi
package on Fedora
I'm not near my machine right now, but do tell if installing that makes a difference for you
from gnome-ext-hanabi.
@VarLad This extension doesn't use anything from Flathub. It is not a Flatpak. So the HW acceleration setup on host must be done correctly.
from gnome-ext-hanabi.
@jeffshee Sorry, I mistyped, but I quickly edited the issue :3
Again, I'd recommend to see if installinggstreamer1-vaapi
package in Fedora makes a difference.
from gnome-ext-hanabi.
Again, I'd recommend to see if installing
gstreamer1-vaapi
package in Fedora makes a difference.
Yep, we should experiment with this first. Did installing gstreamer1-vaapi
work for you? (It doesn't work for me tho)
from gnome-ext-hanabi.
Didn't work for me either :<
On the other hand, @jeffshee I found https://gstreamer.freedesktop.org/documentation/tutorials/playback/hardware-accelerated-video-decoding.html?gi-language=c
and I remember that Clapper too had different ranks for hw-decoding too like this
Can you try setting the va plugin in gstreamer (should handle most formats)?
from gnome-ext-hanabi.
The creator of Clapper themselves :D
@Rafostar Any idea why hardware decoding is not working here?
from gnome-ext-hanabi.
@Rafostar, wow thanks for the advice! (I'm a Clapper user too =3)
TBH, Gstreamer stuff is currently outside of my expertise. I managed to generate the dot files but have no idea what info I should look for. I attached mine here if you don't mind taking a look =)
pipeline.zip
GTK media stream is a very simple and basic example for testing video playback within GTK4, but in order to maximize performance and reduce CPU usage as much as possible, what you really should do is use custom GStreamer pipeline instead.
It's indeed very sad to know that GTK4's implementation sometimes lacks this and that. Is that mean I have to follow the path that Clapper is doing (Gstreamer -> OpenGL -> Gtk4)? How much work needs to be done, if all I need is just a simple window playing video in a loop, with HW acceleration? Any documentation suggestions?
from gnome-ext-hanabi.
@jeffshee
Thanks for the dot files. As I suspected, it IS using hardware acceleration for decoding π. The example you send was using VP9 hw accel with Nvidia binary driver (nvvp9dec
element).
The problem here is that it does not go with GLMemory
but raw frames. Basically it goes like this: frame is hw decoded, then it is downloaded from GPU memory to system memory (which already puts pressure on system memory bandwidth), later software colorspace conversion happens and its uploaded back to GPU within GTK4 so it can be shown. All this results in high CPU usage that you see.
There is no much control from user side over GTK media stream implementation "auto" behaviour as it doesn't have access to underlying GStreamer pipeline at all (BTW: I do not use it in Clapper either). But do you even need a GUI toolkit for anything here? AFAIK you just want to spawn a process that shows video only, no?
from gnome-ext-hanabi.
@Rafostar
Thanks for the detailed explanation, that makes a lot of sense! π
But do you even need a GUI toolkit for anything here? AFAIK you just want to spawn a process that shows video only, no?
Yep, probably a process that allows some basic control over its spawned window, is enough. For the extension to work, the window needs to have a magic string @io.github.jeffshee.hanabi_renderer!0,0;H
as its title. Some basic window operations such as minimize, full-screen, and disable window decoration, are also desirable.
from gnome-ext-hanabi.
Some basic window operations such as minimize, full-screen, and disable window decoration, are also desirable.
@jeffshee
That's quite a few features for a wallpaper π. For something easy to implement and universal (works on both xorg, wayland) without much code changes, one way would be using for e.g. GstPlay
API + some video sink. There is video overlay renderer iface in there that uses window handle, but AFAIK GJS does not have bindings for this functionality AND this is not working with Wayland.
An easy to do replacement here would be using GstPlay
API + clapper sink, but this would introduce a Clapper dependency to your extension, something that you probably do not want doing. Other then that, you could investigate why GtkMedia sink decides to use raw frames. But as I mentioned earlier, its very limited and (unlike GstPlay) it doesn't give you option to not decode audio.
from gnome-ext-hanabi.
@Rafostar
Thanks! I think I should go through the Gstreamer docs and look at Clapper's codes to understand better how things work under the hood. I might ask you some questions in Clapper's Matrix if you don't mind! =3
An easy to do replacement here would be using GstPlay API + clapper sink, but this would introduce a Clapper dependency to your extension, something that you probably do not want doing.
I'm not against the idea of ββintroducing a Clapper dependency. If the player stuff can't fit in a simple GJS script, I would rather separate them from the extension part and package them into Flatpaks~ Users will have to install 2 separate components tho, which is... not too bad IMO.
from gnome-ext-hanabi.
@jeffshee
If you are not against option of using Clapper for improved performance, then my suggestion to make this universal would be: do a little code refactor and use Clapper sink when installed otherwise fallback to current GtkMediaStream implementation (your current code).
I can help you with this by doing a MR, so you won't have to reverse engineer how some parts in Clapper work π. Just let me know. I do work on some easy-to-use Clapper player API, its not ready yet, but since Clapper 0.5 the sink alone is installed and publicly available as first step. For now can be used with GStreamer provided GstPlay
API which should be more then enough for your use-case here (looping a video), and maybe even your other project if you decide to port it to GTK4: jeffshee/hidamari#64 instead of bundling VLC there.
Remember, that this is only my suggestion. I do not want to put a pressure/force you into this if you do not want to.
from gnome-ext-hanabi.
I can help you with this by doing a MR, so you won't have to reverse engineer how some parts in Clapper work smile. Just let me know.
That sounds great! I would appreciate it if you could help me out with Gstreamer stuff! π
About the Clapper player API that you mentioned, will that be a library like libvlc
or libmpv
, but for Gtk4? If that is the case, I think it will be an awesome addon to the Gtk4 itself!
from gnome-ext-hanabi.
Related Issues (20)
- Can not play videos on Fedora. HOT 3
- Auto Pause on Window Focus or Maximize HOT 2
- Wallpaper only showing in correct resolution when using ./run.sh renderer HOT 4
- Reload Feature
- Background not looping HOT 2
- [Question] Support for Cinnamon?
- toggle to prioritize intel over nvidia
- Can hear audio, change wallpapers, but can't see anything. Just a black screen. HOT 1
- Pause video during busy hours
- Corners in overview look wrong
- Fedora 40 after codec update black screen HOT 2
- [FEATURE REQUEST] Integrate Wallpaper Engine HOT 1
- Using wine staging makes a window appear.
- gif wallpaper freezes at the end of it HOT 2
- Black screen
- Can't install hanabi in Pop!_OS 22.04
- Corner rounding is bugged in the overview HOT 1
- extension can't turn on HOT 1
- Using a short video will cause memory leaks HOT 1
- Frozen video background on boot
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gnome-ext-hanabi.