accelerated windowsless mode on Windows

[Ivshti]

accelerated windowsless mode on Windows

[jaruba]

I think this is related to this post: nwjs/nw.js#3293 (comment)

[Ivshti]

Thanks, this seems interesting.

Still, passing frames and drawing them in canvas will obviously be terrible in performance.

Drawing in an OpenGL context would be best, and I don't know how the bindings would happen.

[RSATom]

The problem is there are no way combine accelerated output with windowless mode (and have some custom controls over video) on Windows platform. It's exactly the reason why WebChimera was created.

On Mac OS X all is totally different - all UI there is hardware accelerated by design, so windowless NPAPI get advantages from this too.

The app is NW.js-based. I am thinking that if there was a limitation before, currently with the latest Chromium codebase there would certainly be a way to wire in an NPAPI plugin directly into the browser composing, avoiding the need to have it's own window and still having hardware acceleration.

To have accelerated output, we have to use some sort of hardware accelerated drawing engine, but NPAPI on windows could work only with HWND, so we have two choices:

1. use GDI (which was created in 199x and know nothing about hardware acceleration) - and have poor performance;
2. create DirectX/OpenGL view over passed from browser HWND - and have problem with items overlapping;

One possible solution could be pass WebGL context from browser to NPAPI plugin - then browser will know how embed it into html renderer seamlessly - but don't think it will be created ever since NPAPI declared as deprecated.

[Ivshti]

So, from what I understand, Windows has this problem by design.

And how is WebChimera's performance compared to VLC? Is there any overhead there, maybe frame copying related to the Qt integration?

[RSATom]

Obviously it has some overhead, but on desktop platforms it's not significant. Of course some additional optimizations are possible, but since nobody complain on WebChimera performance - I prefer spend time on more important things...

[RSATom]

btw, on Mac platform WebChimera has even more overhead than on Windows - since it renders to texture at least two times: first time for video, second when renders final scene to CALayer (which is some sort of texture on Mac OS X).

[Ivshti]

I would have to disagree here. I don't know how much the overhead is, I am yet to research that, but if it's significant enough, it could be a big hit on energy consumption over watching a whole movie - e.g. consuming 10% on native playback and 50% on a non-optimized video player.

Again, I don't know how significant Webchimera's overhead is.

[RSATom]

Using JS to write application is the first place overhead :)

[RSATom]

so if you mind about energy consumption - it will be better use native application written on C++ or something similar.

[Ivshti]

You're generally right. The design was to use native for every critical component, such as the video - and since HTML5 provides a video tag, this design works fine. Until more codecs and features are required, and it becomes obvious re-compiling with more ffmpeg flags won't do the entire job.

WebChimera is probably the best solution here. Or using libvlc for a new NPAPI plugin, then re-writing the controls natively for every OS, but this would be time consuming.

[RSATom]

Anyway WebChimera adds just a few percents of additional cpu load compared to vlc player. On my system something like 3-5% instead of 2-3% - but this only on some specific movie - I didn't do any special measurements.
If you will do something like this, I will be glad to see your results. And even I think it could be useful to other WebChimera users.

[RSATom]

WebChimera is probably the best solution here.

Maybe, at least I don't know any alternative.

Or using libvlc for a new NPAPI plugin, then re-writing the controls natively for every OS, but this would be time consuming.

Even if you will use native controls over libvlc output window - you could have some glitches. I've already met this issue when did toolbar for Vlc Web Plugins on Windows.

[Ivshti]

Something like 3-5% vs 2-3% seems insignificant and would maybe result in 20-30% battery usage increase, which at the point would not be worth to dig into.

Out of curiosity though - why Qt? Would it be possible to simply overlay a transparent WebKit instance over VLC? Would it be more performant?

[RSATom]

Out of curiosity though - why Qt?

Just because it has QML which allow define additional controls and effects in plain text file, which could be loaded over network for every specific application. I.e. something like html but more simple and hardware accelerated by design.

Would it be possible to simply overlay a transparent WebKit instance over VLC?

I think it's possible, but didn't try. But this approach has one disadvantage: you could not apply any effects to video. For example you could apply huge amount different effects directly to video (transitions, transparency, rotation, moving, scaling, blur and many many others. look http://doc.qt.io/qt-5/graphicaleffects.html ) with WebChimera, not only place additional controls.

Would it be more performant?

It could be, but it's just not so interesting as WebChimera :) And I don't sure how complicated is task to place html engine over OpenGL/DirectX view (libvlc use them for video output too).

[Ivshti]

I see. One more question - where does the overhead in WebChimera come from exactly. A few percent sound too little to be from copying buffers.

I'll soon test playing a 1080p tv show on OS X / Windows with VLC, HTML5 video and WebChimera and post the results (CPU usage)

Btw it's amusing, one or two years ago when we were still in the prototype phase of our app, we found that windowless vlc is extremely heavy, I wrote a Qt + VLC plugin (not firebreath, but qt-npapi or something like that) and used it with QML. Dropped it because HTML5 video seemed sufficient at the time and it was too tiresome to compile every time.

[RSATom]

I see. One more question - where does the overhead in WebChimera come from exactly. A few percent sound too little to be from copying buffers.

Yes, exactly. Libvlc decodes video frames to buffer in main memory, after that I upload them to texture (with texImage2d). One possible optimization is decode directly to GPU memory with something like PBO. You could read following discussion about used in WebChimera approach: https://forum.videolan.org/viewtopic.php?f=32&t=121786

And plus QML engine itself consume some amount of CPU and GPU time.

Btw it's amusing, one or two years ago when we were still in the prototype phase of our app, we found that windowless vlc is extremely heavy, I wrote a Qt + VLC plugin (not firebreath, but qt-npapi or something like that) and used it with QML. Dropped it because HTML5 video seemed sufficient at the time and it was too tiresome to compile every time.

Good ideas have to come to more than one head :) I've started work on WebChimera in december 2013

[Ivshti]

Now this is mega-curious. I expected bigger differences if buffer copying is involved.

OS X
VLC - 33-38%
HTML5 - 40-45%
WebChimera - 40-46%

Windows
VLC - 5-7%
HTML5 - 8-9% (but split between two NW.exe processes, one using 2-3%)
WebChimera - 8-9%

[RSATom]

Funny, so WebChimera even better than we thought :)

Btw what hardware do you have with OS X ?

[Ivshti]

iMac, late 2013, i5, 8GB RAM. I don't know why the % is so high compared to the Windows, which is i7 Dell Vostro laptop from 2013. Maybe just because of software differences. It didn't feel "heavy" and I didn't feel a system slowdown when playing the video - and it played perfectly. The difference was purely in the numbers.

[RSATom]

hm... looks really strange. Unfortunately can't remember what load I had on Mac.

[Ivshti]

It could be related to I/O. I have a fusion drive and apparently Apple implemented it badly, because all of my I/O is really slow.

Also, could be that the OS X activity monitor shows % of the usage of one core/thread, and the Windows one showing the % of usage of the entire processor. In this case we divide all the OS X % by 4 or 8 (depending on core or thread).

Dividing by four - OS X
VLC 8.25-9.5%
HTML5 10-11%
WEbChimera - 10-11%

That makes more sense. Considering the performance difference between i5 and i7 everything fits in.

[RSATom]

@Ivshti, can we close this issue?

[Ivshti]

Sure. It's no longer relevant.