GL_NV_vdpau_interop not available on current context

[i-rinat]

GL_NV_vdpau_interop not available on current context

[i-rinat]

GL_NV_vdpau_interop is an OpenGL extension. It helps to connect VDPAU surfaces to GL textures. An application may want to avoid downloading data from video surface just to upload it to GL texture, and that extension adds functions which helps to avoid unnecessary data transfers. Since it's GL-related, libGL is the library which implements the extension.

When Nvidia's, or Mesa's [1] VDPAU drivers are used, it's possible, since they have control of both libGL.so and libvdpau_drivername.so.1. But with libvdpau-va-gl it's close to impossible, since it has no access to libGL internals.

I thought about hacking into libGL to intercept calls to glGetProcAddress, but never tried to implement it.

[1] Mesa also have that extension implemented for some Gallium3D-based drivers

[Queuecumber]

Interesting, do you know how hard the intercept call is expected to be to implement?

[i-rinat]

Now when I thought about it more, none of the dynamic tricks seems to be feasible. Memory hotpatching will fail due to memory protection. Even if it's possible to make executable memory writable, VDPAU driver can be loaded after application asks libGL for entry points. Creating a wrapper for libGL must proxy all libGL exports, which is insane. There could be also issues with duplicated symbols.

The only viable way is to patch Mesa for "hookbacks", i.e., if application ask any of VDPAU interop extensions, load libvdpau and return one of its predefined entries. libvdpau must be patched too.

Looks fragile.

[Queuecumber]

What about replacing the function pointer for glGetProcAddress when the vdpau library is loaded? I used something similar on windows several years ago to force an application to go through one of my functions whenever it tried to load a dll with the LoadLibrary function, no idea if Linux would allow such a thing.

[i-rinat]

What about replacing the function pointer for glGetProcAddress when the vdpau library is loaded?

Application can try to acquire GL extension functions before it even tries to load libvdpau.

[Queuecumber]

And I'm assuming that function call will fail reporting that the extension is unavailable at that point right?

[i-rinat]

that the extension is unavailable at that point right?

Right.

[i-rinat]

By the way, why do you want that extension? The only application requiring it I know, is Kodi (XBMC). But it have native VA-API support.

[Queuecumber]

Steam in-home streaming but I just got it working with VA-API anyway. It seems you're right that almost everything is supporting both apis at this point so it is probably not necessary