berglie/gstreamer-d3d11-overlay

An rtsp client using GStreamer with Direct3D11/Direct3D9 interop layer

gstreamer-d3d11-overlay

Using GStreamer with Direct3D11/Direct3D9 interop layer in .Net

There is no direct support in WPF to display UI elements over a D3D component due to the airspace issue. This project will tackle this issue by implementing a Direct3D11/Direct3D9 interop layer and by using the D3DImageEx class.

The D3DImageEx class will create a D3D11Texture which is created with the D3D11_RESOURCE_MISC_SHARED. This texture will be used for the SetBackBuffer method and later used for the d3d11videosink element in the gstreamer code. Sample project

Sample project

GStreamer is a framework used for creating streaming media applications. In the sample project included in this repository we are creating a player which can play RTSP streams in WPF with a textbox overlay.

Using the NuGet package

Download the NuGet package from https://www.nuget.org/packages/gstreamer-d3d11-overlay/

Using the native C++ projects

What You Need

• Visual Studio 2015
• DirectX SDK
• Windows SDK

Credits

The Direct3D11/Direct3D9 interop layer is based on the project http://jmorrill.hjtcentral.com/Home/tabid/428/EntryId/437/Direct3D-10-11-Direct2D-in-WPF.aspx by Jeremiah Morrill.

gstreamer-d3d11-overlay's Issues

Video is choppy due to Dispatcher.Invoke method

Change Dispatcher.Invoke to Dispatcher.BeginInvoke to avoid unnecessary UI blocking

Add custom width and height input for d3d11 texture

Force shutdown application on "Output window was closed" error

Application is not properly shutdown when closing window and receiving "Output window was closed".

Create a NuGet package from the src files

Create a NuGet package so that users doesn't have to include the c++ projects to use the source code

Improve CPU usage by using d3d11 as primary video decoder

CPU usage using the default primary video decoder (libav) will drop down from about 3% to 0.5% using the d3d11 decoder d3d11h264dec.

Change SDK from legacy DirectX SDK to Win 10 SDK references

Make D3D11Scene disposable

Unloading / loading the player causes a memory leak. Release objects on finalizer

Sink.Emit error

Please also see link: https://stackoverflow.com/questions/69114775/c-sharp-gstreamer-d3d11-overlay-sink-emit-pointer-error-on-render

I'm developing a C# WPF application using gstreamer-sharp-netcore(Mingw v1.19.1) on Windows. What we would like to achieve is to be able to render video with hardware acceleration. To avoid the airspace issue in WPF, we are using gstreamer-d3d11-overlay(https://github.com/berglie/gstreamer-d3d11-overlay/).

From the sample, there are 2 ways to initiate rendering the video(MainWindow.cs):

1. Show video by giving the MainWindow handle to GStreamer. This will result in video taking over the whole the WPF. This piece of code works, but we won't be able to add any WPF controls.

    var windowHandle = new WindowInteropHelper(System.Windows.Application.Current.MainWindow).Handle;
    _playback = new Playback(windowHandle, _enableOverlay); 

2. Another way what we would like to use is to make a image object in WPF. Create a D3dImageEx and D3D11TestScene and attach it to the image object.

    _d3DImageEx = new D3DImageEx();
    _D3D11Scene = new D3D11TestScene(1920, 1080);
    d3dScene.Source = _d3DImageEx;
    /* Set the backbuffer, which is a ID3D11Texture2D pointer */
    IntPtr renderTarget = _D3D11Scene.GetRenderTarget();
    IntPtr backBuffer = _d3DImageEx.CreateBackBuffer(D3DResourceTypeEx.ID3D11Texture2D, renderTarget);

    _d3DImageEx.Lock();
    _d3DImageEx.SetBackBuffer(D3DResourceType.IDirect3DSurface9, backBuffer, _enableSoftwareFallback);
    _d3DImageEx.Unlock();

    _playback = new Playback(IntPtr.Zero, _enableOverlay);
    _playback.OnDrawSignalReceived += VideoSink_OnBeginDraw;

    CompositionTarget.Rendering += CompositionTarget_Rendering;

When this is successfully initialized, Gstreamer will render the video and fire the OnDrawSignalReceived:

    private void VideoSink_OnBeginDraw(Element sink, GLib.SignalArgs args)
    {
        var sharedHandle = _D3D11Scene.GetSharedHandle();
        _ = sink.Emit("draw", sharedHandle, (UInt32)2, (UInt64)0, (UInt64)0);
    }

On the line: "sink.Emit" we get a error:

System.Reflection.TargetInvocationException
  HResult=0x80131604
  Message=Exception has been thrown by the target of an invocation.
  Source=System.Private.CoreLib
  StackTrace:
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Delegate.DynamicInvokeImpl(Object[] args)
   at Gst.DynamicSignal.OnMarshal(IntPtr closure, Value& retval, UInt32 argc, IntPtr argsPtr, IntPtr ihint, IntPtr data)

  This exception was originally thrown at this call stack:
    Gst.DynamicSignal.Emit(GLib.Object, string, object[])
    Gst.Object.Emit(string, object[])
    TestHacoustoGst2D3D11WpfApp.MainWindow.VideoSink_OnBeginDraw(Gst.Element, GLib.SignalArgs) in MainWindow.xaml.cs
    TestHacoustoGst2D3D11WpfApp.Playback.VideoSink_OnBeginDraw(object, GLib.SignalArgs) in Playback.cs

Inner Exception 1:
ApplicationException: Invalid parameter type: expected , got System.IntPtr

Does anyone know how to solve this issue? That would be greatly appreciated :)

Property not found

I got property not found exception at the following line in Playback.cs.

_videoSink["draw-on-shared-texture"] = true;

I don't think d3d11videosink has draw-on-shared-texture property.

Can you please tell me what I should do for this exception?

Video stream is disappearing when going to lock screen

Enable software rendering fallback when the front buffer gets unavailable.
See https://docs.microsoft.com/en-us/dotnet/api/system.windows.interop.d3dimage?view=net-5.0