Support streaming for RenderTexture and GameView

[unity-technologies]

Support streaming for RenderTexture and GameView

[karasusan]

We will release the new version of the package until the end of April.
WebRTC API is changed a lot, please see the reference and let me know if your problem hasn't solved yet.
https://docs.unity3d.com/Packages/[email protected]/api/Unity.WebRTC.VideoStreamTrack.html

[karasusan]

This issue is reported on Unity Forum. It seems a bug caused by the method, not WebRTC package.
https://forum.unity.com/threads/resulting-screenshot-image-in-game-is-upside-down-and-not-correctly-positioned.848230/

Regarding the support for RenderTexture, the new API for VideoStreamTrack looks great.

Regarding the GameView, I guess it's a bug related to the Engine in general.
In the attached image, you can see that CaptureScreenshotIntoRenderTexture from a GameView of 480*640px to a RenderTexture of the same size, outputs cut and flipped image to the RenderTexture.

[De-Panther]

I didn't test it(need to upgrade my project to 2019.3), but from the API it looks good.
So sure, you can close.

Thanks for adding the feature :)

[sindharta]

Can you please elaborate what you mean by streaming the GameView in the editor ?

In the sample we provided, we are already showing the rendering results on both the browser, and the editor.

[De-Panther]

Streaming the editor tab Game View(And not the game view in a compiled game), to a WebRTC instance(web page, android app, etc...).
The current package only enables to stream a camera, without screen space overlay canvases...
Thanks

[sindharta]

The template does the following

1. Renders to a render texture
2. Streams the render texture to the network (WebRTC instance)
3. Blits the render texture in the game view

So we have the rendered results showing in both the WebRTC instance, and the game view in the editor.

The code to blit the render texture only exists in the template, not in the package, so perhaps this is the part you are looking for ?

[De-Panther]

The constructors of MediaStream are internals:

internal MediaStream(RenderTexture[] rts, IntPtr ptr)
internal MediaStream(IntPtr ptr)

The Context in WebRTC is internal:
internal static Context Context { get { return s_context; } }

I need to be able to call MediaStream(RenderTexture[] rts, IntPtr ptr) from my own code, so I'll be able to stream my own RenderTexture, and not just use
public static MediaStream CaptureStream(this Camera cam, int width, int height)
Which set a RenderTexture on a single camera
cam.targetTexture = rts[0];

I want to be able to stream the game view tab, including canvases that are set to screen space overlay, and not just canvases that are set to screen space camera.
Similar to how Unity Remote apps works.
https://play.google.com/store/apps/details?id=com.unity3d.genericremote

Thanks

[sindharta]

RenderTexture is required to send the rendered result to the WebRTC plugin (which will then be encoded and sent to the network).
Which means that, you still need to render your screen space overlay canvases into a texture.

I think we can probably add another CaptureStream method that accepts RenderTexture directly.
Is this something that will work for you ?

[De-Panther]

Yes. CaptureStream for RenderTexture would be great.
And if it's possible to also add CaptureStream for the GameView tab it'll be nice.

Thanks

[gtk2k]

+1
I also want a CaptureStream that accepts RenderTexture directly Or Public MediaStream.

Case 1
https://assetstore.unity.com/packages/tools/camera/super-cubemapper-360-texture-sharing-154428
This asset renders an Equirectanguler in a RenderTexture. I want to send this RenderTexture.

Case 2
I have implemented a paint function using RenderTexture and want to send it.

[De-Panther]

Regarding the support for RenderTexture, the new API for VideoStreamTrack looks great.

Regarding the GameView, I guess it's a bug related to the Engine in general.
In the attached image, you can see that CaptureScreenshotIntoRenderTexture from a GameView of 480*640px to a RenderTexture of the same size, outputs cut and flipped image to the RenderTexture.

[karasusan]

Would you mind that I close this issue?
Please let me know if anyone has problems related to the issue.