3dstreamingtoolkit/3dstreamingtoolkit

A toolkit for building powerful stereoscopic 3D experiences that run on the cloud and stream to devices.

3dstreamingtoolkit's Issues

MediaEngineUWP fails to build in x64 configuration.

The error message is 'uuid': attribute not found.

StreamingUnityServerPlugin compiled architecture differences

It seems that when running with Unity5.6.3f1 there is a behavior difference between x64 and x86. namely, on x86, UnityPluginLoad(IUnityInterfaces* unityInterfaces) is not being called.

If we inspect the produced binaries, we do see different name mangling in effect:

dumpbin name mangling inspection results

Dump of file C:\Users\begreen\git\3dtoolkit\Build\x86\Release\StreamingUnityServerPlugin\StreamingUnityServerPlugin.dll

File Type: DLL

  Section contains the following exports for StreamingUnityServerPlugin.dll

    00000000 characteristics
    59B80623 time date stamp Tue Sep 12 12:06:59 2017
        0.00 version
           1 ordinal base
           6 number of functions
           6 number of names

    ordinal hint RVA      name

          1    0 000C69E0 Close = _Close
          3    1 000C6BB0 SetInputDataCallback = _SetInputDataCallback
          4    2 000C6C30 SetLogCallback = _SetLogCallback
          2    3 000C6B30 _GetRenderEventFunc@0 = _GetRenderEventFunc@0
          5    4 000C6890 _UnityPluginLoad@4 = _UnityPluginLoad@4
          6    5 000C6AA0 _UnityPluginUnload@0 = _UnityPluginUnload@0

  Summary

      17A000 .data
        1000 .gfids
      15D000 .rdata
       2F000 .reloc
        3000 .rodata
        1000 .rsrc
      647000 .text
        1000 .tls

C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise>dumpbin /exports "C:\Users\begreen\git\3dtoolkit\Build\x64\Release\StreamingUnityServerPlugin\StreamingUnityServerPlugin.dll"
Microsoft (R) COFF/PE Dumper Version 14.10.25019.0
Copyright (C) Microsoft Corporation.  All rights reserved.
                                                                                                                     
Dump of file C:\Users\begreen\git\3dtoolkit\Build\x64\Release\StreamingUnityServerPlugin\StreamingUnityServerPlugin.dll

File Type: DLL

  Section contains the following exports for StreamingUnityServerPlugin.dll

    00000000 characteristics
    59B80A1C time date stamp Tue Sep 12 12:23:56 2017
        0.00 version
           1 ordinal base
           6 number of functions
           6 number of names

    ordinal hint RVA      name

          1    0 000D1ED0 Close = Close
          2    1 000D2060 GetRenderEventFunc = GetRenderEventFunc
          3    2 000D20F0 SetInputDataCallback = SetInputDataCallback
          4    3 000D2190 SetLogCallback = SetLogCallback
          5    4 000D1DB0 UnityPluginLoad = UnityPluginLoad
          6    5 000D1FC0 UnityPluginUnload = UnityPluginUnload

  Summary

      17F000 .data
        1000 .gfids
       3E000 .pdata
      1A9000 .rdata
        B000 .reloc
        3000 .rodata
        1000 .rsrc
      762000 .text
        1000 .tls

As per MSDN this is expected, in that x64 binaries do not get mangled at all.

My current question is - is the difference in behavior a result of unity not being able to find the correct function due to name mangling in x86?

Chrome client needs support for heartbeat

Updates to add heartbeat broke the chrome web client sample, need to add heartbeat functionality.

unity client turn failure, credential mismatch

in the unity sample client turn server configuration will not work, due to a bug in the mapping from the config username and password to the data object username and credential. The client will automatically fallback to stun (using google stun) or lan direct ip communication.

WebRTCUWP clients cannot create/establish connections

UnityClient and HololensUWP are both unable to be the establishing party (the one sending the sdp offer message) - this is likely an issue with h264 codec negotiation, and can be fixed with something like this webclient workaround.

This was surfaced by trying to add autoconnect behavior to the unity client by calling ConnectToPeer() from within the OnPeerConnected event.

We verified this issue is present in the UWP hololens client as well, which leads us to believe the issue is related to WebRTCUWP.

Refactor Nvenc implementation from direct to cuda

In order to support opengl and non-wibdows platforms, especially docker containerization of servers.

iOS client sample stretches the video frame in portrait mode

Once connected to a server, the iOS sample client stretches the video to be contained in the screen width.

The expected behavior is to keep the aspect ratio of the video frame and not stretch the image.

normalize uri configuration key names

for context, see #63 (comment)

we should rename turnServer.provider config value to turnServer.providerUri and authentication.authority to authentication.authorityUri to be more consistent and clear that these are uri values.

This includes updating the documentation in the wiki, the config stubs, and the parsing logic.

Refactor JSON config into isolated library

Currently we are read/writing to JSON configuration files from different places with a lot of duplicated code. We should refactor into a single lib to handle loading, creation and updating from JSON.

1. If json config isn't present, create file with defaults
2. Defaults stored in single location per config
3. Bad/missing elements gracefully default & provide log info

Primarily for nvencconfig & webrtcconfig - should we also abstract data channel payload schemas to json?

document configuring authentication in the wiki

configuring authentication should be pulled into a step-by-step in the wiki.

Related, See #88

setup script extracting zip failure is incorrectly handled

if setup.ps1 is run with invalid permissions, or with an error downloading the zip files, these files cannot be extracted and we do not handle this case well.

We should be able to handle Central directory corrupt exceptions and deal with this issue.

MultiThreadServer crashes with hardware encoding

Hi Team,

Here's another bug.

We have a powerful remoting server running Windows Server 2016.
I went through the setup process and built successfully. Then, I tried running the Server without the Graphic Card Encoding and works really good.

However, when I try the hardware encoding, the server just crashes immediately.

Attached are the captures related.

Thanks!

Wenbo

debug output

'MultithreadedServer.exe' (Win32): Loaded 'C:\Users\Paperspace\Documents\3dtoolkit\Build\x86\Release\MultithreadServer\MultithreadedServer.exe'. Symbols loaded.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\ntdll.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\kernel32.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\KernelBase.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\user32.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\win32u.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\gdi32.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\gdi32full.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\shell32.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\msvcrt.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\cfgmgr32.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\windows.storage.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\combase.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\ucrtbase.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\rpcrt4.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\sspicli.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\cryptbase.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\bcryptprimitives.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\sechost.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\powrprof.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\advapi32.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\shlwapi.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\kernel.appcore.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\SHCore.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\profapi.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\ws2_32.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\winmm.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\ole32.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\WinSxS\x86_microsoft.windows.common-controls_6595b64144ccf1df_5.82.14393.447_none_5507ded2cb4f7f4c\comctl32.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\IPHLPAPI.DLL'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\D3DCompiler_47.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\secur32.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\msdmo.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\winmmbase.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\cryptsp.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\imm32.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\uxtheme.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\msctf.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\oleaut32.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\msvcp_win.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Program Files (x86)\Common Files\Microsoft Shared\ink\tiptsf.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\dwmapi.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\d3d11.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\dxgi.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\DXGIDebug.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\bcrypt.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\System32\DriverStore\FileRepository\nv_dispswi.inf_amd64_f6d875fdd69bdc31\nvwgf2um.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\version.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\clbcatq.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\UIAutomationCore.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\userenv.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\oleacc.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\NapiNSP.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\mswsock.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\dnsapi.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\nsi.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\winrnr.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\nlaapi.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\rasadhlp.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\FWPUCLNT.DLL'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\MMDevAPI.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\propsys.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\devobj.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\winsta.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\rdpendp.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\wtsapi32.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\msacm32.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\MFWMAAEC.DLL'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\mfperfhelper.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\AudioSes.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\WinTypes.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\avrt.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\dhcpcsvc6.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\dhcpcsvc.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\XInput9_1_0.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\XInput1_4.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\nvEncodeAPI.dll'. Cannot find or open the PDB file.
'MultithreadedServer.exe' (Win32): Loaded 'C:\Windows\SysWOW64\dxva2.dll'. Cannot find or open the PDB file.
The thread 0x1614 has exited with code 0 (0x0).
Unhandled exception at 0x548BD4A1 in MultithreadedServer.exe: 0xC0000005: Access violation executing location 0x00000000.

Exception thrown at 0x00000000 in MultithreadedServer.exe: 0xC0000005: Access violation executing location 0x00000000.

issue screenshots

Client crashes on Windows 10 running on MacBook Pro 2013 via Bootcamp

Client crashes on Windows 10 10493 (Anniversary) running on MacBook Pro 2013 via Bootcamp. Debug dump shows WebRTC throwing the error about the audio device is unsupported in Windows core audio. This is trying to use a Cirrus Logic CS4208 (AB 94) audio adapter.

An issue was filed on WebRTC and a patch was merged in the project but would need to verify we are consuming the fixed branch of WebRTC: https://bugs.chromium.org/p/webrtc/issues/detail?id=7220.

In the meantime, if you disable the audio device in Windows, the client will work fine.

"uuid" attribute not found

When you try to build the project out of the box you need to disable being able to build for x64 (cause HoloLens).

test runners crash on supposedly supported hardware

This issue came out of #63 (comment).

On certain systems that should support nvencode with no issue (like my machine), we instead get an error (NV_ENC_ERR_DEVICE_NOT_EXIST ) trying to register the nvencode resource (m_pEncodeAPI->nvEncRegisterResource(m_hEncoder, &registerResParams);) which causes the test runner to fail.

system specs

OS Name	Microsoft Windows 10 Enterprise
Version	10.0.15063 Build 15063
System Type	x64-based PC
Processor	Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz, 3401 Mhz, 4 Core(s), 8 Logical Processor(s)
BIOS Version/Date	American Megatrends Inc. P2.90, 7/11/2013

Name	Intel(R) HD Graphics 3000
PNP Device ID	PCI\VEN_8086&DEV_0122&SUBSYS_01221849&REV_09\3&11583659&0&10
Adapter Type	Intel(R) HD Graphics Family, Intel Corporation compatible
Adapter Description	Intel(R) HD Graphics 3000
Adapter RAM	(2,084,569,088) bytes
Installed Drivers	igdumd64.dll,igd10umd64.dll,igd10umd64.dll,igdumd32,igd10umd32,igd10umd32
Driver Version	9.17.10.4459
INF File	oem38.inf (iSNBD0 section)
Color Planes	Not Available
Color Table Entries	Not Available
Resolution	Not Available
Bits/Pixel	Not Available
Memory Address	0xF7400000-0xF77FFFFF
Memory Address	0xD0000000-0xDFFFFFFF
I/O Port	0x0000FF80-0x0000FFBF
IRQ Channel	IRQ 4294967282
Driver	c:\windows\system32\drivers\igdkmd64.sys (9.17.10.4459, 5.13 MB (5,382,856 bytes), 3/9/2017 12:16 AM)
	
Name	NVIDIA GeForce GTX 1080 Ti
PNP Device ID	PCI\VEN_10DE&DEV_1B06&SUBSYS_85E21043&REV_A1\4&1CDA0A78&0&0008
Adapter Type	GeForce GTX 1080 Ti, NVIDIA compatible
Adapter Description	NVIDIA GeForce GTX 1080 Ti
Adapter RAM	(1,048,576) bytes
Installed Drivers	C:\WINDOWS\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_ce1961376673184c\nvldumdx.dll,C:\WINDOWS\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_ce1961376673184c\nvldumdx.dll,C:\WINDOWS\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_ce1961376673184c\nvldumdx.dll,C:\WINDOWS\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_ce1961376673184c\nvldumdx.dll,C:\WINDOWS\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_ce1961376673184c\nvldumd.dll,C:\WINDOWS\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_ce1961376673184c\nvldumd.dll,C:\WINDOWS\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_ce1961376673184c\nvldumd.dll,C:\WINDOWS\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_ce1961376673184c\nvldumd.dll
Driver Version	22.21.13.8541
INF File	oem42.inf (Section094 section)
Color Planes	Not Available
Color Table Entries	4294967296
Resolution	1920 x 1080 x 59 hertz
Bits/Pixel	32
Memory Address	0xF6000000-0xF70FFFFF
Memory Address	0xE0000000-0xF1FFFFFF
Memory Address	0xF0000000-0xF1FFFFFF
I/O Port	0x0000E000-0x0000EFFF
IRQ Channel	IRQ 16
I/O Port	0x000003B0-0x000003BB
I/O Port	0x000003C0-0x000003DF
Memory Address	0xA0000-0xBFFFF
Driver	c:\windows\system32\driverstore\filerepository\nv_dispi.inf_amd64_ce1961376673184c\nvlddmkm.sys (22.21.13.8541, 14.88 MB (15,600,248 bytes), 8/28/2017 10:26 AM)

User friendly UX for Web Client

The current web client is an experimental interface for development and testing. It could really use some UX improvements to show how a proper web client sample may operate.

Perhaps syncing to the https://appr.tc/ experience?

Accelerometer input does not work for the iOS client in landscape mode

When using the iOS client in landscape mode, the accelerometer input data does not work.

Normal app closure doesn't end session

If you use any of the native clients or native servers, and close the app using Alt-F4, or just clicking the close button, the other peer doesn't end the session and gets stuck in a zombie state.

support oauth2 refresh tokens

Our authentication work (currently in progress, see https://github.com/CatalystCode/3dtoolkit/commits/feature/native-auth) does not implement refresh tokens, so authentication only works for the duration of the initially retrieved access token.

We should support refresh tokens, to ensure connections can persist for longer than the original access token validity window.

Create IMediaSource for Audio Data Only (No Video stream)

Hello, we are using this project to create WebRTC calls from a web client to a HoloLens client. In some cases, we wish to send microphone audio only, no video.

Does the UWP webRTC implementation have any methods for creating an IMediaSource or equivalent for audio data alone? It appears that the only function to do this sort of thing takes a MediaVideoTrack as it's input. At the moment we are having the web client stream a blank video stream in order to get the source into a playable format in UWP webRTC.

Can't build StreamingDirectXHololensClient

The error we see is

3>C:\Program Files (x86)\MSBuild\Microsoft\.NetNative\x86\ilc\IlcInternals.targets(318,5): error : Error: Windows Runtime metadata is invalid in Windows SDK, follow the steps in http://go.microsoft.com/fwlink/?LinkId=733341 to repair your installation.

See the full build log

Restoring NuGet packages...
To prevent NuGet from restoring packages during build, open the Visual Studio Options dialog, click on the Package Manager node and uncheck 'Allow NuGet to download missing packages during build.'
1>------ Rebuild All started: Project: WebRtcWrapper, Configuration: Release x86 ------
2>------ Rebuild All started: Project: DirectxClientComponent, Configuration: Release Win32 ------
2> compilation object save succeeded; see C:\Users\leola\Documents\Medivis\StreamToolKit\3dtoolkit\Build\x86\Release\DirectxClient\GeometryShader.cso
2> compilation object save succeeded; see C:\Users\leola\Documents\Medivis\StreamToolKit\3dtoolkit\Build\x86\Release\DirectxClient\PixelShader.cso
2> compilation object save succeeded; see C:\Users\leola\Documents\Medivis\StreamToolKit\3dtoolkit\Build\x86\Release\DirectxClient\VertexShader.cso
2> compilation object save succeeded; see C:\Users\leola\Documents\Medivis\StreamToolKit\3dtoolkit\Build\x86\Release\DirectxClient\VPRTVertexShader.cso
2> MediaEngine.cpp
2> MediaEnginePlayer.cpp
2> MediaHelpers.cpp
2> CameraResources.cpp
2> DeviceResources.cpp
2> VideoRenderer.cpp
2> HolographicAppMain.cpp
2> AppCallbacks.cpp
1>C:\Users\leola\Documents\Medivis\StreamToolKit\3dtoolkit\Plugins\UnityClientPlugin\WebRtcWrapper\WebRtcControl.cs(30,55,30,80): warning CS0067: The event 'WebRtcControl.OnPeerDataChannelReceived' is never used
1>C:\Users\leola\Documents\Medivis\StreamToolKit\3dtoolkit\Plugins\UnityClientPlugin\WebRtcWrapper\WebRtcControl.cs(39,37,39,48): warning CS0649: Field 'WebRtcControl._ntpService' is never assigned to, and will always have its default value null
1>C:\Users\leola\Documents\Medivis\StreamToolKit\3dtoolkit\Plugins\UnityClientPlugin\WebRtcWrapper\WebRtcControl.cs(37,33,37,48): warning CS0414: The field 'WebRtcControl._peerVideoTrack' is assigned but its value is never used
1> WebRtcWrapper -> C:\Users\leola\Documents\Medivis\StreamToolKit\3dtoolkit\Build\x86\Release\WebRtcWrapper\WebRtcWrapper.dll
1> Target Plugin Path is C:\Users\leola\Documents\Medivis\StreamToolKit\3dtoolkit\Plugins\UnityClientPlugin\WebRtcWrapper......\Samples\Client\Unity\Assets
1> A subdirectory or file C:\Users\leola\Documents\Medivis\StreamToolKit\3dtoolkit\Plugins\UnityClientPlugin\WebRtcWrapper......\Samples\Client\Unity\Assets\Plugins already exists.
1> A subdirectory or file C:\Users\leola\Documents\Medivis\StreamToolKit\3dtoolkit\Plugins\UnityClientPlugin\WebRtcWrapper......\Samples\Client\Unity\Assets\Plugins\WSA already exists.
1> A subdirectory or file C:\Users\leola\Documents\Medivis\StreamToolKit\3dtoolkit\Plugins\UnityClientPlugin\WebRtcWrapper......\Samples\Client\Unity\Assets\Plugins\WSA\x86 already exists.
1> C:\Users\leola\Documents\Medivis\StreamToolKit\3dtoolkit\Build\x86\Release\WebRtcWrapper\WebRtcWrapper.dll
1> 1 File(s) copied
1> C:\Users\leola\Documents\Medivis\StreamToolKit\3dtoolkit\Build\x86\Release\WebRtcWrapper\WebRtcWrapper.pdb
1> 1 File(s) copied
1> C:\Users\leola\Documents\Medivis\StreamToolKit\3dtoolkit\Build\x86\Release\WebRtcWrapper\Org.WebRtc.dll
1> 1 File(s) copied
1> C:\Users\leola\Documents\Medivis\StreamToolKit\3dtoolkit\Build\x86\Release\WebRtcWrapper\Org.WebRtc.winmd
1> 1 File(s) copied
2>........\Plugins\UnityClientPlugin\MediaEngineUWP\Shared\MediaEnginePlayer.cpp(306): warning C4244: '=': conversion from 'float' to 'LONG', possible loss of data
2>........\Plugins\UnityClientPlugin\MediaEngineUWP\Shared\MediaEnginePlayer.cpp(307): warning C4244: '=': conversion from 'float' to 'LONG', possible loss of data
2>........\Plugins\UnityClientPlugin\MediaEngineUWP\Shared\MediaEnginePlayer.cpp(477): warning C4244: 'argument': conversion from 'UINT32' to 'float', possible loss of data
2>........\Plugins\UnityClientPlugin\MediaEngineUWP\Shared\MediaEnginePlayer.cpp(492): warning C4244: 'argument': conversion from 'UINT32' to 'float', possible loss of data
2>........\Plugins\UnityClientPlugin\MediaEngineUWP\Shared\MediaEnginePlayer.cpp(775): warning C4244: '=': conversion from 'float' to 'LONG', possible loss of data
2>........\Plugins\UnityClientPlugin\MediaEngineUWP\Shared\MediaEnginePlayer.cpp(776): warning C4244: '=': conversion from 'float' to 'LONG', possible loss of data
2>........\Plugins\UnityClientPlugin\MediaEngineUWP\Shared\MediaEnginePlayer.cpp(777): warning C4244: '=': conversion from 'float' to 'LONG', possible loss of data
2>........\Plugins\UnityClientPlugin\MediaEngineUWP\Shared\MediaEnginePlayer.cpp(778): warning C4244: '=': conversion from 'float' to 'LONG', possible loss of data
2>c:\users\leola\documents\medivis\streamtoolkit\3dtoolkit\samples\client\directxuwp\directxclientcomponent\AppCallbacks.h(25): warning C4453: 'Windows::ApplicationModel::Core::CoreApplicationView': A '[WebHostHidden]' type should not be used on the published surface of a public type that is not '[WebHostHidden]' (compiling source file AppCallbacks.cpp)
2>c:\users\leola\documents\medivis\streamtoolkit\3dtoolkit\samples\client\directxuwp\directxclientcomponent\AppCallbacks.h(27): warning C4453: 'Windows::UI::Core::CoreWindow': A '[WebHostHidden]' type should not be used on the published surface of a public type that is not '[WebHostHidden]' (compiling source file AppCallbacks.cpp)
2>Content\VideoRenderer.cpp(23): warning C4244: 'initializing': conversion from 'float' to 'int', possible loss of data
2>Content\VideoRenderer.cpp(24): warning C4244: 'initializing': conversion from 'float' to 'int', possible loss of data
2>Content\VideoRenderer.cpp(29): warning C4244: 'argument': conversion from 'int' to 'float', possible loss of data
2>Content\VideoRenderer.cpp(30): warning C4244: 'argument': conversion from 'int' to 'float', possible loss of data
2>Content\VideoRenderer.cpp(32): warning C4244: 'argument': conversion from 'int' to 'float', possible loss of data
2>Content\VideoRenderer.cpp(33): warning C4244: 'argument': conversion from 'int' to 'float', possible loss of data
2>Content\VideoRenderer.cpp(34): warning C4244: 'argument': conversion from 'int' to 'float', possible loss of data
2>Content\VideoRenderer.cpp(37): warning C4244: 'argument': conversion from 'int' to 'float', possible loss of data
2>Content\VideoRenderer.cpp(38): warning C4244: 'argument': conversion from 'int' to 'float', possible loss of data
2>Content\VideoRenderer.cpp(40): warning C4244: 'argument': conversion from 'int' to 'float', possible loss of data
2>Content\VideoRenderer.cpp(41): warning C4244: 'argument': conversion from 'int' to 'float', possible loss of data
2>Content\VideoRenderer.cpp(42): warning C4244: 'argument': conversion from 'int' to 'float', possible loss of data
2>AppCallbacks.cpp(29): warning C4453: 'Windows::ApplicationModel::Core::CoreApplicationView': A '[WebHostHidden]' type should not be used on the published surface of a public type that is not '[WebHostHidden]'
2>AppCallbacks.cpp(35): warning C4453: 'Windows::UI::Core::CoreWindow': A '[WebHostHidden]' type should not be used on the published surface of a public type that is not '[WebHostHidden]'
2> Creating library C:\Users\leola\Documents\Medivis\StreamToolKit\3dtoolkit\Build\x86\Release\DirectxClient\DirectxClientComponent.lib and object C:\Users\leola\Documents\Medivis\StreamToolKit\3dtoolkit\Build\x86\Release\DirectxClient\DirectxClientComponent.exp
2> Generating code
2> All 3918 functions were compiled because no usable IPDB/IOBJ from previous compilation was found.
2> Finished generating code
2> DirectxClientComponent.vcxproj -> C:\Users\leola\Documents\Medivis\StreamToolKit\3dtoolkit\Build\x86\Release\DirectxClient\DirectxClientComponent.dll
2> DirectxClientComponent.vcxproj -> C:\Users\leola\Documents\Medivis\StreamToolKit\3dtoolkit\Build\x86\Release\DirectxClient\DirectxClientComponent.pdb (Full PDB)
2> C:\Users\leola\Documents\Medivis\StreamToolKit\3dtoolkit\Build\x86\Release\DirectxClient\GeometryShader.cso
2> C:\Users\leola\Documents\Medivis\StreamToolKit\3dtoolkit\Build\x86\Release\DirectxClient\PixelShader.cso
2> C:\Users\leola\Documents\Medivis\StreamToolKit\3dtoolkit\Build\x86\Release\DirectxClient\VertexShader.cso
2> C:\Users\leola\Documents\Medivis\StreamToolKit\3dtoolkit\Build\x86\Release\DirectxClient\VPRTVertexShader.cso
2> 4 File(s) copied
3>------ Rebuild All started: Project: StreamingDirectxHololensClient, Configuration: Release x86 ------
3> StreamingDirectxHololensClient -> C:\Users\leola\Documents\Medivis\StreamToolKit\3dtoolkit\Build\x86\Release\DirectxClient\StreamingDirectxHololensClient.exe
3>C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v14.0\AppxPackage\Microsoft.AppXPackage.Targets(1930,5): warning APPX0104: Certificate file 'StreamingDirectxHololensClient_TemporaryKey.pfx' not found.
3>C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v14.0\AppxPackage\Microsoft.AppXPackage.Targets(1930,5): warning APPX0102: A certificate with thumbprint 'A3C0FEEFC5C5EFF2AEE1191F50451266B313CAEE' that is specified in the project cannot be found in the certificate store. Please specify a valid thumbprint in the project file.
3>C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v14.0\AppxPackage\Microsoft.AppXPackage.Targets(1930,5): warning APPX0107: The certificate specified is not valid for signing. For more information about valid certificates, see http://go.microsoft.com/fwlink/?LinkID=241478.
3> Starting .NET Native compilation
3>C:\Program Files (x86)\MSBuild\Microsoft.NetNative\x86\ilc\IlcInternals.targets(318,5): error : Error: Windows Runtime metadata is invalid in Windows SDK, follow the steps in http://go.microsoft.com/fwlink/?LinkId=733341 to repair your installation.
3>C:\Program Files (x86)\MSBuild\Microsoft.NetNative\x86\ilc\IlcIntern

UnityServer closing crash

Unity-FractalStream server sample crashes on close. This is apparently known, on master, and has been present for some time.

unable to select peer in unity samples

Just a question,

If I'm building both the client and server in Unity, how can I select connection since neither of them has the interface ( as in Multithreadserver.exe) to do that.

Thanks,
Wenbo

[docs] set github project description

We should set the github description value to our tagline, and add some relevant topics to improve visibility of this project 😸

Currently it's just No description, website, or topics provided.

Error on long path

This git was cloned into a folder with long names.

When the solution was building, path-related errors occurred.

After re-cloning the repo in root C:, it was GOOD.

User friendly UX for Unity Client

Currently the Unity client UX is pretty broken (and hidden for HoloLens).

Loading, connection properties, and user login will all need to be added shortly to keep it operational with the OAuth identity work.

support https in unity client

currently, the unity client does not support https signaling communications. It should.

Exception thrown: 'System.AggregateException' in mscorlib.ni.dll on Hololens unity build

I try to build the hololens client following the guide.

Build succeeded but can't run on the hololens.

here is the full output from the Unity Client:

'UnityX86.exe' (CoreCLR: DefaultDomain): Loaded 'c:\data\Programs\WindowsApps\Microsoft.NET.CoreRuntime.1.0_1.0.23819.0_x86__8wekyb3d8bbwe\mscorlib.ni.dll'. Cannot find or open the PDB file.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\entrypoint\UnityX86.exe'. Symbols loaded.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\System.Runtime.dll'. Cannot find or open the PDB file.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\WinMetadata\Windows.winmd'. Module was built without symbols.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\UnityPlayer.winmd'. Module was built without symbols.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\System.Runtime.InteropServices.WindowsRuntime.dll'. Cannot find or open the PDB file.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\System.Runtime.WindowsRuntime.dll'. Cannot find or open the PDB file.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\WinRTBridge.winmd'. Cannot find or open the PDB file.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\BridgeInterface.winmd'. Module was built without symbols.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\System.Collections.dll'. Cannot find or open the PDB file.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\UnityEngine.dll'. Symbols loaded.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\Assembly-CSharp.dll'. Symbols loaded.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\Org.WebRtc.winmd'. Module was built without symbols.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\UnityEngine.Networking.dll'. Symbols loaded.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\UnityEngine.HoloLens.dll'. Symbols loaded.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\UnityEngine.UI.dll'. Symbols loaded.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\nunit.framework.dll'. Symbols loaded.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\System.Reflection.dll'. Module was built without symbols.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\System.Globalization.dll'. Module was built without symbols.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\System.IO.dll'. Cannot find or open the PDB file.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\System.Linq.dll'. Cannot find or open the PDB file.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\System.Net.Primitives.dll'. Cannot find or open the PDB file.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\System.Private.Networking.dll'. Cannot find or open the PDB file.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\Microsoft.Win32.Primitives.dll'. Cannot find or open the PDB file.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\System.ObjectModel.dll'. Cannot find or open the PDB file.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\WebRtcWrapper.dll'. Symbols loaded.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\System.Reflection.Primitives.dll'. Module was built without symbols.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\System.Runtime.Extensions.dll'. Cannot find or open the PDB file.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\System.Threading.Tasks.dll'. Cannot find or open the PDB file.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\System.Xml.XDocument.dll'. Cannot find or open the PDB file.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\System.Private.Uri.dll'. Cannot find or open the PDB file.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\System.Xml.ReaderWriter.dll'. Cannot find or open the PDB file.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\System.Text.Encoding.dll'. Module was built without symbols.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\System.Text.RegularExpressions.dll'. Cannot find or open the PDB file.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\UnityEngine.VR.dll'. Symbols loaded.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\System.Runtime.InteropServices.dll'. Cannot find or open the PDB file.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\System.Threading.dll'. Cannot find or open the PDB file.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\System.Diagnostics.Debug.dll'. Cannot find or open the PDB file.
Module information:
Built with Compiler Ver '190023918'
Built from '5.6/release' branch
Version is '5.6.0f3 (497a0f351392)'
Release build
Application type 'D3D11'
Used 'UWP'

OS 'Windows 10 (10.0.14393)'

PlayerConnection initialized from C:/Data/Users/DefaultAccount/AppData/Local/DevelopmentFiles/UnityX86VS.Release_x86.leola/Data (debug = 0)

PlayerConnection initialized network socket : 0.0.0.0 55342

Multi-casting "[IP] 10.0.0.9 [Port] 55342 [Flags] 2 [Guid] 506219090 [EditorId] 4294967295 [Version] 1048832 [Id] MetroPlayerX86(HoloLens-7JN8J) [Debug] 0" to [225.0.0.222:54997]...

Started listening to [0.0.0.0:55342]

Player data archive not found at C:/Data/Users/DefaultAccount/AppData/Local/DevelopmentFiles/UnityX86VS.Release_x86.leola/Data/data.unity3d, using local filesystem
GfxDevice: creating device client; threaded=1

Direct3D:

Version:  Direct3D 11.0 [level 11.1]

Renderer: HoloLens Graphics (ID=0x22b0)

Vendor:   Intel

VRAM:     604 MB

Initialize engine version: 5.6.0f3 (497a0f351392)

'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\UnityEngineProxy.dll'. Cannot find or open the PDB file.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\System.Reflection.Extensions.dll'. Module was built without symbols.
Logical Screen DPI is 144.00.

If your machine has multiple graphics adapters, Unity may have created a WindowContext on the wrong adapter. If you experience a black screen when playing, please restart the Editor.

(Filename: C:/buildslave/unity/build/Runtime/VR/VRDevice.cpp Line: 184)

UnloadTime: 4.841515 ms

'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\System.Collections.Concurrent.dll'. Cannot find or open the PDB file.
'UnityX86.exe' (CoreCLR: CoreCLR_UWP_Domain): Loaded 'C:\Data\Users\DefaultAccount\AppData\Local\DevelopmentFiles\UnityX86VS.Release_x86.leola\System.Threading.Timer.dll'. Module was built without symbols.
Exception thrown: 'System.AggregateException' in mscorlib.ni.dll
Setting up 1 worker threads for Enlighten.

Thread -> id: 874 -> priority: 1

The thread 0x1604 has exited with code 0 (0x0).
The thread 0x1044 has exited with code 0 (0x0).
Holographic Tracking State Lost. New State: 4

(Filename: C:/buildslave/unity/build/Runtime/VR/HoloLens/HoloLensWorldManager.cpp Line: 130)

Holographic Tracking State Active.

(Filename: C:/buildslave/unity/build/Runtime/VR/HoloLens/HoloLensWorldManager.cpp Line: 126)

The program '[1232] UnityX86.exe' has exited with code -1 (0xffffffff).

document building webrtc from source in the wiki

Building WebRTC from source should be pulled into a step-by-step in the wiki, instead of a quick note in the readme.

hololens exception behind proxy

On certain networks, the hololens can fail to connect to the service for signalling, but may still receive a 200 status. This will cause this code to fail and throw an exception. We should try and detect these situations better, but we may be unable to do so (ie: transparent proxy).

For example, our guest network proxy appliance uses the following trick:

<html>
<head>
<meta http-equiv='refresh' content='1; url=http://<signaling_server_address>/sign_in?peer_name=hololens-qfrtq&arubalp=59beda57-187d-4d56-ad05-928b617a58'>
</head>
</html>

Which breaks us.

how does authentication work, and how can we leverage it?

when working with the toolkit a common question is "how do we secure access to our services, and ensure our clients are used only by authenticated users".

We need to answer and document this question.

Refactor ConfigParser for unit tests

We should refactor the ConfigParser library so that it's easy to mock the config values. Some unit tests should also be included.

Stereo Rendering mode unable to reset back to Mono on server samples

Once a server is connected to a Stereo rendering client, it cannot go back to Mono mode. Disconnecting from a stereo client and connecting to a mono client will crash the server.

The same is true in reverse. Ending a mono session and then connecting to a stereo session will crash the server.

Unity client build failure

The unity client fails to build, giving the following error:

Assets\Scripts\ControlScript.cs(336,50): error CS1061: 'Media' does not contain a definition for 'CreateMediaStreamSource' and no extension method 'CreateMediaStreamSource' accepting a first argument of type 'Media' could be found (are you missing a using directive or an assembly reference?)

incorrect build output directory for unknown sub-project

When building, you may see a Build\Debug directory, when you should only see the <architecture>\<configuration> folder only (like Build\x86\Debug)

This needs some investigation to figure out which project is incorrectly creating this folder

do not place unity binaries that are incompatible with unity builds

currently we place all unity binaries into our unity sample project at the end of the build. This is an issue when the build targets x64, because our unity plugins directory (and hololens supported architecture) is x86. We should modify our build to not place x64 binaries if they are built.

azure sdk is a dependency for deployment projects

Right now, if you open the .sln without having the azure sdk installed, you see a warning about deployment projects (DirectX-Multithread for example) not being able to load.

How should we communicate or deal with this?

long running signaling sessions timeout

a long-running signaling session (> 2m) with a server running on any PaaS that terminates open TCP connections after a period of time results in a failure with a pending connection at that point in time.

Repro steps

• launch the client
• launch the server
• connect to any signaling server on azure
• wait 2m
• notice connection is lost

README clarification for new users

• Should point out that the prebuilt binaries are only for those "consuming" the project, vs the build process which is for project contributors.
• Should clarify in what rare instances would rebuilding the WebRTC libraries be required. This is a rare occurrence that is already taken care of in the CI builds and most devs won't need this, and therefore will not need the Windows DDK.
• We should be clear that in order to run the setup script, you should have the execution mode set to unrestricted in PowerShell.
• Replace the "Finished" message during Setup in PowerShell to more clearly say "Step 1 of X completed: insertstepname here".
• The comment about "this patch" should be expanded to explain what the patch is and why it exists (since you called attention to it).

handle temporary turn creds rollover

when using temporary turn credentials today, if the credentials expire during a session, the session is invalidated and the users see an error. this should be handled more gracefully

support multiple ice servers via config

Consumers should be able to specific multiple ICE servers in our configuration files - this is especially important for STUN, where you likely wish to provide a collection so that if one STUN server is down, the application will simply use the next one.

support authentication in unity server

currently, the unity server does not support authentication, nor turn credentials (since they depend on authentication) signaling communications. It should.

The right way to implement this is likely to consume InitializerWrapper in the plugin, much as we do in the native server implementations of InitWebRTC().

UnityHoloClient fails to parse webRtc config file when ran on PC

Hi,

the Holo Unity client has problems finding or parsing the webRTC config file on PC.
It happens when I am building the Unity client project for PC
and running it as PC app. Nevertheless, it should work fine, right?

Here are the exceptions:

Exception thrown: 'System.UriFormatException' in System.Private.Uri.dll
Exception thrown: 'System.UriFormatException' in mscorlib.ni.dll

The path is:
_configFilePath = "ms-appx:///Data/StreamingAssets/webrtcConfig.json"

and this fails (line 270, WebRtcControl.cs)
StorageFile configFile = await StorageFile.GetFileFromApplicationUriAsync( new Uri(_configFilePath)) .AsTask() .ConfigureAwait(false);

Thanks for checking this out

DirectX Client video locks up over SSL

With the feature/native-auth and in the master branch, the client video stream stops functioning after 10-15 seconds and never returns. This occurs over TURN and STUN, and also occurs when connecting the client to older server instances. It is somewhat random, sometimes it will run for minutes, but once the video stream locks up, disconnect and reconnecting will resume video but only for a short period of time

This also happens over both http and https, with and without auth.

auto-enable software encoder when nvenc hardware is missing

Currently if the machine running server app doesn't support nvenc (either it doesn't have the required NVIDIA GPU or the system picks the wrong default graphics card), there is no video render on client although the input still works. In order to fix this, we should either raise the error or set to software encoder automatically if the hardware isn't supported.

UnityServer Stereo Rendering support for HoloLens

The UnityServer sample currently doesn't support mode switching into Stereo rendering mode for HoloLens. It is also not usable because neither the Unity Server nor the Unity HoloLens client have the ability to auto-connect and neither have interactive modes for choosing a peer to connect to.

Using the pattern from Multithreaded Server, implement the same mode switching to stereo rendering on HoloLens.

Automatic texture resolution detection for HoloLens Unity client

Currently client resolution is hard coded at design-time in the Unity client. Instead, resolution should be set based at peer connection time and should be resilient to changes in servers.

unity server sample fails to connect

I follow the steps build the 3dtoolkit solution and also all the plugin for Unity.

Successfully built the standalone app on Windows, run the app with command line to -force-d3d11-no-singlethreaded but I can't find the server from the Client app

StreamingDirectxClient.exe showing white screen on connect

When I launch the MultithreadServer with the StreamingDirectxClient, the client shows up as a white screen (no visible stream). But if I try to navigate as if it were rendering, the server shows the output just fine. Tried relaunching multiple times switching launch order of server/client and it still happened.

[docs] add the config file in the wiki

Today we don't have docs that cover what all the values in the config files are for - we should author that page.