microsoft/mixedrealitycompanionkit

This is a MixedRealityToolkit style repository for code bits and components that may not run directly on Microsoft HoloLens or immersive headsets but instead pair with them to build experiences.

MixedRealityCompanionKit

Mixed Reality Companion Kit -- This is a MixedRealityToolkit style repository for code bits and components that may not run directly on Microsoft HoloLens or immersive headsets but instead pair with them to build experiences targeting Windows Mixed Reality.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

HoloLensCompanionKit contains the following feature areas:

1. Holographic Remoting Host

2. KinectIPD

3. MixedRemoteViewCompositor

4. SpectatorView

5. Windows Mixed Reality Commander (Formerly, HoloLens Commander)

6. Asset Bundler

mixedrealitycompanionkit's Issues

HoloLensCommander: move setting the tag (local name) and ipd off of the context menu

This will make room for the Device Information entry as well as reduce the number of clicks required to update the ipd, etc.

KinectIPD: NFC* classes dont have a copyright header

MRVC PlayBack

I am using 5.5.0f3 personal unity editor to create the LowLatencyMrcSample app from source. When I use the viewer, it is always timing out. I have restarted the app and rebooted the device multiple times, but to no avail. I have put some Logging statements in MrvcManager to see if playback connector ever gets called, but the log statement isn't getting hit. I am wondering if that is because of a different version of Unity Editor, or something wrong that I doing. Any help would be appreciated.

HoloLens Commander: Saving Mixed Reality Capture files is broken

Requires either a custom build of Windows Device Portal Wrapper or waiting for an updated NuGet package (v0.9.2 or later).

The fix is in the Windows Device Portal Wrapper project.

Spectator View: Debug assertion failed in Calibration

Hi,

we are following the instructions in this document:
https://developer.microsoft.com/en-us/windows/holographic/spectator_view

Currently, we are in the Calibration step, and when we run the calibration from Visual Studio, we see the camera image for something like 20 seconds, after which the following error appears:

Also, no files are created in the Documents/CalibrationFiles directory, and the video feed when running Calibration is very jittery (maybe 1 fps), not sure if this is normal.

We are using:

• Canon EOS 5D Mark III
• Elgato HD 60 S USB capture device

Our HoloLens is connected via USB. Developer mode and device portal are enabled, the device portal can be accessed under the given URL and the credentials are set correctly in the configuration files.

Is there anything else we can check to find out why this is happening?

HoloLens Commander: Update to Windows Device Portal Wrapper 0.9.2

There is an issue in Windows Device Portal Wrapper 0.9.1 that prevents getting the Mixed Reality Capture data streams.

When the version 0.9.2 (or later) NuGet package of Windows Device Portal Wrapper is available, update the project.json file to get the fix.

DllNotFoundException

Reopening this because i have the same troubles with the dll under unity not loading the plugin.

"Failed to load '...MixedRemoteViewCompositor.dll' with error 'this operation is only valid in the context of an app container.'

The dll is there, it is in the 64bits folder, with 64bit targets in the properties, built in 64bits from the sources, but it still fails to load in Unity 5.5 64bits. I cannot understand why this plugin won't load. (got the same DllNotFoundException also).

EDIT: Ok, simple mistake, there was the WSA dll trying to load in unity but there was no dll copy of the win32 version into the AddOn Assets automatically.

Thanks!

HoloLens_Mount.stp appears to be corrupted

HoloLens_Mount.stp appears to be corrupted. My team is unable to open it, saying there may be an issue with saving this.

IsPerceptionSimulationSupported Carash

I've follow the 'Add holographic remoting' document to setup the projects , but the project RemotingHostSampleDesktop crashed when press space to connect to HoloLens for
HolographicStreamerHelpers' line 604:
if (!IsPerceptionSimulationSupported())
{
throw ref new Platform::NotImplementedException();
}
and the console print these info:
0x74F03E28 (KernelBase.dll)(RemotingHostSampleDesktop.exe) exception: 0x40080201: WinRT originate error
0x74F03E28 RemotingHostSampleDesktop.exe exception: Microsoft C++ Exception: Platform::NotImplementedException

Thanks for help~

Spectator View: calibration throws unhandled exception

I've followed the documentation to build the Calibration solution, which completes without errors. When I run the solution, the window appears with a working camera feed, but after a couple seconds a breakpoint is triggered by the unhandled exception.

The stack looks like it starts at CalibrationApp::TakeCalibraitonPictureAtInterval and end with an exception from opencv_world310.dll. Let me know what information I can provide to shed light on this issue.

KinectIPD: Classes need summary comments

Right now the class files don't have a summary comment.

This will not allow auto doc generation tools to use those to provide API docs.

RealTimePlayback="True"

In the MixedRealityViewDialog, the playback would be a lot faster if RealTimePlayback were set to true. This would get performance closer to 350 milliseconds or so.

DllNotFoundException: MixedRemoteViewCompositor in Unity

Hi,

When I click the button, "Start Playback", in MRVC in the editor of Unity, there are errors:
DllNotFoundException: MixedRemoteViewCompositor MixedRemoteViewCompositor.Plugin.Awake () (at Assets/AddOns/MixedRemoteViewCompositor/Scripts/Interop/Plugin.cs:64)
DllNotFoundException: MixedRemoteViewCompositor MixedRemoteViewCompositor.Plugin.OnDisable () (at Assets/AddOns/MixedRemoteViewCompositor/Scripts/Interop/Plugin.cs:74)

Could I ask you how to fix the errors?
Thanks,
Woon

Spectator View: failed to record my voice

i used opencv and a Logitech webcam to capture the video, i can record the video, but i can not record my voice through the webcam. I checked the ENCODE_AUDIO is TRUE. I do not known where is wrong.

Multhithreading D3D11 Corruption

Error output:

D3D11 WARNING: ID3D11DeviceContext::OMSetRenderTargets: Resource being set to OM RenderTarget slot 0 is still bound on input! [ STATE_SETTING WARNING #9: DEVICE_OMSETRENDERTARGETS_HAZARD]
D3D11 WARNING: ID3D11DeviceContext::OMSetRenderTargets[AndUnorderedAccessViews]: Forcing PS shader resource slot 1 to NULL. [ STATE_SETTING WARNING #7: DEVICE_PSSETSHADERRESOURCES_HAZARD]
Exception thrown at 0x773224C2 in lowlatencymrcsample.exe: Microsoft C++ exception: _com_error at memory location 0x0DB4C380.
D3D11 CORRUPTION: ID3D11DeviceContext::UpdateSubresource: Two threads were found to be executing functions associated with the same Device[Context] at the same time. This will cause corruption of memory. Appropriate thread synchronization needs to occur external to the Direct3D API (or through the ID3D10Multithread interface). 4668 and 1584 are the implicated thread ids. [ MISCELLANEOUS CORRUPTION #28: CORRUPTED_MULTITHREADING]
Exception thrown at 0x773224C2 (KernelBase.dll) in lowlatencymrcsample.exe: 0x0000087D (parameters: 0x00000000, 0x0470DA8C, 0x0470CEC4).
Exception thrown: 'System.Runtime.InteropServices.SEHException' in Assembly-CSharp.dll
The program '[4836] lowlatencymrcsample.exe' has exited with code 0 (0x0).

The exception triggers at line 564 in PlaybackEngine.cpp, on the spBuffer->Unlock() instruction.

It appears when i set the Hololens to be the playback device while Unity does the capture of the local PC webcam. Streaming from Hololens to Unity seems to be stable enough but streaming from PC to Hololens is very unstable and lead to this error 95% of the time with 5% working properly enough (lagging though, probably because of those warnings).

Would like to figure out myself what is wrong with the critical section and D3D11 multithreading issue but since you worked on this you probably have a better idea of what's going on there.

Thanks for your support.

Remoting Player: Mixed Reality Capture messes stereo images

Hi,
the right-eye image shown on the HoloLens gets messed up when streaming 3d content from a host application to the Holographic Remoting Player and starting mixed reality capture from the website. I first thought it's an application error but it also happens with the sample from https://github.com/Microsoft/HoloLensCompanionKit/tree/master/RemotingHostSample
The left eye seems correct.
The issue is triggered by capturing both Holograms and PV Camera. If i just capture Holograms or just the PV Camera everything is fine.

Use with C# UWP or unity generated project

I tried adding the holographic remoting nugets to a Unity generated C#/XAML/D3D project but those dlls are probably not compatible. I also tried to generate IL2CPP from Unity to get a C++ D3D UWP project - but this companion is barebones C++/D3D. I am lost on how one would actualy run a Unity3D Holographic app on windows 10 box and be able to remote to HoloLens.

Before I start getting into this code and tryng to retrofit it into teh unity generated code, Is this possible and needs some work from my end or are there any clear roadblocks that you guys are aware of?

Windows Mixed Reality Commander: HoloLens emulator appears as an unknown device in the device list

if the device reports as a virtual machine, check to see if its family is Windows.Holographic. if so, change the cached platform to HoloLens.

Missing license

There is no license file associated with the Companion Kit.

HoloLens Commander: Common app list does not auto-update after an install

Workaround is to use the refresh button on the main UI.

HoloLens Commander: Update to WindowsDevicePortalWrapper v0.9.4 or later

as needed features / fixes are released in the NuGet package...

HoloLens Commander: Cannot install files from non-removable drive

Related to Windows Device Portal Wrapper Issue 228 and requires v 0.9.4 or later.

Spectator View: Hardcoded Video Mode

In DeckLinkManager.cpp you have the following section:

           if (!deckLinkDevice->IsCapturing())
            {
                // Note: this will use the resolution that the camera is outputting.  Ensure you set the camera to the settings you want.
                // The capture card will detect when camera display settings are changed, and update accordingly.
                deckLinkDevice->StartCapture(bmdModeHD1080i5994);
                return S_OK;
            }

The capture mode needs to be changed to what the camera is outputting, but instead it is hardcoded to 1080i5994, which will cause access violation if we record 4K content. Of course we can manually update this mode, but it was not mentioned anywhere so we wasted quite a lot of time to troubleshoot. Please either update the document or fix the code.

Unable to connect LowLatencyMRC

Exception thrown: 'System.Runtime.InteropServices.COMException' in mscorlib.ni.dll
WinRT information: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

Not sure if it could be user error that could be causing this, but using the given samples it doesn't seem to work.

3D Printing Sizing Problem with HoloLens_Mount.stl

Hi,

I'm trying to print the HoloLens_Mount.stl file for adapting to the Spectator view. My university uses 3DPrinterOS system for managing 3D printing files. When I opened the file the dimensions of the mount is roughly at 8mm.

What would be the correct sizes to choose for the Mount? Many thanks!

Spectator View: Calibration debugging

Hi guys, I'm trying to Spectator View. but I was failed on Calibration Step.

when I start debugging, it's working for calibrating.

However, press the Enter to finish the calibration, it has debug error code.

also, in "My Documents\CalibrationFiles" I can see only inverted Picture..

plz help! Thx!

SpectatorView: Menu Items Not Appearing

I am having issues getting the Spectator View and HoloToolKit menu items to appear on the menu bar.

I am following the instructions here: https://github.com/Microsoft/HoloLensCompanionKit/blob/master/SpectatorView/Compositor/README.md

and have got as far as 'Load your Unity project' but I don't have the menu item to continue. The HoloToolKit menu item did appear but has now disappeared.

Any ideas on what I have missed?

Thanks in advance

Spectator View: Broken Audio

With default settings, audio generated from apps are broken in recorded files, even with sample code in 240. We integrated Spectator view with a few of our own solutions, same results. Haven't got time to go through what could be the issue.

Cannot build MixedRemoteViewCompositor.sln

When attempting to build the solution, I get an error from MIDL4034 saying:
Failed to load a dependency file. Windows.Winmd (Invalid path for /metadata_dir switch - "C:\Program Files (x86)\Windows Kits\10\References\windows.foundation.foundationcontract\1.0.0.0")

Looking at the directory in question, I have foundationcontract 2.0.0.0, but not 1.0.0.0. Is it possible to switch the version, or to locate foundationcontract 1.0.0.0?

Spectator View: Video bitRate too low causing tearing

The default bitrate calculation for video ( bitRate(frameWidth * frameHeight * 8)) will result in tearing frames with high motion scenes with default h264 encoding codec. Not sure what you guys are using - hardware or software? This should be mentioned in the doc so people can adjust it to make their environment work. Or better let the user set it in the UI.

Error on MixedRemoteViewCompositor

I am using VS 2015 Enterprise. However, when I built the MixedRemoteViewCompositor in Win32 and WSA projects after I open the solution file, I had the following error:
midlrt : error MIDL4034: Failed to load a dependency file. Windows.Winmd (Invalid path for /metadata_dir switch - "C:\Program Files (x86)\Windows Kits\10\References\windows.foundation.foundationcontract\1.0.0.0")

Could you help me the way to fix this error?

Thanks,
Woon

MixedRemoteViewCompositor crashes when stopping capture

when trying to close the capture a crash occurs.

SpectatorView: Build Visual Studio SLN Crash

Following these instructions to build the Hololens app and Unity crashes everytime I try to 'Build Visual Studio SLN'

HoloToolkit -> Build Window -> Build Visual Studio SLN

Audio Support

I noticed a TODO in the audio streaming support in the sources. Any plans on this side in the very near future?
If not, any suggestion tu make it work for a skype like conversation?

Keep up the good work!

Low Latency proview on Hololens

Hi
how could i get a preview of the LowLatencyCapture on Hololens before it is send down the network.
A method to get the current texture2D per update for example during the capture.
In general if the build in networking part can be turned off and a way to expose the frames to unity can be found, this solution would be also suitable for cases where the dev can use his own network code.

Thanks
Dimi

Remoting Player: Anchor Store

Hi,

it looks like the remoting player identifies host application by some means. When storing Anchors from host application A, it's not possible to retrieve the list when connecting with application B to the very same hololens device. SpatialAnchorTransferManager is also not available when using remoting. Is there any way to share anchors (for a single hololens) between 2 host applications? I skimmed all headers of the Microsoft.Holografic.Remoting package but found nothing.

Thanks alot.

MRVC or LowLatencyMRC players for Non Windows devices.

Hello,
I'm trying to set up a Unity app to stream from the hololens to multiple platforms. From my understanding I just need to compile the dlls for the other platforms, but I have no clue if this is achievable in any sort of easy way.

To be able to utilize this I need at least OS X and iOS.

I really just need any way to play the stream on these other platforms.

Remoting Player

Hi there,

is it possible to replicate/extend the functionality of the remoting player? e.g. any plans to open source the remoting player app or provide frameworks wo write an own?
Motivation is that it useful to perform more computations (e.g. surface mesh/raycasting ..) on the hololens while still offloading the main rendering the to the remote player host pc.

Microsoft.Holographic.Remoting seems to contain server functionality - is this supposed to be used non-internally? e.g. CreateHolographicStreamerServer(..)? Are there any samples?

Thanks alot.

HoloLens Commander: Allow specifying dependency files and certificate during app install

Failed to import package with error: Couldn't decompress package

This is the error message I got when try to import the SpectatorView.unitypackage

HoloLens Commander: Update to WindowsDevicePortalWrapper v0.9.3 or greater

when available, update to the latest NuGet package

HoloLensCommander: add information (os, etc) display option

add to the context menu to display the os info, etc for each HoloLens being monitored

'Constant' errors

Hi,

Bit of a noob and trying to follow the instructions but stuck when it comes to building the UnityCompositorInterface.cpp (SpatialPerceptionHelper has built fine) and I have got it down to the below 9 errors which I am not sure how to recitfy.

Error C2143 syntax error: missing ';' before 'constant' UnityCompositorInterface C:\HoloLensCompanionKit-master\HoloLensCompanionKit-master\SpectatorView\Compositor\UnityCompositorInterface\UnityCompositorInterface.cpp 342
Error (active) no instance of overloaded function "DirectXHelper::CreateTexture" matches the argument list CompositorDLL c:\HoloLensCompanionKit-master\HoloLensCompanionKit-master\SpectatorView\Compositor\CompositorDLL\HologramQueue.cpp 17
Error (active) expected a ')' CompositorDLL c:\HoloLensCompanionKit-master\HoloLensCompanionKit-master\SpectatorView\Compositor\CompositorDLL\HologramQueue.cpp 17
Error (active) expected a ')' CompositorDLL c:\HoloLensCompanionKit-master\HoloLensCompanionKit-master\SpectatorView\Compositor\CompositorDLL\HologramQueue.h 32
Error C2059 syntax error: 'constant' CompositorDLL c:\hololenscompanionkit-master\hololenscompanionkit-master\spectatorview\compositor\compositordll\HologramQueue.h 32
Error C2059 syntax error: 'constant' CompositorDLL c:\hololenscompanionkit-master\hololenscompanionkit-master\spectatorview\compositor\compositordll\HologramQueue.h 32
Error C2059 syntax error: 'constant' CompositorDLL C:\HoloLensCompanionKit-master\HoloLensCompanionKit-master\SpectatorView\Compositor\CompositorDLL\HologramQueue.cpp 17
Error C2059 syntax error: 'constant' UnityCompositorInterface c:\hololenscompanionkit-master\hololenscompanionkit-master\spectatorview\compositor\compositordll\HologramQueue.h 32
Error C2143 syntax error: missing ';' before 'constant' UnityCompositorInterface C:\HoloLensCompanionKit-master\HoloLensCompanionKit-master\SpectatorView\Compositor\UnityCompositorInterface\UnityCompositorInterface.cpp 337

Using MS Visual Studio 2015 with Update 3 and Unity 5.6.0b7 Personal (64bit) and building for x64. IN MS Visual Studio 2015, Programming Languages - Common Tools for Visual C++ 2015
Universal Windows App Development Tools - Tools (1.4.1) and Windows 10 SDK(10.0.14393)
Universal Windows App Development Tools - Windows 10 SDK (10.0.10240 & 10586) are all selected and have re-logged in after installing these.

Any help gratefully appreciated.

Spectator View Feature Request - Advanced Compositor Controls (fps)

Dropdown for "Advanced" control in spectator view including ability to change decklink's framerate at runtime.

Spectator View / 3D Printing HoloLens_Mount.stl

The HoloLens_Mount.stl currently contains multiple (= 7) individual objects that are already bolted down in the model. That makes the model difficult to print as it is, also due to the overhang.

Is there the possibility to get the parts as individual stl files instead? Otherwise, everyone who wants to print would need to rework the model (e.g., in Rhino). If we could instead get the individual stl files of the objects, it would make it much easier to print that (e.g., with Cura as used for our 3D printer).

HoloLens Commander: Some long operations can temporarilly stall the UI

The application's UI code is waiting on some REST calls that take some time.

MrvcPlayer.cs - MrvcPlayer failed: -2147024809 - PlaybackEngine.Stop(): result: 0x80070057

Hello,

I'm trying to make for some while to run MrvcPlayer.cs so I can get the Texture2D and applied on cube what I get from capture.

So I've initialized MrvcCapture (like usual from MrvcManager.cs) and also MrvcPlayer.cs when a listener is called, but for some unknown reason in PlaybackEngine.cs at Initialize (227) the Wrapper.exCreate doesn't call the method this.createhandler

What I'm trying to achieve is to get the videoCapture frame and apply it on a quad in unity so what you can see can be seen on a game object in Unity.

Short edit:
Result of the Wrapper.exCreate from Playbackengine is 0, and is library it says it should be correct but for some unknown reason it doesn't do the call to createHandler

P.S
MRVC example how it should work ? I only managed to make it work (the stream) with Viewer example from UWP. And the Connector how should be called or from where more exactly.

Thank you !

LowLatencyMrcSample only works on Debug mode.

Hi,

When I run the sample codes, it only works on Debug mode.
Please help me to fix the problem.

Thanks,
Woon

HoloLens Commander: add ui to specify HoloLens IP address when connecting

loopback connections are only supported during development and (when configured) side-load scenarios. add UI to enable connecting to a HoloLens via a user specified IP address

IsPerceptionSimulationSupported Carash

I've follow the 'Add holographic remoting' document to setup the projects , but the project RemotingHostSampleDesktop crashed when press space to connect to HoloLens for
HolographicStreamerHelpers' line 604:
if (!IsPerceptionSimulationSupported())
{
throw ref new Platform::NotImplementedException();
}
and the console print these info:
0x74F03E28 (KernelBase.dll)(RemotingHostSampleDesktop.exe) exception: 0x40080201: WinRT originate error
0x74F03E28 RemotingHostSampleDesktop.exe exception: Microsoft C++ Exception: Platform::NotImplementedException

Thanks for help~