stereokit/stereokit

An easy-to-use XR engine for building AR and VR applications with C# and OpenXR!

StereoKit is an easy-to-use open source mixed reality library for building HoloLens and VR applications with C# and OpenXR! Inspired by libraries like XNA and Processing, StereoKit is meant to be fun to use and easy to develop with, yet still quite capable of creating professional and business ready software.

The getting started guide can be found here!

Interested in news and updates about StereoKit? Maybe just looking for some extra help?

• Submit bugs on the Issues tab, and ask questions in the Discussions tab!
• Follow koujaku on Twitter for development news.
• Drop into the Discord channel for discussion, help and updates.
• Check out this blog for occasional substantial updates!

StereoKit Features

• Mixed Reality inputs like hands and eyes are trivial to access
• Easy and powerful UI and interactions
• Model formats: .gltf, .glb, .obj, .stl, ASCII .ply
• Texture formats: .jpg, .png, .tga, .bmp, .psd, .gif, .hdr, .pic, .qoi, cubemaps
• Flexible shader/material system with built-in PBR
• Performance-by-default instanced render pipeline
• Skeletal/skinned animation
• Flat screen MR simulator with input emulation for easy development
• Builds your application to device in seconds, not minutes
• Runtime asset loading and cross-platform file picking
• Physics
• Documentation generated directly from the source code, including screenshots

Platform Support

StereoKit supports HoloLens 2, Oculus Quest, Windows Mixed Reality, Oculus Desktop, SteamVR, Varjo, Monado, and essentially everywhere OpenXR is!

These are the binaries that currently ship in the NuGet package. StereoKit developers working from C/C++ should be fine to build any architecture from this list of platforms.

Platform	x64	ARM	ARM64
Windows (Desktop/Win32)	X		X
Windows (HoloLens/UWP)	X	X	X
Linux	X		X
Android	X		X

Architecture support has focused on 64 bit architectures, with exceptions for certain platforms. (If you require an additional architecture in the NuGet, please propose it as an Issue!) Here, UWP's ARM builds much faster than UWP's ARM64, and WASM only comes in 32 bit flavors.

Getting started

Follow this guide for a detailed introduction! This repository is the raw source for those who wish to build StereoKit themselves, the Visual Studio templates and the NuGet packages referenced in the guide are how most people should build their applications!

StereoKit focuses on getting you productive with the least amount of code possible. You can actually do most tasks with a single line of code, including UI! Here's hello world with StereoKit, this is all you need to get up and running!

using StereoKit;

SK.Initialize();

Model helmet = Model.FromFile("Assets/DamagedHelmet.gltf");

SK.Run(() => {
    helmet.Draw(Matrix.TS(Vec3.Zero, 0.1f));
});

FAQ

• How do I get started using StereoKit?

  For the majority of users, go here, this is a Visual Studio centric introduction to StereoKit, and contains the most resources for getting started! For VS Code users, go here, and for C/C++ users, see here!

• How do I use this repository?

  This is where StereoKit's core SDK code resides, and the majority of users will not need to use it directly. C# developers will rely on prebuilt packages from NuGet and should refer to the getting started guides for instructions. If you wish to build StereoKit or the samples from scratch, see the build guide!

• I would like to contribute to StereoKit! How do I start?

  First off, thank you! :) Get comfortable with building StereoKit from source, and then check out our contributor's guide. Our maintainers are easily available on the StereoKit Discord, as well as GitHub Issues/Discussions! We'll be happy to help you out with whatever's on your mind!

• Why is the MIT License appropriate for this project?

  The MIT License is a copyright license that is easy to read and has a long history in open source. The MIT License is considered to be a low-friction choice for multiple downstream licensing models, including incorporation into commercial code.

Dependencies

Just like all software, StereoKit is built on the shoulders of incredible people! Here's a list of the libraries StereoKit uses to get things done.

• OpenXR
• ReactPhysics3D - Physics.
• cgltf - Gltf format support.
• Sean Barrett's stb libraries - Image and font format support.
• QOI - Image format support.
• miniaudio - Audio playback.
• sokol_time - Cross platform high performance time.

stereokit's Issues

[CS] There is no way of knowing if a correct xr_session has been created

Issue: With C# it's not possible to know if the OpenXR initialization has succeeded. The application will just run silently.

Steps to reproduce:

• Don't attach a Windows MR headset
• Run the StereoKitTest application

Result: The application starts normally, however nothing happens. As a developer I have no way of knowing if the application has a correct OpenXR xr_session running.

Expected: There is either an exception if no xr_session could be created or some other way of knowing.

Details: With CPP it is possible to know if the initialization has failed as sk_init() returns a bool. With C# there is no such thing as the constructor is just calling sk_init() and not listening to any return.

Suggestion: Change StereoKit class constructor to:

        public StereoKitApp(string name, Runtime runtime)
        {
            if (!sk_init(name, runtime))
            {
                throw new Exception("StereoKit: Could not initialize " + runtime.ToString() + " runtime");
            }
        }

Proposal: Add exceptions to StereoKitC library to let them properly bubble up, regardless of CPP or C#. At least for the fatal ones (like creating xr_session, events, actions, rendering etc.)

Device Input API

Right now, inputs are hands only. This is great for MR, but limiting for debug/flatscreen. Also, MR devices can still have mice and keyboards! This will also open up stronger compatibility with ImGui.

We'll limit this to just Keyboard + Mouse for now.

This is not part of an Action system, though it may function as a lower layer for one later on.

Use correct LUID when creating OpenXR session

I tried StereoKit with the Oculus runtime and ran into the issue at openxr.cpp#L165 because the laptop I was using has Optimus and creates an integrated rather than discrete graphics device.

PLY file format support

I like .ply, it's an easy format that supports vertex colors! .ply files are often used for point clouds, so it could be good to consider point cloud rendering when importing .ply files as well.

Might be able to find a nice library for this on here:
https://github.com/nothings/single_file_libs

ISpatialAudioClient for audio on WMR platforms

ISpatialAudioClient is the recommended method for rendering audio on a HoloLens! It does spatial audio and mixing on the HPU, so is largely free to use as far as the app is concerned. Win-win all around.

Reference implementation is here:
https://github.com/microsoft/spatialaudio-unity/tree/develop/Source/Spatializer/HoloLens2

Fix Warnings

There's a lot of warnings right now. Ooops, bad form! It's on the ToDo list now.

Full render to texture

It's currently possible to blit from texture to texture with a Material. We need to be able to render a full scene to the texture as well.

This could be implemented as a render queue stack, so you can push and pop render queues. This would also need to work well with the Hierarchy as well, perhaps pushing and popping the whole hierarchy stack too.

More hand visualization options

• Toggle hand visualization on/off.
• Let user swap to user defined materials.
• User defined meshes too?

Spherical Harmonics and PBR Reflections

Reflections are a super important component of PBR! Without it, metals and shiny objects just don't look right.

StereoKit currently has some basic PBR support with directional light sources, but still needs a fair bit of cubemap and spherical harmonics work before reflections are in.

Sooo, cubemaps! Spherical harmonics! And mip-mapping, since it's quite related. Existing work can be found here: b4b0120

NuGet Package

Once 0.1 is stabilized a bit, it needs to be easy to use and create a project with! NuGet is a great way to do this. Paired with a project template, it should be simple to get started.

DLL not found exception when trying to build x64 for HoloLens 2 Emulator

Hi and at first - thanks for the nice work!

I have a problem, when I try to run the example in the HoloLens 2 Emulator. As I unfortunately don't have a HL2 available yet I wanted to have look at you framework in the emulator.

If I am right the emulator for HL2 does not use ARM64 like the real device, but x64 or x86 to run the application. For that reason I choose the config Debug, x64, StereoKitTest_UWP, HoloLens 2 Emulator. But when starting this, I am getting an exception complaining about a missing DLL file StereoKitC.dll.
Building and running it as x64 for Local Machine works as expected. (Beside the fact I would expect to run the app in the Windows MR Portal, instead of a UWP Desktop application - but that's not related to my question here ;) )

Did I miss something or do I need to run some scripts first?

UI - Design Revision

Did a design review with Dan recently and got a great chunk of feedback to try out! The UI needs at least a cursory pass at these things before we can call it ready. Has to look at least a little bit good in order to get people to bite :)

• Color Scheme! Light text on darker background. Needs some palette work too.
• Finger Glow. Finger proximity feedback can be helpful for understanding distance to interface.
• Interaction Feedback: More feedback for interaction, colors, visual effects, noises, etc.
• Round Buttons: They look more professional :)
• Button Frames: Something a little extra around the base of the buttons, make them feel like they're not just boxes.

API sk_ prefix

Functions need a consistent prefix throughout the project!

Input - Far Interactions

Currently, we've got some pointer code! But this was written pretty early, and I haven't used it for much. It needs review, updating, and integration with the UI system for far interactions.

This is dependent on the intersections API.

Multi-Architecture Support

A C# binary works pretty much everywhere, but a C++ binary is platform and architecture specific! The StereoKit library binds to a native DLL to gain access to lower level functionality, so it needs to be referencing the correct native DLL for the system that it's currently on!

It's possible to manually load a DLL in advance (at least on Windows?) and references will just use the manually loaded DLL instead of trying to load it again. This allows us to decide and load a platform specific DLL before anything starts making calls to it!

Some initial work for this is is here: 8d8d489

Model edit/info functions

Right now, Model is the most basic of 3d object containers! It would be good to flesh this out with additional information about the model, the ability to parse through a model loaded from file in greater detail, and assemble more complicated models.

Multi-Platform Rendering

OpenXR is supported on a number of different platforms, so it would be advantageous to be able to render to those platforms too! Right now StereoKit is based on DX11, but this only gives us the Windows platform.

For full compatibility, Metal support, as well as Vulkan or OpenGL would also be needed. That's at least two more renderers that'll need to be written.

Alternatively, using a rendering library such as bgfx (https://github.com/bkaradzic/bgfx) could be really good! Would be great to survey around a bit and see what other options would work as well.

This topic also has a big impact on how materials and shaders look in the future.

Comment Docs

Need to finish adding comment docs to C++ and C#.

Would be great to have this in a way where it's visible both to intellisense, -and- the documentationn generator, while avoiding docs duplication, but this might be an uphill battle.

• Material
• Mesh
• Model
• Shader
• Solid
• Sprite
• Tex
• Plane
• Pose
• Quat
• Ray
• Sphere
• Vec2
• Vec3
• Vec4
• Input
• Log
• Renderer
• UI
• Color
• StereoKitApp
• Native Types

Chaining initialization calls Fail

Currently chaining the Initialization() calls fails (for example calling it once with Runtime.MixedReality and if that fails with Runtime.Flatscreen) because of some internal configuration that already happened. Ideally this should work.

As per @DirkSonguer's observation.

SDF text

Current text implementation is raster, SDF (Signed Distance Field) could increase visual quality! Also may allow for inline bold text :)

Demos & Guides v0.1

Docs need guides! And in StereoKit, we do that by writing demos! So here's some demos we still need:

• Getting Started
• User Interface
• Input

Texture Compression

StereoKit currently uploads raw RGBA data to the GPU, which is generally slow and inefficient. Textures should be compressed if they're not being regularly altered.

Basis Universal could be an excellent intermediate or preferred format for compressed textures.
https://github.com/binomialLLC/basis_universal

Here's a basisu sample implementation over here in the Sokol samples:
https://github.com/floooh/sokol-samples/tree/master/libs/basisu

And another implementation in the Wicked engine:
https://github.com/turanszkij/WickedEngine/blob/master/WickedEngine/Utility/utility_common.cpp

FBX File format support

Pretty common to see .fbx files around!

Here's a nice library for it:
https://github.com/nem0/OpenFBX

HL2 Best Practices

There's still some remaining recommendations for OpenXR performance I haven't gotten to for StereoKit. Don't need to get them all in for 0.1, just need to review and snag the low hanging fruit for now.

Recommendations are all here:
https://github.com/microsoft/OpenXR-SDK-VisualStudio#understand-openxr-samples-and-best-practices-for-hololens-2

• Rendertarget color format
• Texture array rendertarget
• Unbound reference space
• MRC alpha blend

Improved text flow

It should be possible to specify a containing rectangle when rendering text! A bounding box that lets text wrap, align to different sides, and clip at the bottom.

Worth considering multi-column text too, as long as we're at it. But not a requisite for this ticket!

[CS] Scene Graph / Prefab System

Something for managing a scene, swapping out scenes, and loading scenes from files! This could probably be mixed with some kind of prefab system, since a prefab is really just a small scene. Bonus points for something that already has an editor?

This could be a core item for a UI framework later on as well, especially if it pulls double duty as a prefab system?

If this is also a prefab, then having a system for promoting child parameters to the root interface would be essential. For example, if there was a complex prefab hierarchy with models, text, icons, etc. like a button, the text field should be promoted to the root so it's easily visible and changeable without having knowledge of the button's prefab hierarchy.

Async Texture Load

Asset loading is the most common form of blocking operation, so lets make it not block! Since we're doing all our asset pipeline work during runtime, this could end up being a lot, especially for big, non-optimal files.

This will allow us to do things like a staggered loading system:

• Begin loading, set a 'fallback' texture while we're grabbing the initial file.
• Load the image, set that as the active object so it can be seen right away.
• Compress the texture, and again replace the existing object.

UI - Model visuals

UI should have the ability to display 3D models as part of the visuals. This can be as replacement for buttons, or as their own UI elements.

UI models are largely for use with UI interactions, like grab/move/scale UI interactions.

quat_from_angles needs to be right-handed

Currently using left handed values. Should be easy enough, XM library says to just negate each value:https://docs.microsoft.com/en-us/windows/win32/api/directxmath/nf-directxmath-xmquaternionrotationrollpitchyaw

Pre-compile shader assets

Currently, shaders are compiled to the application's temporary folder at first launch. But this can cause a long pause on the very first run: not great for first impressions!

Since shaders are StereoKit specific creations anyhow, not generic assets, it would be fine to just pre-compile them, and use the compiled shader as the final asset. This could be a command-line utility executed through something like the documentation tool, or a standalone tool/IDE for building shaders that outputs a compiled file. Command-line may be a good stop-gap until a shader editor can be built.

This should also consider multi-platform rendering, so a pre-compiled shader asset should contain binary data for DirectX as well as OpenGL or Vulkan, whatever we settle on for rendering on non-windows platforms. Doesn't need to be in until we have multi-platform rendering, but it's good to consider when writing it.

DirectX matrix multiplication glitches on x86

x86 builds won't display properly with this glitch present. Looks like the problem may be fixed somewhere upstream, so I'll put this here for revisiting later.

https://developercommunity.visualstudio.com/content/problem/562932/directxmath-matrix-multiplication-incorrect-when-r.html

StereoKit Visual Studio Template requires visual studio 2019

Following getting started, I downloaded the vs template, but was unable to open it, with the error:

"This extension is not installable on any currently installed products."

I only had Visual Studio 2017 installed, but it turns out StereoKit needs Visual Studio 2019. It would be nice to mention this in the documentation.

Debug builds in x64/ARM64 UWP directly from the git repository.

There's a DLL issue when running x64/ARM64 UWP builds in Debug mode. This only happens in the github example project, not projects built through the template!

Normal map support

Currently, shaders don't have enough information to use normal maps. Meshes don't have tangents! So that'll need calculated and added to the mesh information.

Alternatively, there may be methods for doing normals without precalculated tangents?
http://www.thetenthplanet.de/archives/1180

Shader Variants

Right now, a shader is just a single shader. It's often quite useful to multi-compile a shader to remove costly operations that aren't being used! Like normal mapping.

This will require some kind of sub-shader resource that contains the actual compiled shader.

Compiling all variants at startup could be costly for large shaders (see Unity compile time) so it could be advantageous to delay shader variant compile until it actually gets used?

UI - Text Input

A simple text input that will receive characters from the keyboard.

For now we'll limit functionality to just typing characters. No in-MR virtual keyboard or text selection.

Depth Buffer Sharing

Splitting this out from HL2 Best Practices, since I'm pushing it to v0.2. Depth buffer is important, but most important at low frame rate. StereoKit has been extremely fast so far, any performance issues right now are typically blocking IO operations.

Depth sharing recommendation details are here:
https://github.com/microsoft/OpenXR-SDK-VisualStudio#understand-openxr-samples-and-best-practices-for-hololens-2

Basic Intersection API

A small API for detecting intersections between basic shapes! This will largely be used by the UI system for detecting interaction with inputs.

Should include functions for the following:
Rays/Points/Lines/AABBs/Spheres intersect with
Planes/AABBs/Spheres

Where Spheres are low priority.

Update main page doc for 0.1

Main page needs to reflect all the great work that's been happening lately!

• Getting started guide
• Link to NuGet and VS templates
• Link to new docs
• Current feature list update
• Roadmap
• More???

[CS] Entity Component System

A little further down the line at the moment, but still worth putting on the roadmap.

A component system for implementing quick and re-usable behaviors! C# should do a much better job of this than C, so I think it's an excellent place to start moving over into the C# world. This would probably look a lot like what Unity has in many ways.

I'd love to have components stored in large arrays of structs for cache locality.

SpatialCoordinateSystem Translation

Need functionality for translating SpatialCoordinateSystem into a usable position for use with OpenXR.

Physics

Physics is critical to realistic interactions, and the heart of MR UI!

Looking to integrate Bullet (https://pybullet.org/wordpress/) as a core component of StereoKit. Bullet looks like one of the better options, but I'm still pretty open to alternatives!

Physics joints

StereoKit has Solid physics objects, but I only ever got a basic joint system implemented. Might be sitting in a stash or branch somewhere? This needs to be revisited, polished up, and exposed to C#!

Graphics on separate thread

It would be great to offload graphics to a separate thread! Multi-threading is definitely optimal and wonderful, but the primary objective here is to try and prevent users from slowing down the rendering.

If the user ever does anything on the main thread that blocks execution, the render thread should be able to continue along, rendering the most recently provided list using newer head/hand position information.

UI - Interactions

A set of MR interactions where you'll be able to grab UI elements and move/rotate/scale objects. This is similar to the current sk_ui_affordance function, but with an expanded set of capabilities.

HL1 Compatability

Hello,

This looks like a great project. I see that it currently is only compatible to build for ARM64, which is 64-bit and thus only works on the HL2. Is there planned availability for the 32-bit HL1?

Thanks in advance, and I look forward to hopefully being able to use StereoKit soon.

techtide

AccessViolation when attempting to set a texture to Default.Material.Copy()

I noticed this in a project I'm working on; if I take a copy of the default material for a mesh, then attempt to set a texture to that material, it ends in an AccessViolationException. If I don't copy the material everything is fine, except all the models in my scene inherit that texture.

Here's some code that you can drop into your test project to repro:

public class DemoTextureCrash : IDemo
{
    private Model _Model;

    public void Initialize()
    {
        _Model = Model.FromMesh(Mesh.GenerateCylinder(10, 1, Vec3.Up), Default.Material.Copy());            
    }

    public void Shutdown()
    {

    }

    private bool _DoOnce;

    public void Update()
    {
        if (!_DoOnce)
        {
            var mat = _Model.GetMaterial(0);
            mat.SetTexture("diffuse", Tex.FromFile("Floor.png"));

            _DoOnce = true;
        }

        _Model.Draw(Matrix.T(Vec3.Zero));
    }
}

The exception is on this line in render.cpp during a call to Step:

d3d_context->VSSetShader(shader->vshader, nullptr, 0)

Support spatial anchors

Anchors are important for hologram stability, so this is an important detail for people that want to create a quality application. Could consider adding this as part of the Hierarchy system? But could still be better as an individual item.

See the OpenXR spec for spatial anchors here:
https://www.khronos.org/registry/OpenXR/specs/1.0/html/xrspec.html#XR_MSFT_spatial_anchor

Project Template

StereoKit needs a C# template for getting started. This might need a NuGet package first? Needs some research. This is definitely dependent on 0.1 being relatively stable first.

[Proposal] Updating the folder structure

As I am going through the code, I would love to propose some (very opinionated) updates of the file structure of the project:

Collect examples in their own topical subfolders with CPP and CS underneath
Reason: This would give us the ability to add more focused examples as we go along.

• Create a /Examples folder in the root directory
• Add a folder for each topic, currently a /GettingStarted one and add simple README for folder
• Move /StereoKitCTest to /GettingStarted/CPP
• Move StereoKitTest to /GettingStarted/CS

Move topics in the core to respective subfolders
Reason: The current prefixes help to identify topics, however the number of files will grow and splitting them into concerns will keep the project manageable.

• Mirror the current VS project structure as folders by adding a /Platform folder and adding openxr.cpp and .h, win32.cpp and .h and win32_input.cpp and .h there
• Start creating topical folders for things that have more than 2 files, for example /Input or /AssetManagement

What do you think?