microsoft/mixedreality-sharing

Portable APIs and components for networking and state synchronization

NOTE: This project is no longer maintained

MixedReality-Sharing

MixedReality-Sharing is a cross-platform solution to enable rich multiuser experiences. In particular most mixed reality experiences need the following:

• Matchmaking - find and join experiences
• Streaming - transmit and receive audio/video/data
• Localization - agree a shared reference frame (anchor)

MixedReality-Sharing provides library packages to fulfil each of the above needs. The packages are independent, meaning they can be adopted or replaced incrementally. Integration libraries are provided so that the packages work together seamlessly, both with each other and with external libraries.

Matchmaking

The Matchmaking.Discovery is based on a simple advertisement/discovery protocol. The transport is pluggable and there are implementations for peer-to-peer UDP broadcast and dedicated server.

Streaming

We recommend MixedReality-WebRTC for streaming audio, video and data. As with any WebRTC solution, an external signaling mechanism is required to establish a connection. We provide an in-process signaling solution to make this easy.

Localization

On augmented reality (AR) platforms, “anchors” are a common frame of reference for enabling multiple users to place digital content in the same physical location, where it can be seen on different devices in the same position and orientation relative to the environment.

SpatialAlignment allows a variety of backends such as Azure Spatial Anchors or Fiduciary Markers such as QR codes.

Getting Started

Installation

• NuGet Packages
  • NuGet packages are our recommended way to consume MixedReality-Sharing libraries.
• Download
  • Packages can be downloaded directly from the Releases Page
• Build Source
  • Clone the github repo

Documentation

...

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

mixedreality-sharing's Issues

PeerDiscoveryAgent multiply inherits IDisposable

Once through IDiscoveryAgent and again through IDisposableBase.

Implement the C++ internals of the versioned storage library

This should be a separate static library (consumed by the main StateSync library) with a very low-level interface, so that all low-level elements can be tested separately without exposing them through the main library's interface. Features that should go in first:

• Two-level ({key:sybkey}=>value) map-like versioned storage.
• Support for atomic transactions that validate and modify multiple keys and subkeys.
• Support for multiple immutable snapshots of different versions, that are not influenced by ongoing modifications.
• Deterministically ordered iteration over keys and subkeys within keys.

Future commits:

• Subscribing to changes (of individual keys and subkeys).
• Fast-forwarding the state of one instance of the storage to a newer state of another instance of storage (will be used by the StateSync directly).

Make sure that CI tests are also being executed on ARM CPUs.

Certain parts of the back end are CPU-specific. Currently the ARM implementation for them is missing, but it will have to be added at some point to support HL2.

It would be great to have a CI running the same tests on an ARM64 CUP (so that we can be sure that everything works).

Use short license boilerplate globally

https://docs.opensource.microsoft.com/content/releasing/copyright-headers.html?q=license%20header

// Copyright (c) Microsoft Corporation.
// Licensed under the MIT License.

Implement WebRTC signaling on top of matchmaking API

Implement the ISignaler interface from https://github.com/microsoft/MixedReality-WebRTC on top of a joined IRoom. Support connecting to a specific participant or to anyone available.

Test (and fix) room timeout

We should verify that rooms disappear from discovery lists when they time out - from my casual usage they seem not to - and add a unit test. This means that we should make the timeout configurable - at least for the whole matchmaking service - or we'll have very long-running tests.

Stop publishing/abandon individual resources

There is no way at the moment to stop publishing a single resource - only the whole discovery agent can be disposed.

Stopping a resource in the UDP implementation is trivial. OTOH room-based matchmaking services usually only drop a room when no one is in it anymore - they provide no way to close/hide a room forcefully from a client.

Maybe we can add a StopPublishing/Abandon method that only guarantees that the resource will be stopped/destroyed eventually.

Setup docs pipeline

Sequence UDP server packets

At the moment we risk later announces to be delivered before earlier ones, potentially writing attributes in the wrong order, and announces being delivered after bye-byes, making rooms live longer than necessary.

Unity binding for room discovery

For starters this can be a couple of classes exposing the configuration to the inspector and dispatching discovery events to the Update loop.

[MM] Design discovery protocol

As said IRL this could be simplified into having room owners just announce their rooms in a loop.

Originally posted by @fibann in #35

Renaming/cleanup in Matchmaking projects

At the moment what it contains is more a simple, generic room discovery-announcement that can be used in P2P networks (but also with a single server in theory). So it should be Matchmaking.Simple, Matchmaking.Peer or something like that, and get proper documentation around the public API and extension points.

Add retries to unreliable network transports

e.g. SSDP suggests sending each UDP packet 3 times, staggered by a few hundred ms + random.

To reduce retry bloat (1 query -> 3 query packets -> 3 replies -> 9 reply packets) we could introduce an 'etag' so that duplicates can be ignored if already handled.

Add basic documentation

Refactor the solution properties

#114 was forced to introduce an MRS_DevOptOut parameter because the property block in Directory.Build.props wasn't suitable for a project.
Let's be explicit and flip that so projects opt-in instead.

Catch exceptions thrown when parsing UDP packets

StartDiscovery(categoryX) doesn't send query packet the second time

A query packet is sent by StartDiscovery only if the category is not present in infoFromCategory, but categories are never removed from the map.

[MM] Sensible strategy for room IDs

The current version of matchmaking doesn't use room IDs - it identifies rooms by their connection string on the basis that two rooms cannot have the same connection string at the same time. This is simplistic since we might want to a) change connection string after room creation and/or b) reuse the same connection string for two different rooms existing in different moments. We can definitely improve on this but it's not really clear what's best.

A better solution is assigning user-specified room IDs to each new room (like Photon), keeping the connection string separate and mutable. This also enables scenarios where e.g. a host and a client can agree on the room ID offline and use it as a key to connect to each other (notably, this is what SpectatorView could do). The problems with having user-specified IDs is that the application must now care about passing unique IDs, and the backend must enforce uniqueness - which is very difficult if the backend doesn't have a central authority. Also, some backends (e.g. Steam) do not allow users to specify room IDs.

The former API version had string IDs generated by the matchmaking implementation. This solves the uniqueness problem but forces users to implement an additional mechanism to share the generated room ID at runtime, which is inconvenient especially given that matchmaking is used specifically to get a way to share info at runtime.

A user-specified ID can be "emulated" by using a user-specified attribute instead. So a client looking for a specific room would search for a specific attribute value. Here users have full responsibility of enforcing uniqueness - the backend won't help nor care. Also this only works if all rooms are public and searchable by attributes.

Update room attributes after creation

e.g. connection string & property bag.

Add basic examples

A small command-line .NET app should be sufficient.

[MM] Reduce copying in network layer

Fix inefficiencies, e.g. messages created on the heap for every send, multiple buffers created/copied/discarded.

SpatialAlignment Release

Remove IDisposable from IDiscoveryTask (and IMatchmakingService?)

It seems IDisposable is intended only if you have unmanaged resources, or have a field which is disposable. For managed objects, the c# idiom is to have an explicit close or shutdown methods instead.

Add expiry and refresh announcements

Announcements currently are considered alive until we receive an explicit server shutdown message. However, we may fail to receive that message for many reasons. Announcements should carry an expiration and clients should purge rooms which expire.
If we reannounce at say 49% of expiration time, clients will have two chances to see refreshes.

[StateSync] Implement StateSyncAPI.cs to interface with C++

The StateSync change that went in, is the C# layer with stubbed out StateSyncAPI.cs for the calls to the C++.

wyhash has been updated to V3

Dear Microsoft:
I feel happy as you includes wyhash. recently wyhash has been upgrades to V3, with doubles speed. Please check the latest version at github.com/wangyi-fudan/wyhash :-)

Setup Internal NuGet publishing

Setup internal NuGet publishing for matchmaking component. This will be internal at this point to get the flow in, we will have to update this in the future with code signing, etc.

Set this up for:

• SpatialAlignment
• StateSync
• Matchmaking

Make MemoryPeerNetwork.PumpNetwork thread safe

PumpNetwork()

Probably doesn't work, if I understand correctly what it's supposed to do.

Thread 1: just done with the processing, about to do "pumpingNetwork_ = false;"
Thread 2: Calling PumpNetwork(). pumpingNetwork_ is true? kthxbye.
Thread 1: pumpingNetwork_ = false. Done.

Result: no one is processing the message in the queue.

Originally posted by @ivte-ms in #35

[MM] Handle repeated broadcasts of the same room

Periodically broadcast UDP discovery queries

Some devices (e.g. Android) discard incoming UDP broadcast/multicast packets by default to save energy, and some do not support multicast at all. Any such device won't be able to receive local UDP announcements from room hosts, only replies to query messages.

At the moment client devices send one query message when starting the discovery phase and that's it, so devices that cannot receive announcements will miss any host coming online after the start of discovery. We can work around this by periodically sending queries while the discovery is active.

Periodic queries would make periodic announcements partially redundant so we might think of removing them or making them optional (they might be needed if it's the host devices that cannot receive broadcast messages).

Threading rules of PeerDiscoveryAgent are confusing and undocumented

We assume that no code from any thread calls methods directly on a PeerDiscoveryAgent after the point where Dispose is called, but at the same time we want to allow DiscoverySubscriptions to be disposed after their agent/at the same time, even from multiple threads. This is unexpected and should be either simplified or documented.

[MM] Remove usage of IAsyncResult

It is not portable to .Net Core.

Currently used in UdpPeerNetwork.cs
https://devblogs.microsoft.com/dotnet/migrating-delegate-begininvoke-calls-for-net-core/

Handle resource data that doesn't fit a UDP packet

Split from #40. Connection string/attributes can be larger than MTU/UDP size limit. We need a fallback e.g. using a TCP connection like in #25 .

Starting/stopping transport in PeerDiscoveryAgent is not thread-safe

See #95 (comment). Starting/stopping the transport should be under lock.

Port SpatialAlignment feature from SpectatorView

As on tin.

Discovery tests which interleave creation & query

The current discovery tests create rooms and then start discovery.
We should have tests which start discovery and then create rooms.
And a mixture of the above.

Implement UDP matchmaking over local network

Write an implementation of the matchmaking API that performs room discovery/advertisement using UDP broadcast/multicast. The scope is demos/small experiences, scalability is not a concern for now.

ServiceShutdownRemovesRooms fails non-deterministically

It seems to be a non-deterministic bug and it's there with the previous commit - digging into it but it's not this change.

Originally posted by @fibann in #53 (comment)

It seems that calling socket_.Dispose() when shutting down the MM service might drop the ByeBye message. Adding socket.Close(10) seems to fix this - but Socket.Close is not in the UWP API and won't compile on Hololens, so we might need to add a Thread.Sleep.

EDIT: To be precise, I am having problems with Socket.Close only on Unity 2018.4, .NET build (IL2CPP and normal UWP VS2019 compile it fine).

Disposing a DiscoveryTask from its handler may cause deadlock

If the task is the only one running, disposing it will call network_.Stop() which will wait for the receiving thread to be finished - but the receiving thread is the one executing the handler.
This should be documented, raise a hard error, or work as expected.

[MM] Replace ObservableCollection in interfaces with something better

We want to update the discovered rooms from a network task. But ObservableCollection isn't threadsafe. Locking seems inadequate since we don't want to block writers because of slow readers. Some kind of lightweight snapshot + notification seems like it could work better.