ksdot/hololens

This is a Unity Application for the Hololens 2, part of the AR practical course. Multiplayer game, where each player can influence the shape of the world by placing pre-defined objects in it to construct a physically based world. After creating a blueprint of the world, players can start the physics simulation part of it so that everything behaves as it would in the real world.

Prerequisites

Make sure you have the following:

1. Unity 2020.3.x (LTS) or similar with UWP as a targetable platform installed.

And for building and deploying you need: (Add these with the Visual Studio Installer)

2. Windows SDK 10.0.220000 or similar
3. C++ v143

Installation

Clone this repository, open Unity Hub and click on Open > Open project from disk and navigate to your clone path.

Wait for Unity to import all packages, this might take a couple of minutes.

Once the editor is open, make sure the project is properly configured for the Hololens by clicking on Mixed Reality > Project > Apply recommended settings for the Hololens 2.

Deploying to the Hololens

If you want to build and deploy to the Hololens right away, follow the steps here.

Holographic Remoting

This vital development feature only works when your device and the hololens are connected to the same Network. For instance, if a PC has a WiFi adapter and is connected to the Internet through Ethernet, you can create a Hotspot and connect the Hololens to it. Alternatively, you can connect both devices to the same router, either through ethernet or wifi.

Make sure your Hololens is connected to the right wifi network and that it has the holographic remoting application installed. Open the holographic remoting app and note the displayed IP address. In the Unity Editor click on Mixed Reality > Remoting > Holographic Remoting for Play Mode. A new tab called should open and there under Remote Host Name write the IP address displayed in the Holographic Remoting app in the Hololens. Finally, Click on Enable Holographic Remoting for Play Mode in the same window and now Unity will try to couple with your Hololens to stream the Camera view of the editor to it.

Project Overview

VirtualArchitects is a construction simulator. The project is a sim game, with the possibility to be used by professionals for prototyping or visualizing designs. When using this application as a game, one could construct structures, akin to playing with Legos or Minecraft. For professionals, entire city mockups can be built and displayed in this application.

Current Status

Currently, we have a functioning building block simulator with accurate physics. These are integrated within the Hololenses AR suite, e.g. hand gestures, voice commands, and spatial awareness. Additionally, we have implemented a novel interaction method based on a toolbox. The first steps to fully networking the game have been taken, users can log in to a remote server and see each other in the scene and primitives can be initialized over the network but without custom properties.

Future Work

To have a shared workspace the primitives need to be instantiated with the same properties. Also, to allow for collaboration on the same blocks, ownership transfer of blocks also needs to be implemented. To better support co-located usage, either shared spatial awareness or a QR code as anchor need to be implemented still. We also want to support passive observers that interface with the project using WebXR. This would allow, for instance, phones and tablets to see the buildings.

The physics simulation needs to be run authoritatively on the server in order for the simulations results to be shared. One feature that the project could do well with is the saving and loading of models. With this one could implement exporting these saved models in a 3D printer compatible format.

Known issues

For singleplayer we use Microsofts Spatial Awareness to generate a mesh corresponding to the real world objects surrounding the player. Sometimes the resulting mesh within our application has errors in the form of holes and unexpected geometry. We believe these errors stem from Microsofts mesh generation algorithm.

There is a known discrepancy in performance between holographic remoting and deployment in the Hololens 2 during multiplayer. This problem is exacerbated when the server is set to use the TCP protocol. This leads us to believe that our Networking solution (PUN) is not properly optimized to run on a device like the Hololens 2.

Scenes

There are 2 scenes in the project, Singleplayer, and Multiplayer. The only difference in the hierarchy is that the HandContainer > BlockBox[Object Manager]. Using Networking is disabled for single-player. Additionally, the Objects Network Lobby and Shared Playground are disabled in single-player.

Prefabs

There are 2 classes of prefabs in the Prefabs directory, networking and UI prefabs. The networking prefabs include the NetworkLobby, the PhotonUser, and the SharedPlayground. NetworkLobby needs to be added to a scene for it to be managed as a networked room. The PhotonUser contains the visual representation of a player through the network and also contains the PhotonUser Component, where networked code used to manage player assets is located. The SharedPlayground prefab contains some example code for dealing with Azures Spatial Anchors over the network, although this functionality is currently not supported.

The HandContainer prefab is responsible for most UI functions. It handles object spawning, color and material selection. The 3D Button prefab is present within the HandContainer.

The HandMenu_Small prefab is a hand-gesture summoned UI which contains the Start/Stop, Save and Load buttons.

The Folder ReferenceObjects contains a prefab for each of the supported primitives. Each defines the mesh corresponding to a specific tag.

The tutorial folder contains the UI elements that walk through the games functionality. It includes the buttons and the information.

The WindowExplorer prefab is a placeholder for a future Save/Load Menu. Th

Scripts

Our scripts are all in Assets > Scripts folder. For more information, we have thoroughly commented the code.

Networking

These are all the scripts that interface the PUN API with our project. Most of these have been taken from the PUN Networking tutorial and fitted here for our purposes. Here you can work on expanding the networking features, for example, shared play areas, Azures Spatial Anchors, and so on.

Physics

These scripts handle the collisions between the building blocks, themselves, and some UI elements. For example, the deletion of objects is handled in Deletion Collider If you want to introduce nonrigid body physics to the application, for instance, some elastic ropes, here is the appropriate part to place your scripts.

UI and Interactions

We built some custom UI elements for this application. Especially the building block box features novel interaction patterns. You can find and work on these features here.