xrlabpomerania/hololens-2-object-detection

Hololens 2 Object Detection with Unity

A template for Microsoft Hololens 2 Object Recognition with Unity. This project is part of the Interreg collaboration.

Description

Our goal is to create a Microsoft Hololens 2 template which uses the Locatable Camera to do Image Classification and Object Recognition by using ONNX models and the Barracuda plugin. No Microsoft Azure needed.

Getting Started

Dependencies

• Unity 2021.3.1f1
• Burst 1.6.6 (preinstalled)
• Barracuda 3.0.0 (preinstalled)
• MRT Feature Tool (Mixed Reality Toolkit Foundation 2.8.0, Mixed Reality Toolkit Standard Assets 2.8.0, Mixed Reality OpenXR Plugin 1.4.0) (preinstalled)

Installation & Setup

When opening the project, the MRTK Toolkit Configurator should open automatically. If not, go to Mixed Reality > Toolkit > Utilities > Configure Project for MRTK. Make sure that Universal Windows Platform is selected and MRTK and OpenXR are checkmarked. Then go to File > Build Settings and select Architecture ARM64-bit, D3D Project and click Build. Save the Build where you prefer. If your compilation process does not succeed, its probably because Burst depends on some workloads which are not installed in Visual Studio. Check the Error in Unity to see if you have to install Visual Studio 2022 or if you can add the workloads to your Visual Studio version. The Workloads and individual components you probably need to add are:

• Universal Windows Platform Development Workload
• Universal Windows Platform Support for vNNN build tools (ARM64)
• MSVC vNNN - VS NNNN C++ ARM build tools (latest)
• MSVC vNNN - VS NNNN C++ ARM64 build tools (latest)

Choosing between models

You can choose between the BlazeFace detection model and the efficientnet lite model. Note that the latter has no bounding box, so the labels wont be placed appropiately. In order to switch between both models, set the useBlazeFace property inside the CustomVisionAnalyzer.cs to true or false.

More information about dependencies & versions

If you have any trouble with this project, here are some information and resources which might help you:

• This project is built using Unity Version 2021.3.1f1. We may switch or add a 2020 Version, which will probably be uploaded as a new repository.
• To add Barracuda 3.0.0 (not 2.0.0) to your project, follow the steps of A__'s answer here: https://stackoverflow.com/questions/68460374/cant-find-barracuda-package-in-unity-registry
• Get the MRTK feature Tool here: https://www.microsoft.com/en-us/download/details.aspx?id=102778
• When opening Unity, you may need to go into the hierarchy in the top left, under MixedRealityPlayspace > Main Camera
  • in the Inspector on the right, scroll to the script Custom Vision Analyzer and add the "efficientnet-lite4-11 (NN Model)", "labels_map" and the BlazeFace model from the Assets/Models folder and the Cube Object from the hierarchy into the respective boxes in the Inspector panel.
• if you get a null pointer exception on the castRay variables inside the CustomVisionAnalyzer, you have to select the Main Camera object inside the Hololens Scene Hierarchy and drag and drop it into the Scene Organizer component (into the Cast Ray field) of the Main Camera. The same principle applies to the Analyzer object.

Preview (BlazeFace model used)

Background

This project is the result a collaboration of members of the University of Greifswald (https://www.instagram.com/xrlabpomerania/), Szeczin.LAB (https://szczecinlab.pl/) and Makerspace Greifswald e.V. (https://www.instagram.com/makerspacehgw/).

Dieses Projekt wird durch die Europäische Union aus Mitteln des Europäischen Fonds für Regionale Entwicklung unterstützt (Fonds für kleine Projekte Interreg V A Mecklenburg-Vorpommern / Brandenburg / Polen in der Euoregion Pomerania).

hololens-2-object-detection's Issues

Unity critical error

When I open this project using Unity 2021.3.12f1 (which should be compatible with 2021.3.1), it encounters a critical error. I opened it in safe mode and figured that these issues may be related. Any suggestions on this? Thank you.

Deploy Yolov5 of HoloLens2

I want to deploy the Yolov5 for the purpose of the object detection on the HoloLens2, where can I find the code that tested and working for that goal?

Thanks in advance.

Custom model

Hi @yoKurt94
I tried this project and it worked well on my hololens.Can I use my custom model? I tried the model I trained in azure Custom Vision, but it didn't work. And why do you need two models (efficientnet-lite4-11 and BlazeFace model) in the project? I'm new to this field, and I'd appreciate it if you could answer me.

Yolo Model

Hello @yoKurt94
I have tried this project and it worked on my HL2 Device. I wanna add a custom yolo model and I am not sure of how to do so. I saw a script called YoloObjectDetector and I am wondering if you already did that and if so how it works.
Thank you so much in advance