davide-cas/holohelp

HoloHelp: HoloLens Object Detection for a Guided Interaction

HoloHelp: HoloLens Object Detection for a Guided Interaction

My bachelor's degree thesis project @ University of Catania, 2021.

The thesis has the primary aim of helping, assisting and supporting people in the usage of a specific object observed or interacted with.

Starting from Microsoft HoloLens 2, the idea was successfully implemented with a customized Object Detector trained in cloud via Microsoft Azure. To do that, we have used 9 specific classes of the COCO dataset to upload over 1000 images.

The goal is to make the tool as much manageable and easy to use as possible. Therefore, the idea is to use a single, simple voice command, HoloHelp. Nothing more. Once pronounced, HoloLens will immediately take a picture, saving the eye gaze coordinates of the object that the user is looking at. Once saved the picture with those information, an API request will be sent to Microsoft Azure Custom Vision, and a small audio and AR video guide that explains the usage of the object detected will appear.

Credits

The thesis has been supervised by the professor Antonino Furnari, in collaboration with NEXT VISION.