microsoft/gestures-samples

Project Prague Code Samples

Overview

This repository contains code samples demonstrating how to use the Project Prague hand gestures SDK. For a quick introduction to programming with our SDK, please visit our overview page on docs.microsoft.com. In order to build and run the code samples provided in this repository, you must first set up Project Prague on your machine. Note that you will need either an Intel® RealSense™ SR300 camera or a Kinect for Windows v2 camera to enable our gesture detection runtime.

Code Samples

After cloning this repository, please open the Microsoft.Gestures.Samples soulution in Visual Studio. You will find 12 projects in this solution. Each project is a stand-alone code sample that can be built and run independently of the other projects. The table below lists the samples ordered from the least to the most advanced:

Sample	Use case	What will you learn?
ConsoleCore, ConsoleNative	Test gesture detection in a command-line window	Create several simple gestures and register them to the Gestures Service
RotateSample, RotateSampleUwpManaged, RotateSampleUwpNative	Rotate an image by 90°	Create a simple gesture made up of two hand poses
VideoPlayerGestureControl, VideoPlayerGestureControlUwpNative	Pause and rewind audio\video playback	Create a simple gesture made up of multiple hand poses and a hand motion
RockPaperScissors	Play Rock-Paper-Scissors against an invincible bot	Create a gesture made up of multiple hand poses connected in a complex structure
GrabAndPushUwpNative	Scale up\down a UWP control using gestures	Create a simple gesture and consume hand skeleton stream
Camera3D	Manipulate camera in a 3D scene	Create a simple gesture and consume hand skeleton stream
GesturesPowerPointPlugin (requires Office Tools for Visual Studio)	Create and present a slideshow in PowerPoint using gestures	Create multiple simple gestures and wire them to activate various PowerPoint features
CarGestures	Control a virtual car's infotainment-system with gestures	Create a complete WPF UI with multiple gestures of varying complexity
Unity Package and Tutorial Projects	Follow our Unity tutorial	Integrate gestures and hand-skeleton input in your Unity games

Sample Projects - Naming Conventions

Suffix	Meaning
No suffix	Standard WPF project
*Native	Native project complient with C++ 11 standard (portable to Linux)
*Core	.NET Core 2.0 project (supported on Linux)
*UwpManaged	Universal Windows Platform C# project
*UwpNative	Universal Windows Platform C++/CX project

Contributing

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

gestures-samples's Issues

Librealsense or SDK 1 is required? SR300 not detected

Before run the application is required to instal LibRealsense or Intel Realsense SDK 1?
I'm using SR300 on Windows 10 FCU but the camera is not detected.
D400 cameras will be supported?

Cannot parse "1.0" to its numeric equivalent

Hi.

Can you help me guys to fix this? Please! Thanks!

[2018.06.13 11:31:10.239] [WorkflowManager] Failed processing frame: Moriah workflow failed processing a frame System.TypeInitializationException: The type initializer for 'Microsoft.MSR.ATL.ATLI.Moriah.MapleSkeleton' threw an exception. ---> System.TypeInitializationException: The type initializer for 'Microsoft.MSR.ATL.ATLI.Moriah.HandSkeleton.SkeletonBase' threw an exception. ---> Microsoft.Gestures.GesturesFrameworkObjectValidationException: Invalid value assigned to [value]:
Cannot parse "1.0" to its numeric equivalent
   at Microsoft.MSR.ATL.ATLI.Moriah.HandSkeleton.SatisfactionParameters.ParseToNumeric[T](Object value)
   at CallSite.Target(Closure , CallSite , SatisfactionParameters , Object )
   at System.Dynamic.UpdateDelegates.UpdateAndExecute2[T0,T1,TRet](CallSite site, T0 arg0, T1 arg1)
   at Microsoft.MSR.ATL.ATLI.Moriah.HandSkeleton.SatisfactionParameters..ctor(Object parameters)
   at CallSite.Target(Closure , CallSite , Type , Object )
   at System.Dynamic.UpdateDelegates.UpdateAndExecute2[T0,T1,TRet](CallSite site, T0 arg0, T1 arg1)
   at Microsoft.MSR.ATL.ATLI.Moriah.HandSkeleton.SkeletonBase..cctor()
   --- End of inner exception stack trace ---
   at Microsoft.MSR.ATL.ATLI.Moriah.HandSkeleton.SkeletonBase.OverrideSatisfactionParameters(Object overridingConfig)
   at System.Dynamic.UpdateDelegates.UpdateAndExecute2[T0,T1,TRet](CallSite site, T0 arg0, T1 arg1)
   at Microsoft.MSR.ATL.ATLI.Moriah.MapleSkeleton..cctor()
   --- End of inner exception stack trace ---
   at Microsoft.MSR.ATL.ATLI.Moriah.MapleSkeleton..ctor(MapleEstimation mapleEstimation)
   at Microsoft.MSR.ATL.ATLI.Moriah.Runtime.Workflow.SensorDataProcessors.Maple.MapleRapidDataProcessor.<>c.<ActualProcessFrame>b__14_2(MapleEstimation est)
   at System.Linq.Enumerable.WhereSelectArrayIterator`2.MoveNext()
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at Microsoft.MSR.ATL.ATLI.Moriah.Runtime.Workflow.SensorDataProcessors.Maple.MapleRapidDataProcessor.ActualProcessFrame(SensorFrame frame)
   at Microsoft.Gestures.Runtime.Workflow.Manager.DataModel.WorkflowElementBase`2.ProcessFrame(InputFrameType inputFrame)
   at Microsoft.Gestures.Runtime.Workflow.Manager.WorkflowWorker.DoWorkflowIteration()
   at Microsoft.Gestures.Runtime.Workflow.Manager.WorkflowWorker.TryWorkflowIteration()

This repo is missing important files

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

Azure Kinect DK and RealSense D435

Any chance the service will be updated to support the latest camera SDKs?

MicrosoftGesturesInstallDir environment variable not existing

This case should be handled and a user-friendly comment should be displayed when the build breaks for this reason.

Support to Kinect V1 sensor

Don't let my Kinect V1 sensor be left in the dust, please add support to more depth cameras (including the first Kinect)

UWP Support

Please support UWP ASAP. Would be nice to see Microsoft as committed to UWP as Microsoft wants the rest of the world to be. This is an important project, and it's kind of remarkable that the first public release is lacking a UWP implementation.

Microsoft Gestures Service does not detect any gestures

I started Microsoft Gestures Service. I can see the GUI and on the GUI, I see my hands with color markers on each fingers. However, nothing is displayed under gesture, current, or timeline. Any ideas why no gestures are detected?

Many thanks for your kind reply!

SDK Doesn't install anymore

Hi,

When installing the SDK on Windows 10 / 11 it quickly shows the screen, then closes the app and nothing happens anymore.

Gesture Detection Stopped (Kinect V2 sensor not detected)

In a previous Win 10 build (around April 2019) gestures service and Kinect V2 could integrate seamlessly. However, following an upgrade to Win 10 (version 1903; build 18362:10024), the gestures service no longer detects Kinect V2. Kinect sensor integrates with Unity3D seamlessly. Any assistance accorded towards resolving this is highly appreciated. NB, downgrading the win build is presently not possible. Thank you.

Linux Support for Gesture Service

The documentation on the repository says the .NET Core version is supported on Linux, but I couldn't find any source to download the SDK or Gesture Service for Linux. Is the .NET Core version self contained and does not require the service to be running parallel?

Intel RealSense SR305

Since I was not able to purchase an SR300 camera, I am trying to use Project Gesture with the successor model SR305.

Unfortunately, the Gesture Service doesn't look like it would support it as it throws the following exception:

[2020-03-04 12:22:49.011] [WorkflowManager] Workflow not loaded: Microsoft.Gestures.Runtime.Workflow.Manager.Exceptions.InvalidWorkflowException: None of the preconfigured workflows loaded successfully
	None of the preconfigured sensors loaded successfully
	Sensor plug-in Intel(R) RealSense(TM) Camera SR300 Depth is disconnected or failed to initialize.
	Sensor plug-in Intel(R) RealSense(TM) Camera SR300 Depth is disconnected or failed to initialize.
	Sensor plug-in Intel(R) RealSense(TM) Camera SR300 Depth is disconnected or failed to initialize.
	Sensor plug-in Intel(R) RealSense(TM) Camera SR300 Depth is disconnected or failed to initialize.
	None of the preconfigured sensors loaded successfully
	Sensor plug-in Intel(R) RealSense(TM) 410 Depth is disconnected or failed to initialize.
	Sensor plug-in Intel(R) RealSense(TM) 410 Depth is disconnected or failed to initialize.
	None of the preconfigured sensors loaded successfully
	Sensor plug-in WDF KinectSensor Interface 0 is disconnected or failed to initialize.
	Sensor plug-in WDF KinectSensor Interface 0 is disconnected or failed to initialize.
	None of the preconfigured sensors loaded successfully
	Sensor plug-in Intel(R) RealSense(TM) 3D Camera (Front F200) Depth is disconnected or failed to initialize.
	Sensor plug-in Intel(R) RealSense(TM) 3D Camera (Front F200) Depth is disconnected or failed to initialize. ---> System.AggregateException: One or more errors occurred. ---> Microsoft.Gestures.Runtime.Workflow.Manager.Exceptions.InvalidWorkflowException: None of the preconfigured sensors loaded successfully
	Sensor plug-in Intel(R) RealSense(TM) Camera SR300 Depth is disconnected or failed to initialize.
	Sensor plug-in Intel(R) RealSense(TM) Camera SR300 Depth is disconnected or failed to initialize.
	Sensor plug-in Intel(R) RealSense(TM) Camera SR300 Depth is disconnected or failed to initialize.
	Sensor plug-in Intel(R) RealSense(TM) Camera SR300 Depth is disconnected or failed to initialize. ---> System.AggregateException: One or more errors occurred. ---> Microsoft.Gestures.Runtime.Workflow.Manager.Exceptions.InvalidWorkflowException: Sensor plug-in Intel(R) RealSense(TM) Camera SR300 Depth is disconnected or failed to initialize.
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at Microsoft.Gestures.Runtime.Workflow.Manager.WorkflowLoader.SpecificWorkflowConfigurationLoad(String sensorFriendlyName, String sensorDataProcessorFriendlyName, String gestureDetectorFriendlyName, Object dynamicConfig, IAvailableWorkflowElements availableWorkflowElements, IDeviceManagerListener deviceManagerListener, TryingToSetupWorkflowEventHandler handler)
   at Microsoft.Gestures.Runtime.Workflow.Manager.WorkflowLoader.Load(IEnumerable`1 configurations, Object dynamicConfig, IAvailableWorkflowElements availableWorkflowElements, IDeviceManagerListener deviceManagerListener, TryingToSetupWorkflowEventHandler handler)
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at Microsoft.Gestures.Runtime.Workflow.Manager.WorkflowLoader.Load(IEnumerable`1 configurations, Object dynamicConfig, IAvailableWorkflowElements availableWorkflowElements, IDeviceManagerListener deviceManagerListener, TryingToSetupWorkflowEventHandler handler)
   at Microsoft.Gestures.Runtime.Workflow.Manager.WorkflowLoader.<>c__DisplayClass4_0.<LoadAsync>b__0()
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.Tasks.Task.Execute()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at CallSite.Target(Closure , CallSite , Object )
   at Microsoft.Gestures.Runtime.Workflow.Manager.WorkflowManager.<InitializeWorkflow>d__86.MoveNext()

Is this by design or am I doing something wrong in setting up the camera?

.NET 6.0+ support?

Dear authors of the fantastic Project Prague, I wonder if there are updated bits for using Microsoft Gestures in the app built on top of .NET 6.0 and up?

I've tried to decompile .NET Core versions of the Microsoft.Gestures* assemblies just to figure if I could try things for myself first before writing here, but there seems to be a lot of dynamic code used that decompilers can't figure out, making .NET Core-optimized assemblies of no use in a .NET 6.0 app :(

Please advise!

Some stock gestures and poses are internal

According to https://docs.microsoft.com/en-us/dotnet/api/microsoft.gestures.stock.gestures.swipegesture?view=gestures-1.0 the SwipeGesture class should be public. But since I wasn't able to use it, I have taken a look at the assembly. Here it says the class is internal.

// Type: Microsoft.Gestures.Stock.Gestures.SwipeGesture
// Assembly: Microsoft.Gestures, Version=1.0.17339.1, Culture=neutral, PublicKeyToken=null
// MVID: 3B751A2D-7053-4B44-A0F0-5ACB9ECA5A0F
// Assembly location: C:\Users\dzoech\AppData\Roaming\Microsoft\Prague\PragueVersions\LatestVersion\Samples\DiscoveryClient\Microsoft.Gestures.dll
// Metadata token values is shown

using Microsoft.Gestures.Stock.HandPoses;
using System;
using System.ComponentModel;
using System.Linq;
using System.Linq.Expressions;

namespace Microsoft.Gestures.Stock.Gestures
{
  internal sealed class SwipeGesture : StockGesture
  {
      ...

Have I referenced the wrong DLL?
Also, what's the difference between the .NET Core DLLs and the .NET Framework DLLs? Since the docs says to include the .NET Framework DLLs. Are the Core ones working nevertheless?

Skeleton palm position return wrong values

When I call to skeleton.PalmPosition, return <-41,179400,16> where the element Y is wrong.

Have someone had this problem?

Thanks

Issue with Stock Hand Poses

The .NET Core versions of UWP throw 'Unsupported Segment type' exception when registering a gesture with Stock Hand Poses.

Moriah workflow failed processing a frame System.TypeInitializationException

Whenever I launch the Microsoft Gestures Service I always get the following error and Can't use the app nor the framework at all. Please seet the full details below:

1. [2017/07/13 10:31:10.065] [WorkflowManager] Trying to setup workflow [Intel(R) RealSense(TM) Camera SR300 Depth]->[MapleRapidDataProcessor]->[SimpleGestureDetector]...

2. [2017/07/13 10:31:11.386] [WorkflowManager] Workflow loaded successfully [Intel(R) RealSense(TM) Camera SR300 Depth]->[MapleRapidDataProcessor]->[SimpleGestureDetector]...

3. [2017/07/13 10:31:11.818] [WorkflowManager] Failed processing frame: Moriah workflow failed processing a frame System.TypeInitializationException: The type initializer for 'Microsoft.MSR.ATL.ATLI.Moriah.MapleSkeleton' threw an exception. ---> System.TypeInitializationException: The type initializer for 'Microsoft.MSR.ATL.ATLI.Moriah.HandSkeleton.SkeletonBase' threw an exception. ---> Microsoft.CSharp.RuntimeBinder.RuntimeBinderException: Cannot implicitly convert type 'string' to 'float'
   at CallSite.Target(Closure , CallSite , Object )
   at System.Dynamic.UpdateDelegates.UpdateAndExecute1[T0,TRet](CallSite site, T0 arg0)
   at Microsoft.MSR.ATL.ATLI.Moriah.HandSkeleton.SatisfactionParameters..ctor(Object parameters)
   at CallSite.Target(Closure , CallSite , Type , Object )
   at System.Dynamic.UpdateDelegates.UpdateAndExecute2[T0,T1,TRet](CallSite site, T0 arg0, T1 arg1)
   at Microsoft.MSR.ATL.ATLI.Moriah.HandSkeleton.SkeletonBase..cctor()
   --- End of inner exception stack trace ---
   at Microsoft.MSR.ATL.ATLI.Moriah.HandSkeleton.SkeletonBase.OverrideSatisfactionParameters(Object overridingConfig)
   at System.Dynamic.UpdateDelegates.UpdateAndExecute2[T0,T1,TRet](CallSite site, T0 arg0, T1 arg1)
   at Microsoft.MSR.ATL.ATLI.Moriah.MapleSkeleton..cctor()
   --- End of inner exception stack trace ---
   at Microsoft.MSR.ATL.ATLI.Moriah.MapleSkeleton..ctor(MapleEstimation mapleEstimation)
   at Microsoft.MSR.ATL.ATLI.Moriah.Runtime.Workflow.SensorDataProcessors.Maple.MapleRapidDataProcessor.<>c.b__13_2(MapleEstimation est)
   at System.Linq.Enumerable.WhereSelectArrayIterator2.MoveNext() at System.Linq.Buffer1..ctor(IEnumerable1 source) at System.Linq.Enumerable.ToArray[TSource](IEnumerable1 source)
   at Microsoft.MSR.ATL.ATLI.Moriah.Runtime.Workflow.SensorDataProcessors.Maple.MapleRapidDataProcessor.ActualProcessFrame(SensorFrame frame)
   at Microsoft.Gestures.Runtime.Workflow.Manager.DataModel.WorkflowElementBase`2.ProcessFrame(InputFrameType inputFrame)
   at Microsoft.Gestures.Runtime.Workflow.Manager.WorkflowWorker.DoWorkflowIteration()
   at Microsoft.Gestures.Runtime.Workflow.Manager.WorkflowWorker.TryWorkflowIteration()

Minimal distance or duration for motion

Is there a possibility to define a minimum distance or duration for a HandMotion or PalmMotion?

To be specific, I'm trying to implement a swipe gesture which has no concrete ending HandPose and should only trigger if the hand is moved for at least a short distance.

How to setup the environment for this demo?

I don't know which lib to install before use this demo and there is no tips in readme.