dngoins/kinectv2heartrate

Kinectv2HeartRate

Kinect for Windows v2 Heart Rate Library

This application is a .Net WPF application which uses the R Statistical programming language engine version > 3.12. This application requires the R engine to be installed on the system running the application. R can be installed from here: http://cran.r-project.org/ The WPF application utilizes the Kinect RGB, IR, and Face streams of data to determine a region around the face and calculate a spatially averaged brightness over time. The averaged values are then divided by their respective standard deviations to provide a unit variance value. These values are required for feeding into ICA algorithms. The values are saved into a csv file for processsing with other Machine Learning techniques and algorithms.

The basic approach is simple. When a person's heart pumps blood, the volume of blood is pushed through various veins and muscles. As the blood pumps through the muscles, particularly the face, the more light is absorbed, and the less brightness the a web camera sensor picks up. This change in brightness value is very minute and can be extracted using matematical tricks. The change in brightness is periodic. In otherwords, a signal or wave. If we can match the signal/wave to that of a blood pulse, we can calculate the heart rate.

In order to match the change in brightness to a blood pulse we use the Independent Component Analysis (ICA) concept. This concept is the cocktail party concept and is the basis for finding hidden signals within a set of mixed signals. If you have two people talking in a crowded room, and you have microphones placed at various locations around the room, ICA algorithms let you take a mixed sample of signals, such as sound waves, and calculates an estimated separattion mixture of components. If you match the separate components to the orignal signal of a person speaking you have found that person in the crowded room.

This ICA concept is also known as blind source separation, and this project uses the JADE algorithm for R, to provide the separation matrix of commponents for the R,G, B, IR mixture of data. The separate components then have their signals extracted using a fast Fourier transform to find a matching frequency range of a heart rate.

kinectv2heartrate's Issues

The JADE package is not automatically installed

I had to install it manually in the R console using the command install.packages("JADE")

RDotNet.EvaluationException: 'Error: package or namespace load failed for 'JADE'

RDotNet.EvaluationException: 'Error: package or namespace load failed for 'JADE' in inDL(x, as.logical(local), as.logical(now), ...):
unable to load shared object 'C:/Program Files/R/R-3.4.3/library/stats/libs/i386/stats.dll':
LoadLibrary failure: The specified module could not be found.

I am getting this error.

The program cannot start because MSVCR120D.dll is missing

I got this message when trying to run the program. The program cannot start because MSVCR120D.dll is missing from your computer. Try reinstalling the program to fix this problem. I have tried reinstalling visual studio redistributable but nothing seems to work.

'System.ApplicationException' occurred in Microsoft.Kinect.Face.dll

Hi,
I am using your code with Kinect v2, Windows 8.1 and visual studio 2013 ultimate version. At first, when I ran the code, the window opens but the heart rate did not calculate and the timer did not start. After a few days when I executed the code again, the following error/exception happens and the program gets stuck/freezes.

"An unhandled exception of type 'System.ApplicationException' occurred in Microsoft.Kinect.Face.dll
Additional information: Failed to load NuiDatabase"

I checked the csv file created. It has data.

Can you please guide me how to overcome this error? I don't know why this error comes suddenly. I did not change any piece of code.

Thanks.

Cool but wildly inaccurate

Was hoping to integrate this into a kiosk for a boxing robot I'm working on (www.overthrowrobotics.com) but I get readings that are way off. 100 bpm after light exercise on my Samsung S6 sensor showed 44 on this. Tried multiple people with multiple readings and the numbers are way off. Sat right in front of the camera, didn't move with good lighting.

Exception in ProcessData "Error in lowHzIndex:highHzIndex : NA/NaN argument"

I'm receiving strange behavior. Fairly often, I receive an exception at this line in ProcessData:
engine.Evaluate(string.Format("source('{0}/RScripts/KinectHeartRate_JADE.r')", currentDir));

"Error in lowHzIndex:highHzIndex : NA/NaN argument"

Not familiar with the JADE algorithm in the RScript, so its extra difficult to debug. Also, the measured heart rate I receive is all over the place. Im measuring myself and Im usually resting about 75-85bpm, but the app fairly consistently places me between 46-56bpm. Once (out of about 30 runs) it got me at 77bpm so I know its possible, but something is definitely off.

Additionally, my measured heart rate seems to drop each run until I get this error (in the app, not irl). Usually when the measured rate is around 51 to 48 then it seems more likely that the exception will occur in the following run.

Culture variance messes up with the CSV file

On the line:

string data = string.Format("{0},{1},{2},{3},{4},{5}\n", m_secondsElapsed.ElapsedMilliseconds, norm_alpha, norm_red, norm_green, norm_blue, norm_ir);

C# converts the numbers to string, but as I'm from Brazil and we use commas as float separator, it messes up with the CSV file, I hat to put a CultureInfo as the first parameter.
This is how I did with my version:

var enUS = new System.Globalization.CultureInfo("en-US");
.........
string data = string.Format(enUS, "{0},{1},{2},{3},{4},{5}\n", m_secondsElapsed.ElapsedMilliseconds, norm_alpha, norm_red, norm_green, norm_blue, norm_ir);

Empty csv file

For some reason I'm generating empty csv files, it appears that I'm not detecting any faces. Is there a time limit that is need for facial detection? Or could this be some other problem? Sorry I'm new to the kinect environment.

jade package is not find

as the title mentioned, i receive this error when generating a solution.

engine.Evaluate("install.packages('JADE', repos='http://cran.rstudio.com/bin/windows/contrib/3.2/JADE_1.9-92.zip'");
engine.Evaluate("library(JADE)");

Thanks for your return.

BPM

Hi Mr. Goins,

I did all setup, kinect detect my face and I´m pressing the botton calculate rate but the number off BPM not appear on the screen and the application displays the following errors:

• An exception of type 'RDotNet.EvaluationException' occurred in RDotNet.dll but was not handled in user code
• An exception of type 'RDotNet.EvaluationException' occurred in mscorlib.dll but was not handled in user code

Could you please help me?

Problem csv file

Hello,

I have detected a little problem, in the header of the csv file you have

 string header = "nMillisecondsElapsed,nBlue,nGreen,nRed,nAlpha,nIr\n";

And you append you text in this order

string data = string.Format("{0},{1},{2},{3},{4},{5}\n", m_secondsElapsed.ElapsedMilliseconds, norm_alpha, norm_red, norm_green, norm_blue, norm_ir);

System.FormatException in mscorlib.dll

Hello! I'm trying to run the program but this message appears when the InitializeR() is invokated , do you know how to fix it?

thanks in advance!

After the 30 second countdown an RDotNet.EvaluationException is thrown

Exception thrown: 'RDotNet.EvaluationException' in RDotNet.dll

Error in read.table(file = file, header = header, sep = sep, quote = quote,  : 
  more columns than column names

I have no experience at all with R, but can look up more details if needed.