shoeffner/gaze

Gaze tracking using ordinary webcams (eye tracking works, gaze tracking not). My master's thesis project.

gaze's Issues

Implement eye model

This will allow to accurately project the detected pupil into 3D space to eventually cast a ray onto the screen.

Implement some Head pose estimation

E.g. Vatahska et al. (2007): Feature-based Head Pose Estimation from Images.

https://www.hrl.uni-bonn.de/papers/vatahska07humanoids.pdf

However, this paper uses some trained classifiers which are not available online (maybe they were in the past, see http://hrl.informatik.uni-freiburg.de/publications/archive/ which links to the dead http://planning.cs.cmu.edu/humanoids07/index.shtml (Archive's 2011 version https://web.archive.org/web/20110612050832/http://planning.cs.cmu.edu:80/humanoids07/papers.shtml does not have links to the models either, it seems). The conference website is also missing (https://www.ieee.org/conferences_events/conferences/conferencedetails/index.html?Conf_ID=13447 links to humainoids2007.org ).

• Write to authors.
• Find alternatives.

Python bindings

Create Python bindings to use gaze within Python scripts.

Find proper test framework

• Find framework
• Find evaluation criteria
• Generate test set and test calculations
• Setup CI (semaphore? circle ci?)

Implement target regions

As a start I shall focus on a 4x3 grid or similar.

Update README

Dependencies:

• OpenCV 3.3.0 / 3.4.0
• CMake 3.10.0-rc2
• Boost 1.65.1
• C++17
• Doxygen
• Dlib 19.8

Build steps:

• Install dependencies
• ./configure.sh
• make install

Usage instructions:

• Configuration
• Example programs
• Library usage

Explanations:

• Goal
• Thesis
• Results / Expectations

Other:

• Point to other projects (e.g. https://github.com/trishume/eyeLike)
• Add references
• Minimal example! (+ Results)

Known issues:

• Segfault in debug mode -> only on shutdown, only in debug window mode Resolved.
• where_people_look: GLib-GObject-CRITICAL **: g_object_unref: assertion 'G_IS_OBJECT (object)' failed -> seems to be unimportant
• HighSierra issue (probably fixed until thesis submission):
  objc[52475]: Class FIFinderSyncExtensionHost is implemented in both /System/Library/PrivateFrameworks/FinderKit.framework/Versions/A/FinderKit (0x7fffb5c01a70) and /System/Library/PrivateFrameworks/FileProvider.framework/OverrideBundles/FinderSyncCollaborationFileProviderOverride.bundle/Contents/MacOS/FinderSyncCollaborationFileProviderOverride (0x128c08cd8). One of the two will be used. Which one is undefined. Might be fixed (2017-12-28)

Deleted.

Implement camera parameters

Make it possible to configure how the camera is positioned in relation to the screen.

First step might be to hard-code a macbook's camera position.

Stop after video is processed

With the version from 7a20c35 onwards, video files will never end.

• Build a way to stop once a video is processed.

Implement a reader

Implement a reader to read the data from #8. This should allow for offline debugging. (Just use visualization steps and no processing steps).

Create configuration file

Configuration file:

• Screen size (or find programmatically?)
• Webcam position relative to screen
• Capture source
• Capture dimensions
• Target resolution (start with 4x3)

Optional:

• Model file for dlib's face detector

Stretch:

• Pipeline steps?

In general:

• all options for pipeline steps should be configurable by file, especially if the pipeline itself should become configurable.

Notes for thesis

In this issue I will write down some notes for the thesis, especially decisions during the implementation process.

Forward compatibility: decltype(...) for loops

Use the following to keep up with changes in APIs automatically:

for (auto i = decltype(boundary_var){0}; i < boundary_var; ++i) {

}

This will avoid having to figure out what type boundary_var has (int, unsigned long, unsigned int, ...) and in case of const decltypes drop the const due to moving to an auto type. See https://stackoverflow.com/a/19236169/3004221 and the interesting post linked there: http://herbsutter.com/2013/08/12/gotw-94-solution-aaa-style-almost-always-auto/ .

Implement result writer

• research eye tracking data formats (e.g. check PyGaze)
• Implement a simple writer
• dump all data

Implement Timm and Barth (2011).

Timm and Barth (2011): Accurate eye centre localisation by means of gradients.

See for an implementation e.g. https://github.com/trishume/eyeLike .

Documentation to gh-pages

Add Doxygen tag files for other libraries

Libraries not covered yet:

• Boost – see e.g. http://charette.no-ip.com:81/programming/doxygen/boost/doxygen_notes.html , but it's slow and marks all const/void keywords as boost keywords...
• Gtk
• dlib

Calculate gaze point

Calculate gaze point using pupil position #3, head pose estimation #4, camera parameters #5.

• #3
• #4
• #5
• #7
• #21

Use configure_file in cmake/DocsTarget.cmake

CMake's configure_file can change the Doxyfile already. Currently (d5ea815) the docs target uses a Python script and multiple generated Doxyfiles to achieve the same.

Evaluate dlib for face tracking

http://dlib.net/imaging.html#get_frontal_face_detector seems to be a good option for face pose tracking (thanks to @ahoereth and Jonas Kindler.) Evaluate its usability.