eyetrackvr/eyetrackvr

Open Source and Affordable, Virtual Reality Eye Tracking Platform.

Picture courtesy of Wackalope#6737

• EyeTrackVR - This project is in development and is not fully finished
  • Hardware
  • About IR Emitter Safety - Make sure you are using NON-focused emitters and at around 5ma total power
  • Firmware
  • Headset support
    • Contact
• Licenses

Documentation

EyeTrackVR

Open source and affordable VR eye tracker platform for VRChat via OSC and UDP protocol.

Note: This project is in development and is not fully finished

Documentation

Please check out our documentation site.

Hardware

3d files for mounting brackets will be found here IR emitter files are also located there. For more info please reference our documentation site

ESP-CAM Firmware

Current work has been with our official firmware by lorow, found here.

Headset support

Any headset that can fit the camera and LEDs can likely be supported, however, mounts may not have been made for it. Please reference to our parts list for details.

About IR Emitter Safety

Please exercise extreme caution when messing around with IR emitters. Once safety testing has been completed links and files will be provided for the emitters. Please make sure you know what you are doing when assembling the IR emitters. DO NOT BYPASS (OR NOT DO) ANY SAFETY FEATURES PUT IN PLACE. This can result in irreversible bodily harm. The safety measures were put in place to REDUCE the potential failure risk. All further safety responsibilities are on the user. This includes visually checking with an IR camera that the brightness is correct and making sure you do not feel warmth.

Warning: Make sure you are using NON-focused emitters and at power less than 8mW cm^2 total per eye. Please read our LED safety page for a breakdown of math for out V3 and V4 LED kits here

Effect of infrared radiation on the lens

Training-library Nir Stds

AN002_Details on photobiological safety of LED light sources

Contact

Please join our Discord for updates and any questions.

Licenses

All software is under the MIT License. All documentation, including the Wiki, is under the Creative Commons CC-BY-SA-4.0 license.

Top

eyetrackvr's Issues

Add eye openness support

On installation not working on Windows

Downloaded the .exe file from github, but only a blank cmd window is displayed on running the application and

No settings file found, using base settings displayed in cmd

Clean up code and make comments

Have you researched the eye tracking solution of Apple Vision Pro?

Eyebrow tracking

Add support for eyebrow tracking via tensorflow model

Safety docs are missing

PROBLEM DESCRIPTION

The 3 documentation documents about the safety of the IR Emitters are missing and the links are leading to a 404 page.

REQUESTED INFORMATION

Make sure your have performed every step and checked the applicable boxes before submitting your issue. Thank you!

• Read the Contributing Guide and Policy and the Code of Conduct
• Searched the problem in issues
• Searched the problem in discussions
• Searched the problem in the docs
• Searched the problem in the chat

SCREENSHOTS

Can't open documentation

The document cannot be opened. return 404

Upload esp firmware through tracker program

Allow main program to upload firmware to esp cam

Eye track crashes when detection box is too small

when you adjust the detection box too small the program crashes

Blob tracking can return multiple times

The for loop in the blob fallback for the detector will keep running after returning a frame, meaning we can stack frames for the GUI. This causes a memory leak and overflows OSC output.

Use dual eye tracking to make up for movement extremity tracking loss in the headset

We should failover to tracking movement from a single valid eye when eye tracking fails for the other.

Add app support for dual camera tracking

Add optional support for 2 camera tracking, probably by just making the main window itself a widget and putting 2 of them side by side. This will require extracting the OSC sender, so that the two cameras share the output port queue.

Eyetracking wont start without having Hsrac and crashes with any other algo

soooo i just tried this tool as the pimax crystal doesnt provide openness value's buuut it seems ive ran into an issue. hsrac does work but is extremely unstable even after calibrating so i tried others. leap seems to work but just crashes leaving behind this message

it will only work if i select hsrac again altough leap is providing way better results. is there any way i can fix this?

im running on 0.2.0 BETA 8

Add dual eye support and export/main

Allow for both eyes to be tracked with data in a single program for export to vrc mod and any future supported software

Github pages

PROBLEM DESCRIPTION

A clear and concise description of what the problem is.
Github pages link isn't in the about section making it hard for people to find docs at first glance

REQUESTED INFORMATION

Make sure your have performed every step and checked the applicable boxes before submitting your issue. Thank you!

• [ x] Read the Contributing Guide and Policy and the Code of Conduct
• [x ] Searched the problem in issues
• [ x] Searched the problem in discussions
• [ x] Searched the problem in the docs
• [ x] Searched the problem in the chat
• [ x] Device used (e.g., ESP32): _____
• [x ] EyeTrackVR binary firmware version number used: _____
  • [x ] Pre-compiled
  • [ x] Self-compiled
• [x ] Flashing tools used: _____
• [ x] Provide the output of command: Backlog Template; Module; GPIO 255:

  Configuration output here:
NONE

• If using rules, provide the output of this command: Backlog Rule1; Rule2; Rule3:

  Rules output here:

NONE

• Provide the output of this command: Status 0:

  STATUS 0 output here:
NONE

• Set weblog to 4 and then, when you experience your issue, provide the output of the Console log:

  Console output here:
NONE

TO REPRODUCE

Steps to reproduce the behavior:
look at the github page

EXPECTED BEHAVIOUR

A clear and concise description of what you expected to happen.
I wanted to open this repo and instantly go to the pages, instead because it's not easy at first glance, it may be harder to get people the help they need.

SCREENSHOTS

If applicable, add screenshots to help explain your problem.

ADDITIONAL CONTEXT

Add any other context about the problem here.

(Please, remember to close the issue when the problem has been addressed)

The automated release is failing 🚨

🚨 The automated release from the docs branch failed. 🚨

I recommend you give this issue a high priority, so other packages depending on you can benefit from your bug fixes and new features again.

You can find below the list of errors reported by semantic-release. Each one of them has to be resolved in order to automatically publish your package. I’m sure you can fix this 💪.

Errors are usually caused by a misconfiguration or an authentication problem. With each error reported below you will find explanation and guidance to help you to resolve it.

Once all the errors are resolved, semantic-release will release your package the next time you push a commit to the docs branch. You can also manually restart the failed CI job that runs semantic-release.

If you are not sure how to resolve this, here are some links that can help you:

• Usage documentation
• Frequently Asked Questions
• Support channels

If those don’t help, or if this issue is reporting something you think isn’t right, you can always ask the humans behind semantic-release.

---

Invalid publishCmd option.

The publishCmd option is required and must be a non empty String.

Your configuration for the publishCmd option is ``.

---

Good luck with your project ✨

Your semantic-release bot 📦🚀

Connect to VRC face tracking mod

Connection to tcp://192.168.4.1:80 failed: Error number -138 occurred

Connection to tcp://192.168.4.1:80 failed: Error number -138 occurred

Camera address settings need explicit save/reset buttons in app

Right now the camera address entry saves on every letter typed. This should actually be a button that the user hits to save explicitly, so it's obvious that things are saving.

Better error output and no more requirement of launching gui first to not crash

Build capabilities for in-world calibration triggering

Build an OSC listener in the EyeTrackApp that will allow for triggering of re-centering/re-framing in world.

Smooth eye data

Provide a method for smoothing out eye data for more steady output.

OSC Messages only queued when not blinking

OSC message queuing happens in a logic branch only run if we're not currently blinking.

Corrupted config causes crash

PROBLEM DESCRIPTION

The application crashes on startup if the config gets currupted.

REQUESTED INFORMATION

Make sure your have performed every step and checked the applicable boxes before submitting your issue. Thank you!

• Read the Contributing Guide and Policy and the Code of Conduct
• Searched the problem in issues
• Searched the problem in discussions
• Searched the problem in the docs
• Searched the problem in the chat
• Device used (e.g., ESP32): -
• EyeTrackVR binary firmware version number used: v0.1.8.1
  • Pre-compiled
  • Self-compiled
• Flashing tools used: -
• Provide the output of command: Backlog Template; Module; GPIO 255:

  Configuration output here: -

• If using rules, provide the output of this command: Backlog Rule1; Rule2; Rule3:

  Rules output here: -

• Provide the output of this command: Status 0:

  STATUS 0 output here: -

• Set weblog to 4 and then, when you experience your issue, provide the output of the Console log:

  Console output here: -

TO REPRODUCE

Steps to reproduce the behavior:

eyetrack_settings.json
Replace the config with the corrupted config I had. It got corrupted due to the perfect timing of a bluescreen.

EXPECTED BEHAVIOUR

A clear and concise description of what you expected to happen.

Showing an error. (And maybe asking the user if he wants to get the config deleted.)

SCREENSHOTS

If applicable, add screenshots to help explain your problem.

ADDITIONAL CONTEXT

Add any other context about the problem here.

(Please, remember to close the issue when the problem has been addressed)

Foveated rendering support

Add some form of foveated rendering support either through mod, or directly. Requires lower latency of camera feed transmission.

Hardware mounts for other headsets

Create hardware mounts for other headsets like the Index, Vive Pro, HP Reverb G2 etc. Help would be appreciated if you have a unsupported headset.

github pages link

Put github pages link in about section

Allow app users to pull a single frame from a camera, versus only allowing streaming

When figuring out camera addresses and setting initial ROI, it would be nice to be able to send a one-shot frame request, versus constantly streaming. Streaming can cause massive memory backups if, say, a 4k camera is accidentally selected.

Graph should change color on events (failure, blink)

PySimpleGUI graph isn't updating on events. It should show blue when blinking, red during failure.

Create documentation

Create docs with hardware needed, setup instructions and info on the project.

USB Cameras with similar VID/PID can collide

Trying to hook 2 wired OV5640 cameras to the same USB bus causes collision issues. The cameras can be seen by the host machine, but any effort to stream video from both at the same time ends up in only the first one connected receiving any frames. This may be an issue with how UVC is processed in OpenCV.

Make color image display selectable in app

Color image isn't really important once tracking is going, so it's taking up valuable UI real estate.

Auto camera resolution detection and set

Allow the search box to be auto configured to create an easier setup process.

USB UVC cameras may need settings (Gain/WB) adjustment

USB UVC cameras may have gain and auto white balance turned on by default. This can cause issues with threshold boundaries inside the headset, if settings change for relative light/dark.