saharsh1005/nod-o-meter

Ask a question to the user, track head movement via webcam, classify nod as Yes or No. Track human gesture

Nod-o-meter

Nod-o-meter is a high-tech nod tracking app that uses your webcam to answer questions by nodding your head. It's not your average nodding game; it's designed to uncover the deepest secrets of your nods! Are you ready to nod your way to enlightenment? Let the nodding begin!

Demo Video Link

Watch the video

Usage

1. Clone this repo.
2. Create conda env with conda create --name test_nodometer python=3.8.18, conda activate test_nodometer install requirements pip install -r requirements.txt
3. Run app python web/app.py
4. Click Start to begin logging, then click on Play to start playing nod-o-meter.
5. Nod your head vertically to answer "yes" and shake your head to answer "no".
6. The app will log your first response in the log table below.
7. To continue playing click on reset followed by play again.
8. Enjoy :p

Features

• RealTime Webcam nod tracking: Uses live webcam feed for nod tracking
• Question Prompt: Provides a variety of questions to answer
• Logging: Logs your responses for each question
• Documentation ReadMe with added Issues(features enhancements, bugs etc) + demo video
• Dockerize the app for easier deployment (wip - rectify link to webcam from container)

Application Components

Further Enhancements

• Improve the computer vision algorithm for more accurate nod tracking (TBD)
• Generate questions using a large language model (LLM) for more personalized experiences (TDB)
• Improve the user interface

Bug Reports and Open Source Contribution

Please report any bugs you find to the issues page. We welcome open source contributions!

Contributor

_Saharsh

Repo Link

Github repo link: https://github.com/Saharsh1005/nod-o-meter

Reference

Git-gist

nod-o-meter's Issues

Feature: Logging logic

Present implementation logs the first valid (Yes or No) movement as the response for the question. Can we make this more robust?
Log based on the longer response, some confidence threshold

Computer Vision: Improve gesture tracking algorithm

Currently, detecting a point in the face (near forehead) and using its movements across frames defining the gesture.
Incremental suggestions:

1. Instead of simply taking any point, take key feature points (nose or eyes or mean pt of their coordinates)
2. Use DL based head_pose_estimation model (DL or yaw/roll/pitch movement -> classification) : [link1] [link2]

LLM: Enhance question generation

Present version utilizes a pool of questions and selects a question from that, instead can we use LLM (API or local) to generate a question?

Method 1: OPENAPI [link to get started]

Bug: Docker container - unable to find webcam

Upon running the app from a docker container, the app is unable to find the webcam.

$ docker run -p 5000:5000 human_nod
[ WARN:[email protected]] global cap_v4l.cpp:997 open VIDEOIO(V4L2:/dev/video0): can't open camera by index
[ERROR:[email protected]] global obsensor_uvc_stream_channel.cpp:159 getStreamChannelGroup Camera index out of range
Traceback (most recent call last):
  File "web/app.py", line 42, in <module>
    socketio.run(app, debug=True)
  File "/usr/local/lib/python3.8/site-packages/flask_socketio/__init__.py", line 640, in run
    raise RuntimeError('The Werkzeug web server is not '
RuntimeError: The Werkzeug web server is not designed to run in production. Pass allow_unsafe_werkzeug=True to the run() method to disable this error.

Try: docker run -it --rm --device=/dev/video0 -p 5000:5000 my-flask-app

docker: Error response from daemon: error gathering device information while adding custom device "/dev/video0": no such file or directory.

Feature: Improve frontend - no need to click reset after each quesiton

Currently, to ensure logging happens correctly for each quesiton posed, user needs to reset after each question.