thohemp/nitec

NITEC: Versatile Hand-Annotated Eye Contact Dataset for Ego-Vision Interaction (WACV24)

NITEC: Versatile Hand-Annotated Eye Contact Dataset for Ego-Vision Interaction (WACV24)

Citing

If you find our work useful, please cite the paper:

@INPROCEEDINGS{10484276,
  author={Hempel, Thorsten and Jung, Magnus and Abdelrahman, Ahmed A. and Al-Hamadi, Ayoub},
  booktitle={2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)}, 
  title={NITEC: Versatile Hand-Annotated Eye Contact Dataset for Ego-Vision Interaction}, 
  year={2024},
  pages={4425-4434},
  doi={10.1109/WACV57701.2024.00438}}

Paper

Thorsten Hempel, Magnus Jung, Ahmed A. Abdelrahman and Ayoub Al-Hamadi, "NITEC: Versatile Hand-Annotated Eye Contact Dataset for Ego-Vision Interaction", WACV 2024.

Abstract

Eye contact is a crucial non-verbal interaction modality and plays an important role in our everyday social life. While humans are very sensitive to eye contact, the capabilities of machines to capture a person's gaze are still mediocre. We tackle this challenge and present NITEC, a hand-annotated eye contact dataset for ego-vision interaction. NITEC exceeds existing datasets for ego-vision eye contact in size and variety of demographics, social contexts, and lighting conditions, making it a valuable resource for advancing ego-vision-based eye contact research. Our extensive evaluations on NITEC demonstrate strong cross-dataset performance, emphasizing its effectiveness and adaptability in various scenarios, that allows seamless utilization to the fields of computer vision, human-computer interaction, and social robotics. We make our NITEC dataset publicly available to foster reproducibility and further exploration in the field of ego-vision interaction.

Quick Usage:

pip install face_detection@git+https://github.com/elliottzheng/face-detection
pip install nitec

Example usage:

from nitec import NITEC_Classifier, visualize
import cv2

nitec_pipeline = NITEC_Classifier(
    weights= CWD / 'models' / 'nitec_rs18_e20.pth',
    device=torch.device('cuda') # or 'cpu'
)

cap = cv2.VideoCapture(0)

_, frame = cap.read()    
# Process frame and visualize
results = nitec_pipeline.predict(frame)
frame = visualize(frame, results, confidence=0.5)

Train / Test

NITEC Dataset

Prepare the dataset as explained here.

Snapshots

Download from here: https://drive.google.com/drive/folders/1zc6NZZ6yA4NJ52Nn0bgky1XpZs9Z0hSJ?usp=sharing

Train

 python train.py \
 --gpu 0 \
 --num_epochs 50 \
 --batch_size 64 \
 --lr 0.0001 \

Test

 python test.py \
 --snapshot models/nitec_rs18_20.pth \
 --gpu 0 \