Giter Site home page Giter Site logo

epic-kitchens / epic-kitchens-55-annotations Goto Github PK

View Code? Open in Web Editor NEW
148.0 14.0 26.0 30.11 MB

🍴 Annotations for the EPIC KITCHENS-55 Dataset.

Home Page: https://epic-kitchens.github.io/2020

License: Other

Python 100.00%
egocentric-action-recognition action-recognition object-detection dataset deep-learning video first-person-video egocentric

epic-kitchens-55-annotations's Introduction

CircleCI GitHub release arXiv-1804.02748 Data

EPIC-KITCHENS-55 is the largest dataset in first-person (egocentric) vision; 55 hours of multi-faceted, non-scripted recordings in native environments - i.e. the wearers' homes, capturing all daily activities in the kitchen over multiple days. Annotations are collected using a novel `live' audio commentary approach.

Authors

Dima Damen (1) Hazel Doughty (1) Giovanni Maria Farinella (3) Sanja Fidler (2) Antonino Furnari (3) Evangelos Kazakos (1) Davide Moltisanti (1) Jonathan Munro (1) Toby Perrett (1) Will Price (1) Michael Wray (1)

  • (1 University of Bristol)
  • (2 University of Toronto)
  • (3 University of Catania)

Contact: [email protected]

Citing

When using the dataset, kindly reference:

@INPROCEEDINGS{Damen2018EPICKITCHENS,
   title={Scaling Egocentric Vision: The EPIC-KITCHENS Dataset},
   author={Damen, Dima and Doughty, Hazel and Farinella, Giovanni Maria  and Fidler, Sanja and 
           Furnari, Antonino and Kazakos, Evangelos and Moltisanti, Davide and Munro, Jonathan 
           and Perrett, Toby and Price, Will and Wray, Michael},
   booktitle={European Conference on Computer Vision (ECCV)},
   year={2018}
} 

(Check publication here)

Dataset Details

Ground Truth

We provide ground truth for action segments and object bounding boxes.

  • Objects: Full bounding boxes of narrated objects for every annotated frame.
  • Actions: Split into narrations and action labels:
    • Narrations containing the narrated sentence with the timestamp.
    • Action labels containing the verb and noun labels along with the start and end times of the segment.

Dataset Splits

The dataset is comprised of three splits with the corresponding ground truth:

  • Training set - Full ground truth.
  • Seen Kitchens (S1) Test set - Start/end times only.
  • Unseen Kitchens (S2) Test set - Start/end times only.

Initially we are only releasing the full ground truth for the training set in order to run action and object challenges.

Important Files

Additional Files

We direct the reader to RDSF for the videos and rgb/flow frames.

We provide html and pdf alternatives to this README which are auto-generated.

Files Structure

EPIC_train_action_labels.csv

CSV file containing 14 columns:

Column Name Type Example Description
uid int 6374 Unique ID of the segment.
video_id string P03_01 Video the segment is in.
narration string close fridge English description of the action provided by the participant.
start_timestamp string 00:23:43.847 Start time in HH:mm:ss.SSS of the action.
stop_timestamp string 00:23:47.212 End time in HH:mm:ss.SSS of the action.
start_frame int 85430 Start frame of the action (WARNING only for frames extracted as detailed in Video Information).
stop_frame int 85643 End frame of the action (WARNING only for frames extracted as detailed in Video Information).
participant_id string P03 ID of the participant.
verb string close Parsed verb from the narration.
noun string fridge First parsed noun from the narration.
verb_class int 3 Numeric ID of the parsed verb's class.
noun_class int 10 Numeric ID of the parsed noun's class.
all_nouns list of string (1 or more) ['fridge'] List of all parsed nouns from the narration.
all_noun_classes list of int (1 or more) [10] List of numeric IDs corresponding to all of the parsed nouns' classes from the narration.

Please note we have included a python pickle file for ease of use. This includes a pandas dataframe with the same layout as above. This pickle file was created with pickle protocol 2 on pandas version 0.22.0.

EPIC_train_invalid_labels.csv

CSV file containing 14 columns:

Column Name Type Example Description
uid int 6374 Unique ID of the segment.
video_id string P03_01 Video the segment is in.
narration string close fridge English description of the action provided by the participant.
start_timestamp string 00:23:43.847 Start time in HH:mm:ss.SSS of the action.
stop_timestamp string 00:23:47.212 End time in HH:mm:ss.SSS of the action.
start_frame int 85430 Start frame of the action (WARNING only for frames extracted as detailed in Video Information).
stop_frame int 85643 End frame of the action (WARNING only for frames extracted as detailed in Video Information).
participant_id string P03 ID of the participant.
verb string close Parsed verb from the narration.
noun string fridge First parsed noun from the narration.
verb_class int 3 Numeric ID of the parsed verb's class.
noun_class int 10 Numeric ID of the parsed noun's class.
all_nouns list of string (1 or more) ['fridge'] List of all parsed nouns from the narration.
all_noun_classes list of int (1 or more) [10] List of numeric IDs corresponding to all of the parsed nouns' classes from the narration.

Please note we have included a python pickle file for ease of use. This includes a pandas dataframe with the same layout as above. This pickle file was created with pickle protocol 2 on pandas version 0.22.0.

EPIC_train_action_narrations.csv

CSV file containing 5 columns:

Note: The start/end timestamp refers to the start/end time of the narration, not the action itself.

Column Name Type Example Description
participant_id string P03 ID of the participant.
video_id string P03_01 Video the segment is in.
start_timestamp string 00:23:43.847 Start time in HH:mm:ss.SSS of the narration.
stop_timestamp string 00:23:47.212 End time in HH:mm:ss.SSS of the narration.
narration string close fridge English description of the action provided by the participant.

EPIC_train_object_labels.csv

CSV file containing 6 columns:

Column Name Type Example Description
noun_class int 20 Integer value representing the class in noun-classes.csv.
noun string bag Original string name for the object.
participant_id string P01 ID of participant.
video_id string P01_01 Video the object was annotated in.
frame int 056581 Frame number of the annotated object.
bounding_boxes list of 4-tuple (0 or more) "[(76, 1260, 462, 186)]" Annotated boxes with format (<top:int>,<left:int>,<height:int>,<width:int>).

EPIC_train_object_action_correspondence.csv

CSV file containing 5 columns:

Column Name Type Example Description
participant_id string P01 ID of participant.
video_id string P01_01 Video the frames are part of.
object_frame int 56581 Frame number of the object detection image from object_detection_images.
action_frame int 56638 Frame number of the corresponding image in the released frames for action recognition in frames_rgb_flow.
timestamp string 00:00:00.00 Timestamp in HH:mm:ss.SS corresponding to the frame.

Please note we have included a python pickle file for ease of use. This includes a pandas dataframe with the same layout as above. This pickle file was created with pickle protocol 2 on pandas version 0.22.0.

EPIC_test_s1_object_action_correspondence.csv

CSV file containing 5 columns:

Column Name Type Example Description
participant_id string P01 ID of participant.
video_id string P01_11 Video containing the object s1 test frames.
object_frame int 33601 Frame number of the object detection image from object_detection_images.
action_frame int 33635 Frame number of the corresponding image in the released frames for action recognition in frames_rgb_flow.
timestamp string 00:09:20.58 Timestamp in HH:mm:ss.SS corresponding to the frames.

Please note we have included a python pickle file for ease of use. This includes a pandas dataframe with the same layout as above. This pickle file was created with pickle protocol 2 on pandas version 0.22.0.

EPIC_test_s2_object_action_correspondence.csv

CSV file containing 5 columns:

Column Name Type Example Description
participant_id string P09 ID of participant.
video_id string P09_05 Video containing the object s2 test frames.
object_frame int 15991 Frame number of the object detection image from object_detection_images.
action_frame int 16007 Frame number of the corresponding image in the released frames for action recognition in frames_rgb_flow.
timestamp string 00:04:26.78 Timestamp in HH:mm:ss.SS corresponding to the frames.

Please note we have included a python pickle file for ease of use. This includes a pandas dataframe with the same layout as above. This pickle file was created with pickle protocol 2 on pandas version 0.22.0.

EPIC_test_s1_object_video_list.csv

CSV file listing the videos used to obtain the object s1 test frames. The frames can be obtained from RDSF under object_detection_images/test. Please test all frames from this folder for the videos listed in this csv.

Column Name Type Example Description
video_id string P01_11 Video containing the object s1 test frames.
participant_id string P01 ID of the participant.

EPIC_test_s2_object_video_list.csv

CSV file listing the videos used to obtain the object s2 test frames. The frames can be obtained from RDSF under object_detection_images/test. Please test all frames from this folder for the videos listed in this csv.

Column Name Type Example Description
video_id string P01_11 Video containing the object s2 test frames.
participant_id string P01 ID of the participant.

EPIC_test_s1_timestamps.csv

CSV file containing 7 columns:

Column Name Type Example Description
uid int 1924 Unique ID of the segment.
participant_id string P01 ID of the participant.
video_id string P01_11 Video the segment is in.
start_timestamp string 00:00:00.000 Start time in HH:mm:ss.SSS of the action.
stop_timestamp string 00:00:01.890 End time in HH:mm:ss.SSS of the action.
start_frame int 1 Start frame of the action (WARNING only for frames extracted as detailed in Video Information).
stop_frame int 93 End frame of the action (WARNING only for frames extracted as detailed in Video Information).

Please note we have included a python pickle file for ease of use. This includes a pandas dataframe with the same layout as above. This pickle file was created with pickle protocol 2 on pandas version 0.22.0.

EPIC_test_s2_timestamps.csv

CSV file containing 7 columns:

Column Name Type Example Description
uid int 15582 Unique ID of the segment.
participant_id string P09 ID of the participant.
video_id string P09_01 Video the segment is in.
start_timestamp string 00:00:01.970 Start time in HH:mm:ss.SSS of the action.
stop_timestamp string 00:00:03.090 End time in HH:mm:ss.SSS of the action.
start_frame int 118 Start frame of the action (WARNING only for frames extracted as detailed in Video Information).
stop_frame int 185 End frame of the action (WARNING only for frames extracted as detailed in Video Information).

Please note we have included a python pickle file for ease of use. This includes a pandas dataframe with the same layout as above. This pickle file was created with pickle protocol 2 on pandas version 0.22.0.

EPIC_noun_classes.csv

CSV file containing 3 columns:

Note: a colon represents a compound noun with the more generic noun first. So pan:dust should be read as dust pan.

Column Name Type Example Description
noun_id int 2 ID of the noun class.
class_key string pan:dust Key of the noun class.
nouns list of string (1 or more) "['pan:dust', 'dustpan']" All nouns within the class (includes the key).

EPIC_verb_classes.csv

CSV file containing 3 columns:

Column Name Type Example Description
verb_id int 3 ID of the verb class.
class_key string close Key of the verb class.
verbs list of string (1 or more) "['close', 'close-off', 'shut']" All verbs within the class (includes the key).

EPIC_descriptions.csv

CSV file containing 4 columns:

Column Name Type Example Description
video_id string P01_01 ID of the video.
date string 30/04/2017 Date on which the video was shot.
time string 13:49:00 Local recording time of the video.
description string prepared breakfast with soy milk and cereals Description of the activities contained in the video.

EPIC_many_shot_verbs.csv

CSV file containing the many shot verbs. A verb class is considered many shot if it appears more than 100 times in training. (NOTE: this file is derived from EPIC_train_action_labels.csv, checkout the accompanying notebook demonstrating how we compute these classes)

Column Name Type Example Description
verb_class int 1 Numeric ID of the verb class
verb string put Verb corresponding to the verb class

EPIC_many_shot_nouns.csv

CSV file containing the many shot nouns. A noun class is considered many shot if it appears more than 100 times in training. (NOTE: this file is derived from EPIC_train_action_labels.csv, checkout the accompanying notebook demonstrating how we compute these classes)

Column Name Type Example Description
noun_class int 3 Numeric ID of the noun class
noun string tap Noun corresponding to the noun class

EPIC_many_shot_actions.csv

CSV file containing the many shot actions. An action class (composed of a verb class and noun class) is considered many shot if BOTH the verb class and noun class are many shot AND the action class appears in training at least once. (NOTE: this file is derived from EPIC_train_action_labels.csv, checkout the accompanying notebook demonstrating how we compute these classes)

Column Name Type Example Description
action_class (int, int) (9, 84) Numeric Pair of IDs, first the verb, then the noun
verb_class int 9 Numeric ID of the verb class
verb string move Verb corresponding to the verb class
noun_class int 84 Numeric ID of the noun class
noun string sausage Noun corresponding to the noun class

EPIC_video_info.csv

CSV file containing information for each video

Column Name Type Example Description
video (string) P01_01 Video ID
resolution (string) 1920x1080 Resolution of the video, format is WIDTHxHEIGHT
duration (float) 1652.152817 Duration of the video, in seconds
fps (float) 59.9400599400599 Frame rate of the video

File Downloads

Due to the size of the dataset we provide scripts for downloading parts of the dataset:

Note: These scripts will work for Linux and Mac. For Windows users a bash installation should work.

These scripts replicate the folder structure of the dataset release, found here.

If you wish to download part of the dataset instructions can be found here.

Video Information

Videos are recorded in 1080p at 59.94 FPS on a GoPro Hero 5 with linear field of view. There are a minority of videos which were shot at different resolutions, field of views, or FPS due to participant error or camera. These videos identified using ffprobe are:

  • 1280x720: P12_01, P12_02, P12_03, P12_04.
  • 2560x1440: P12_05, P12_06
  • 29.97 FPS: P09_07, P09_08, P10_01, P10_04, P11_01, P18_02, P18_03
  • 48 FPS: P17_01, P17_02, P17_03, P17_04
  • 90 FPS: P18_09

The GoPro Hero 5 was also set to drop the framerate in low light conditions to preserve exposure leading to variable FPS in some videos. If you wish to extract frames we suggest you resample at 60 FPS to mitigate issues with variable FPS, this can be achieved in a single step with FFmpeg:

ffmpeg -i "P##_**.MP4" -vf "scale=-2:256" -q:v 4 -r 60 "P##_**/frame_%010d.jpg"

where ## is the Participant ID and ** is the video ID.

Optical flow was extracted using a fork of gpu_flow made available on github. We set the parameters: stride = 2, dilation = 3, bound = 25 and size = 256.

License

All files in this dataset are copyright by us and published under the Creative Commons Attribution-NonCommerial 4.0 International License, found here. This means that you must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use. You may not use the material for commercial purposes.

Disclaimer

EPIC-KITCHENS-55 and EPIC-KITCHENS-100 were collected as a tool for research in computer vision, however, it is worth noting that the dataset may have unintended biases (including those of a societal, gender or racial nature).

Changelog

See release history for changelog.

epic-kitchens-55-annotations's People

Contributors

antoninofurnari avatar hazeld avatar mwray avatar willprice avatar yassersouri avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

epic-kitchens-55-annotations's Issues

evaluation classes

Hi,

thank you for grate dataset, I'd like to eval my trained model with your test data.
I know EPIC-Kitchens Object Detection challenge has submission form, but I don't know what classes should be detected.

Should my model detect all noun classes(784 classes) or 15 most frequent object classes?

thanks.

How do you parse nouns from narrations?

Hi,

I am curious to know the exact method you used to gather nouns from narrations. I have used the following code:


nlp = spacy.load('en') 
sentence =  "place bottle"
doc = nlp(sentence)
pos = [tag for tag in doc.noun_chunks]

However, the place gets incorrectly classified as a Noun. How do you overcome this?

Concerning the separate parts of the dataset

You mention that "Due to the size of the dataset we provide three scripts for downloading the videos, frames or object annotation images separately". Would it be possible to display the size of each of these separate parts of the dataset?
I am interested in applying a model to extract saliency maps from your data, but the 1TB is too large in size and also to compute, potentially.

Missing classes

Hi,

Thanks for the dataset. I started to use your dataset for my experiments but just found that some of the classes (noun or verb) are missing! Did I miss something?

Thanks,

Mohammadreza

How to evaluate our model

Hello.

I'm going to try Action Recognition by using this dataset, but annotations for test set is not published(I know this is because in order to run challenge).
So, how can I test our model ?

Is bounding boxes order in README.md correct?

Hello,
I'm really interested in the dataset, but I have a question in the dataset annotation.

In the description of bounding boxes, its order is (<top:int>,<left:int>,<width:int>,<height:int>), but is it correct?
I tried to visualize the bounding boxes, but it seems weird as below.
I thought (<top:int>,<left:int>,<height:int>,<width:int>) is the correct order, but Im not sure.
I want to make sure the bounding boxes order in the annotation.

vis_epic_kitchens

Mistaken values?

I'm parsing your EPIC_train_object_labels.csv, but I found mistaken annotation.

In video_id:P04_03, frame: 15541, bounding_boxes indicated [(872, 958, 210, 140)].
So It should be top:int + height:int = 1082 but this video resolution is 1920x1080.

question about the frame annotation

Hi,Thanks for the release of this dataset.
In EPIC_video_info.csv,duration of P29_03 is 2074.672583,and fps is 59.9400599400599. So total frame is duration*fps= 124355.But in EPIC_train_action_labels.csv,uid 35757,35758 and 35759,their start_frame and stop_frame is bigger than 124355.The same thing happens in other videos.Is this a mistake?

Annotations: test set

Hi,

Thanks for the release of this dataset, this is absolutely awesome for the community.
When do you plan to release the annotations for the test set? Or at least open the leaderboard?
Thanks,
Best

Submission failed

I've finished training with your train dataset, and testing the detection with test video frame.
Once I've submitted my detection result to CodaLab, an error occurred ValueError: Image: P01_12_2761 has 0 bounding boxes not 300.

In my inference by trained model, no bounding boxes were detected in P01_12/0000002761.jpg.
So please tell me how to output the result for no-bounding boxes in JSON object?

Best,

Official annotations for Anticipation

Actually, I think it should be better to provide the "official annotations" for action anticipation along with action recognition task. For example, it is the 1s time stamp starting and ending of the anticipation segment.

Of course, we are able to modify the code for extracting these frame segments. However, It would be fairer if we have an "exact frame segments" provided by the dataset.

Pretrained Faster R-CNN inference over Test sets having much lower evaluation metrics

I have used the pretrained Faster R-CNN model you provided and the TF object detection API to inference over the seen and unseen test sets and submitted the results to the CodaLab platform. My evaluation metrics are way lower than yours, stated in the paper. I have included to the submitted json files all the 300 proposed regions for each frame, with their respective confidence scores even thought the vast majority of them have very low scores. Shouldn't I have my evaluation metrics being close to yours? Am i missing something?

Thank you in advance

How to apply to participate in the EPIC-KITCHENS-55 Object Detection leaderboard?

Hello, I have applied to participate in the EPIC-KITCHENS-55 Object Detection leaderboard and would like to test on the EPIC-KITCHENS-55 test set.
The CodaLab website shows "Your request to participate in this challenge has been received and a decision is pending.", but I have not received a response for a long time.
I can't find the ground truth file of test dataset in this repository and the competition EPIC-KITCHENS-55 Object Detection has ended, so how can I get the test accuracy of action recognition benchmark? I want to get the test accuracy of my model both on epic-kitchens-55 datasets.

How can I perform action recognition experiments on a subset of the published test data?

Hello there! Thank you for creating and maintaining a great project!
In my case I am working with a subset ,i.e., some of the kitchens, of the epic-kitchens-55 dataset. I think I need ground truth test action labels, since I am not using the entire dataset to perform my experiments. Basically, I want to calculate accuracy on the actions from certain kitchens and not the whole dataset, so I guess the epic-55 leaderboard option will not do. How can I get the the test accuracy in my case using a subset of the published test data?

evaluation categories in object detection

Hi, thank you for the charming work of the dataset.
I have a question about the classes evaluated:
The class_map provided in your object detection pre-trained model has only 290 catogories. So I am confused about what collection of categories would be evaluated.

Ref:

Thank you for your interest in the dataset. Your model should detect all clustered noun classes which appear in the training set (of which there are 295). You can find the noun classes for each bounding box in the noun_class column of the EPIC_train_object_labels.csv. The class label and names of objects contained in each class can be found in EPIC_noun_classes.csv.

Originally posted by @hazeld in #14 (comment)

Object Detection Submission Failed

As recommended on CodaLab website, for the object detection challenge, I'm submitting 300 boxes per image (each of them rounded to 5 decimals). However, my submission fails without providing any qualitative errors. Here's the text from the log of the submission file:

elapsedTime: 93.22970509529114
exitCode: 137
ingestionExitCode: 0

or another one that took much longer to fail:

elapsedTime: 8046.879673957825
exitCode: 137
ingestionExitCode: 0

I have successfully evaluated some submissions before. However, I noticed that sometimes the submissions fail, if there are too many bounding boxes included (even if it's < 300 boxes per image). How do I fix this issue?

I also noticed that sometimes I get "Execution time limit exceeded!" error.

Where are the multiple language narrations?

Hi,
Where is the download link for multi-language narrations or the corresponding data source? As mentioned in the paper or the homepage of the dataset, one of the characteristics is Multi-language Narrations.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.