m-and-ms/localizingmoments

Github for my ICCV 2017 paper: "Localizing Moments in Video with Natural Language"

Localizing Moments in Video with Natural Language.

Hendricks, Lisa Anne, et al. "Localizing Moments in Video with Natural Language." ICCV (2017).

Find the paper here and the project page here.

@inproceedings{hendricks17iccv, 
        title = {Localizing Moments in Video with Natural Language.}, 
        author = {Hendricks, Lisa Anne and Wang, Oliver and Shechtman, Eli and Sivic, Josef and Darrell, Trevor and Russell, Bryan}, 
        booktitle = {Proceedings of the IEEE International Conference on Computer Vision (ICCV)}, 
        year = {2017} 
}

License: BSD 2-Clause license

Running the Code

Preliminaries: I trained all my models with the BVLC caffe version. Before you start, look at "utils/config.py" and change any paths as needed (e.g., perhaps you want to point to a Caffe build in a different folder).

Evaluation

Look at "utils/eval.py" if you would like to evaluate a model that you have trained. Below are instructions to eval the models I proposed in my paper:

• Download data/models with "download/get_models.sh". This should download models I trained and pre-extracted features. Note that I retrained my models before releasing and the numbers are slightly different than those reported in the paper.
• Run "test_network.sh". This will run both RGB and flow models on the val and test sets. It will also produce the scorse for the fusion model.

You should get the following outputs:

	Rank@1	Rank@5	mIOU
RGB val	0.2442	0.7540	0.3739
Flow val	0.2626	0.7839	0.4015
Fusion val (lambda 0.5)	0.2765	0.7961	0.4191
RGB test	0.2312	0.7336	0.3549
Flow test	0.2583	0.7540	0.3894
Fusion test (lambda 0.5)	0.2708	0.7853	0.4053

Training

Use "run_job_rgb.sh" to train an RGB model and "run_job_flow.sh" to train a flow model. You should be able to rerun these scripts and get simiar numbers to those reported in the paper.

Dataset

Annotations

To access the dataset, please look at the json files in the "data" folder. Our annotations include descriptions which are temporally grounded in videos. For easier annotation, each video is split into 5-second temporal chunks. The first temporal chunk correpsonds to seconds 0-5 in the video, the second temporal chunk correpsonds to seconds 5-10, etc. The following describes the different fields in the json files:

• annotation_id: Annotation ID for description
• description: Description for a specific video segment
• video: Video name
• times: Ground truth time points marked by annotators. The time points indicate which chunk includes the start of the moment and which chunk includes the end of the moment. An annotation of (3,3) indicates that a moment starts at 3x5=15 seconds and ends at (3+1)x5=20 seconds. An annotation of (1,4) indicates that a moment starts at 1x5=5 seconds and ends at (4+1)x5=20 seconds. Note that the last segment is not always 5 seconds long. For example, for a video which is 28.2 seconds long, the annotation (5,5) will correpsond to 5x5=25 seconds to min((5+1)x5 seconds, video length) = 28.2 seconds. Some videos are longer than 30 seconds. These videos were truncated to 30 seconds during annotation.
• download_link: A download link for the video. Unfortunately, this download link does not work for many Flickr videos anymore. See "Getting the Videos" for more details.
• num_segments: Some videos are a little shorter than 25 seconds, so were split into five temporal chunks instead of six.

Getting the Videos

1. Download videos from AWS (preferred method). YFCC100M images and videos are stored on AWS here. Because many videos have been deleted off of Flickr since I collected the dataset, it is best to access the videos stored on AWS instead of trying to download directly from Flickr. To download the videos used in my dataset use the script download_videos_AWS.py:

python download_videos_AWS.py --download --video_directory DIRECTORY

There are 13 videos which are not on AWS which you may download from my website here (I don't have enough space to store all the videos on my website -- Sorry!)

2. Download videos directly from Flickr. This is what I did when collecting the dataset, but now many Flickr videos have been deleted and many people have had issues running my download script. To download videos directly from Flickr:

Use the script download_videos.py: python download_videos.py --download --video_directory DIRECTORY

When I originally released the dataset, ~3% of the original videos had been deleted from Flickr. You may access them here. If you find that more videos are missing, please download the videos via the AWS links above.

3. Download from Google Drive.

You can view the Creative Commons licenses in "video_licenses.txt".

Pre-Extracted Features

You can access preextracted features for RGB here and for flow here. These are automatically downloaded in "download/get_models.sh". To extract flow, I used the code here.

I provide re-extracted features in the Google Drive above. You can use this script to create a dict with averaged RGB features and this script. The average features will be a bit different than the original release, but did not influence any trends in the results.