About generate_descriptions_lists.py

[lvapeab]

About generate_descriptions_lists.py

[lvapeab]

Hello,
caption.id.en contains the captions from each video, with the following format:
video_id#number_of_caption----caption

The ---- is an arbitrary separator that can be changed in generate_descriptions_lists.py. I provide an insight of my file:

mv89psg6zh4_33_46#1----A bird is bathing in a sink.
mv89psg6zh4_33_46#2----A bird is splashing around under a running faucet.
mv89psg6zh4_33_46#3----A bird is bathing in a sink.
[...]
mv89psg6zh4_33_46#28----A bird is drinking out of the sink.
ZbzDGXEwtGc_6_15#0----a plane flying in the sky
ZbzDGXEwtGc_6_15#1----A commercial plane flying.
ZbzDGXEwtGc_6_15#2----A jet is flying.

The *_list.txt files, contain each a list of the video_id belonging to each split. The default splits of the MSVD task contain 1200, 100 and 670 samples for the train, val and test splits, respectively. Therefore, the *_list.txt files should have these numbers of lines each. Each line relating to a video_id. E.g.:

-DRy7rBg0IQ_31_37
0lh_UWF9ZP4_79_82
Vn0Xv0fb1z4_18_30

train_descriptions_counts.npy contains what you said. I take your suggestion and I'll try to upload some sample files in a few days.

Cheers!

[Hsiao-hsu-Li]

Thank you for your reply.

I have another question about the subsample_frames_features.py.

• In line 52 to 57 in this script,

`# Get chosen frames

    for p in pick_pos:
        of.write(feats[p])
        oc.write(str(count_pick)+'\n')
        if count_pick != n_frames_per_video_subsample:
            print "different", count_videos
            print "num", count_pick

This means every time we write a frame feat of a video to file, we will write the total number of the video to the file once. Should line 54-57 be outdented so that only after we write all frame feats of a video to file, we will write the total number of the video to the count file?

[lvapeab]

Hello,
This is the format required by generate_features_lists.py. This script generates the features in the final format required by the system, and makes what you said.

Cheers!

[Hsiao-hsu-Li]

In generate_features_lists.py, only after a video was processed entirely, the total frame number of a video will be stored in 'train_feat_counts.txt'.

But in subsample_frames_features.py, each time we write a feat of one frame of a video, the total frame number of a video will be stored in the out_features_counts text.

Therefore, I want to know whether in the feature.csv file, each row is a feature vector of one frame of a video?

[lvapeab]

The subsample_frames_features.py can be seen as a preprocessing stage, for alleviating some computational overhead of the following steps.

Therefore, both scripts recieve a .csv file with each row representing one frame of a video.

The subsample_frames_features.py will convert a .csv file with features from every frame from each video to another .csv with a fixed number of frames per video.

For instance, I show you my data from the MSVD training set. It contains 1,200 videos. I extracted features with ImageNet for each frame. Therefore, I have a large .csv with one feature per frame:

$ wc -l train_ImageNet_features.csv
317881 train_ImageNet_features.csv

If we subsample the features to 26 frames per video, subsample_frames_features.py will output a .csv with 31,200 rows, each of them also representinga frame of each video (26 * 1,200 = 31,200):

$ wc -l train_ImageNet.csv
31200 train_ImageNet.csv

Both .csvs can be processed by the generate_features_lists.py, but to do it with the one with all frames is excessively costly.

In a nutshell, the preprocessing pipeline is:

full_frames_features.csv -> subsample_frames_features.py  ->  subsampled_frames_features.csv  -> generate_features_lists.py