runzhouge/mac

MAC: Mining Activity Concepts for Language-based Temporal Localization

MAC

By Runzhou Ge, Jiyang Gao, Kan Chen, Ram Nevatia.

University of Southern California (USC).

Introduction

This repository contains the code for the WACV 2019 paper, MAC: Mining Activity Concepts for Language-based Temporal Localization. arXiv

Requirements

• Python 2.7
• Tensorflow 1.0 or higher
• others

Download

The code is for Charades-STA dataset.

After cloning this repo, please donwload:

• visual features
• visual activity concepts
• ref_info

ref_info contains Charades-STA annotations, semantic activity concepts, checkpoints and others. After downloading ref_info.tar, untar it and move the folder to the root/ directory of this repo.

Please also change the visual feature and visual activity concepts directories in the main.py.

Training

For the paper results on Charades-STA dataset, run

python main.py --is_only_test True \
--checkpoint_path ./ref_info/charades_sta_wacv_2019_paper_ACL_k_results/trained_model.ckpt-10000 \
--test_name paper_results

You will get similar results listed in the row "ACL-K" of the following table.

Model	R@1,IoU=0.7	R@1,IoU=0.5	R@5,IoU=0.7	R@5,IoU=0.5
CTRL	7.15	21.42	26.91	59.11
ACL-K	12.20	30.48	35.13	64.84

To train the model from scratch, run

python main.py

The results and checkpoints will appear in root/results_history/ and root/trained_save/, respectively.

Results Visualization

Citation

If you find this work is helpful, please cite:

@InProceedings{Ge_2019_WACV,
  author = {Ge, Runzhou and Gao, Jiyang and Chen, Kan and Nevatia, Ram},
  title = {MAC: Mining Activity Concepts for Language-based Temporal Localization},
  booktitle = {The IEEE Winter Conference on Applications of Computer Vision (WACV)},
  month = {January},
  year = {2019}
}

License

MIT License

Acknowledgements

This research was supported, in part, by the Office of Naval Research under grant N00014-18-1-2050 and by an Amazon Research Award.

mac's Issues

Details for Getting Files in ref_info

Thanks for your brilliant work. Could you share more details on how files in ref_info are generated from video clips and texts? Thank you!

about the charades_sta_train_clip-sentvec_o0.5_l10_activity_nodup.pkl in ref_info

The train_clip and test_clip in ref_info have the same size. The train_clip have stored the content of test_clip. I wonder if you have chosen the wrong file when uploading. Could you re-upload the ref_info?
Look forward your reply.

the details get the visual features

Thanks for the nice job! @runzhouge
Could you please tell me the details about getting the visual features?
Such as C3D features are extracted using this？ https://github.com/facebook/C3D/tree/master/C3D-v1.0
The frames are resized to 128171 and then crop 112112 as the input?

about the dataset

Hello,

I wanna to know the fps when you process the datasets, namely how many frames one second?

Looking forward to your reply.

training problem

hello!Why did my loss is quite low when I start training,and in the end the loss didn't change too much?
like:
Step 0: loss = 0.46 (1.248 sec)
Step 5: loss = 0.47 (0.273 sec)
Step 10: loss = 0.48 (0.405 sec)
Step 15: loss = 0.48 (0.325 sec)
Step 20: loss = 0.50 (0.372 sec)
Step 25: loss = 0.50 (0.297 sec)
Step 30: loss = 0.54 (0.295 sec)
Step 35: loss = 0.53 (0.320 sec)
Step 40: loss = 0.55 (0.313 sec)
Step 45: loss = 0.52 (0.289 sec)
Step 50: loss = 0.49 (0.303 sec)
...but my test results are similar to those in the paper......

Confirm Python 3 support

Python 2.7 is just a year away from being permanently retired. It is surprising that it was used. Please ensure the code works with Python 3, and confirm the same.