coldmanck/acar-net

Actor-Context-Actor Relation Network for Spatio-temporal Action Localization

ACAR-Net

Actor-Context-Actor Relation Network for Spatio-temporal Action Localization - 1st place solution in AVA-Kinetics Crossover Challenge 2020 .

Code and model will come soon!

Junting Pan	Siyu Chen	Zheng Shou	Jing Shao	Hongsheng Li

Abstract

Localizing persons and recognizing their actions from videos is a challenging task towards high-level video understanding. Recent advances have been achieved by modeling either “actor-actor” or “actorcontext” relations. However, such direct first-order relations are not sufficient for localizing actions in complicated scenes. Some actors might be indirectly related via objects or background context in the scene. Such indirect relations are crucial for determining the action labels but are mostly ignored by existing work. In this paper, we propose to explicitly model the Actor-Context-Actor Relation, which can capture indirect high-order supportive information for effectively reasoning actors’ actions in complex scenes. To this end, we design an Actor-ContextActor Relation Network (ACAR-Net) which builds upon a novel Highorder Relation Reasoning Operator to model indirect relations for spatiotemporal action localization. Moreover, to allow utilizing more temporal contexts, we extend our framework with an Actor-Context Feature Bank for reasoning long-range high-order relations. Extensive experiments on AVA dataset validate the effectiveness of our ACAR-Net. Ablation studies show advantages of modeling high-order relations over existing first-order relation reasoning methods. The proposed ACAR-Net is also the core module of our 1st place solution in AVA-Kinetics Crossover Challenge 2020.

CVPR 2020 AVA-Kinetics Challenge

Find slides and video presentation of our winning solution on [Google Slides] [Youtube Video] [Bilibili Video] (Starting from 18:20).

Preprint

Find our work on Arxiv.

Please cite with the following Bibtex code:

@article{pan2020actorcontextactor,
  title={Actor-Context-Actor Relation Network for Spatio-Temporal Action Localization},
  author={Junting Pan and Siyu Chen and Zheng Shou and Jing Shao and Hongsheng Li},
  journal={arXiv preprint arXiv:2006.07976},
  year={2020}
}

You may also want to refer to our publication with the more human-friendly Chicago style:

Junting Pan, Siyu Chen, Zheng Shou, Jing Shao, Hongsheng Li. "Actor-Context-Actor Relation Network for Spatio-Temporal Action Localization." Arxiv 2020.

Models

ACAR Net Architecture

Contact

If you have any general doubt about our work or code which may be of interest for other researchers, please use the public issues section on this github repo. Alternatively, drop us an e-mail at [email protected] and [email protected] .