shuida/long-term-visual-tracking

Long-term Visual Tracking:

This page focuses on watching the state-of-the-art performance for the long-term tracking task (if you are interested in the short-term tracking task, please visit here).

Recent Long-term Trackers

• LTMU: Kenan Dai, Yunhua Zhang, Dong Wang, Jianhua Li, Huchuan Lu, Xiaoyun Yang.
  "High-Performance Long-Term Tracking with Meta-Updater." CVPR (2020). [paper] [code] VOT2019-LT Winner🌟 This work is an improved version of the VOT2019-LT winner, [LT_DSE].

• Siam R-CNN: Paul Voigtlaender, Jonathon Luiten, Philip H.S. Torr, Bastian Leibe.
  "Siam R-CNN: Visual Tracking by Re-Detection." ArXiv (2019). [paper] [code] [project]

• DAL: Yanlin Qian, Alan Lukežič, Matej Kristan, Joni-Kristian Kämäräinen, Jiri Mata
  "DAL - A Deep Depth-aware Long-term Tracker" ArXiv (2019). [paper] RGB-D Long-term

• GlobalTrack: Lianghua Huang, Xin Zhao, Kaiqi Huang.
  "GlobalTrack: A Simple and Strong Baseline for Long-term Tracking." AAAI (2020). [paper] [code]

• SPLT: Bin Yan, Haojie Zhao, Dong Wang, Huchuan Lu, Xiaoyun Yang.
  "Skimming-Perusal' Tracking: A Framework for Real-Time and Robust Long-Term Tracking." ICCV (2019). [paper] [code]

• flow_MDNet_RPN: Han Wu, Xueyuan Yang, Yong Yang, Guizhong Liu.
  "Flow Guided Short-term Trackers with Cascade Detection for Long-term Tracking." ICCVW (2019). [paper]

• OTR: Ugur Kart, Alan Lukezic, Matej Kristan, Joni-Kristian Kamarainen, Jiri Matas.
  "Object Tracking by Reconstruction with View-Specific Discriminative Correlation Filters." CVPR (2019). [paper] [code] RGB-D Long-term

• SiamRPN++: Bo Li, Wei Wu, Qiang Wang, Fangyi Zhang, Junliang Xing, Junjie Yan. "SiamRPN++: Evolution of Siamese Visual Tracking with Very Deep Networks." CVPR (2019). [paper] [code]

• MBMD: Yunhua Zhang, Dong Wang, Lijun Wang, Jinqing Qi, Huchuan Lu.
  "Learning regression and verification networks for long-term visual tracking." Arxiv (2018). [paper] [code] VOT2018-LT Winner🌟

• MMLT: Hankyeol Lee, Seokeon choi, Changick Kim.
  "A Memory Model based on the Siamese Network for Long-term Tracking." ECCVW (2018). [paper] [code]

• FuCoLoT: Alan Lukežič, Luka Čehovin Zajc, Tomáš Vojíř, Jiří Matas and Matej Kristan.
  "FuCoLoT - A Fully-Correlational Long-Term Tracker." ACCV (2018). [paper] [code]

Long-term Trackers modified from Short-term Ones

• SiamDW: Zhipeng Zhang, Houwen Peng.
  "Deeper and Wider Siamese Networks for Real-Time Visual Tracking." CVPR (2019). [paper] [code] VOT2019 RGB-D Winner🌟 Denoted as "SiamDW_D" "SiamDW_LT", see the VOT2019 official report [vot2019code]

• DaSiam_LT: Zheng Zhu, Qiang Wang, Bo Li, Wei Wu, Junjie Yan, Weiming Hu.
  "Distractor-Aware Siamese Networks for Visual Object Tracking." ECCV (2018). [paper] [code] VOT2018-LT Runner-up🌟

Early Long-term Trackers (before 2018)

• PTAV: Heng Fan, Haibin Ling.
  "Parallel Tracking and Verifying: A Framework for Real-Time and High Accuracy Visual Tracking." ICCV (2017). [paper] [supp] [project] [code]

• EBT: Gao Zhu, Fatih Porikli, Hongdong Li.
  "Beyond Local Search: Tracking Objects Everywhere with Instance-Specific Proposals." CVPR (2016). [paper] [exe]

• LCT: Chao Ma, Xiaokang Yang, Chongyang Zhang, Ming-Hsuan Yang.
  "Long-term Correlation Tracking." CVPR (2015). [paper] [project] [github]

• MUSTer: Zhibin Hong, Zhe Chen, Chaohui Wang, Xue Mei, Danil Prokhorov, Dacheng Tao.
  "MUlti-Store Tracker (MUSTer): a Cognitive Psychology Inspired Approach to Object Tracking." CVPR (2015). [paper] [project]

• CMT: Georg Nebehay, Roman Pflugfelder.
  "Clustering of Static-Adaptive Correspondences for Deformable Object Tracking." CVPR (2015). [paper] [project]
  [github]

• SPL: James Steven Supančič III, Deva Ramanan.
  "Self-paced Learning for Long-term Tracking." CVPR (2013). [paper] [github]

• TLD: Zdenek Kalal, Krystian Mikolajczyk, Jiri Matas.
  "Tracking-Learning-Detection." TPAMI (2012). [paper] [project]

Benchmark

• VOT: . [Visual Object Tracking Challenge]

  • [VOT2020LT][Coming Soon]
  • [VOT2019LT][Report]
  • [VOT2018LT][Report]

• OxUvA: Jack Valmadre, Luca Bertinetto, João F. Henriques, Ran Tao, Andrea Vedaldi, Arnold Smeulders, Philip Torr, Efstratios Gavves.
  "Long-term Tracking in the Wild: a Benchmark." ECCV (2018). [paper] [project]

• TLP: Abhinav Moudgil, Vineet Gandhi.
  "Long-term Visual Object Tracking Benchmark." ACCV (2018). [paper] [project]

• CDTB: Alan Lukežič, Ugur Kart, Jani Käpylä, Ahmed Durmush, Joni-Kristian Kämäräinen, Jiří Matas, Matej Kristan.
  "CDTB: A Color and Depth Visual Object Tracking Dataset and Benchmark." ICCV (2019). [paper] [project] RGB-D Long-term

• LaSOT: Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, Haibin Ling.
  "LaSOT: A High-quality Benchmark for Large-scale Single Object Tracking." CVPR (2019). [paper] [project]
  The LaSOT dataset is not a typical long-term dataset. But it is a good choice for connecting long-term and short-term trackers. Usually, short-term trackers drift very easily in the long-term datasets since they have no re-detection module. Long-term trackers also achieve unsatisfactory performance in the short-term datasets, since the tested sequences are often very short and the evaluation criterion pay less attention to the re-detection capability (especially VOT' EAO). LaSOT is a large-scale, long-frame dataset with precision and succuess criterion. Thus, it is a good choice if you want to fairly compare the performance of long-term and short-term trackers in one figure/table.

• UAV20L: Matthias Mueller, Neil Smith and Bernard Ghanem.
  "A Benchmark and Simulator for UAV Tracking." ECCV (2016). [paper] [project] [dataset]
  All 20 videos of UAV20L have been included in the VOT2018LT dataset.

Measurement&Discussion:

• Alan Lukežič, Luka Čehovin Zajc, Tomáš Vojíř, Jiří Matas, Matej Krista. "Performance Evaluation Methodology for Long-Term Visual Object Tracking." ArXiv (2019). [paper]

• Alan Lukežič, Luka Čehovin Zajc, Tomáš Vojíř, Jiří Matas, Matej Kristan. "Now You See Me: Evaluating Performance in Long-term Visual Tracking." ArXiv (2018). [paper]

• Shyamgopal Karthik, Abhinav Moudgil, Vineet Gandhi. "Exploring 3 R's of Long-term Tracking: Re-detection, Recovery and Reliability." WACV (2020). [paper]

Resources:

• "Paper, Benchmark, Researchers, Teams" maintained by Qiang Wang:
  https://github.com/foolwood/benchmark_results

• "pysot [SiamRPN++, SiamMask, DaSiamRPN, SiamRPN]":
  https://github.com/STVIR/pysot

• "pytracking [PrDIMP, SuperDIMP, DIMP, ATOM]":
  https://github.com/visionml/pytracking

Benchmark Results:

• VOT2019-LT/VOT2020-LT:

  Tracker	F-Score	Speed (fps)	Paper/Code
  LTMU (CVPR20)	0.697	13 (RTX 2080Ti)	Paper/Code
  LT_DSE	0.695	N/A	N/A
  CLGS	0.674	N/A	N/A
  SiamDW_LT	0.665	N/A	N/A
  SPLT (ICCV19)	0.587	26 (GTX 1080Ti)	Paper/Code
  mbdet	0.567	N/A	N/A
  SiamRPNsLT	0.556	N/A	N/A
  Siamfcos-LT	0.520	N/A	N/A
  CooSiam	0.508	N/A	N/A
  ASINT	0.505	N/A	N/A
  FuCoLoT	0.411	N/A	N/A

  • Most results are obtained from the original VOT2019_LT report.

• VOT2018-LT:

  Tracker	F-Score	Speed (fps)	Paper/Code
  LTMU (CVPR20)	0.690	13 (RTX 2080Ti)	Paper/Code
  Siam R-CNN (CVPR20)	0.668	5 (Tesla V100)	Paper/Code
  SiamRPN++	0.629	35 (Titan XP)	Paper/Code
  SPLT (ICCV19)	0.622	26 (GTX 1080Ti)	Paper/Code
  MBMD (Arxiv)	0.610	4 (GTX 1080Ti)	Paper/Code
  DaSiam_LT (ECCV18)	0.607	110 (TITAN X)	Paper/Code

  • MBMD and DaSiam_LT is the winner and runner-up in the original VOT2018_LT report.

• OxUvA:

  Tracker	MaxGM	Speed (fps)	Paper/Code
  LTMU (CVPR20)	0.751	13 (RTX 2080Ti)	Paper/Code
  Siam R-CNN (CVPR20)	0.723	5 (Tesla V100)	Paper/Code
  SPLT (ICCV19)	0.622	26 (GTX 1080Ti)	Paper/Code
  GlobalTrack (AAAI20)	0.603	6 (GTX TitanX)	Paper/Code
  MBMD (Arxiv)	0.544	4 (GTX 1080Ti)	Paper/Code
  SiamFC+R (ECCV18)	0.454	52 (Unkown GPU)	Paper/Code

  • OxUvA Leaderboard: https://competitions.codalab.org/competitions/19529#results
  • SiamFC+R is the best tracker in the original OxUvA paper.

• TLP:

  Tracker	Success Score	Speed (fps)	Paper/Code
  LTMU (CVPR20)	0.571	13 (RTX 2080Ti)	Paper/Code
  GlobalTrack (AAAI20)	0.520	6 (GTX TitanX)	Paper/Code
  SPLT (ICCV19)	0.416	26 (GTX 1080Ti)	Paper/Code
  MDNet (CVPR16)	0.372	5 (GTX 1080Ti)	Paper/Code

  • MDNet is the best tracker in the original TLP paper.

• LaSOT:

  Tracker	Success Score	Speed (fps)	Paper/Code
  Siam R-CNN (CVPR20)	0.648	5 (Tesla V100)	Paper/Code
  PrDiMP50 (CVPR20)	0.598	30 (Unkown GPU)	Paper/Code
  LTMU (CVPR20)	0.572	13 (RTX 2080Ti)	Paper/Code
  DiMP50 (ICCV19)	0.568	43 (GTX 1080)	Paper/Code
  SiamAttn (CVPR20)	0.560	45 (RTX 2080Ti)	Paper/Code
  SiamFC++GoogLeNet (AAAI20)	0.544	90 (RTX 2080Ti)	Paper/Code
  MAML-FCOS (CVPR20)	0.523	42 (NVIDIA P100)	Paper/Code
  GlobalTrack (AAAI20)	0.521	6 (GTX TitanX)	Paper/Code
  ATOM (CVPR19)	0.515	30 (GTX 1080)	Paper/Code
  SiamBAN (CVPR20)	0.514	40 (GTX 1080Ti)	Paper/Code
  SiamCar (CVPR20)	0.507	52 (RTX 2080Ti)	Paper/Code
  SiamRPN++ (CVPR19)	0.496	35 (Titan XP)	Paper/Code
  ROAM++ (CVPR20)	0.447	20 (RTX 2080)	Paper/Code
  SPLT (ICCV19)	0.426	26 (GTX 1080Ti)	Paper/Code
  MDNet (CVPR16)	0.397	5 (GTX 1080Ti)	Paper/Code

  • MDNet is the best tracker in the original LaSOT paper.

All Tracking Datasets:

• List:

  Datasets	#videos	#total/min/max/average frames	Absent Label
  OTB-2015	100	59K/71/3,872/590	No
  TC-128	128	55K/71/3,872/429	No
  NUS-PRO	365	135K/146/5,040/371	No
  UAV123	123	113K/109/3,085/915	No
  TB70	70	XXXX	No
  ALOV300++	315	8.9K/XXXX/XXXX/284	No
  NfS	100	383K/169/20,665/3,830	No
  GOT-10k	train-10k, val-180, test-180	1.5M	No
  LaSOT	1,400 (I-all-1,400/II-test-280)	3.52M/1,000/11,397/2,506	Yes
  VOT2019-LT/VOT2020-LT	50	XXXX/XXXX/XXXX/XXXX	Yes
  TLP	50	XXXX/XXXX/XXXX/XXXX	No
  OxUvA	366 (dev-200/test-166)	XXXX/XXXX/XXXX/XXXX	Yes

  • OTB-2013 is a subset of OTB-2015.
  • UAV-20L has been included in VOT2018-LT/VOT2019-LT/VOT2020-LT.
  • VOT2018-LT is a subset of VOT2019-LT/VOT2020-LT. VOT2019-LT and VOT2020-LT are same.