syscv/tet

Implementation of Tracking Every Thing in the Wild, ECCV 2022

Tracking Every Thing in the Wild [ECCV2022]

This is the offical implementation of paper Tracking Every Thing in the Wild.

Our project website contains more information: vis.xyz/pub/tet.

Abstract

Current multi-category Multiple Object Tracking (MOT) metrics use class labels to group tracking results for per-class evaluation. Similarly, MOT methods typically only associate objects with the same class predictions. These two prevalent strategies in MOT implicitly assume that the classification performance is near-perfect. However, this is far from the case in recent large-scale MOT datasets, which contain large numbers of classes with many rare or semantically similar categories. Therefore, the resulting inaccurate classification leads to sub-optimal tracking and inadequate benchmarking of trackers. We address these issues by disentangling classification from tracking. We introduce a new metric, Track Every Thing Accuracy (TETA), breaking tracking measurement into three sub-factors: localization, association, and classification, allowing comprehensive benchmarking of tracking performance even under inaccurate classification. TETA also deals with the challenging incomplete annotation problem in large-scale tracking datasets. We further introduce a Track Every Thing tracker (TETer), that performs association using Class Exemplar Matching (CEM). Our experiments show that TETA evaluates trackers more comprehensively, and TETer achieves significant improvements on the challenging large-scale datasets BDD100K and TAO compared to the state-of-the-art.

TETA

TETA builds upon the HOTA metric, while extending it to better deal with multiple categories and incomplete annotations. TETA evaluate trackers based on a novel local cluster design. TETA consists of three parts: a localization score, an association score, and a classification score, which enable us to evaluate the different aspects of each tracker properly.

Install TETA metric:

 python -m pip install git+https://github.com/SysCV/tet.git/#subdirectory=teta  

TETer

TETer follows an Associate-Every-Thing (AET) strategy. Instead of only associating objects in the same class, we associate every object in neighboring frames. We introduce Class Exemplar Matching (CEM), where the learned class exemplars incorporate valuable class information in a soft manner. In this way, we effectively exploit semantic supervision on large-scale detection datasets while not relying on the often incorrect classification output.

Main results

Our method outperforms the states of the art on BDD100K, and TAO benchmarks.

BDD100K val set

Method	backbone	pretrain	mMOTA	mIDF1	TETA	LocA	AssocA	ClsA	IDF1	MOTA	config	model
QDTrack(CVPR21)	ResNet-50	ImageNet-1K	36.6	51.6	47.8	45.9	48.5	49.2	71.5	63.5	-	-
TETer (Ours)	ResNet-50	ImageNet-1K	39.1	53.3	50.8	47.2	52.9	52.4	71.1	64.4	cfg	google drive

BDD100K test set

Method	backbone	pretrain	mMOTA	mIDF1	TETA	LocA	AssocA	ClsA	IDF1	MOTA	config	model
QDTrack(CVPR21)	ResNet-50	ImageNet-1K	35.7	52.3	49.2	47.2	50.9	49.2	72.3	64.3	-	-
TETer (Ours)	ResNet-50	ImageNet-1K	37.4	53.3	50.8	47.0	53.6	50.7	71.8	64.8	cfg	google drive

TAO val set

Method	backbone	pretrain	TETA	LocA	AssocA	ClsA	config	model	preds
QDTrack(CVPR21)	ResNet-101	ImageNet-1K	30.0	50.5	27.4	12.1	-	-	-
TETer (Ours)	ResNet-101	ImageNet-1K	33.3	51.6	35.0	13.2	cfg	google drive	coming soon
TETer-HTC (Ours)	ResNeXt-101-64x4d	ImageNet-1K	36.9	57.5	37.5	15.7	-	-	-
TETer-SwinT (Ours)	SwinT	ImageNet-1K	34.6	52.1	36.7	15.0	cfg	google drive	coming soon
TETer-SwinS (Ours)	SwinS	ImageNet-1K	36.7	54.2	38.4	17.4	cfg	google drive	coming soon
TETer-SwinB (Ours)	SwinB	ImageNet-22K	38.8	55.6	40.1	20.8	cfg	google drive	coming soon
TETer-SwinL (Ours)	SwinL	ImageNet-22K	40.1	56.3	39.9	24.1	cfg	google drive	google drive

Installation

Please refer to INSTALL.md for installation instructions.

Usages

Please refer to GET_STARTED.md for dataset preparation and running instructions.

Citation

@InProceedings{trackeverything,
  title = {Tracking Every Thing in the Wild},
  author = {Li, Siyuan and Danelljan, Martin and Ding, Henghui and Huang, Thomas E. and Yu, Fisher},
  booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
  month = {Oct},
  year = {2022}
}

tet's Issues

Testing results on small validation set

Hi,
Firstly thanks a lot for great work. Your model is basis for my current internship project!

I have done testing on validation split set containing 200 video sequences and I got the same results as mentioned in your paper as shown below:

I want to try testing on small validation split to save time. Hence I only chose first ten validation sequences, make the corresponding annotations using python -m bdd100k.label.to_coco command and run the testing. For the small validation split, the metrics like TETA, LoacA etc were heavily decreased. Even the MOTA is coming up in negative values (see figure below).

I am not sure about the reason for such low metric values for a smaller validation split. I only changed the ann_file, scalabel_gt and img_prefix paths to respective small validation split folder in cem_bdd.py configuration file.

Can you guide me in this issue. Is this the expected output or I am missing something because of which my metrics are coming very low on small validation split. Any suggestion would be helpful for me.

Best Regards,
Rehman

Different components of The TETer on the TAO open set using our TETA metrics

Hi, I would like to reproduce the result of QDTrack+CAL in table 4 of the paper, which achieves 30.00(TETA)，50.53(LocA)， 27.36（AssocA） 12.11（ClsA）。Thus I use the rep(https://github.com/SysCV/qdtrack) with the command "sh ./tools/dist_train.sh ./configs/tao/qdtrack_frcnn_r101_fpn_24e_lvis_1230_cls.py 8" and then get the trained model (24th checkpoint) to the next step, with the command "sh ./tools/dist_train.sh ./configs/tao/qdtrack_frcnn_r101_fpn_12e_tao_ft.py 8". In particular, I modified the following code：
1. “num_classes=482“ => “num_classes=1230“
2. the rcnn test config as following:

after completing the two steps training, I got the model with only 25.97(TETA). I wonder can you tell me if I did something wrong? or can tell me how you get the baseline model?

Using detector only

Hi,
I hope everyone will be fine.

Is it possible to first use detector only and calculate the evaluation metrics e.g. mAP ?
I actually want to conduct an experiment showing that tracker also improves detection results. So I need to run the detection part first and then detection + tracking part.

and the detection results we get from test.py script (see figure below), are they before applying tracker or after?

Kindly guide or share any possible suggestions. It will be much helpful for me.

Performance on LVIS mAP is low

We have reproduced mAP on TAO dataset, and found that the performance is quite low.

AP	AP50	AP75	track3D mAP
8.8717	14.0362	9.4615	7.7657

The numbers are generated with the block here
, while track3D mAP is calculated with the tao toolkit.

Could you provide the mAP numbers you get on TAO? We need to further verify it there is any bugs here

Without file of lvis_classes.txt

Hi, When I try to test this tracker with pre-trained model. It shows the following errors:

Traceback (most recent call last):
File "tools/test.py", line 164, in
main()
File "tools/test.py", line 111, in main
dataset = build_dataset(cfg.data.test)
File "/home/xd1/.local/lib/python3.8/site-packages/mmdet/datasets/builder.py", line 82, in build_dataset
dataset = build_from_cfg(cfg, DATASETS, default_args)
File "/home/xd1/.local/lib/python3.8/site-packages/mmcv/utils/registry.py", line 55, in build_from_cfg
raise type(e)(f'{obj_cls.name}: {e}')
FileNotFoundError: TaoDataset: [Errno 2] No such file or directory: 'data/lvis/annotations/lvis_classes.txt'

I used the following commands:

CONFIG=configs/tao/tracker_swinL_tao.py
CHECKPOINT=pretrain/teter_tao_swinL_1X_20220807_220424.pth
GPUS=1
PORT=25000

export PYTHONPATH="${PYTHONPATH}:$(dirname $0)/.."

python -m torch.distributed.launch --nproc_per_node=$GPUS --master_port=$PORT \
    tools/test.py $CONFIG $CHECKPOINT --launcher pytorch ${@:5} \
    --eval track --eval-options resfile_path=results/teter_tao_results/

Could you share the file of lvis_classes.txt with me? Or how to generate this file?
Thank you very much

checkpoint of the model trained LVISv0.5 + COCO dataset

Hi, I wonder can you provide the checkpoint of the model trained on LVISv0.5 + COCO dataset? many thanks.