xingyizhou/unidet

Object detection on multiple datasets with an automatically learned unified label space.

Simple multi-dataset detection

An object detector trained on multiple large-scale datasets with a unified label space; Winning solution of ECCV 2020 Robust Vision Challenges.

Simple multi-dataset detection,
Xingyi Zhou, Vladlen Koltun, Philipp Krähenbühl,
CVPR 2022 (arXiv 2102.13086)

Contact: [email protected]. Any questions or discussions are welcomed!

Features at a glance

• We trained a unified object detector on 4 large-scale detection datasets: COCO, Objects365, OpenImages, and Mapillary, with state-of-the-art performance on all of them.

• The model predicts class labels in a learned unified label space.

• The model can be directly used to test on novel datasets outside the training datasets.

• In this repo, we also provide state-of-the-art baselines for Objects365 and OpenImages.

Main results

• RVC challenge

COCO test-challenge	OpenImages public test	Mapillary test	Objects365 val
52.9	60.6	25.3	33.7

Results are obtained using a Cascade-RCNN with ResNeSt200 trained in an 8x schedule.

• Unified model vs. ensemble of dataset-specific models with known test domains.

	COCO	Objects365	OpenImages	mean.
Unified	45.4	24.4	66.0	45.3
Dataset-specific models	42.5	24.9	65.7	44.4

Results are obtained using a Cascade-RCNN with Res50 trained in an 8x schedule.

• Zero-shot cross dataset evaluation

	VOC	VIPER	CityScapes	ScanNet	WildDash	CrowdHuman	KITTI	mean
Unified	82.9	21.3	52.6	29.8	34.7	70.7	39.9	47.3
Oracle models	80.3	31.8	54.6	44.7	-	80.0	-	-

Results are obtained using a Cascade-RCNN with Res50 trained in an 8x schedule.

More models can be found in our MODEL ZOO.

Installation

Our project is developed on detectron2. Please follow the official detectron2 installation.

Demo

We use the same inference API as detectorn2. To run inference on an image folder using our pretrained model, run

python demo.py --config-file configs/Unified_learned_OCIM_R50_6x+2x.yaml --input images/*.jpg --opts MODEL.WEIGHTS models/Unified_learned_OCIM_R50_6x+2x.pth

If setup correctly, the output should look like:

*The sample image is from WildDash dataset.

Note that the model predicts all labels in its label hierarchy tree (for example, both vehicle and car for a car), following the protocol in OpenImages.

Benchmark evaluation and training

After installation, follow the instructions in DATASETS.md to setup the (many) datasets. Then check REPRODUCE.md to reproduce the results in the paper.

License

Our code is under Apache 2.0 license.

Citation

If you find this project useful for your research, please use the following BibTeX entry.

@inproceedings{zhou2021simple,
  title={Simple multi-dataset detection},
  author={Zhou, Xingyi and Koltun, Vladlen and Kr{\"a}henb{\"u}hl, Philipp},
  booktitle={CVPR},
  year={2022}
}

unidet's Issues

Cannot import SplAtConv2d

Hi Xingyi,

Thanks for sharing your models. However, when I tried to run inferences with model 'Unified_learned_OCIM_RS200_6x+2x', I got an exception on 'from .splat import SplAtConv2d'. As I cannot find 'splat' in the codebase, I wonder if the splat.py file is missing or I missed something when installing the dependencies?

Wenchang

RuntimeError: Expected nested_tensorlist[0].size() > 0 to be true, but got false.

I encountered an issue while trying to train the model on my custom dataset. The error message I received is as follows:
Traceback (most recent call last):
File "/mnt/d/Pycharm_Projects/UniDet/train_net.py", line 302, in
launch(
File "/mnt/d/Pycharm_Projects/detectron2-main/detectron2/engine/launch.py", line 84, in launch
main_func(*args)
File "/mnt/d/Pycharm_Projects/UniDet/train_net.py", line 295, in main
do_train(cfg, model, resume=args.resume)
File "/mnt/d/Pycharm_Projects/UniDet/train_net.py", line 200, in do_train
optimizer.step()
File "/root/anaconda3/envs/Detectron2/lib/python3.10/site-packages/torch/optim/lr_scheduler.py", line 68, in wrapper
return wrapped(*args, **kwargs)
File "/mnt/d/Pycharm_Projects/detectron2-main/detectron2/solver/build.py", line 73, in optimizer_wgc_step
per_param_clipper(p)
File "/mnt/d/Pycharm_Projects/detectron2-main/detectron2/solver/build.py", line 46, in clip_grad_value
torch.nn.utils.clip_grad_value_(p, cfg.CLIP_VALUE)
File "/root/anaconda3/envs/Detectron2/lib/python3.10/site-packages/torch/nn/utils/clip_grad.py", line 122, in clip_grad_value_
grouped_grads = _group_tensors_by_device_and_dtype([grads])
File "/root/anaconda3/envs/Detectron2/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/root/anaconda3/envs/Detectron2/lib/python3.10/site-packages/torch/utils/_foreach_utils.py", line 42, in _group_tensors_by_device_and_dtype
torch._C._group_tensors_by_device_and_dtype(tensorlistlist, with_indices).items()
RuntimeError: Expected nested_tensorlist[0].size() > 0 to be true, but got false. (Could this error message be improved? If so, please report an enhancement request to PyTorch.)

Why some labels are duplicated in the unified detector?

For example, nightstand appears twice in the labels file learned_mAP.csv:

line255: _nightstand_,,,nightstand,
line693: Nightstand__,/m/02z51p,Nightstand,,

But some labels from different datasets are normalized to one label. Why?
Is it due to the When there are different label granularities, we keep them all in our label-space, and expect to predict all of them in the paper?

Outputlayers has no attributions: self.pred_class_logits

rm the (self. ) before the attribution, this para was sent as para in func

RuntimeError: result type Float can't be cast to the desired output type Long

hi , it seems the problem caused by the diff of torch version, I just want to get the right torch version

result type Float can't be cast to the desired output type Long

when i use 1 GPU(v100)
"python projects/UniDet/train_net.py --config-file projects/UniDet/configs/Partitioned_COI_R50_2x.yaml --num-gpus 1",batch=16, there is no problem.

When I use two GPUs( 2x v100), an error occurred

python projects/UniDet/train_net.py --config-file projects/UniDet/configs/Partitioned_COI_R50_2x.yaml --num-gpus 1",batch=32

How can I solve this problem? Thank you！

how to deal with machine-generated labels of openimages ?

how to deal with machine-generated labels of openimages ?

[Learning a unified label space]: FileNotFoundError: [Errno 2] No such file or directory for "json.load(open(unified_label_file, 'r'))"

Hi,

thank you for this awesome idea on this nice multi-dataset-object-detectror.

I've trained a "Partitioned detector" on coco, oid and obejcts365-V2. (for V2, because MEGVII have updated their dataset, we could not download v1 anymore.)

When I tried the second step: Learning a unified label space:
I followed the tutorial and run:

python projects/UniDet/train_net.py \
--config-file projects/UniDet/Partitioned_COI_R50_2x.yaml \
--num-gpus 1 \
--eval-only MULTI_DATASET.UNIFIED_EVAL True

It raised a "FileNotFoundError". When I checked config file, I found that the key "UNIFIED_LABEL_FILE" in the config file has been set as ' ' (None), as shown in images below.

Could you give me some hints about how could I generate this this file or label space?

many thanks!

FileNotFoundError:

config.yaml:

Mapillary object detection dataset

Thank you for that great work. I would like to know which dataset of mapillary datasets has you used to train your model. I have check the website and I am not sure which one contains the object detection bounding boxes.

Could you please provide a link to download the dataset.

Question about RPN

Hello, I am very interested in your work, but I have a question to ask you. In the RPN stage, a proposal may be a positive sample in dataset A but a negative sample in dataset B. Will this produce ambiguity? Thanks！

DETESETS.md

Hello ,autors, thanks for your paper firstly.
I want to reproduce the code，but I found a question:

In your README docs/DATASETS.md ，you said "For unified evaluation, we converted the annotation to COCO format for convenience. Our converted annotation can be found here."

But these folders don't have annotations files in this link, please help me.

Thank you!
Best wishes.

Does the model use data augmentation strategies when training?

Thank you for your hard work and spiritual welfare. Does the model use data augmentation strategies when training? How to increase the data augmentation strategy?

Thank you very much.

Could you provide the files in the label_space folder

Hi Xingyi,

Thanks for you wonderful and easy to use work! I have managed to do some testing with the models and they performed quite well.

However I am not able to find the files under label_spaces folder to train, e.g., datasets/label_spaces/learned_mAP+M.json
Could you provide those?

Also as a suggestion, maybe you can put the model_zoo in the main read_me. They are quite hard to find at a glance.

Thanks

• Links to the relevant documentation/comment:

AssertionError: Attribute 'thing_classes' in the metadata of 'objects365_train' cannot be set to a different value!

📚 Documentation

I arrange the dataset as guided in the documentation, but get this error when trying to train split detectors.

• Links to the relevant documentation/comment:

6x+2x meaning

📚 Documentation

• Links to the relevant documentation/comment:
  Hello, Thanks for your work again. I have a question regarding the config file with '6x+2x'. The question is about the different training strategy during different stage, i.e., 6x stage and 2x stage. As I can see from the config file, The difference may be the use of EQL_LOSS; however, this loss depends the file 'datasets/oid/annotations/openimages_challenge_2019_train_v2_cat_info.json', which I do not find. Can you elaborate this a little bit? Many thanks. @xingyizhou

Convert annotations of OpenImages to COCO format

Hi Xingyi,

Thanks for you wonderful and easy to use work! I have prepare OpenImages dataset.

However, when I use projects/UniDet/tools/convert_datasets/convert_oid.py to convert training annotations.

There is an error:
sets train
converting train data
Traceback (most recent call last):
File "projects/UniDet/tools/convert_datasets/convert_oid.py", line 276, in
original_image_annotations = csvread(os.path.join(base_dir, 'annotations', image_label_sourcefile))
NameError: name 'image_label_sourcefile' is not defined

'image_label_sourcefile' is not defined when converting training annotations, I don't know how to solve it?
Could you give me the solution?
Thanks

• Links to the relevant documentation/comment:
  https://github.com/xingyizhou/UniDet/blob/master/projects/UniDet/unidet_docs/DATASETS.md

How to display label name like the paper? objects365 - Dog like this

📚 Documentation

• Links to the relevant documentation/comment:
  I want to detect a lot of objects and need to get specific category names. plz tell me how to get that

can't download preprocessed annotations

📚 Documentation

The preprocessed annotations link is invalid, could you please reupload them?
https://drive.google.com/drive/folders/1dDLiQfjEE0PqRlb7gtja-ermW2HJK5Tz?usp=sharing

import error 'FastRCNNOutputs' from 'detectron2.modeling.roi_heads.fast_rcnn'

$ python projects/UniDet/demo/demo.py --config-file projects/UniDet/configs/Unified_learned_OCI_R50_6x.yaml --input images/34501842524_3c858b3080_k.jpg --opts MODEL.WEIGHTS models/Unified_learned_OCI_R50_6x.pth
Traceback (most recent call last):
  File "projects/UniDet/demo/demo.py", line 18, in <module>
    from unidet.config import add_unidet_config
  File "projects/UniDet/unidet/__init__.py", line 17, in <module>
    from .modeling.roi_heads.custom_roi_heads import CustomROIHeads, CustomCascadeROIHeads
  File "projects/UniDet/unidet/modeling/roi_heads/custom_roi_heads.py", line 22, in <module>
    from .custom_fast_rcnn import CustomFastRCNNOutputLayers
  File "projects/UniDet/unidet/modeling/roi_heads/custom_fast_rcnn.py", line 16, in <module>
    from detectron2.modeling.roi_heads.fast_rcnn import FastRCNNOutputLayers, FastRCNNOutputs
ImportError: cannot import name 'FastRCNNOutputs' from 'detectron2.modeling.roi_heads.fast_rcnn' (/ProjectRoot/pyprojects/venv/test2/lib/python3.7/site-packages/detectron2/modeling/roi_heads/fast_rcnn.py)

I install detectron2 v0.6 built from https://dl.fbaipublicfiles.com/detectron2/wheels/cu101/torch1.8/index.html . The reason is detectron2 change its codebase, while ours is older one. So i follow the https://issueexplorer.com/issue/facebookresearch/unbiased-teacher/44 to fix the bug perfectly. I hope this can help someone who faces it.

Access to the inference results for reproduction of learning unified label space

Hi, xingyi,
Thanks for your work.
I would like to reproduce the label space results, but it seems my inference json files are not appropriate for learning the unified space. Would you please help provide some examples of your inference results (json format) so that I can try if my codes went wrong. For example, the data path in </tools/UniDet_learn_labelspace_mAP.ipynb>:

DATA_PATH = {
    'objects365': ROOT_PATH + 'logits/{}/inference_objects365_val/unified_instances_results.json'.format(EXP_NAME),
    'coco': ROOT_PATH + 'logits/{}/inference_coco_2017_val/unified_instances_results.json'.format(EXP_NAME),
    'oid': ROOT_PATH + 'logits/{}/inference_oid_val_expanded/unified_instances_results.json'.format(EXP_NAME),
}

I appreciate it a lot if you can share the three json files above. Thank you very much!

Custom labels

Hi

Thanks for releasing UniDet!

Is there a way to specify custom labels similar to Detic (using clip) ?

Thanks,
Subham

AttributeError: 'CustomFastRCNNOutputs' object has no attribute '_log_accuracy'

AttributeError: 'CustomFastRCNNOutputs' object has no attribute '_log_accuracy'
when i run train "python projects/UniDet/train_net.py --config-fil│
e projects/UniDet/configs/Base-CRCNN-COCO.yaml --num-gpus 8"

onnx model

how to convert model weight into onnx

question on the model settings in Tab. 5

Hi Xingyi, thanks for your wonderful work! I have a question regarding the model setting, in Tab. 5, for the naive merge, how is the merge conducted? and for the retrained, in my understanding, the box labels of multiple datasets are converted to the unified space and used for training a unified model, but still, the objects in one dataset may not be labeled in another, so we could still face such background confusion in supervision? is my understanding correct?

Thanks in advance!

Loss without gradients

Great work!
I don't really understand how in the unified version you apply the loss on all the detection heads, even when there are no gradients because we didn't pass the input through them. What am I missing and where can I find it in the code?