opengvlab/gv-benchmark

General Vision Benchmark, GV-B, a project from OpenGVLab

gv-benchmark's Issues

Where to download the 10% list

Where can I find these 10% lists (such as VOC det, VOC seg)? The address from this page is a private address (10.xxx.xxx.xxx).
Thanks.

ModuleNotFoundError: No module named 'gvbenchmark'

CMD RUN: mim train mmcls configs/cls/linear_probe/mnb4_Up-E-C_pretrain_flowers_10p.py

LOG:

Training command is python c:\users\xukang\miniconda3\envs\torch1.7.0\lib\site-packages\mmcls.mim\tools\train.py configs/cls/linear_probe/mnb4_Up-E-C_pretrain_flowers_10p.py --gpus 1 --launcher none.
Traceback (most recent call last):
File "c:\users\xukang\miniconda3\envs\torch1.7.0\lib\site-packages\mmcv\utils\misc.py", line 73, in import_modules_from_strings
imported_tmp = import_module(imp)
File "c:\users\xukang\miniconda3\envs\torch1.7.0\lib\importlib_init_.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1006, in _gcd_import
File "", line 983, in _find_and_load
File "", line 953, in _find_and_load_unlocked
File "", line 219, in _call_with_frames_removed
File "", line 1006, in _gcd_import
File "", line 983, in _find_and_load
File "", line 953, in _find_and_load_unlocked
File "", line 219, in _call_with_frames_removed
File "", line 1006, in _gcd_import
File "", line 983, in _find_and_load
File "", line 953, in _find_and_load_unlocked
File "", line 219, in _call_with_frames_removed
File "", line 1006, in _gcd_import
File "", line 983, in _find_and_load
File "", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'gvbenchmark'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "c:\users\xukang\miniconda3\envs\torch1.7.0\lib\site-packages\mmcls.mim\tools\train.py", line 203, in
main()
File "c:\users\xukang\miniconda3\envs\torch1.7.0\lib\site-packages\mmcls.mim\tools\train.py", line 92, in main
cfg = Config.fromfile(args.config)
File "c:\users\xukang\miniconda3\envs\torch1.7.0\lib\site-packages\mmcv\utils\config.py", line 337, in fromfile
import_modules_from_strings(**cfg_dict['custom_imports'])
File "c:\users\xukang\miniconda3\envs\torch1.7.0\lib\site-packages\mmcv\utils\misc.py", line 80, in import_modules_from_strings
raise ImportError
ImportError
Traceback (most recent call last):
File "c:\users\xukang\miniconda3\envs\torch1.7.0\lib\runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "c:\users\xukang\miniconda3\envs\torch1.7.0\lib\runpy.py", line 85, in run_code
exec(code, run_globals)
File "C:\Users\xukang\miniconda3\envs\torch1.7.0\Scripts\mim.exe_main.py", line 7, in
File "c:\users\xukang\miniconda3\envs\torch1.7.0\lib\site-packages\click\core.py", line 829, in call
return self.main(*args, **kwargs)
File "c:\users\xukang\miniconda3\envs\torch1.7.0\lib\site-packages\click\core.py", line 782, in main
rv = self.invoke(ctx)
File "c:\users\xukang\miniconda3\envs\torch1.7.0\lib\site-packages\click\core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "c:\users\xukang\miniconda3\envs\torch1.7.0\lib\site-packages\click\core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "c:\users\xukang\miniconda3\envs\torch1.7.0\lib\site-packages\click\core.py", line 610, in invoke
return callback(*args, **kwargs)
File "c:\users\xukang\miniconda3\envs\torch1.7.0\lib\site-packages\mim\commands\train.py", line 107, in cli
other_args=other_args)
File "c:\users\xukang\miniconda3\envs\torch1.7.0\lib\site-packages\mim\commands\train.py", line 256, in train
cmd, env=dict(os.environ, MASTER_PORT=str(port)))
File "c:\users\xukang\miniconda3\envs\torch1.7.0\lib\subprocess.py", line 363, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['python', 'c:\users\xukang\miniconda3\envs\torch1.7.0\lib\site-packages\mmcls\.mim\tools\train.py', 'configs/cls/linear_probe/mnb4_Up-E-C_pretrain_flowers_10p.py', '--gpus', '1', '--launcher', 'none']' returned non-zero exit status 1.

FileNotFoundError: checkpoints/up-e/c-mnb4.pth can not be found

where to download the up-e/c-mnb4.pth pretrained model weights?

2 questions about the INTERN

Hello,

First, thanks for releasing this great work !

I have 2 questions about the INTERN.
The questions are as follows:

1. What's the difference between UP-G det and UP-G cls ?

   • according to the paper, there was a single benchmark score for UP-G.
   • however, in the released model and configuration, I could find the UP-G cls, det model.

2. Is backbone frozen when expert and general training?

• Due to the amateur training, the backbone may have strong representation. Then, I think we don't have to train the backbone on amateur and expert sessions. Is it right?
• How can I check the detailed information about the training for each stage?

Thanks in advance.

Problems about reproducing INTERN-r50-Up-G-dbn

Thanks for your great work.

My reproduction linear probing result for INTERN-r50-Up-G-dbn on the full VOC07+12 dataset is only 77.5, which is much lower than the paper result 87.7 obtained with only 10% data.

My exp info:

1. INTERN-r50-Up-G-dbn-a4040c9c4.pth.tar is used as pretrained model, which is downloaded from the webside: https://opengvlab.shlab.org.cn/models
2. The file in this repo "./configs/det/linear_probe/faster_rcnn/central_mnb4_fpn_Up-G-D_pretrain_voc0712_10p.py" is used in my exps with two changes:
   (a). _base_ = [..., '../../base/datasets/voc0712.py', ...]
   (b). model = dict(backbone=dict(init_cfg=dict(
   type='Pretrained',
   checkpoint='./INTERN-r50-Up-G-dbn-a4040c9c4.pth.tar',
   ))
3. the used command:
   bash ./tools/dist_train.sh configs/det/linear_probe/faster_rcnn/central_r50_fpn_Up-G-D_pretrain_voc0712_10p.py 8

I have no idea what may cause this.

Some of my training log as follows:

INFO - Set random seed to 1962424783, deterministic: False
INFO - initialize Central_Model with init_cfg {'type': 'Pretrained', 'checkpoint': './INTERN-r50-Up-G-dbn-a4040c9c4.pth.tar'}
INFO - load model from: ./INTERN-r50-Up-G-dbn-a4040c9c4.pth.tar
INFO - load checkpoint from local path: ./INTERN-r50-Up-G-dbn-a4040c9c4.pth.tar
WARNING - The model and loaded state dict do not match exactly

unexpected key in source state_dict: lateral_convs.0.convclear.weight, lateral_convs.0.convclear.bias, fpn_convs.0.conv.weight, fpn_convs.0.conv.bias, lateral_convs.1.conv.weight, ateral_convs.1.conv.bias, fpn_convs.1.conv.weight, fpn_convs.1.conv.bias, lateral_convs.2.convweight, lateral_convs.2.convbias, fpn_convs.2.conv.weight, fpn_convs.2.conv.bias, lateral_convs.3.conv.weight, lateral_convs.3.conv.bias, fpn_convs.3.convweight, fpn_convs.3.convbias, rpn_head.rpn_conv.weight, rpn_head.rpn_conv.bias, roi_head.bbox_head.shared_fcs.0.weight, roi_head.bbox_head.shared_fcs.0.bias, roi_head.bbox_head.shared_fcs.1.weight, roi_head.bbox_head.shared_fcs.1.bias

INFO - initialize FPN with init_cfg {'type': 'Xavier', 'layer': 'Conv2d', 'distribution': 'uniform'}
INFO - initialize RPNHead with init_cfg {'type': 'Normal', 'layer': 'Conv2d', 'std': 0.01}
INFO - initialize Shared2FCBBoxHead with init_cfg [{'type': 'Normal', 'std': 0.01, 'override': {'name': 'fc_cls'}}, {'type': 'Normal', 'std': 0.001, 'override': {'name': 'fc_reg'}}, {'type': 'Xavier', 'distribution': 'uniform', 'override': [{'name': 'shared_fcs'}, {'name': 'cls_fcs'}, {'name': 'reg_fcs'}]}]

Results:
+-------------+------+-------+--------+-------+
| class | gts | dets | recall | ap |
+-------------+------+-------+--------+-------+
| aeroplane | 285 | 787 | 0.923 | 0.836 |
| bicycle | 337 | 992 | 0.941 | 0.851 |
| bird | 459 | 1389 | 0.900 | 0.769 |
| boat | 263 | 1504 | 0.863 | 0.688 |
| bottle | 469 | 1951 | 0.842 | 0.661 |
| bus | 213 | 812 | 0.920 | 0.810 |
| car | 1201 | 3952 | 0.959 | 0.868 |
| cat | 358 | 1142 | 0.958 | 0.864 |
| chair | 756 | 4869 | 0.862 | 0.598 |
| cow | 244 | 794 | 0.959 | 0.857 |
| diningtable | 206 | 1409 | 0.922 | 0.725 |
| dog | 489 | 1612 | 0.973 | 0.849 |
| horse | 348 | 1035 | 0.957 | 0.878 |
| motorbike | 325 | 1032 | 0.923 | 0.836 |
| person | 4528 | 14008 | 0.938 | 0.835 |
| pottedplant | 480 | 2511 | 0.802 | 0.503 |
| sheep | 242 | 800 | 0.884 | 0.760 |

| sofa | 239 | 1301 | 0.933 | 0.742 |
| train | 282 | 1033 | 0.908 | 0.812 |
| tvmonitor | 308 | 1388 | 0.899 | 0.757 |
+-------------+------+-------+--------+-------+
| mAP | | | | 0.775 |
+-------------+------+-------+--------+-------+
2022-03-24 09:57:06,094 - mmdet - INFO - Exp name: central_r50_fpn_Up-G-D_pretrain_voc0712.py
2022-03-24 09:57:06,094 - mmdet - INFO - Epoch(val) [12][619] mAP: 0.7750, AP50: 0.7750

the full logs here: voc.log

Reproduce Issue: Up-A CIFAR-100, Flowers 102

Hi,

When I reproduce the linear-probe classification performance on CIFAR-100, Flowers 102,
I got weird results when Up-A R50 was used for the backbone.

In the paper, Up-A R50 outperforms ImageNet pre-trained R50, however, ImageNet pre-trained R50 outperforms Up-A R50 on CIFAR-100, Flowers 102 cases with large margins.

My configurations as below:
Model

Dataset

Others

I checked whether the pre-trained Up-A was successfully loaded or not.

Thanks in advance.

how to load released MetaNet-B4-Up-G

when I load the released model through the scripts:
data = pickle.load(open('data.pkl', 'rb'))

the error encoutered:
UnpicklingError: A load persistent id instruction was encountered,
but no persistent_load function was specified.

and the version of pytorch is 1.10.2