divadi/solofusion

Time Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detection

solofusion's Issues

Question about BEVDet and BEVDepth

Thanks for the great work 👍
As you mentioned in #6 (comment), the implementations of BEVDepth btw. BEVDepth's official codebase and BEVDet's reproduction are slightly different. And as you know, the reproduction in BEVDet shows lower accuracy.

After comparing the two codebases, I notice that the DepthNet and LSS are identical from the two implementations,
but not for the detection head (BEVDet's reproduction uses CenterPoint, not BEVDepthHead).

I think the main reason is the difference in the detection head.
Furthermore, the mAVE of BEVDet's reproduction that I evaluated is way worse than the official BEVDepth, and as far as I know, the mAVE metric is related to the detection head.

ANY opinions on my assumption? I wonder about your thoughts.

Results on Test set

Hi authors, Thank you for your great work.
I would like to know the details of the test model config. Did you only replace the backbone, image resolution, bev grid size, and train on train+val set ?

Compute the group correlation

Dear author,

Hello,
I would like to know why use "*" to measure the correlation between "curr_prev_sampled_feats_padded" and "curr_curr_stereo_feats_valid_padded".

About GPU memory used during training

Could you please provide more information about the training GPU memory? Just as a reference. Thank you! 🙏

why detach history feat？

SOLOFusion/mmdet3d/models/detectors/solofusion.py

Line 377 in 683edce

self.history_bev = feats_cat[:, :-self.single_bev_num_channels, :, :].detach().clone()

Performance mismatch

I use bs=4x8, lr=1e-4 for training and the cbgs result is:

Evaluating bboxes of pts_bbox
mAP: 0.4166
mATE: 0.5791
mASE: 0.2850
mAOE: 0.5265
mAVE: 0.2424
mAAE: 0.1887
NDS: 0.5261
Eval time: 72.3s

Enlarge perception range

Dear author,

Hello,
I would like to know which variables should be changed if I want to train solofusion in 150x150m perception range.
I have changed the following:
point_cloud_range = [-153.6, -153.6, -5.0, 153.6, 153.6, 5.0];
grid_config={
'xbound': [-153.6, 153.6, 0.8],
'ybound': [-153.6, 153.6, 0.8],
'zbound': [-5.0, 5.0, 10.0],
'dbound': [2.0, 158.0, 0.5],}
grid_size=[3072, 3072, 50],
If there is any other variable I missed? Thank you for your time.

nuScenes val submmision json

Hello, dear authors
I am doing analysis and demonstration of related work recently, can you publish the prediction result file of the nuScenes val dataset? Thank you very much.

Why lr_config is None?

I find that lr_config=None. Is that mean lr is not declaying during training ?
It's strange to me. Hope for your reply!

inference error

python tools/swap_ema_and_non_ema.py work_dirs/r50-fp16_phase2/r50-fp16-cbgs_phase2_ema.pth
Traceback (most recent call last):
File "tools/swap_ema_and_non_ema.py", line 34, in
main()
File "tools/swap_ema_and_non_ema.py", line 31, in main
print(swap_ema_and_non_ema(sys.argv[1]))
File "tools/swap_ema_and_non_ema.py", line 16, in swap_ema_and_non_ema
ckpt['state_dict'][k] = ckpt['state_dict'][ema_name]
KeyError: 'ema_pts_bbox_head_shared_conv_conv_weight'

grad norm inf or nan in training of phase 1

Hi, @Divadi

Thanks for your great work!

I'm using this command to train in phase 1:
./tools/dist_train.sh configs/solofusion/r50-fp16_phase1.py 1 --autoscale-lr --no-validate

But I get nan or inf gran norm like follows. What's the problem?

2023-08-05 20:33:00,438 - mmdet - INFO - Iter [50/5274] lr: 2.500e-05, eta: 1:52:54, time: 1.297, data_time: 0.079, memory: 15212, loss_depth: 17.8610, task0.loss_xy: 0.2953, task0.loss_z: 0.2632, task0.loss_whl: 0.7729, task0.loss_yaw: 0.3595, task0.loss_vel: 0.0873, task0.loss_heatmap: 119.4384, task1.loss_xy: 0.2598, task1.loss_z: 0.2394, task1.loss_whl: 0.8032, task1.loss_yaw: 0.3510, task1.loss_vel: 0.0875, task1.loss_heatmap: 4587.3349, task2.loss_xy: 0.2673, task2.loss_z: 0.2936, task2.loss_whl: 1.1154, task2.loss_yaw: 0.3380, task2.loss_vel: 0.0352, task2.loss_heatmap: 8788.5075, task3.loss_xy: 0.2540, task3.loss_z: 0.3332, task3.loss_whl: 0.3935, task3.loss_yaw: 0.3192, task3.loss_vel: 0.0254, task3.loss_heatmap: 486.4620, task4.loss_xy: 0.2952, task4.loss_z: 0.3318, task4.loss_whl: 0.4240, task4.loss_yaw: 0.3640, task4.loss_vel: 0.0736, task4.loss_heatmap: 7132.7133, task5.loss_xy: 0.2553, task5.loss_z: 0.2510, task5.loss_whl: 0.4125, task5.loss_yaw: 0.3699, task5.loss_vel: 0.0489, task5.loss_heatmap: 667.0299, loss: 21809.0666, grad_norm: inf
2023-08-05 20:34:02,149 - mmdet - INFO - Iter [100/5274] lr: 2.500e-05, eta: 1:49:07, time: 1.234, data_time: 0.037, memory: 15212, loss_depth: 19.2391, task0.loss_xy: 0.2312, task0.loss_z: 0.2441, task0.loss_whl: 0.5329, task0.loss_yaw: 0.3376, task0.loss_vel: 0.0765, task0.loss_heatmap: 61.9678, task1.loss_xy: 0.2024, task1.loss_z: 0.2205, task1.loss_whl: 0.7115, task1.loss_yaw: 0.3442, task1.loss_vel: 0.0696, task1.loss_heatmap: 2507.1604, task2.loss_xy: 0.2393, task2.loss_z: 0.3086, task2.loss_whl: 1.0194, task2.loss_yaw: 0.3265, task2.loss_vel: 0.0498, task2.loss_heatmap: 3067.5153, task3.loss_xy: 0.2045, task3.loss_z: 0.2860, task3.loss_whl: 0.2993, task3.loss_yaw: 0.3214, task3.loss_vel: 0.0216, task3.loss_heatmap: 145.4959, task4.loss_xy: 0.2468, task4.loss_z: 0.2349, task4.loss_whl: 0.3451, task4.loss_yaw: 0.3188, task4.loss_vel: 0.0486, task4.loss_heatmap: 5200.2193, task5.loss_xy: 0.2096, task5.loss_z: 0.2415, task5.loss_whl: 0.3624, task5.loss_yaw: 0.3523, task5.loss_vel: 0.0431, task5.loss_heatmap: 472.5930, loss: 11482.6411, grad_norm: 88765.7708
2023-08-05 20:35:03,848 - mmdet - INFO - Iter [150/5274] lr: 2.500e-05, eta: 1:47:10, time: 1.234, data_time: 0.036, memory: 15212, loss_depth: 19.0185, task0.loss_xy: 0.1844, task0.loss_z: 0.2168, task0.loss_whl: 0.3578, task0.loss_yaw: 0.3301, task0.loss_vel: 0.0696, task0.loss_heatmap: 26.7677, task1.loss_xy: 0.1574, task1.loss_z: 0.2252, task1.loss_whl: 0.5558, task1.loss_yaw: 0.3282, task1.loss_vel: 0.0593, task1.loss_heatmap: 1187.7882, task2.loss_xy: 0.1787, task2.loss_z: 0.3269, task2.loss_whl: 0.8107, task2.loss_yaw: 0.3191, task2.loss_vel: 0.0355, task2.loss_heatmap: 1002.1995, task3.loss_xy: 0.1629, task3.loss_z: 0.2478, task3.loss_whl: 0.1829, task3.loss_yaw: 0.3149, task3.loss_vel: 0.0170, task3.loss_heatmap: 82.1107, task4.loss_xy: 0.2011, task4.loss_z: 0.2375, task4.loss_whl: 0.2725, task4.loss_yaw: 0.3130, task4.loss_vel: 0.0489, task4.loss_heatmap: 2316.9863, task5.loss_xy: 0.1666, task5.loss_z: 0.2272, task5.loss_whl: 0.2627, task5.loss_yaw: 0.3322, task5.loss_vel: 0.0422, task5.loss_heatmap: 193.5049, loss: 4835.5605, grad_norm: 42493.9716

Performance mismatch

Dear author,

I have executed the training command provided in the readme file to train the R50-SOLOFusion model. However, upon inference, I observed that the model's output exhibited relatively large values for mASE and mAOE compared to the provided weights.

I did not make any modifications to the code, except for changing the batch size from 16 to 8. I seek guidance on how to reproduce the expected results under these circumstances.

Thank you for your time.

Question about long-term fusion performance

What's the difference in the setting between the two experiments about long-term fusion? I think their results should be the same.

Some fp16 problems when extending solofusion to the large backbone.

Dear author,

When I extend SOLOFusion to ConvNext-B, it can't converge well according to your provided config settings. However, when I canceled the fp16 hook and re-trained it, the loss tendency was normal. But there is no doubt that the re-implementation performance will differ from yours. Is there any revision when training the model of the large version?

Performance mismatch

Notice

There are several common situations in the reimplementation issues as below

1. Reimplement a model in the model zoo using the provided configs

Describe the issue

My results mismatch with the repo reported.

Reproduction

1. What command or script did you run?

The MMDet3D test.py.

2. What config dir you run?

The provided r50-fp16-cbgs_phase2.py and r50-longonly-fp16_phase2.py

3. Did you make any modifications on the code or config? Did you understand what you have modified?

Only the dataset path.

5. What dataset did you use?

nuScenes dataset.

Environment

1. Please run python mmdet3d/utils/collect_env.py to collect necessary environment infomation and paste it here.

Details

sys.platform: linux
Python: 3.8.16 (default, Mar  2 2023, 03:21:46) [GCC 11.2.0]
CUDA available: True
GPU 0,1,2,3,4,5,6,7: NVIDIA A100-SXM4-80GB
CUDA_HOME: /nvme/share/cuda-11.3/
NVCC: Build cuda_11.3.r11.3/compiler.29920130_0
GCC: gcc (GCC) 10.1.0
PyTorch: 1.10.1+cu111
PyTorch compiling details: PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.2.3 (Git Hash 7336ca9f055cf1bfa13efb658fe15dc9b41f0740)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX512
  - CUDA Runtime 11.1
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
  - CuDNN 8.0.5
  - Magma 2.5.2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.10.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, 

TorchVision: 0.11.2+cu111
OpenCV: 4.7.0
MMCV: 1.3.16
MMCV Compiler: GCC 7.3
MMCV CUDA Compiler: 11.3
MMDetection: 2.14.0
MMSegmentation: 0.14.1
MMDetection3D: 0.17.2+229e79e

Results

Configuration: r50-fp16-cbgs_phase2.py. Checkpoint: solofusion-r50-fp16-cbgs_phase2_ema.pth.

Details

Evaluating bboxes of pts_bbox
mAP: 0.2757                                                                                                                           
mATE: 0.6761
mASE: 0.2874
mAOE: 0.4937
mAVE: 0.2744
mAAE: 0.2093
NDS: 0.4438
Eval time: 101.9s

Per-class results:
Object Class    AP      ATE     ASE     AOE     AVE     AAE
car     0.576   0.439   0.163   0.154   0.289   0.207
truck   0.333   0.590   0.217   0.133   0.224   0.212
bus     0.314   0.754   0.212   0.129   0.439   0.246
trailer 0.117   1.198   0.253   0.333   0.201   0.200
construction_vehicle    0.030   1.067   0.521   1.271   0.126   0.345
pedestrian      0.409   0.661   0.291   0.723   0.373   0.199
motorcycle      0.281   0.589   0.261   0.779   0.388   0.261
bicycle 0.259   0.511   0.259   0.806   0.156   0.004
traffic_cone    0.049   0.490   0.411   nan     nan     nan
barrier 0.390   0.462   0.287   0.115   nan     nan

Configuration: r50-longonly-fp16_phase2.py. Checkpoint: solofusion-r50-longonly-fp16_phase2_ema.pth.

Details

Evaluating bboxes of pts_bbox
mAP: 0.2532                                                                                                                           
mATE: 0.7142
mASE: 0.2892
mAOE: 0.6561
mAVE: 0.3579
mAAE: 0.2162
NDS: 0.4032
Eval time: 111.4s

Per-class results:
Object Class    AP      ATE     ASE     AOE     AVE     AAE
car     0.505   0.496   0.173   0.231   0.398   0.222
truck   0.271   0.668   0.223   0.204   0.282   0.225
bus     0.321   0.682   0.215   0.272   0.564   0.243
trailer 0.081   1.050   0.233   0.549   0.265   0.219
construction_vehicle    0.031   1.133   0.506   1.441   0.138   0.333
pedestrian      0.357   0.716   0.291   0.857   0.467   0.248
motorcycle      0.250   0.674   0.270   0.911   0.584   0.232
bicycle 0.280   0.650   0.272   1.272   0.166   0.008
traffic_cone    0.010   0.550   0.425   nan     nan     nan
barrier 0.427   0.524   0.282   0.166   nan     nan

Question about history_bev

Hi Author,

I have a question about history_bev.
The history_bev is calculated from last iter and is detached.
However, the weight of img branch is changed ever iter, the history_bev stored previous and the history_bev calculate by img_branch(current iter) is different, will this bring issue to the model?

Question about resnet50 results

Hi, in your table, resnet-50 mAP/NDS: 42.7/53.4，but According to the calculation rules of NDS, the real NDS is 54.4. Is it a printing error?

autoscale_lr only works in non-distributed training

autoscale_lr only works in non-distributed training，because args.gpus and args.gpu_ids are all none in distributed training. But i found your training log, lr is 2e-4, which is scaled by number of gpus, is there anything wrong?

Question about relation between 【localization potential】 and【low-resolution-Long-term Fusion】

Great work! I have tried to read carefully, and got some question.
low-resolution-Long-term Fusion is performed after depth prediction, why can it still benefit from localization potential with long term history frame?
Maybe there are some misunderstanding, but it realy confuse me yet.

SOLOFusion Test Model + Config

Hi authors,
Thank you for your great work and releasing the Val models. I wanted to build on top of SOLOFusion. However, the model zoo does not contain the nuScenes test model with AP=54.0 and NDS=61.9 as reported in Table 3 of the paper.

It would be great if you could release the test model and configs.

checkpoint of Resnet?

Hi, thanks for the great work! May I ask which version of pytorch you used? And could you please provide the checkpoint url of the resnet50 (in1k)? We found that it's difficult to reproduce your results.

Code release time

Great Works! When will the code be released?

About fps of r101 model

Hello author, can you provide the fps of your r101 model?

SOLOFusion with python3.7.10, torch1.9, cudatoolkit 11.1 bug fixed

Hi Divadi, I've found that execute SOLOFusion project with python3.7.10, torch1.9, cudatoolkit 11.1 as your project mentioned, there is a small bug.

mmdet3d/models/necks/view_transformer_solofusion.py
meshgrid = torch.stack(torch.meshgrid(torch.arange(stereo_W), torch.arange(stereo_H), indexing="xy"), dim=2)
should changed as
meshgrid = torch.stack(torch.meshgrid(torch.arange(stereo_H), torch.arange(stereo_W)), dim=2)

which is mainly because pytorch with version 1.9, the meshgrid method didn't support indexing. The problem further results in the reverse of stereo_H and stereo_W, hope that can help other people

Environment

Python: 3.7.10 (default, Feb 26 2021, 18:47:35) [GCC 7.3.0]
CUDA available: True
GPU 0,1: NVIDIA GeForce RTX 3090
CUDA_HOME: /usr/local/cuda
NVCC: Build cuda_11.1.TC455_06.29190527_0
GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
PyTorch: 1.9.0
PyTorch compiling details: PyTorch built with:

• GCC 7.3
• C++ Version: 201402
• Intel(R) oneAPI Math Kernel Library Version 2021.2-Product Build 20210312 for Intel(R) 64 architecture applications
• Intel(R) MKL-DNN v2.1.2 (Git Hash 98be7e8afa711dc9b66c8ff3504129cb82013cdb)
• OpenMP 201511 (a.k.a. OpenMP 4.5)
• NNPACK is enabled
• CPU capability usage: AVX2
• CUDA Runtime 11.1
• NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
• CuDNN 8.0.5
• Magma 2.5.2
• Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.9.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,

TorchVision: 0.10.0
OpenCV: 4.7.0
MMCV: 1.3.16
MMCV Compiler: GCC 7.3
MMCV CUDA Compiler: 11.1
MMDetection: 2.14.0
MMSegmentation: 0.14.1
MMDetection3D: 0.17.2+683edce

Bug fix
If you have already identified the reason, you can provide the information here. If you are willing to create a PR to fix it, please also leave a comment here and that would be much appreciated!

bev resolution

Dear author,

Hello,
I would like to know which variables should be changed in config file to use 256x256 bev resolution feature.
Thank you for your time.

Can solofusion trained with one phase?

Now solofusion are trained in 2 phases, can it be simplied to only 1 phase?