divadi / solofusion Goto Github PK
View Code? Open in Web Editor NEWTime Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detection
License: Apache License 2.0
Time Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detection
License: Apache License 2.0
Thanks for the great work 👍
As you mentioned in #6 (comment), the implementations of BEVDepth btw. BEVDepth's official codebase and BEVDet's reproduction are slightly different. And as you know, the reproduction in BEVDet shows lower accuracy.
After comparing the two codebases, I notice that the DepthNet and LSS are identical from the two implementations,
but not for the detection head (BEVDet's reproduction uses CenterPoint, not BEVDepthHead).
I think the main reason is the difference in the detection head.
Furthermore, the mAVE of BEVDet's reproduction that I evaluated is way worse than the official BEVDepth, and as far as I know, the mAVE metric is related to the detection head.
ANY opinions on my assumption? I wonder about your thoughts.
Hi authors, Thank you for your great work.
I would like to know the details of the test model config. Did you only replace the backbone, image resolution, bev grid size, and train on train+val set ?
Dear author,
Hello,
I would like to know why use "*" to measure the correlation between "curr_prev_sampled_feats_padded" and "curr_curr_stereo_feats_valid_padded".
Could you please provide more information about the training GPU memory? Just as a reference. Thank you! 🙏
I use bs=4x8, lr=1e-4 for training and the cbgs result is:
Evaluating bboxes of pts_bbox
mAP: 0.4166
mATE: 0.5791
mASE: 0.2850
mAOE: 0.5265
mAVE: 0.2424
mAAE: 0.1887
NDS: 0.5261
Eval time: 72.3s
Dear author,
Hello,
I would like to know which variables should be changed if I want to train solofusion in 150x150m perception range.
I have changed the following:
point_cloud_range = [-153.6, -153.6, -5.0, 153.6, 153.6, 5.0];
grid_config={
'xbound': [-153.6, 153.6, 0.8],
'ybound': [-153.6, 153.6, 0.8],
'zbound': [-5.0, 5.0, 10.0],
'dbound': [2.0, 158.0, 0.5],}
grid_size=[3072, 3072, 50],
If there is any other variable I missed? Thank you for your time.
Hello, dear authors
I am doing analysis and demonstration of related work recently, can you publish the prediction result file of the nuScenes val dataset? Thank you very much.
I find that lr_config=None. Is that mean lr is not declaying during training ?
It's strange to me. Hope for your reply!
python tools/swap_ema_and_non_ema.py work_dirs/r50-fp16_phase2/r50-fp16-cbgs_phase2_ema.pth
Traceback (most recent call last):
File "tools/swap_ema_and_non_ema.py", line 34, in
main()
File "tools/swap_ema_and_non_ema.py", line 31, in main
print(swap_ema_and_non_ema(sys.argv[1]))
File "tools/swap_ema_and_non_ema.py", line 16, in swap_ema_and_non_ema
ckpt['state_dict'][k] = ckpt['state_dict'][ema_name]
KeyError: 'ema_pts_bbox_head_shared_conv_conv_weight'
Hi, @Divadi
Thanks for your great work!
I'm using this command to train in phase 1:
./tools/dist_train.sh configs/solofusion/r50-fp16_phase1.py 1 --autoscale-lr --no-validate
But I get nan or inf gran norm like follows. What's the problem?
2023-08-05 20:33:00,438 - mmdet - INFO - Iter [50/5274] lr: 2.500e-05, eta: 1:52:54, time: 1.297, data_time: 0.079, memory: 15212, loss_depth: 17.8610, task0.loss_xy: 0.2953, task0.loss_z: 0.2632, task0.loss_whl: 0.7729, task0.loss_yaw: 0.3595, task0.loss_vel: 0.0873, task0.loss_heatmap: 119.4384, task1.loss_xy: 0.2598, task1.loss_z: 0.2394, task1.loss_whl: 0.8032, task1.loss_yaw: 0.3510, task1.loss_vel: 0.0875, task1.loss_heatmap: 4587.3349, task2.loss_xy: 0.2673, task2.loss_z: 0.2936, task2.loss_whl: 1.1154, task2.loss_yaw: 0.3380, task2.loss_vel: 0.0352, task2.loss_heatmap: 8788.5075, task3.loss_xy: 0.2540, task3.loss_z: 0.3332, task3.loss_whl: 0.3935, task3.loss_yaw: 0.3192, task3.loss_vel: 0.0254, task3.loss_heatmap: 486.4620, task4.loss_xy: 0.2952, task4.loss_z: 0.3318, task4.loss_whl: 0.4240, task4.loss_yaw: 0.3640, task4.loss_vel: 0.0736, task4.loss_heatmap: 7132.7133, task5.loss_xy: 0.2553, task5.loss_z: 0.2510, task5.loss_whl: 0.4125, task5.loss_yaw: 0.3699, task5.loss_vel: 0.0489, task5.loss_heatmap: 667.0299, loss: 21809.0666, grad_norm: inf
2023-08-05 20:34:02,149 - mmdet - INFO - Iter [100/5274] lr: 2.500e-05, eta: 1:49:07, time: 1.234, data_time: 0.037, memory: 15212, loss_depth: 19.2391, task0.loss_xy: 0.2312, task0.loss_z: 0.2441, task0.loss_whl: 0.5329, task0.loss_yaw: 0.3376, task0.loss_vel: 0.0765, task0.loss_heatmap: 61.9678, task1.loss_xy: 0.2024, task1.loss_z: 0.2205, task1.loss_whl: 0.7115, task1.loss_yaw: 0.3442, task1.loss_vel: 0.0696, task1.loss_heatmap: 2507.1604, task2.loss_xy: 0.2393, task2.loss_z: 0.3086, task2.loss_whl: 1.0194, task2.loss_yaw: 0.3265, task2.loss_vel: 0.0498, task2.loss_heatmap: 3067.5153, task3.loss_xy: 0.2045, task3.loss_z: 0.2860, task3.loss_whl: 0.2993, task3.loss_yaw: 0.3214, task3.loss_vel: 0.0216, task3.loss_heatmap: 145.4959, task4.loss_xy: 0.2468, task4.loss_z: 0.2349, task4.loss_whl: 0.3451, task4.loss_yaw: 0.3188, task4.loss_vel: 0.0486, task4.loss_heatmap: 5200.2193, task5.loss_xy: 0.2096, task5.loss_z: 0.2415, task5.loss_whl: 0.3624, task5.loss_yaw: 0.3523, task5.loss_vel: 0.0431, task5.loss_heatmap: 472.5930, loss: 11482.6411, grad_norm: 88765.7708
2023-08-05 20:35:03,848 - mmdet - INFO - Iter [150/5274] lr: 2.500e-05, eta: 1:47:10, time: 1.234, data_time: 0.036, memory: 15212, loss_depth: 19.0185, task0.loss_xy: 0.1844, task0.loss_z: 0.2168, task0.loss_whl: 0.3578, task0.loss_yaw: 0.3301, task0.loss_vel: 0.0696, task0.loss_heatmap: 26.7677, task1.loss_xy: 0.1574, task1.loss_z: 0.2252, task1.loss_whl: 0.5558, task1.loss_yaw: 0.3282, task1.loss_vel: 0.0593, task1.loss_heatmap: 1187.7882, task2.loss_xy: 0.1787, task2.loss_z: 0.3269, task2.loss_whl: 0.8107, task2.loss_yaw: 0.3191, task2.loss_vel: 0.0355, task2.loss_heatmap: 1002.1995, task3.loss_xy: 0.1629, task3.loss_z: 0.2478, task3.loss_whl: 0.1829, task3.loss_yaw: 0.3149, task3.loss_vel: 0.0170, task3.loss_heatmap: 82.1107, task4.loss_xy: 0.2011, task4.loss_z: 0.2375, task4.loss_whl: 0.2725, task4.loss_yaw: 0.3130, task4.loss_vel: 0.0489, task4.loss_heatmap: 2316.9863, task5.loss_xy: 0.1666, task5.loss_z: 0.2272, task5.loss_whl: 0.2627, task5.loss_yaw: 0.3322, task5.loss_vel: 0.0422, task5.loss_heatmap: 193.5049, loss: 4835.5605, grad_norm: 42493.9716
Dear author,
I have executed the training command provided in the readme file to train the R50-SOLOFusion model. However, upon inference, I observed that the model's output exhibited relatively large values for mASE and mAOE compared to the provided weights.
I did not make any modifications to the code, except for changing the batch size from 16 to 8. I seek guidance on how to reproduce the expected results under these circumstances.
Thank you for your time.
Dear author,
When I extend SOLOFusion to ConvNext-B, it can't converge well according to your provided config settings. However, when I canceled the fp16 hook and re-trained it, the loss tendency was normal. But there is no doubt that the re-implementation performance will differ from yours. Is there any revision when training the model of the large version?
Notice
There are several common situations in the reimplementation issues as below
Describe the issue
My results mismatch with the repo reported.
Reproduction
The MMDet3D
test.py
.
The provided
r50-fp16-cbgs_phase2.py
andr50-longonly-fp16_phase2.py
Only the dataset path.
nuScenes
dataset.
Environment
python mmdet3d/utils/collect_env.py
to collect necessary environment infomation and paste it here.sys.platform: linux
Python: 3.8.16 (default, Mar 2 2023, 03:21:46) [GCC 11.2.0]
CUDA available: True
GPU 0,1,2,3,4,5,6,7: NVIDIA A100-SXM4-80GB
CUDA_HOME: /nvme/share/cuda-11.3/
NVCC: Build cuda_11.3.r11.3/compiler.29920130_0
GCC: gcc (GCC) 10.1.0
PyTorch: 1.10.1+cu111
PyTorch compiling details: PyTorch built with:
- GCC 7.3
- C++ Version: 201402
- Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v2.2.3 (Git Hash 7336ca9f055cf1bfa13efb658fe15dc9b41f0740)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- LAPACK is enabled (usually provided by MKL)
- NNPACK is enabled
- CPU capability usage: AVX512
- CUDA Runtime 11.1
- NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
- CuDNN 8.0.5
- Magma 2.5.2
- Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.10.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,
TorchVision: 0.11.2+cu111
OpenCV: 4.7.0
MMCV: 1.3.16
MMCV Compiler: GCC 7.3
MMCV CUDA Compiler: 11.3
MMDetection: 2.14.0
MMSegmentation: 0.14.1
MMDetection3D: 0.17.2+229e79e
Results
Configuration: r50-fp16-cbgs_phase2.py
. Checkpoint: solofusion-r50-fp16-cbgs_phase2_ema.pth
.
Evaluating bboxes of pts_bbox
mAP: 0.2757
mATE: 0.6761
mASE: 0.2874
mAOE: 0.4937
mAVE: 0.2744
mAAE: 0.2093
NDS: 0.4438
Eval time: 101.9s
Per-class results:
Object Class AP ATE ASE AOE AVE AAE
car 0.576 0.439 0.163 0.154 0.289 0.207
truck 0.333 0.590 0.217 0.133 0.224 0.212
bus 0.314 0.754 0.212 0.129 0.439 0.246
trailer 0.117 1.198 0.253 0.333 0.201 0.200
construction_vehicle 0.030 1.067 0.521 1.271 0.126 0.345
pedestrian 0.409 0.661 0.291 0.723 0.373 0.199
motorcycle 0.281 0.589 0.261 0.779 0.388 0.261
bicycle 0.259 0.511 0.259 0.806 0.156 0.004
traffic_cone 0.049 0.490 0.411 nan nan nan
barrier 0.390 0.462 0.287 0.115 nan nan
Configuration: r50-longonly-fp16_phase2.py
. Checkpoint: solofusion-r50-longonly-fp16_phase2_ema.pth
.
Evaluating bboxes of pts_bbox
mAP: 0.2532
mATE: 0.7142
mASE: 0.2892
mAOE: 0.6561
mAVE: 0.3579
mAAE: 0.2162
NDS: 0.4032
Eval time: 111.4s
Per-class results:
Object Class AP ATE ASE AOE AVE AAE
car 0.505 0.496 0.173 0.231 0.398 0.222
truck 0.271 0.668 0.223 0.204 0.282 0.225
bus 0.321 0.682 0.215 0.272 0.564 0.243
trailer 0.081 1.050 0.233 0.549 0.265 0.219
construction_vehicle 0.031 1.133 0.506 1.441 0.138 0.333
pedestrian 0.357 0.716 0.291 0.857 0.467 0.248
motorcycle 0.250 0.674 0.270 0.911 0.584 0.232
bicycle 0.280 0.650 0.272 1.272 0.166 0.008
traffic_cone 0.010 0.550 0.425 nan nan nan
barrier 0.427 0.524 0.282 0.166 nan nan
Hi Author,
I have a question about history_bev.
The history_bev is calculated from last iter and is detached.
However, the weight of img branch is changed ever iter, the history_bev stored previous and the history_bev calculate by img_branch(current iter) is different, will this bring issue to the model?
autoscale_lr only works in non-distributed training,because args.gpus and args.gpu_ids are all none in distributed training. But i found your training log, lr is 2e-4, which is scaled by number of gpus, is there anything wrong?
Great work! I have tried to read carefully, and got some question.
low-resolution-Long-term Fusion
is performed after depth prediction, why can it still benefit from localization potential
with long term history frame?
Maybe there are some misunderstanding, but it realy confuse me yet.
Hi authors,
Thank you for your great work and releasing the Val models. I wanted to build on top of SOLOFusion. However, the model zoo does not contain the nuScenes test model with AP=54.0 and NDS=61.9 as reported in Table 3 of the paper.
It would be great if you could release the test model and configs.
Hi, thanks for the great work! May I ask which version of pytorch you used? And could you please provide the checkpoint url of the resnet50 (in1k)? We found that it's difficult to reproduce your results.
Great Works! When will the code be released?
Hello author, can you provide the fps of your r101 model?
Hi Divadi, I've found that execute SOLOFusion project with python3.7.10, torch1.9, cudatoolkit 11.1 as your project mentioned, there is a small bug.
mmdet3d/models/necks/view_transformer_solofusion.py
meshgrid = torch.stack(torch.meshgrid(torch.arange(stereo_W), torch.arange(stereo_H), indexing="xy"), dim=2)
should changed as
meshgrid = torch.stack(torch.meshgrid(torch.arange(stereo_H), torch.arange(stereo_W)), dim=2)
which is mainly because pytorch with version 1.9, the meshgrid method didn't support indexing. The problem further results in the reverse of stereo_H and stereo_W, hope that can help other people
Environment
Python: 3.7.10 (default, Feb 26 2021, 18:47:35) [GCC 7.3.0]
CUDA available: True
GPU 0,1: NVIDIA GeForce RTX 3090
CUDA_HOME: /usr/local/cuda
NVCC: Build cuda_11.1.TC455_06.29190527_0
GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
PyTorch: 1.9.0
PyTorch compiling details: PyTorch built with:
TorchVision: 0.10.0
OpenCV: 4.7.0
MMCV: 1.3.16
MMCV Compiler: GCC 7.3
MMCV CUDA Compiler: 11.1
MMDetection: 2.14.0
MMSegmentation: 0.14.1
MMDetection3D: 0.17.2+683edce
Bug fix
If you have already identified the reason, you can provide the information here. If you are willing to create a PR to fix it, please also leave a comment here and that would be much appreciated!
Dear author,
Hello,
I would like to know which variables should be changed in config file to use 256x256 bev resolution feature.
Thank you for your time.
Now solofusion are trained in 2 phases, can it be simplied to only 1 phase?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.