leaf1170124460/mask3d-shift

This repository provides a multi task benchmark for instance segmentation, depth estimation, and 3D object detection.

mask3d-shift's Issues

the error when batch size is increased.

I appreciate your efforts. I have a question. When I set the batch size to 1, there are no issues, but when I increase it to 2, I encounter an error related to collate_fn. Are you aware of this problem? Can you provide any tips for resolving it?

  File "tools/train.py", line 262, in <module>                                                                                                                                                                     
    main()                                                                                                                                                                                                         
  File "tools/train.py", line 250, in main                                                                                                                                                                         
    train_model(                                                                                                                                                                                                   
  File "/mnt/ywlee/2023_hyundai/Mask3D-SHIFT/projects/mmdet3d_plugin/apis/train.py", line 346, in train_model                                                                                                      
    train_detector(                                                                                                                                                                                                
  File "/mnt/ywlee/2023_hyundai/Mask3D-SHIFT/projects/mmdet3d_plugin/apis/train.py", line 321, in train_detector                                                                                                   
    runner.run(data_loaders, cfg.workflow)
  File "/opt/conda/envs/shift/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 127, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/opt/conda/envs/shift/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 47, in train
    for i, data_batch in enumerate(self.data_loader):
  File "/opt/conda/envs/shift/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()               
  File "/opt/conda/envs/shift/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 561, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/opt/conda/envs/shift/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch
    return self.collate_fn(data)
  File "/opt/conda/envs/shift/lib/python3.8/site-packages/mmcv/parallel/collate.py", line 79, in collate
    return {                 
  File "/opt/conda/envs/shift/lib/python3.8/site-packages/mmcv/parallel/collate.py", line 80, in <dictcomp>
    key: collate([d[key] for d in batch], samples_per_gpu)
  File "/opt/conda/envs/shift/lib/python3.8/site-packages/mmcv/parallel/collate.py", line 84, in collate
    return default_collate(batch)   
  File "/opt/conda/envs/shift/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 56, in default_collate
    return torch.stack(batch, 0, out=out)
RuntimeError: stack expects each tensor to be equal size, but got [4, 6] at entry 0 and [14, 6] at entry 1

error running the test code

I was runing the dist_test.sh,
bash tools/dist_test.sh projects/configs/detr3d_r101_with_seg_depth_shift_minival.py 1 --checkpoint ./epoch_10.pth --out ./output_pkl.pkl --show-dir output_show --format-only
got the following error:

Traceback (most recent call last):
  File "tools/test.py", line 252, in <module>
    main()
  File "tools/test.py", line 180, in main
    init_dist(args.launcher, **cfg.dist_params)
  File "/root/miniconda3/lib/python3.8/site-packages/mmcv/utils/config.py", line 513, in __getattr__
    return getattr(self._cfg_dict, name)
  File "/root/miniconda3/lib/python3.8/site-packages/mmcv/utils/config.py", line 49, in __getattr__
    raise ex
AttributeError: 'ConfigDict' object has no attribute 'dist_params'

could you pls help

missing link

Hi @leaf1170124460 @suniique https://github.com/suniique/SHIFT-3D-challenge/blob/challenge/annotations/SHIFT_challenge2023_multitask.zip is not valid now?

usage question: Mask3D - SHIFT

Can i use the same way on lidar cloud point data ?

The model and loaded state dict do not match exactly

Hello, thank you for providing great code.

After I downloaded 'detr3d_resnet101.pth' and ran the code
'bash tools/dist_train.sh projects/configs/detr3d_r101_with_seg_depth_shift.py 1 --auto-resume',
I got an error like below.

I don't know why there some mismatch between parameters.

Thanks.

> Loading annotations...
> /home/user/anaconda3/envs/shift3D2/lib/python3.8/site-packages/mmdet3d/utils/misc.py:30: UserWarning: There are no checkpoints in the path.
>   warnings.warn('There are no checkpoints in the path.')
> 2024-04-05 00:11:55,817 - mmdet - INFO - load checkpoint from local path: ckptdir/detr3d_resnet101.pth
> 2024-04-05 00:11:56,077 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.0.conv2 is upgraded to version 2.
> 2024-04-05 00:11:56,079 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.1.conv2 is upgraded to version 2.
> 2024-04-05 00:11:56,081 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.2.conv2 is upgraded to version 2.
> 2024-04-05 00:11:56,082 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.3.conv2 is upgraded to version 2.
> 2024-04-05 00:11:56,084 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.4.conv2 is upgraded to version 2.
> 2024-04-05 00:11:56,085 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.5.conv2 is upgraded to version 2.
> 2024-04-05 00:11:56,087 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.6.conv2 is upgraded to version 2.
> 2024-04-05 00:11:56,088 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.7.conv2 is upgraded to version 2.
> 2024-04-05 00:11:56,090 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.8.conv2 is upgraded to version 2.
> 2024-04-05 00:11:56,091 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.9.conv2 is upgraded to version 2.
> 2024-04-05 00:11:56,093 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.10.conv2 is upgraded to version 2.
> 2024-04-05 00:11:56,094 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.11.conv2 is upgraded to version 2.
> 2024-04-05 00:11:56,096 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.12.conv2 is upgraded to version 2.
> 2024-04-05 00:11:56,097 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.13.conv2 is upgraded to version 2.
> 2024-04-05 00:11:56,099 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.14.conv2 is upgraded to version 2.
> 2024-04-05 00:11:56,101 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.15.conv2 is upgraded to version 2.
> 2024-04-05 00:11:56,102 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.16.conv2 is upgraded to version 2.
> 2024-04-05 00:11:56,104 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.17.conv2 is upgraded to version 2.
> 2024-04-05 00:11:56,105 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.18.conv2 is upgraded to version 2.
> 2024-04-05 00:11:56,107 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.19.conv2 is upgraded to version 2.
> 2024-04-05 00:11:56,108 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.20.conv2 is upgraded to version 2.
> 2024-04-05 00:11:56,110 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.21.conv2 is upgraded to version 2.
> 2024-04-05 00:11:56,111 - root - INFO - ModulatedDeformConvPack img_backbone.layer3.22.conv2 is upgraded to version 2.
> 2024-04-05 00:11:56,113 - root - INFO - ModulatedDeformConvPack img_backbone.layer4.0.conv2 is upgraded to version 2.
> 2024-04-05 00:11:56,116 - root - INFO - ModulatedDeformConvPack img_backbone.layer4.1.conv2 is upgraded to version 2.
> 2024-04-05 00:11:56,119 - root - INFO - ModulatedDeformConvPack img_backbone.layer4.2.conv2 is upgraded to version 2.
> 2024-04-05 00:11:56,123 - mmdet - WARNING - The model and loaded state dict do not match exactly
> 
> 1size mismatch for pts_bbox_head.cls_branches.0.6.weight: copying a param with shape torch.Size([10, 256]) from checkpoint, the shape in current model is torch.Size([6, 256]).
> 1size mismatch for pts_bbox_head.cls_branches.0.6.bias: copying a param with shape torch.Size([10]) from checkpoint, the shape in current model is torch.Size([6]).
> 1size mismatch for pts_bbox_head.cls_branches.1.6.weight: copying a param with shape torch.Size([10, 256]) from checkpoint, the shape in current model is torch.Size([6, 256]).
> 1size mismatch for pts_bbox_head.cls_branches.1.6.bias: copying a param with shape torch.Size([10]) from checkpoint, the shape in current model is torch.Size([6]).
> 1size mismatch for pts_bbox_head.cls_branches.2.6.weight: copying a param with shape torch.Size([10, 256]) from checkpoint, the shape in current model is torch.Size([6, 256]).
> 1size mismatch for pts_bbox_head.cls_branches.2.6.bias: copying a param with shape torch.Size([10]) from checkpoint, the shape in current model is torch.Size([6]).
> 1size mismatch for pts_bbox_head.cls_branches.3.6.weight: copying a param with shape torch.Size([10, 256]) from checkpoint, the shape in current model is torch.Size([6, 256]).
> 1size mismatch for pts_bbox_head.cls_branches.3.6.bias: copying a param with shape torch.Size([10]) from checkpoint, the shape in current model is torch.Size([6]).
> 1size mismatch for pts_bbox_head.cls_branches.4.6.weight: copying a param with shape torch.Size([10, 256]) from checkpoint, the shape in current model is torch.Size([6, 256]).
> 1size mismatch for pts_bbox_head.cls_branches.4.6.bias: copying a param with shape torch.Size([10]) from checkpoint, the shape in current model is torch.Size([6]).
> 1size mismatch for pts_bbox_head.cls_branches.5.6.weight: copying a param with shape torch.Size([10, 256]) from checkpoint, the shape in current model is torch.Size([6, 256]).
> 1size mismatch for pts_bbox_head.cls_branches.5.6.bias: copying a param with shape torch.Size([10]) from checkpoint, the shape in current model is torch.Size([6]).
> 1size mismatch for pts_bbox_head.query_embedding.weight: copying a param with shape torch.Size([900, 512]) from checkpoint, the shape in current model is torch.Size([200, 512]).
> missing keys in source state_dict: pts_bbox_head.mask_fcs.0.weight, pts_bbox_head.mask_fcs.1.weight, pts_bbox_head.mask_fcs.1.bias, pts_bbox_head.fc_mask.weight, pts_bbox_head.fc_mask.bias, pts_bbox_head.depth_fcs.0.weight, pts_bbox_head.depth_fcs.1.weight, pts_bbox_head.depth_fcs.1.bias, pts_bbox_head.fc_depth.weight, pts_bbox_head.fc_depth.bias, pts_bbox_head.conv_depth.weight, pts_bbox_head.conv_depth.bias

...
File "/home/user/anaconda3/envs/shift3D2/lib/python3.8/site-packages/mmcv/ops/modulated_deform_conv.py", line 101, in forward
    ext_module.modulated_deform_conv_forward(
RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)`
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 121525) of binary

usage question: detr3d for SHIFT

@leaf1170124460 Hi Chengxiang, thanks for sharing the Mask3D-SHIFT code, I'm wondering why the DETR3D is used for SHIFT dataset, could you give some reference (e.g. papers) or explanation? As DETR-3D was designed for multi-view 3D detection and did not sepcifically designed depth prediction.

Thanks!

no depth img in minival set

There is no depth img in minival set.
When I downloaded minival set, I can't match the 0000200, 0000300, 0000400 img between img and depth.
There are 0000200, 0000300, 0000400 in img, but There is not in depth folder. Can you check this issue?
When i reproduce your code in my local, It is important for submit good result during 2023 ICCV challenge.

And also, I want to ask to check the minival ground truth file. It is not available to download, now.

Thank you for your attention.