trailab/caddn

Categorical Depth Distribution Network for Monocular 3D Object Detection (CVPR 2021 Oral)

CaDDN

CaDDN is a monocular-based 3D object detection method. This repository is based off of [OpenPCDet].

Categorical Depth Distribution Network for Monocular 3D Object Detection
Cody Reading, Ali Harakeh, Julia Chae, and Steven L. Waslander
[Paper]

Overview

• Changelog
• Model Zoo
• Installation
• Getting Started
• Citation

Changelog

[2021-03-16] CaDDN v0.3.0 is released.

Introduction

What does CaDDN do?

CaDDN is a general PyTorch-based method for 3D object detection from monocular images. At the time of submission, CaDDN achieved first 1st place among published monocular methods on the Kitti 3D object detection benchmark. We welcome contributions to this project.

CaDDN design pattern

We inherit the design pattern from [OpenPCDet].

• Data-Model separation with unified point cloud coordinate for easily extending to custom datasets:

• Unified 3D box definition: (x, y, z, dx, dy, dz, heading).

Model Zoo

KITTI 3D Object Detection Baselines

Selected supported methods are shown in the below table. The results are the 3D detection performance of Car class on the val set of KITTI dataset.

• All models are trained with 2 Tesla T4 GPUs and are available for download.
• The training time is measured with 2 Tesla T4 GPUs and PyTorch 1.4.

	training time	Easy@R40	Moderate@R40	Hard@R40	download
CaDDN	~76 hours	23.77	16.07	13.61	model-774M

Installation

Please refer to INSTALL.md for the installation of CaDDN.

Getting Started

Please refer to GETTING_STARTED.md to learn more usage about this project.

License

CaDDN is released under the Apache 2.0 license.

Acknowledgement

CaDDN is an open source project for monocular-based 3D scene perception. We would like to thank the authors of OpenPCDet for their open-source release of their 3D object detection codebase.

Citation

If you find this project useful in your research, please consider citing:

@article{CaDDN,
    title={Categorical Depth DistributionNetwork for Monocular 3D Object Detection},
    author={Cody Reading and
            Ali Harakeh and
            Julia Chae and
            Steven L. Waslander},
    journal = {CVPR},
    year={2021}
}

Contribution

Welcome to be a member of the CaDDN development team by contributing to this repo, and feel free to contact us for any potential contributions.

caddn's Issues

Generation of labels on depth_2 file

Hi

Thank you very much for sharing your great work with us.

I am a big fun of your use of semantic segmentation and pcdet in 3D detection.

Could I ask detatils about how you trasform the sparse depth labels provided in kitti to the denth labels in depth_2?

I would really appreciate if you could help.

Best.

How to get the depth map?

I've downloaded the depth map you provided, but I don't know how to generate it

How can I convert the Lidar point cloud data into sparse depth map as mentioned in the paper, and then use the sparse depth to complete the dense depth map?

Can you provide the code?

About depth map file

Wonderfull job!
But i still have a personal question, how to produce the depth maps ground true for kitti training set?
In an another word, how i reproduce this depth maps ground true to supervise my network for other dataset?
The download file may not belong to the official KITTI file.
Could you help me?

KITTI Dataset
Please download the official KITTI 3D object detection dataset and organize the downloaded files as follows:
Download the precomputed depth maps for the KITTI training set

Question about voxel size and point cloud range

Hi, thanks for sharing your great work!
I want to enlarge voxel size for less memory usage. I notice that there are two factors affect 3d voxel shape: VOXEL_SIZE and POINT_CLOUD_RANGE. In POINT_CLOUD_RANGE: [2, -30.08, -3.0, 46.8, 30.08, 1.0], the forward range is limited into (2,46.8), is it mean that the far away objects are ignored from training data? What's the intuition behind it?
Thanks!

How to draw the smear effect

Thanks for sharing the codes. Just wonder how to draw the smear effect as Figure 1 show in the papers?

Problem the provided pretrained model

Hi,

I tried to evaluate the pre-rained model provided on the GitHub homepage.
My evaluation of this model on the KITTI 3D validation dataset shows a moderate 3D ap(R40) of 15.87.
However, a moderate 3D ap (R40) of 16.31 is illustrated in table 3 and table 4 in your essay.
Do you have any idea about it?

On the other hand, as I found on prevent issues, you mentioned that you have refactored your model. May I ask what specific changes have you made?

Best
Nelson

result on waymo dataset

Thanks for your contribution!

Waymo dataset is quite large and more diverse. Why didn't you use a look-around camera for detection on the Waymo？

about dist_test.sh

Thanks for your contribution!

When I run the script dist_test.sh, what value should be set about PY_ARGS.

How to generate a depth map for the Kitti dataset?

Thanks for your help on the previous issue.
Was the depth map of the Kitti dataset synthesized from 5 frames of LiDAR Scan? Because Lidar Scan and depth map data looked different.

Training on custom dataset

Hello Thank you for your great work.

I am trying to train your model on a custom dataset where we have 3d bounding boxes for images. But we don't have depth maps or sparse LIDAR maps. Can I still use your model for 3d object detection on our dataset?

Thanks again for your help.

result on NuScenes dataset

How about the result on nuscenes dataset?

Which version of Waymo did you use?

Thanks for sharing your great work! Could you tell me that which version of Waymo did you use in your paper, V1.0, V1.1, or V1.2?

Thank you!

RuntimeError: Error compiling objects for extension

Traceback (most recent call last): File "setup.py", line 105, in <module> 'src/sampling_gpu.cu', File "/home/CN/zizhang.wu/anaconda3/envs/CaDDN/lib/python3.7/site-packages/setuptools/__init__.py", line 153, in setup return distutils.core.setup(**attrs) File "/home/CN/zizhang.wu/anaconda3/envs/CaDDN/lib/python3.7/distutils/core.py", line 148, in setup dist.run_commands() File "/home/CN/zizhang.wu/anaconda3/envs/CaDDN/lib/python3.7/distutils/dist.py", line 966, in run_commands self.run_command(cmd) File "/home/CN/zizhang.wu/anaconda3/envs/CaDDN/lib/python3.7/distutils/dist.py", line 985, in run_command cmd_obj.run() File "/home/CN/zizhang.wu/anaconda3/envs/CaDDN/lib/python3.7/site-packages/setuptools/command/develop.py", line 34, in run self.install_for_development() File "/home/CN/zizhang.wu/anaconda3/envs/CaDDN/lib/python3.7/site-packages/setuptools/command/develop.py", line 136, in install_for_development self.run_command('build_ext') File "/home/CN/zizhang.wu/anaconda3/envs/CaDDN/lib/python3.7/distutils/cmd.py", line 313, in run_command self.distribution.run_command(command) File "/home/CN/zizhang.wu/anaconda3/envs/CaDDN/lib/python3.7/distutils/dist.py", line 985, in run_command cmd_obj.run() File "/home/CN/zizhang.wu/anaconda3/envs/CaDDN/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 79, in run _build_ext.run(self) File "/home/CN/zizhang.wu/anaconda3/envs/CaDDN/lib/python3.7/distutils/command/build_ext.py", line 340, in run self.build_extensions() File "/home/CN/zizhang.wu/anaconda3/envs/CaDDN/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 580, in build_extensions build_ext.build_extensions(self) File "/home/CN/zizhang.wu/anaconda3/envs/CaDDN/lib/python3.7/distutils/command/build_ext.py", line 449, in build_extensions self._build_extensions_serial() File "/home/CN/zizhang.wu/anaconda3/envs/CaDDN/lib/python3.7/distutils/command/build_ext.py", line 474, in _build_extensions_serial self.build_extension(ext) File "/home/CN/zizhang.wu/anaconda3/envs/CaDDN/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 196, in build_extension _build_ext.build_extension(self, ext) File "/home/CN/zizhang.wu/anaconda3/envs/CaDDN/lib/python3.7/distutils/command/build_ext.py", line 534, in build_extension depends=ext.depends) File "/home/CN/zizhang.wu/anaconda3/envs/CaDDN/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 423, in unix_wrap_ninja_compile with_cuda=with_cuda) File "/home/CN/zizhang.wu/anaconda3/envs/CaDDN/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1140, in _write_ninja_file_and_compile_objects error_prefix='Error compiling objects for extension') File "/home/CN/zizhang.wu/anaconda3/envs/CaDDN/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1413, in _run_ninja_build raise RuntimeError(message) RuntimeError: Error compiling objects for extension
When I runs the python setup.py develop, I met the probelm.
My environment is that
+-----------------------------------------------------------------------------+ | NVIDIA-SMI 440.82 Driver Version: 440.82 CUDA Version: 10.2 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================|
and my pytorch installed is conda install pytorch==1.5.0 torchvision==0.6.0 cudatoolkit=10.2 -c pytorch

ImportError: cannot import name 'iou3d_nms_cuda' from 'pcdet.ops.iou3d_nms' (unknown location)

Hi~

Thank you very much for sharing your great work with us.

When I generate data info in kitti format, the following error occurred:
ImportError: cannot import name 'iou3d_nms_cuda' from 'pcdet.ops.iou3d_nms' (unknown location)

Then I search for 'iou3d_nms_cuda' file in 'pcdet.ops.iou3d_nms' , but it is not found. Is this file missing?How should I solve this problem?

My dataset is in KITTI format, and I execute the following code:
python -m pcdet.datasets.kitti.kitti_dataset create_kitti_infos tools/cfgs/dataset_configs/kitti_dataset.yaml

I would really appreciate if you could help.

Best.

Training program

Thank you for your help in advance.
When I run the scipts bash scripts/dist_train.sh 2 --batch_size 4 --cfg_file /newnfs/zzwu/08_3d_code/CaDDN-master/tools/cfgs/kitti_models/CaDDN.yaml
The program was stuck here.
`(CaDDN05) zizhang.wu@shaxbw06:/newnfs/zzwu/08_3d_code/CaDDN-master/tools$ sh train.sh

• NGPUS=2
• PY_ARGS='--batch_size 4 --cfg_file /newnfs/zzwu/08_3d_code/CaDDN-master/tools/cfgs/kitti_models/CaDDN.yaml'
• python -m torch.distributed.launch --nproc_per_node=2 train.py --launcher pytorch --batch_size 4 --cfg_file /newnfs/zzwu/08_3d_code/CaDDN-master/tools/cfgs/kitti_models/CaDDN.yaml

---

Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.

---

`

Question about waymo model in your paper

Hi. Is your model on waymo dataset train for vehicle, cyclist and pedestrian simultaneously?

Questions about training time

Hello,

Thanks for releasing the code of this great work! I have a question regarding the training time and I would appreciate it if you can offer some advice.

I'm training with the default config for KITTI with 8 V100 GPUs and one epoch takes about 1 hours and 10 minutes. When I reduce the number of GPU to 1 it further increases to 9 hours. So the total training time is much longer than what you posted here. Do you have some suggestions for that? Thanks!

allow_unreachable=True) # allow_unreachable flag RuntimeError: derivative for to_sparse is not implemented

When I added sparse conv to CaDDN, the error is below.

Traceback (most recent call last):
File "train.py", line 215, in
main()
File "train.py", line 185, in main
merge_all_iters_to_one_epoch=args.merge_all_iters_to_one_epoch
File "/newnfs/zzwu/08_3d_code/CaDDN_CenterPoint/tools/train_utils/train_utils.py", line 95, in train_model
dataloader_iter=dataloader_iter
File "/newnfs/zzwu/08_3d_code/CaDDN_CenterPoint/tools/train_utils/train_utils.py", line 42, in train_one_epoch
loss.backward()
File "/home/CN/zizhang.wu/anaconda3/envs/CaDDN_CenterPoint/lib/python3.7/site-packages/torch/tensor.py", line 221, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/home/CN/zizhang.wu/anaconda3/envs/CaDDN_CenterPoint/lib/python3.7/site-packages/torch/autograd/init.py", line 132, in backward
allow_unreachable=True) # allow_unreachable flag
RuntimeError: derivative for to_sparse is not implemented

How much performance degrade with lighter backbone ?

Hello !

The backbone used in this paper is heavy which leads to the slower inference time.
Have you ever tried lighter backbone such as Mobile Deeplab ?

I'd like to figure out how much performance degrade when using lighter backbone network.

Thanks a lot :)

Is the depth prediction network pre-trained on a larger depth dataset other than KITTI itself ?

HELLO, good job!
I have been looking into different mono 3D detectors recently, and i noticed some methods surpass CaDDN by 2-3 points in the KITTI test set up to now. By reading the article , i noticed a significant step in these methods is to use a lager depth dataset other than KITTI , e.g. DDAD to pre-train the depth prediction network and then fine-tune it to the KITTI , which results in a huge improvement in AP.
SO i'd like to know if the backbone of CaDDN is pre-trained on other lager depth dataset . If not, i do not think these 2 methods can be compared fairly, and why not try it i.e. pre-traing CaDDN on a lager depth dataset, and lift CaDDN to a new level.
I have some new idea on CaDDN and am trying, but first i think i should control the resource uesd to be same compared with the other methods, i.e. i'd like to use CaDDN to pre-train on a large dataset to see how CaDDN can perform. Otherwise i do not think i can surpass those methods using extra large dataset .

Cuda Out of Memory with Cam150 image (above 2M image size)

I train CaDDN on A100 with Kitti Dataset it works fluently.
But with our private dataset (captured with lidar and camera 150 degree) (after conversion to Kitti), it produces errors:
File "/home/ubuntu/CaDDN/CaDDN/pcdet/models/backbones_3d/ffe/depth_ffe.py", line 93, in create_frustum_features
frustum_features = depth_probs * image_features
RuntimeError: CUDA out of memory. Tried to allocate 5.53 GiB (GPU 0; 39.59 GiB total capacity; 32.48 GiB already allocated; 4.92 GiB free; 32.99 GiB reserved in total by PyTorch).
depth probs: torch.Size([2, 1, 80, 302, 480])
image feature shape: torch.Size([2, 64, 1, 302, 480])

Pytorch 1.9.0
cudatoolkit 11.1
torchaudio 0.9.0
torchvision 0.5.0
It is the same error with cudatoolkit 11.0, pytorch 1.7.1 .
Thank you very much !!!

Depth Map Completion Setup

Hi, from the paper, it mentioned that you use https://github.com/kujason/ip_basic for depth completion. Which version of the algorithm are you using for KITTI and Waymo? Gaussian, bilateral, multiscale? Thanks.

how to select the best model ?

Hi, after finishing training the KITTI 3D detection train + val set for 80 Epochs, which one of the trained models would be the best for submitting to the KITTI official?

pcdet/ops/iou3d_nms/src/iou3d_cpu.cpp:12:10: fatal error: cuda.h: No such file or directory

Hi, when I install the pcdet library with the command:python setup.py develop, there is a problem appealed:

pcdet/ops/iou3d_nms/src/iou3d_cpu.cpp:12:10: fatal error: cuda.h: No such file or directory

so, could you please give me some suggestions?

How to use ip_basic get dense depth map on waymo dataset

Thank for your great work! I'm confuse about how to generate depth map on waymo dataset. First I project lidar point to image plane and get sparse depth map. Then I use ip_basic to get the dense depth map, but it looks not good, just as follow:

Questions about MaptoBEV operation

Hello!
Thanks for your sharing great mono3d detection work CaDDN! After I read the code, I have a simple question about the Map-to-BEV operation. I found two operations here 'Conv2DCollapse' and 'HeightCompression', and they do the same thing to collapse voxel-wise features into BEV features. I want to know where is the difference between these two operations. And the code of reshape operation '.view' and '.flatten' , is there any difference between these two ways?

Thanks a lot!

The results I test on the validation set are different from those provided in the paper?

Thanks for your contribution!
The results I tested on the validation set is bellow:

but, the results provided in this paper is:

Rendering 3D bounding boxes onto test images

Hello,

Thanks for your help on the previous issue. I'm running the tools/test.py script with the intent to generate 3D bounding boxes on the test images as a result image set. I'm using the KITTI dataset. Is there a flag I can set for test.py, or use another software package, or follow some other additional steps to be able to render those 3D bounding boxes?

Also, my next step is to try using a custom generated imageset. I'll be using the pre-trained model even though it's from the perspective of a street light. They would be frames from a video capture. I was going to try to follow the structure of the KITTI dataset and see if I can repurpose the script that generated the datainfos. Do you have any suggestions for the best way to go about using a custom set of images like this?

Ultimately, it's my goal to see 3D bounding box rendering working on those images.

Thanks!

Could you please provide the code for visualization?

Result gap between my own training and published paper

Hi, Cody! Thanks very much for your great work!

I had trained with your released code in both repo(CaDDN and OpenPCDet) several times, but can’t match the result in your paper with a gap about 3 point lower in R40 easy, 1 point lower in R40 moderate and hard.

My best R40 val result in the total 80 epochs is 20.6976, 15.2527, 12.8991.

I am wonder if there are any details that I am missing?

How to generate the 'depth_2' dataset?

Thanks for your help on the previous issue. I want to generate depth_2 dataset in my data. How can I do?

Question about DISCRETIZE in caddn.yaml

Hi~ thanks for sharing your great work!

I want to train CaDDN on custom dataset. I notice that the values of depth_min and depth_max in the DISRETIZE in caddn.yaml match the range of x in POINT_CLOUD_RANGE. However, the -y direction of the custom dataset in the lidar coordinate system is the forward direction. Therefore,

1. Do depth_min and depth_max refer to the distances in the advancing direction under the camera? Does it have to be numerically consistent with the range of the heading of POINT_CLOUD_RANGE?
2. How is the number of num_bins obtained? Is it set based on different datasets?

Thanks!

Acquisition of depth data from waymo dataset

Is it possible to provide scripts for depth map acquisition in waymo dataset?
Including how to convert lidar data into sparse depth map?
And how to complete sparse depth map to dense depth map？
Thanks！

how to understand the rec_to_lidar function?

def rect_to_lidar(self, pts_rect):
"""
:param pts_lidar: (N, 3)
:return pts_rect: (N, 3)
"""
pts_rect_hom = self.cart_to_hom(pts_rect) # (N, 4)
R0_ext = np.hstack((self.R0, np.zeros((3, 1), dtype=np.float32))) # (3, 4)
R0_ext = np.vstack((R0_ext, np.zeros((1, 4), dtype=np.float32))) # (4, 4)
R0_ext[3, 3] = 1
V2C_ext = np.vstack((self.V2C, np.zeros((1, 4), dtype=np.float32))) # (4, 4)
V2C_ext[3, 3] = 1

    pts_lidar = np.dot(pts_rect_hom, np.linalg.inv(np.dot(R0_ext, V2C_ext).T))
    return pts_lidar[:, 0:3]`

`

Errors found in running ./build.sh

Thank you for sharing your code.
During the last step of docker building, I met with the following problem:

My local machine is CUDA 11.2.

I am not sure if it is related to my local cuda.

PyTorch exceptions when running train.py or test.py

Hello,

Thank you for maintaining this project. I am trying out the test.py and train.py scripts with the kitti dataset from NVIDIA Jetson AGX Xavier and running into an error I am having trouble resolving. Shortly after I get to these log lines, an exception is thrown:

2021-06-07 22:59:42,230   INFO  **********************Start training kitti_models/CaDDN(default)**********************
epochs:   0%|                                                                                                     | 0/80 [00:02<?, ?it/s]

This is the command: python train.py --cfg_file cfgs/kitti_models/CaDDN.yaml --batch_size 2

That exception is this:

Traceback (most recent call last):                                                                              | 0/1856 [00:00<?, ?it/s]
  File "train.py", line 198, in <module>
    main()
  File "train.py", line 170, in main
    merge_all_iters_to_one_epoch=args.merge_all_iters_to_one_epoch
  File "/home/dpwolfe/repo/CaDDN/tools/train_utils/train_utils.py", line 93, in train_model
    dataloader_iter=dataloader_iter
  File "/home/dpwolfe/repo/CaDDN/tools/train_utils/train_utils.py", line 19, in train_one_epoch
    batch = next(dataloader_iter)
  File "/home/dpwolfe/miniforge3/envs/open-pcdet/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 363, in __next__
    data = self._next_data()
  File "/home/dpwolfe/miniforge3/envs/open-pcdet/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 989, in _next_data
    return self._process_data(data)
  File "/home/dpwolfe/miniforge3/envs/open-pcdet/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 1014, in _process_data
    data.reraise()
  File "/home/dpwolfe/miniforge3/envs/open-pcdet/lib/python3.6/site-packages/torch/_utils.py", line 395, in reraise
    raise self.exc_type(msg)
KeyError: Caught KeyError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/dpwolfe/miniforge3/envs/open-pcdet/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 185, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/dpwolfe/miniforge3/envs/open-pcdet/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/dpwolfe/miniforge3/envs/open-pcdet/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/dpwolfe/repo/OpenPCDet/pcdet/datasets/kitti/kitti_dataset.py", line 424, in __getitem__
    data_dict = self.prepare_data(data_dict=input_dict)
  File "/home/dpwolfe/repo/OpenPCDet/pcdet/datasets/dataset.py", line 129, in prepare_data
    'gt_boxes_mask': gt_boxes_mask
  File "/home/dpwolfe/repo/OpenPCDet/pcdet/datasets/augmentor/data_augmentor.py", line 112, in forward
    data_dict = cur_augmentor(data_dict=data_dict)
  File "/home/dpwolfe/repo/OpenPCDet/pcdet/datasets/augmentor/data_augmentor.py", line 84, in random_image_flip
    images = data_dict["images"]
KeyError: 'images'

While going through the setup process, I've needed to use some slightly different versions of dependencies declared in the requirements file. I've done this to use available pre-built versions for aarch64 and avoid having to build them myself. Those are:

numpy 1.19.5 instead of 1.20.1
scikit-image 0.18.0rc1 instead of 0.18.1
scipy 1.5.4 instead of 1.6.1
tifffile 2020.9.3 instead of 2021.2.26

I've also needed to make a couple major version changes that I'm concerned might be the problem:

kornia 0.5.3 instead of 0.2.2 since 0.2.2 was not readily available for aarch64
torch 1.6.0 since kornia 0.5.3 requires >= 1.6.0

Hardware and Environment:
NVIDIA Jetson AGX Xavier
Jetpack 4.5 (Ubuntu 18.04)
Python 3.6 (using miniforge)
CUDA Version: 10.2

My conda list output

# packages in environment at /home/dpwolfe/miniforge3/envs/open-pcdet:
#
# Name                    Version                   Build  Channel
_openmp_mutex             4.5                       1_gnu    conda-forge
absl-py                   0.12.0                   pypi_0    pypi
ca-certificates           2018.03.07                    0    c4aarch64
cachetools                4.2.1                    pypi_0    pypi
certifi                   2020.12.5                pypi_0    pypi
chardet                   4.0.0                    pypi_0    pypi
cycler                    0.10.0                   pypi_0    pypi
dataclasses               0.8                      pypi_0    pypi
decorator                 4.4.2                    pypi_0    pypi
easydict                  1.9                      pypi_0    pypi
future                    0.18.2                   pypi_0    pypi
google-auth               1.27.1                   pypi_0    pypi
google-auth-oauthlib      0.4.3                    pypi_0    pypi
grpcio                    1.36.1                   pypi_0    pypi
idna                      2.10                     pypi_0    pypi
imageio                   2.9.0                    pypi_0    pypi
importlib-metadata        4.5.0                    pypi_0    pypi
kiwisolver                1.3.1                    pypi_0    pypi
kornia                    0.5.3                    pypi_0    pypi
ld_impl_linux-aarch64     2.35.1               h02ad14f_2    conda-forge
libblas                   3.8.0               17_openblas    conda-forge
libcblas                  3.8.0               17_openblas    conda-forge
libffi                    3.3                  h884eca8_2    conda-forge
libgcc-ng                 9.3.0               he1ea209_19    conda-forge
libgfortran-ng            7.3.0                h6bc79d0_0    c4aarch64
libgomp                   9.3.0               he1ea209_19    conda-forge
liblapack                 3.8.0               17_openblas    conda-forge
libllvm10                 10.0.1               he513fc3_3    conda-forge
libllvm12                 12.0.0               h6293a0b_1    conda-forge
libopenblas               0.3.10          pthreads_hb3c22a3_4    conda-forge
libstdcxx-ng              9.3.0               h1ed1776_19    conda-forge
llvm-tools                12.0.0               h6293a0b_1    conda-forge
llvmdev                   12.0.0               h6293a0b_1    conda-forge
llvmlite                  0.35.0           py36h2826d25_1    conda-forge
markdown                  3.3.4                    pypi_0    pypi
matplotlib                3.3.4                    pypi_0    pypi
ncurses                   6.2                  h7fd3ca4_4    conda-forge
networkx                  2.5                      pypi_0    pypi
numba                     0.52.0           py36ha63b481_0    conda-forge
numpy                     1.19.5           py36hdc1b780_1    conda-forge
oauthlib                  3.1.0                    pypi_0    pypi
openssl                   1.1.1k               hf897c2e_0    conda-forge
pcdet                     0.3.0+150b7ba             dev_0    <develop>
pillow                    8.1.2                    pypi_0    pypi
pip                       21.1.2             pyhd8ed1ab_0    conda-forge
protobuf                  3.15.3                   pypi_0    pypi
pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
pyparsing                 2.4.7                    pypi_0    pypi
python                    3.6.13          h468538b_0_cpython    conda-forge
python-dateutil           2.8.1                    pypi_0    pypi
python_abi                3.6                     1_cp36m    conda-forge
pywavelets                1.1.1                    pypi_0    pypi
pyyaml                    5.4.1                    pypi_0    pypi
readline                  8.1                  h1a49cc3_0    conda-forge
requests                  2.25.1                   pypi_0    pypi
requests-oauthlib         1.3.0                    pypi_0    pypi
rsa                       4.7.2                    pypi_0    pypi
scikit-image              0.18.0rc1                pypi_0    pypi
scipy                     1.5.4                    pypi_0    pypi
setuptools                49.6.0           py36h704843e_3    conda-forge
six                       1.15.0                   pypi_0    pypi
spconv                    1.2.1                    pypi_0    pypi
sqlite                    3.35.5               h43e6a2a_0    conda-forge
tensorboard               2.4.1                    pypi_0    pypi
tensorboard-plugin-wit    1.8.0                    pypi_0    pypi
tensorboardx              2.1                      pypi_0    pypi
tifffile                  2020.9.3                 pypi_0    pypi
tk                        8.6.10               ha99a2a3_1    conda-forge
torch                     1.6.0                    pypi_0    pypi
torchvision               0.9.1                    pypi_0    pypi
tqdm                      4.58.0                   pypi_0    pypi
typing-extensions         3.7.4.3                  pypi_0    pypi
urllib3                   1.26.3                   pypi_0    pypi
werkzeug                  1.0.1                    pypi_0    pypi
wheel                     0.36.2             pyhd3deb0d_0    conda-forge
xz                        5.2.5                h6dd45c4_1    conda-forge
yolk3k                    0.9                      pypi_0    pypi
zipp                      3.4.1                    pypi_0    pypi
zlib                      1.2.11               h7b6447c_2    c4aarch64

Also, if I run python test.py --cfg_file cfgs/kitti_models/CaDDN.yaml --batch_size 2 --ckpt ../checkpoints/caddn.pth I will get the following:

2021-06-07 23:16:09,131   INFO  Loading KITTI dataset
2021-06-07 23:16:09,576   INFO  Total samples for KITTI dataset: 3769
2021-06-07 23:16:13,328   INFO  ==> Loading parameters from checkpoint ../checkpoints/caddn.pth to GPU
2021-06-07 23:16:16,395   INFO  ==> Checkpoint trained from version: pcdet+0.3.0+0000000
2021-06-07 23:16:18,141   INFO  ==> Done (loaded 229/229)
2021-06-07 23:16:18,239   INFO  *************** EPOCH no_number EVALUATION *****************
eval:   0%|                                                                                                     | 0/1885 [00:00<?, ?it/s]

Followed by this error:

Traceback (most recent call last):
  File "test.py", line 199, in <module>
    main()
  File "test.py", line 195, in main
    eval_single_ckpt(model, test_loader, args, eval_output_dir, logger, epoch_id, dist_test=dist_test)
  File "test.py", line 63, in eval_single_ckpt
    result_dir=eval_output_dir, save_to_file=args.save_to_file
  File "/home/dpwolfe/repo/CaDDN/tools/eval_utils/eval_utils.py", line 57, in eval_one_epoch
    pred_dicts, ret_dict = model(batch_dict)
  File "/home/dpwolfe/miniforge3/envs/open-pcdet/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/dpwolfe/repo/OpenPCDet/pcdet/models/detectors/caddn.py", line 11, in forward
    batch_dict = cur_module(batch_dict)
  File "/home/dpwolfe/miniforge3/envs/open-pcdet/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/dpwolfe/repo/OpenPCDet/pcdet/models/backbones_2d/map_to_bev/conv2d_collapse.py", line 34, in forward
    voxel_features = batch_dict["voxel_features"]
KeyError: 'voxel_features'
Exception ignored in: <bound method tqdm.__del__ of <tqdm.std.tqdm object at 0x7f849ebb70>>
Traceback (most recent call last):
  File "/home/dpwolfe/miniforge3/envs/open-pcdet/lib/python3.6/site-packages/tqdm/std.py", line 1143, in __del__
  File "/home/dpwolfe/miniforge3/envs/open-pcdet/lib/python3.6/site-packages/tqdm/std.py", line 1297, in close
  File "/home/dpwolfe/miniforge3/envs/open-pcdet/lib/python3.6/site-packages/tqdm/std.py", line 1490, in display
  File "/home/dpwolfe/miniforge3/envs/open-pcdet/lib/python3.6/site-packages/tqdm/std.py", line 1146, in __str__
  File "/home/dpwolfe/miniforge3/envs/open-pcdet/lib/python3.6/site-packages/tqdm/std.py", line 1448, in format_dict
TypeError: 'NoneType' object is not iterable

Have you seen these errors before or do you know if they're caused by the change in one of the dependencies, such as pytorch 1.6.0 instead of 1.4.0?

I greatly appreciate your help. The next path for me otherwise is to unwind my setup and rebuild it with torch 1.4.0 (NVIDIA provides it) since the exception originates from torch. I'll also have to see if I can build kornia 0.2.2 for aarch64.

Thank you!

when training, the loss and depth loss are not convergent

Thanks for your brilliant work!

However, I print the loss and depth_loss when training and they have no tendency to converge. And I want to know what is wrong with my operation.

Thanks in advance for any help you can offer.

train: 55%|█████▌ | 1022/1856 [18:19<14:59, 1.08s/it, total_it=1021]
epochs: 0%| | 0/80 [18:19<?, ?it/s, loss=10.9, lr=0.000101]
loss is tensor(19.3046, device='cuda:0')
loss_depth is 16.68878936767578

train: 55%|█████▌ | 1023/1856 [18:20<14:59, 1.08s/it, total_it=1022]
epochs: 0%| | 0/80 [18:20<?, ?it/s, loss=19.3, lr=0.000101]
loss is tensor(8.4508, device='cuda:0')
loss_depth is 6.085319995880127

train: 55%|█████▌ | 1024/1856 [18:21<14:58, 1.08s/it, total_it=1023]
epochs: 0%| | 0/80 [18:21<?, ?it/s, loss=8.45, lr=0.000101]
loss is tensor(7.5888, device='cuda:0')
loss_depth is 4.7100067138671875

train: 55%|█████▌ | 1025/1856 [18:22<14:56, 1.08s/it, total_it=1024]
epochs: 0%| | 0/80 [18:22<?, ?it/s, loss=7.59, lr=0.000101]
loss is tensor(7.5831, device='cuda:0')
loss_depth is 5.188798904418945

train: 55%|█████▌ | 1026/1856 [18:23<14:56, 1.08s/it, total_it=1025]
epochs: 0%| | 0/80 [18:23<?, ?it/s, loss=7.58, lr=0.000101]
loss is tensor(11.0090, device='cuda:0')
loss_depth is 8.783016204833984

train: 55%|█████▌ | 1027/1856 [18:24<14:55, 1.08s/it, total_it=1026]
epochs: 0%| | 0/80 [18:24<?, ?it/s, loss=11, lr=0.000101]
loss is tensor(9.2116, device='cuda:0')
loss_depth is 6.269783020019531

train: 55%|█████▌ | 1028/1856 [18:25<14:55, 1.08s/it, total_it=1027]
epochs: 0%| | 0/80 [18:26<?, ?it/s, loss=9.21, lr=0.000101]
loss is tensor(7.4908, device='cuda:0')
loss_depth is 4.79773473739624

train: 55%|█████▌ | 1029/1856 [18:26<14:52, 1.08s/it, total_it=1028]
epochs: 0%| | 0/80 [18:27<?, ?it/s, loss=7.49, lr=0.000101]
loss is tensor(17.2144, device='cuda:0')
loss_depth is 14.055488586425781

train: 55%|█████▌ | 1030/1856 [18:27<14:52, 1.08s/it, total_it=1029]
epochs: 0%| | 0/80 [18:28<?, ?it/s, loss=17.2, lr=0.000101]
loss is tensor(7.1547, device='cuda:0')
loss_depth is 4.631966590881348

train: 56%|█████▌ | 1031/1856 [18:28<14:50, 1.08s/it, total_it=1030]
epochs: 0%| | 0/80 [18:29<?, ?it/s, loss=7.15, lr=0.000101]
loss is tensor(14.0801, device='cuda:0')
loss_depth is 10.272579193115234

train: 56%|█████▌ | 1032/1856 [18:29<14:50, 1.08s/it, total_it=1031]
epochs: 0%| | 0/80 [18:30<?, ?it/s, loss=14.1, lr=0.000101]
loss is tensor(8.0110, device='cuda:0')
loss_depth is 4.475630283355713

train: 56%|█████▌ | 1033/1856 [18:30<14:48, 1.08s/it, total_it=1032]
epochs: 0%| | 0/80 [18:31<?, ?it/s, loss=8.01, lr=0.000101]
loss is tensor(10.5358, device='cuda:0')
loss_depth is 7.642397880554199

train: 56%|█████▌ | 1034/1856 [18:32<14:49, 1.08s/it, total_it=1033]
epochs: 0%| | 0/80 [18:32<?, ?it/s, loss=10.5, lr=0.000101]
loss is tensor(10.3984, device='cuda:0')
loss_depth is 7.659360408782959

train: 56%|█████▌ | 1035/1856 [18:33<14:48, 1.08s/it, total_it=1034]
epochs: 0%| | 0/80 [18:33<?, ?it/s, loss=10.4, lr=0.000101]
loss is tensor(7.7434, device='cuda:0')
loss_depth is 4.804980278015137

train: 56%|█████▌ | 1036/1856 [18:34<14:46, 1.08s/it, total_it=1035]
epochs: 0%| | 0/80 [18:34<?, ?it/s, loss=7.74, lr=0.000101]
loss is tensor(11.9586, device='cuda:0')
loss_depth is 9.924271583557129

Question of evaluate result on Waymo dataset

Hello! I wonder know whether you remove the objects that out of the front camera’s field of view in validation set when you evaluate your model on waymo dataset.

The problem with the training process.

Thank you for your help in advance!
When I run the script
python train.py --cfg_file /newnfs/zzwu/08_3d_code/CaDDN-master/tools/cfgs/kitti_models/CaDDN.yaml

I met the error below.

2021-06-03 11:28:38,987 INFO **********************Start training newnfs/zzwu/08_3d_code/CaDDN-master/tools/cfgs/kitti_models/CaDDN(default)********************** epochs: 0%| | 0/80 [00:01<?, ?it/s] Traceback (most recent call last): | 0/3712 [00:00<?, ?it/s] File "train.py", line 201, in <module> main() File "train.py", line 173, in main merge_all_iters_to_one_epoch=args.merge_all_iters_to_one_epoch File "/newnfs/zzwu/08_3d_code/CaDDN-master/tools/train_utils/train_utils.py", line 93, in train_model dataloader_iter=dataloader_iter File "/newnfs/zzwu/08_3d_code/CaDDN-master/tools/train_utils/train_utils.py", line 38, in train_one_epoch loss, tb_dict, disp_dict = model_func(model, batch) File "/newnfs/zzwu/08_3d_code/CaDDN/pcdet/models/__init__.py", line 39, in model_func ret_dict, tb_dict, disp_dict = model(batch_dict) File "/home/CN/zizhang.wu/anaconda3/envs/CaDDN/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__ result = self.forward(*input, **kwargs) File "/newnfs/zzwu/08_3d_code/CaDDN/pcdet/models/detectors/caddn.py", line 11, in forward batch_dict = cur_module(batch_dict) File "/home/CN/zizhang.wu/anaconda3/envs/CaDDN/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__ result = self.forward(*input, **kwargs) File "/newnfs/zzwu/08_3d_code/CaDDN/pcdet/models/backbones_3d/ffe/depth_ffe.py", line 51, in forward ddn_result = self.ddn(images) File "/home/CN/zizhang.wu/anaconda3/envs/CaDDN/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__ result = self.forward(*input, **kwargs) File "/newnfs/zzwu/08_3d_code/CaDDN/pcdet/models/backbones_3d/ffe/ddn/ddn_template.py", line 114, in forward x = self.model.classifier(x) File "/home/CN/zizhang.wu/anaconda3/envs/CaDDN/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__ result = self.forward(*input, **kwargs) File "/home/CN/zizhang.wu/anaconda3/envs/CaDDN/lib/python3.7/site-packages/torch/nn/modules/container.py", line 100, in forward input = module(input) File "/home/CN/zizhang.wu/anaconda3/envs/CaDDN/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__ result = self.forward(*input, **kwargs) File "/home/CN/zizhang.wu/anaconda3/envs/CaDDN/lib/python3.7/site-packages/torchvision/models/segmentation/deeplabv3.py", line 92, in forward res.append(conv(x)) File "/home/CN/zizhang.wu/anaconda3/envs/CaDDN/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__ result = self.forward(*input, **kwargs) File "/home/CN/zizhang.wu/anaconda3/envs/CaDDN/lib/python3.7/site-packages/torchvision/models/segmentation/deeplabv3.py", line 61, in forward x = mod(x) File "/home/CN/zizhang.wu/anaconda3/envs/CaDDN/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__ result = self.forward(*input, **kwargs) File "/home/CN/zizhang.wu/anaconda3/envs/CaDDN/lib/python3.7/site-packages/torch/nn/modules/batchnorm.py", line 107, in forward exponential_average_factor, self.eps) File "/home/CN/zizhang.wu/anaconda3/envs/CaDDN/lib/python3.7/site-packages/torch/nn/functional.py", line 1666, in batch_norm raise ValueError('Expected more than 1 value per channel when training, got input size {}'.format(size)) ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 256, 1, 1])

Question: inference time

Really nice work! Thank you for releasing the code.
I wonder how large is the network in total, and how long does it take for inference?

Training Time Problem

Hello

I would like to ask about a training time issue. I am trying to train this model with 3 RTX 3090s and the batch size is set to be 9. However, it shows that the training time is going to be around 370 hours which is much longer than what you posted.

I would really appreciate it if you could help with it.

Many thanks

How can I transform depth map to real depth ?

I notice that you turn the depth map to real depth by dividing 256.
But when I change the depth to point cloud, I found this value not correct.
Could you tell me why you divide 256 to get the depth?

Question about the transformation from grid coordinates to image coordinates?

Thanks for your help on the previous issue.
The depth map is downsampling 4x, but the intrinsic is the same as the original one.
I think the transformation is below that.

Question about the grid_to_lidar_unproject function

I have read your paper and it is really a great job.

Since I'm a new one in this field, I have a question about the code.

In the below code, what does the ' Calculate grid to LiDAR unprojection for each plane' actually means?
In my opinion, x_size/y_size/z_size means one point in the grid represents x_size meters in the lidar coordinates, is that right?
So what's the relationship between x_size/y_size/z_size and the unproject tensor?

CaDDN/pcdet/models/backbones_3d/f2v/frustum_grid_generator.py

Line 45 in 5a96b37

def grid_to_lidar_unproject(self, pc_min, voxel_size):

Thank you very much. Looking forward to your reply!

Out of memory, can not train on 11G machine

Hi,I set batch_size = 2, and I can not train resnet50 on 2080ti due to out of memory.Do you have any way to reduce the memory usage of the code? I tried to reduce the number of blocks in resnet50,but loss=nan, and error:

WARNING:root:NaN or Inf found in input tensor.

Thank you.

Somethings wrong when generate the kitti data infos

When I do dataset preparation according to GETTING_STARTED.md and run commond python -m pcdet.datasets.kitti.kitti_dataset create_kitti_infos tools/cfgs/dataset_configs/kitti_dataset.yaml, the program crash with info"'EasyDict' object has no attribute 'DEPTH_MAP'". Finally I add DEPTH_MAP from CaDDN.yaml to kitti_dataset.yaml and it works, but I don't konw if it's correct.

How many gpus were used ?

hi,
Thank you for sharing your work.

when training on Kitti benchmark, do you mean by batch size of 4 per GPU?
which equals to batch size of 8 when using two GPUs? Am I getting this right..?

IS there any pretrained model on Waymo ?

HELLO,
I have some new ideas on the CaDDN network, and want to test it on the Waymo val dataset. However, the dataset is too large and my device is limited. So i'd like to know if there is any pretrained model on Waymo to ease the process. Even not very well trained ones would be very helpful.
Thank you.

Run test on pretrained model

i am getting the following error by running this command:

python ./tools/test.py --cfg_file ./tools/cfgs/kitti_models/CaDDN.yaml --launcher pytorch --eval_all

error:

Traceback (most recent call last):
  File "./tools/test.py", line 199, in <module>
    main()
  File "./tools/test.py", line 190, in main
    model = build_network(model_cfg=cfg.MODEL, num_class=len(cfg.CLASS_NAMES), dataset=test_set)
  File "/home/mm/MasterThesis/CaDDN/pcdet/models/__init__.py", line 14, in build_network
    model = build_detector(
  File "/home/mm/MasterThesis/CaDDN/pcdet/models/detectors/__init__.py", line 23, in build_detector
    model = __all__[model_cfg.NAME](
  File "/home/mm/MasterThesis/CaDDN/pcdet/models/detectors/caddn.py", line 7, in __init__
    self.module_list = self.build_networks()
  File "/home/mm/MasterThesis/CaDDN/pcdet/models/detectors/detector3d_template.py", line 50, in build_networks
    module, model_info_dict = getattr(self, 'build_%s' % module_name)(
  File "/home/mm/MasterThesis/CaDDN/pcdet/models/detectors/detector3d_template.py", line 74, in build_ffe
    ffe_module = ffe.__all__[self.model_cfg.FFE.NAME](
  File "/home/mm/MasterThesis/CaDDN/pcdet/models/backbones_3d/ffe/depth_ffe.py", line 25, in __init__
    self.ddn = ddn.__all__[ddn_cfg.NAME](
  File "/home/mm/MasterThesis/CaDDN/pcdet/models/backbones_3d/ffe/ddn/ddn_deeplabv3.py", line 21, in __init__
    super().__init__(constructor=constructor, **kwargs)
  File "/home/mm/MasterThesis/CaDDN/pcdet/models/backbones_3d/ffe/ddn/ddn_template.py", line 34, in __init__
    self.model = self.get_model(constructor=constructor)
  File "/home/mm/MasterThesis/CaDDN/pcdet/models/backbones_3d/ffe/ddn/ddn_template.py", line 63, in get_model
    pretrained_dict = self.filter_pretrained_dict(model_dict=model_dict, pretrained_dict=pretrained_dict)
  File "/home/mm/MasterThesis/CaDDN/pcdet/models/backbones_3d/ffe/ddn/ddn_template.py", line 88, in filter_pretrained_dict
    pretrained_num_classes = pretrained_dict["classifier.4.weight"].shape[0]
KeyError: 'classifier.4.weight'

to be more precise: i downloaded the cadnn.pth from the link in README.md and set the path in the config file.
also pretrained_dict only contains: pretrained_dict : dict_keys(['epoch', 'it', 'model_state', 'optimizer_state', 'version'])

installation is based on what you mentioned in INSTALL.md

About the Coordinate Space

When building the voxel, it is constructed in the point cloud coordinate system. So the output heatmap map is also in the point cloud coordinate system (so during training, the 3D box is converted to the lidar coordinate system as the label).
But the final 3D Box in the inference stage is based on lidar coordinate, so it still needs to be converted to camera coodinate?

Do I understand correctly?