tarashakhurana/4d-occ-forecasting

CVPR 2023: Official code for `Point Cloud Forecasting as a Proxy for 4D Occupancy Forecasting'

Point Cloud Forecasting as a Proxy for 4D Occupancy Forecasting

By Tarasha Khurana*, Peiyun Hu*, David Held, and Deva Ramanan

* equal contribution

project page | 5-min summary

Citing us

If you find our work useful, please consider citing:

@inproceedings{khurana2023point,
  title={Point Cloud Forecasting as a Proxy for 4D Occupancy Forecasting},
  author={Khurana, Tarasha and Hu, Peiyun and Held, David and Ramanan, Deva},
  booktitle={IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2023},
}

Setup

• Download nuScenes, KITTI-Odometry and ArgoVerse2.0 (code supports the LiDAR dataset, but the change to Sensor dataset is minor). (Tip: See the python scripts to see how to send the file paths.)
• Create a conda environment with the given environment.yml. Additionally, install the chamferdist package given inside utils/chamferdist by navigating to that directory and doing pip install ..
• All trained model checkpoints for all three datasets for both 1s and 3s forecasting are available in the models/ folder.
• The given code has been tested with python3.8, CUDA-11.1.1, CuDNN-v8.0.4.30, GCC-5.5 and NVIDIA GeForce RTX 3090.

🆕 Differentiable Voxel Rendering implemented as a layer in PyTorch

The original code supported training with only L1, L2, or AbsRel losses. For increased flexibility in the choice of loss, we have added differentiable voxel rendering as a layer in PyTorch. Note that we do not use it ourselves, because it incurs a huge memory footprint (as gradients for the entire voxel grid are now retained in memory).

You can import the layer with:

from utils.layers.differentiable_voxel_rendering import DifferentiableVoxelRendering

This layer is expected to be used (without any initialization) in model.py in place of dvr.render and dvr.render_forward like below:

pred_dist, gt_dist = DifferentiableVoxelRendering(
                        sigma,
                        output_origin,
                        output_points,
                        output_tindex
                     )

CVPR '23 Argoverse challenge evalkit released!

If participating in the CVPR '23 Argoverse2.0 4D Occupancy Forecasting challenge, please see the eval-kit.

Training

Refer to train.sh.

Testing

Refer to test.sh for executing the ray-based evaluation on all points, and test_fgbg.sh for evaluation separately on foreground and background points (only supported for nuScenes).

Ray tracing baseline

The ray tracing baseline is implemented and evaluated by raytracing_baseline.sh and raytracing_baseline_fgbg.sh.

Cross-sensor generalization

In order to test a model trained on X on a dataset other than X, change the dataset field in the respective model's config.

Acknowledgments

The chamferdist package shipped with this codebase is a version of this package. Voxel rendering is an adaptation of the raycasting in our previous work.

4d-occ-forecasting's Issues

About the input occupancy

Hi, thanks for sharing the code

A quick question:

What is the exactly used method to transform the input point clouds to occupancy voxels as the network input?

From the implementation, if a voxel has at least one point inside of it, then the voxel is marked as 1. So no ray-casting is used here. However, the comment here

4d-occ-forecasting/model.py

Line 240 in ff98608

# -1: freespace, 0: unknown, 1: occupied

says the ray-casting is used and the space status is triple instead of binary.

Having trouble reproducing paper results with provided checkpoints

Hi there,

I downloaded the NuScenes checkpoints (for both 1s and 3s prediction) and ran the evaluation code for the checkpoints. I am having trouble reproducing the reported results in the paper from the checkpoints. More specifically, the reported / reproduced results are:

• 1s prediction:

  • L1: 1.40 / 1.463
  • AbsRel: 10.37 / 10.97
  • Chamfer Distance Near-Field: 1.41 / 1.4859
  • Chamfer Distance Vanilla: 2.81 / 2.9902

• 3s prediction:

  • L1: 1.71 / 1.903
  • AbsRel: 13.48 / 15.54
  • Chamfer Distance Near-Field: 1.40 / 1.7462
  • Chamfer Distance Vanilla: 4.31 / 5.2942

Thanks for your help!

2024 wad challenge submission.

Thank you for your great work!

I want to participate in the CVPR'24 4D occupancy forecasting challenge.

If I want to save the output of a pretrained 4D-OCC-forecasting baseline model in the format specified as /path/to/submission, what code should I use? If there isn't any, could you provide some reference code that I could refer to?

Script to compute the metrics in the CVPR '23 paper and the leaderboard for the 4D occupancy forecasting challenge. Usage: python evaluate.py --annotations /path/to/annotations --submission /path/to/submission.

Another question, I can't acess the submission guidance url(https://eval.ai/web/challenges/challenge-page/1977/submission). Error message: Page Not Found 404.

Question about 2024 wad challenge

Hi, I have a question.

Currently, the provided dataloader does not load intensity data. Is it okay to load it and use it as an input for the model?

loss drops to 0

Hello, I noticed a strange thing, when training stationary scenes (such as waiting for a traffic light), the loss will quickly drop to 0 and lose vitality

Can we extend when input resolution(voxel_size) is different in xyz dimensions?

I see both gt_dist and pred_dist are multiplied with voxel_size after, what if voxel size is different on xyz dimensions?

visualization for 4D occupany

Hi,
Thanks for the great work. Would you please release the code for visualizing the 4D occupancy as well?

Evalai Leaderboard doesn't run the submission file.

Hello,

I have submitted the test set result to Argoverse 2: 4D Occupancy Forecasting Challenge leaderboard last week. But the submission has not been evaluated yet and keep submitted status for more than 3 days.

Is there any issue on the evalai server?

Regards

Visualization code issue

I'd like to ask how to write visualization code shown in the paper. How the 4D occupancy is visualized. Thankful if you may share the code or you give instructions on how to write the code.

Loss function - How can I use L1 loss train the network?

Hello @tarashakhurana ,

I want to use L1 loss to train the network as your mentioned in your paper. But in this repo, L1 loss is not used.

Could you provide the code for using L1 loss for training?

Question about occupancy probablity

Hi @tarashakhurana,

In model.py the occupancy probablity is calculated in line pog = 1 - torch.exp(-sigma), what's the reason behind this function 1-exp(-sigma)? And I found in dvr.cu, the way to get occupancy probablity is p[count] = 1 - exp(-sd), where sd = _sigma * _delta, why there is a * _delta involved?

Label of lidar points

Hi, congrats for this great work. I have a minor question here.
I noticed that in nuScenes, lidar points in class [1--23] are set to 1 (occupied), and the rest are 0 (empty).

4d-occ-forecasting/data/nusc.py

Line 133 in 74549e2

fg_labels = np.logical_and(1 <= lidarseg_labels, lidarseg_labels <= 23)

While the class [24--31] are:

24: 'flat.driveable_surface'
25: 'flat.other'
26: 'flat.sidewalk'
27: 'flat.terrain'
28: 'static.manmade'
29: 'static.other'
30: 'static.vegetation'
31: 'vehicle.ego'

Shouldn't points indicating these classes also be set to 1 (occupied) since the rays end here?

About the ray direction

Hi, thanks for your great work. I noticed that in dvr.cu , only setp>0 will be considered as the next voxel border. Will this be okay since in the Lidar axis, there are 360 directions which also contains the negative directions of rays? Looking forward to your help.

        // Distance along the ray to the next voxel border from the current position (tMaxX, tMaxY, tMaxZ).
        const double next_voxel_boundary_x = vx + (stepX < 0 ? 0 : 1);
        const double next_voxel_boundary_y = vy + (stepY < 0 ? 0 : 1);
        const double next_voxel_boundary_z = vz + (stepZ < 0 ? 0 : 1);

CUDA error: invalid configuration argument

Hi, thanks for your great work.
I tried using another model to predict 4d occupancy and use dvr.cu for rendering. But after training a few iterations, the following problem occurs, can you provide some suggestions to help solve this bug.

2023-05-30 09:50:43,139 - mmdet - INFO - Epoch [1][6/27]        lr: 1.700e-04, eta: 0:07:41, time: 0.924, data_time: 0.030, memory: 7264, loss_heatmap: 8808.0361, layer_-1_loss_cls: 44.7087, layer_-1_loss_bbox: 11.4984, matched_ious: 0.0000, loss_l1: 15.6480, loss_l2: 294.7178, loss_absrel: 0.8354, loss: 9175.4443, grad_norm: 34641.0859
2023-05-30 09:50:44,053 - mmdet - INFO - Epoch [1][7/27]        lr: 1.707e-04, eta: 0:06:52, time: 0.910, data_time: 0.062, memory: 7264, loss_heatmap: 6989.6382, layer_-1_loss_cls: 42.0245, layer_-1_loss_bbox: 8.0230, matched_ious: 0.0054, loss_l1: 13.2696, loss_l2: 222.9359, loss_absrel: 0.7619, loss: 7276.6528, grad_norm: 31704.3477
Traceback (most recent call last):
  File "./tools/train.py", line 269, in <module>
    main()
  File "./tools/train.py", line 258, in main
    train_model(
  File "/home/mmdet3d/apis/train.py", line 344, in train_model
    train_detector(
  File "/home/mmdet3d/apis/train.py", line 319, in train_detector
    runner.run(data_loaders, cfg.workflow)
  File "/home/anaconda3/envs/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 136, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/home/anaconda3/envs/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 53, in train
    self.run_iter(data_batch, train_mode=True, **kwargs)
  File "/home/anaconda3/envs/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 31, in run_iter
    outputs = self.model.train_step(data_batch, self.optimizer,
  File "/home/anaconda3/envs/lib/python3.8/site-packages/mmcv/parallel/distributed.py", line 63, in train_step
    output = self.module.train_step(*inputs[0], **kwargs[0])
  File "/home/anaconda3/envs/lib/python3.8/site-packages/mmdet/models/detectors/base.py", line 248, in train_step
    losses = self(**data)
  File "/home/anaconda3/envs/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/mmdet3d/models/detectors/***.py", line 74, in forward
    return self.forward_train(**kwargs)
  File "/home/mmdet3d/models/detectors/***.py", line 250, in forward_train
    losses_occ = self.forward_train(
  File "/home/mmdet3d/models/dense_heads/***.py", line 400, in forward_train
    invalid = torch.isnan(grad_sigma)
RuntimeError: CUDA error: invalid configuration argument
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 2861194) of binary: /home/anaconda3/envs/bin/python

CVPR Eval Kit Challenge Bug

Hello,

From what I understand about the challenge, the queries are meant to be at these 5 timesteps in the future:
{0.6 s, 1.2 s, 1.8 s, 2.4 s, 3.0 s}
However the current dataloader implemented in the CVPR eval kit loads queries at the following timesteps:
{1.2 s, 2.4 s, 3.6 s, 4.8s , 6.0s},
which I think is incorrect.

See cvpr23-evalkit/data/av2.py on line 72, where the stride is self.input_step + self.output_step = 12. This is implemented correctly in the other dataloader data/av2.py on line 78, where the stride is just self.input_step.

Thanks for any clarification and help.

Training & inference time

Thanks for the exciting work! Could you let me know the approximate training and inference time? How much memory does it take for training and on what device? Thanks in advance!

Chamferdis module reported an error

Is there any exception in the provided chamferdis module version 1.0.0? The error is pointing to this module

The following is the error message
“Traceback (most recent call last):
File "test.py", line 327, in
test(args)
File "test.py", line 283, in test
metrics["chamfer_distance"] += compute_chamfer_distance(pred_pcd, gt_pcd, device)
File "/home/gdyanfa1/tmh/tmh_lidar_occ/4d-occ-forecasting-main/utils/evaluation.py", line 207, in compute_chamfer_distance
cd_forward, cd_backward, CD_info = chamfer_distance(
File "/home/gdyanfa1/miniconda3/envs/tmh_lidar_occ/lib/python3.8/site-packages/torch/_tensor.py", line 607, in iter
raise TypeError('iteration over a 0-d tensor')
TypeError: iteration over a 0-d tensor”