Why the profile results are different with yours?

[brummi]

Why the profile results are different with yours?

[Brummi]

Hi!
thank you for being interested in our work!
The figures in the paper were created applying the model trained on KITTI-360 on the KITTI Raw data. Your results seem to come from the KITTI Raw model.
The model trained on KITTI-360 is better at predicting geometry in occluded areas because KITTI-360 has additional fisheye cameras.

Which pretrained model did you use to obtain those results?

Best,
Felix

[zhanghm1995]

Thanks for your quick response!

The test data I used is the data/KITTI-Raw/2011_09_26/2011_09_26_drive_0005_sync/image_02/data/0000000002.png, and the model I used is out/kitti_raw/pretrained/training-checkpoint.pt.

I just tried to use the KITTI-360 model to apply the same image, and obtained the results like this:

They seem much better, but still have some differences.

[Brummi]

Hi!
It looks like the profile you generated covers a much bigger area. You can try to reduce the size to obtain the same results as in the paper.
Best,
Felix

[zhanghm1995]

I see, I just used the default settings of your codes that generating 256x256 profile images with distances between 3 and 21. It seems the profiles in your paper are closer.

BTW, I am curious about the depth processing details. When I try to convert the depth map to point cloud, I found that the converted point cloud is so strange, especially for the upper region (maybe the sky region in the image). I found the reasons may be that you predict the depth for every pixel in the images with the depth constraints that z_near=3 and z_far=80.

Thus, the depth of the sky regions is not correct.

After reading the codes more carefully, I found that you set hard_alpha_cap=1, and that you set the farthest alphas = 1:

if self.hard_alpha_cap:
    alphas[:, -1] = 1

It seems that this part would have effects on the training process, but I am not very sure.

So could you help me understand this phenomenon and are there any solutions to filter the depth for the sky regions?

[Brummi]

Setting this hard alpha cap makes training more stable. Otherwise, the network could predict zero density along the entire ray (maybe because this pixel is occluded in all other images -> no matching color -> reduce color on all samples along the ray).
This would cause numerical problems and is prevented by this measure.

When generating pointclouds: The depth maps are only a 2D visualization of the predicted density field. I would advice so sample random points from the frustum and then keeping / discarding them based on the predicted density at that point. This allows you to also obtain a meaningful pointcloud in occluded areas.

Best,
Felix