Reproducibility on real dataset

[facebookresearch]

Reproducibility on real dataset

[yxie20]

A note on parsing colmap camera K Rt: I used the the handy script provided by the NeRF research group (https://github.com/Fyusion/LLFF/tree/master/llff/poses). The negation (compared to the NeRF/LLFF convention) of the 2nd and 3rd column of camera extrinsics is accounted for.

Ok. We are using OpenCV style coordinates for extrinsics. Do you mean we need to manually change the code to run your Colmap data?

No, not at all! Everything is accounted for and you only need to copy, paste and run!

[yxie20]

Thank you for such prompt reply! I've uploaded the data here: https://github.com/yxie20/lego_nsvf. To use it, simply copy the following: pose, bbox.txt, intrinsics.txt to the root folder. The original NSVF folder structure is followed here.
I'll look into your comment 2 and 3. Thanks!

Can you also share some comparison with NeRF where you observe the clear difference?
Thanks!

Here's the NeRF training progress for 40 minutes or 5k steps, with colmap poses on Lego:

[kondela]

I have similar experience as @yxie20.

NSVF - default configuration, 62.5k steps

NSVF - enabled trainable_extrinsics and lowered learning rate to 0.0001 as suggested by @MultiPath . 25k steps then OOM

nerf (@kwea123 implementation) - original, validation image, and depth map.

I used same images and poses from COLMAP for all experiments.

[kondela]

Hey @MultiPath, you might be right that poses and images are not aligned properly.

I am using @yxie20 script (https://github.com/yxie20/lego_nsvf/blob/master/poses/pose_utils.py) to generate poses, intrinsics and bounding box as well, and it seems that there's assumption that cameras.bin from COLMAP contains sorted poses based on image file names (which it does not). This would explain why both of us have poor results on real datasets.

In original LLFF they use np.argsort() to account for it (see variable perm in https://github.com/Fyusion/LLFF/blob/c6e27b1ee59cb18f054ccb0f87a90214dbe70482/llff/poses/pose_utils.py#L33). I tried to fix it and ran training using default config, however, it seems that results are still poor.

You can download the new pose.zip here.

I will take a look at it again, there might be another bug that causes improper alignment between images and poses or something similar.

Meanwhile, if you have time it would be awesome if you could provide your own script to convert COLMAP's model files (cameras.bin, images.bin, points3D.bin) to format that NSVF expects, i.e. bbox.txt, intrinsics.txt and poses folder.

[MultiPath]

Hi, thanks for posting this! It is interesting that NSVF will be more sensitive to camera errors compared to the original NeRF. In my view, they should face the same learning issue as the ray-marching part is the same.
Also we did not observe the same phenomena on the real datasets we have tested compared to the original NeRF.

1. Can you please share the three Lego dataset for us to take a look?
2. In our code, we also implement an option to fix the camera pose errors (while this function is sometimes not stable)
   by adding --trainable-extrinsics in the training arguments.
3. Decreasing the learning rate might also be helpful. To speed up learning, we usually used lr=0.001. You can try training with lr=0.0001.

[yxie20]

Thank you for such prompt reply! I've uploaded the data here: https://github.com/yxie20/lego_nsvf. To use it, simply copy the following: pose, bbox.txt, intrinsics.txt to the root folder. The original NSVF folder structure is followed here.

I'll look into your comment 2 and 3. Thanks!

[yxie20]

A note on parsing colmap camera K Rt: I used the the handy script provided by the NeRF research group (https://github.com/Fyusion/LLFF/tree/master/llff/poses). The negation (compared to the NeRF/LLFF convention) of the 2nd and 3rd column of camera extrinsics is accounted for.

[MultiPath]

Thank you for such prompt reply! I've uploaded the data here: https://github.com/yxie20/lego_nsvf. To use it, simply copy the following: pose, bbox.txt, intrinsics.txt to the root folder. The original NSVF folder structure is followed here.

I'll look into your comment 2 and 3. Thanks!

Can you also share some comparison with NeRF where you observe the clear difference?
Thanks!

[MultiPath]

A note on parsing colmap camera K Rt: I used the the handy script provided by the NeRF research group (https://github.com/Fyusion/LLFF/tree/master/llff/poses). The negation (compared to the NeRF/LLFF convention) of the 2nd and 3rd column of camera extrinsics is accounted for.

Ok. We are using OpenCV style coordinates for extrinsics. Do you mean we need to manually change the code to run your Colmap data?

[yxie20]

To help us better pinpoint the problem, I added my pose parsing code to https://github.com/yxie20/lego_nsvf. This code takes in colmap .bin files and output pose/, bbox.txt and intrinsics.txt in NSVF standard. The initial voxel size (side length) is automatically set to 1/8 of the total volume side length.

[yxie20]

Here I provide some results with --trainable-extrinsics enabled. The result is still far from that achieved by ground truth poses.

Here is the loss curve comparison (purple is gt poses, blue is colmap+trainable-extrinsics)

[kwea123]

Telling from the depth map, it seems that it doesn't learn the correct structure either...
Haven't tried NSVF yet, but I have a doubt on your NeRF result, because in my experiments, it works pretty well on real world objects for which the poses are also estimated by colmap.
From your very first picture already, I doubt that the reconstructed poses are not correct at all, because the gt poses are distributed rather uniformly on the upper hemisphere if I remember correctly like in Fig. 1 ; your reconstructed poses seem very irregular. I suspect that it is due to the white background which makes the pose reconstruction difficult. I would suggest that you align the gt poses and the reconstructed poses to see how much they differ, before digging into either NeRF or NSVF.

[yxie20]

Telling from the depth map, it seems that it doesn't learn the correct structure either...
Haven't tried NSVF yet, but I have a doubt on your NeRF result, because in my experiments, it works pretty well on real world objects for which the poses are also estimated by colmap.
From your very first picture already, I doubt that the reconstructed poses are not correct at all, because the gt poses are distributed rather uniformly on the upper hemisphere if I remember correctly like in Fig. 1 ; your reconstructed poses seem very irregular. I suspect that it is due to the white background which makes the pose reconstruction difficult. I would suggest that you align the gt poses and the reconstructed poses to see how much they differ, before digging into either NeRF or NSVF.

Thanks for the comment. If you do decide to run NSVF, I look forward to hearing how it goes! Note that the same colmap poses were used in NeRF and NSVF, that's the key of this thread here.

[MultiPath]

Hi I haven't had time looking into it yet. I will take a look at this issue this weekend

[tau-yihouxiang]

@yxie20 Have you solve this issue? I decided to run colmap on real-world dataset by myself.

[yxie20]

No... Still waiting on the author @MultiPath to provide some insights.

[MultiPath]

Sorry about that. I was busy with other papers recently. will debug this for a bit

[MultiPath]

Hi @kondela can you also share your dataset? if possible

[kondela]

Hey @MultiPath, sure! You can download it from my google drive. I used first 142 images for training and left last 2 for validation.

[MultiPath]

Hi @kondela, just for double check.. Are you sure the pose files and the images are aligned? In my code, I sorted the poses and images separately by string order.

Also, how do you get your boundingbox?

[dedoogong]

I also met this problem. I think, to reproduce the original result with other datasets, I would like to ask author to give the code for that( camera.bin/image.bin to bbox.txt, intrinsic.txt)

Please help us! thank you~!