Great Work!!!!!! Few Queries

[apple]

Great Work!!!!!! Few Queries

[Xiaoming-Zhao]

-1.0400704145431519 is quite close to -1, which is the boundary for minimum value. It may be due to precision error on different machines. To resolve this, you can increase the value of spatial_enlarge_factor a little bit, e.g., to 1.01 or 1.1:

ml-gmpi/configs/gmpi.yml

Line 92 in 6b538e8

spatial_enlarge_factor: 1.001

[VIROBO-15]

I have made spatial_enlarge_factor : 1.1 but still I am getting this error

#Epoch - 0
0%|▏ | 92/69987 [00:42<7:16:10, 2.67it/s]
pos: tensor([[-0.6541, -0.0062, 0.2436]], device='cuda:0')

dir: tensor([[ 0.5646, -0.1026, 0.8190]], device='cuda:0')

u: tensor([-0.1839, -0.1839, -0.1839, -0.1839], device='cuda:0')

v: tensor([-0.5088, -0.5048, -0.5008, -0.4968], device='cuda:0')

yaws: [[0.7130070924758911]]

pitches: [[-0.004669070243835449]]

<class 'AssertionError'>Traceback (most recent call last):

File "/proj/cvl/users/x_fahkh/mn/3D-generation/exps/ddpm-improved-concat-poses-conditioning/models/mpi.py", line 107, in homography
assert torch.max(u) <= 1, f"Ray's U direction goes out of plane at {distance}, max val {torch.max(u)}"
AssertionError: Ray's U direction goes out of plane at tensor([[1.1200]], device='cuda:0'), max val 1.0676274299621582

[Xiaoming-Zhao]

I see. I think I get it:

pass the same pose as given by the dataloader([flat_w2c_mats_real] line 419 train.py ) to self.mpi() function

This will cause issues. Due to MPI's intrinsic properties, it cannot be rendered for 360 degrees. Therefore, we have truncated Gaussian:

ml-gmpi/gmpi/utils/cam_utils.py

Line 522 in 566e9c5

elif sample_method == "truncated_gaussian":

The number of standard deviations is set here:

ml-gmpi/configs/gmpi.yml

Line 94 in 6b538e8

cam_pose_n_truncated_stds: 2

If you use all estimated poses from real images, some of the poses will go out of the truncated range, which will cause problems.

BTW, truncated Gaussian is common in 3D generative models. For example, pi-GAN.

[VIROBO-15]

Got it !!!!!!!
Is there any way out to use all the images by setting some hyperparamteres?

[Xiaoming-Zhao]

If you really want to use all real images, you can:

1. compute the real imges's pose range, i.e., minimum and maximum value of horizontal and vertical angle. Assume they are pose_min and pose_max;
2. Set the h_stddev and v_stddev to the largest deviation from the pose's mean. I.e.,
   max(abs(pose_min - pose_mean), abs(pose_max - pose_mean))
3. Assign those h_stddev and v_stddev in the curriculum.py:
   

   ml-gmpi/gmpi/curriculums.py

   Lines 113 to 114 in 6b538e8

   	'h_stddev': 0.289,
   	'v_stddev': 0.127,

4. Set cam_pose_n_truncated_stds to one or some values slightly larger than one to avoid precision issue, e.g., 1.05:
   

   ml-gmpi/configs/gmpi.yml

   Line 94 in 6b538e8

   cam_pose_n_truncated_stds: 2

Essentially, you want the truncated Gaussian to cover all real images's poses. Hope this helps.

[VIROBO-15]

After Calculating h_stddev and v_stddev for real image it comes out to be

'h_stddev': 1.4915, 
 'v_stddev': 0.9975,

and also after settinh

spatial_enlarge_factor: 1.1
cam_pose_n_truncated_stds: 1.05

But I am still get this

pos: tensor([[-0.9283, -0.3449, 0.8610]], device='cuda:0')

dir: tensor([[0.9383, 0.2390, 0.2504]], device='cuda:0')

u: tensor([0.0191, 0.0190, 0.0189, 0.0189], device='cuda:0')

v: tensor([-0.0484, -0.0480, -0.0476, -0.0471], device='cuda:0')

yaws: [[1.4311634302139282]]

pitches: [[-0.05110371112823486]]

Traceback (most recent call last):
File "/proj/cvl/users/x_fahkh/mn/debug-dmpi/gmpi/core/mpi.py", line 107, in homography
assert torch.max(u) <= 1, f"Ray's U direction goes out of plane at {distance}, max val {torch.max(u)}"
AssertionError: Ray's U direction goes out of plane at tensor([[1.1200]], device='cuda:0'), max val 3.882153272628784

[Xiaoming-Zhao]

The problem comes from that the h_stddev and v_stddev are too large, which may not be suitable for MPI representation.

MPI (as in the name) is a set of planes. It cannot handle extremely wide range of camera poses. If you have horizontal angle of 1.4915, the rays from camera are almost parallel to the MPI plane. This is too extreme for such a representation.

I recommend filtering out real image whose camera poses are too extreme. You may need to tune the h_stddev and v_stddev a little bit.