Training on wild videos

[nianticlabs]

Training on wild videos

[mdfirman]

Thanks for the question! @mrharicot is probably best placed to answer this.

[mrharicot]

Hi,
We were originally doing experiments with COLMAP, which gave us a focal length for some of the sequences. I then used an average of these and set the principal point to the center of the image.
For reference here is the intrinsics we used:

self.K = np.array([[0.6,     0,     0.5,       0],
                   [0,   1.066,     0.5,       0],
                   [0,       0,       1,       0],
                   [0,       0,       0,       1]], dtype=np.float32)

We assumed the camera to be linear, but they actually show quite a bit of distortion.
I hope this helps!

[WANG-KX]

Thanks for your kind replay!

Actually, I have tested some method to find the intrinsic parameters of a wild monocular video. Basically, I followed a bundle adjustment method to solve the parameters by optimizing the reprojection of feature points. However, it can be proved that when the camera is moving without rotation (which is a common case in these videos) the focal length is not observable. With small rotations, it also cannot be solved very well.

On the other hand, since the rotation of adjacent frames is small, the focal length actually does not matter too much in the error term.

I do not know how to better solve the intrinsic parameters when trained on wild videos. If you have any other thoughts, please let me know.

Thanks again
Kaixuan

[mrharicot]

I would look at Google's recent paper where they also learn the camera calibration parameters
https://arxiv.org/abs/1904.04998

[WANG-KX]

Yes... Thanks for the sharing.
If we can solve it using multiview geometry, maybe it is better to solve it explicitly than learning. Thanks, maybe the community needs more research on this topic.

[hemangchawla]

Can the calibration parameters be set somehow during inference? For example, if the correct params for the sequence are computed using Colmap and then set during inference to improve the estimation?