menandro/vfs

Variational Fisheye Stereo

vfs's Issues

Function time

Hi, thanks again for making this code available.
I have it running on windows, Cuda 10.0, and am testing the test_t265_data function with the provided data.

It runs, and looks good, but the function takes 15 seconds.

When I adjust the :

int nWarpIters = 20;	// Change to reduce processing time
int nSolverIters = 20;	// Change to reduce processing time

It takes 843 milliseconds.

Is this the expected time? Is it possible for this code to run in realtime?

thanks!

Distortion parameters for dataset

Hi Menandro,

Thanks for the great resource. I want to use your fisheye stereo dataset to test one of my algorithms but can't find the distortion parameters of the camera you used.

Do you know where I can find them?

Thanks!

encoding scale of depth groundtruth

Hi,
Thanks for your code and dataset.
I have a question, what is the encoding scale of the depth groundtruth file? In your dataset, the groudtruth is a 16 bit png file, what's value means, Millimetre or something else?
Best regards

Update of dual variable p

In the VFS paper (and in the paper it cites) :

In contrast, in e.g. https://openaccess.thecvf.com/content_iccv_2013/papers/Ferstl_Image_Guided_Depth_2013_ICCV_paper.pdf and https://link.springer.com/chapter/10.1007/978-3-319-46475-6_36 , it's rather

However, in VFS's TGV solver, the way p is updated seems to follow these other papers :

// NOTE: details/checks skipped for clarify.

 // u_x = dxp(u_) - v_(:, : , 1);
u_x = u_[right] - u_pos - v_pos.x;
 ...
 // du_tensor_x = a.*u_x + c.*u_y;
float du_tensor_x = a[pos] * u_x + c[pos] * u_y;
...
// p(:, : , 1) = p(:, : , 1) + alpha1*sigma / eta_p.*du_tensor_x;
  ppos.x = p[pos].x + (alpha1 * sigma / eta_p) * du_tensor_x;
...

Whereas I would have expected something like

// du_x = dxp(u_);
du_x = u_[right] - u_pos;
 ...
// du_tensor_x = a.*u_x + c.*u_y;
float du_tensor_x = a[pos] * du_x + c[pos] * du_y;
...
// p(:, : , 1) = p(:, : , 1) + alpha1*sigma / eta_p.*(du_tensor_x - v_(:, : , 1));
  ppos.x = p[pos].x + (alpha1 * sigma / eta_p) * (du_tensor_x - v_pos.x);
...

Would you be able to comment on that? Am I missing something?
Thanks

Run in real time with live image feed

Hi again,

after I got the VFS-Project to work, I noticed that it takes a rather long time to display a image. Is that normal? (I am using CUDA 10.2 and a Nvidia GTX970)
Since the paper about it stated that it is able to run in realtime, I was wondering how to use it with a live video feed? Because now it seems like that you have to always specify a single image-pair.

Thanks!
Tom

Point cloud from depth?

Hi, and thank you very much for making this code available! I have been fighting with getting accurate depth from a T265 for a week or so.

Is it possible to create a point cloud from the depth maps created by your code?
Would I need to apply undistortion to do this?

Thanks again!

depth scale error

Hi, we use our own t265 camera, calibrated with kb4 camera model. Then, "generateVectorFieldNewton.m" was used to build *.flo file. Finally, we test the program with binocular fish eye image, but the scale of the depth image genearted by "copyStereoToHost()" is error.

calibrate fisheye and Lidar

Hi!
Thanks for your excellent work. I am running your algorithm, but I do not know how to calibrate fisheye and Lidar.
could you please tell me how do you calibrate and which algorithm you use to finish calibration?

Many thanks! :)

pointcloud with color？

Hi menandro, Thank you for sharing the great work! I have tried to generate pointcloud with color, but the result isn't like the test.ply, which was provided by you. could you give me some suggestions?

Run return test_t265_data() Error

Your work is amazing! I try to run the "testvfs" demo, but failed. Error Message : CUDA error at /home/super/Vision/vfs/testVFS/downscale.cu:92 code=18(cudaErrorInvalidTexture) "cudaBindTexture2D(0, texFine, src, width, height, stride * sizeof(float))". Have you encountered this problem? If not, do you have any ideas about it? thank you!

Real time performance on jetson nx

Hello,what a great work you have done,thanks for sharing. But I see you have done real world test on gtx 1060 runing 30 fps. So i wander the real time performance ,can VFS be used on jetson embeded systems like jetson nx? Thanks for advice.

down load raw data

Hi I downloaded your data icra_dataset but I found that the image2 and proj depth images are rectified, not the raw image. Where can I download the raw image and groundtruth like your Sample Results ? Thank you

About calibration and trajectory filed

Hi @menandro,

Thank you for sharing the great work!
I would like to know how to generate the calibration and trajectory field.
Could you share the code about it?

Thanks.
Yuki

Adapting to a different camera model

Hi, thanks for making this repo available!
I've been trying to get this to work for a different camera model (cv.omnidir in OpenCV contrib), so I've spent quite a bit of time trying to understand what's going on overall and generating the vector fields. I'm nearly there, but I have a few questions that are still puzzling me. If you can help clear that up a bit, that would be very appreciated.

1. In https://gist.github.com/menandro/b829667f616e72aded373479aca61770, when generating vectorxforward and vectorxbakward, a mix of K0 and D1 is used and that confuses me. I don't understand the paper 100%, but I would have thought the second camera's distortion was already included in the calibration field. Note however, that using any combination of D0,D1,K0,K1 has relatively little impact on the disparity results (aka I've got bigger problems :P)

thetad = theta.*(1 + D1(1)*theta.^2 + D1(2)*theta.^4 + D1(3)*theta.^6 + D1(4)*theta.^8);
...
u1 = K0(1,1) * xprime1 + K0(1,3); % Should be K0 because surface is already corrected

2. In https://gist.github.com/menandro/cd5f4b5309f16f1a0f1987fcb2baf057, the translateScale is not the same in the forward and backward direction. I don't understand the logic behind that. In my case, I used the same scale in forward/backward direction, but using either a scale of 1 or 0.01 also has little impact on disparity maps.

%% Forward direction
translateScale = 1.0;
...
%% Backward direction
translateScale = 0.01;

3. Just to make sure, does the disparity relate to the raw (i.e. distorted) images? (e.g. a given point in the first image at position (u,v) will be at (u+disp.x, v+disp.y) in the second image?). Asking because for the equidistant dataset, the disparities I get are coherent with that. However, both for the t265 "robot" image (Kannala-Brandt model) and for my own data (Mei model), the disparity maps look pretty good in general, but the values themselves seem off. E.g. for a large patch of an image where I expect a x disparity of ~ -40, the results are smooth and coherent, but the disparity is ~ -20. I've tried quite a few variations of parameters (nLevel, fScale, nWarpIters, nSolverIters, lambda, limitRange) and this remains.

4. In https://github.com/menandro/vfs/blob/master/stereotgv/conversion.cu, I don't quite understand how the triangulation is done, but it feels like

	float xprime0 = (u0 - focalx) / cx;
	float yprime0 = (v0 - focaly) / cy;
...
	float xprime1 = (u1 - focalx) / cx;
	float yprime1 = (v1 - focaly) / cy;

Should rather be xprime0 = (u0 - cx) / focalx and so on. Is that correct?

Calibrate custom camera

Hi there,

I really like the VFS-Project and would like to use it with a custom camera and a 180 degree fisheye lens.
How would I go about this? What would I have to do in regards to calibration and setup?

Thanks,
Tom