andyzeng/3dmatch-toolbox

3DMatch - a 3D ConvNet-based local geometric descriptor for aligning 3D meshes and point clouds.

3dmatch-toolbox's Issues

Point clouds and transformation matrices

Hi
Thanks for the toolbox
I am new in Ubuntu and I just want to compare my method with the 3Dmatch, actually I need point clouds and transformation matrices of this dataset for training.
is it possible that I have point clouds and transformation matrices directly?

Thanks in advance

When I do geometric registration test, I download the evaluation files? There are some gt files including transform matrix of two pairs?How can I compute these matrices?

Question about the keypoint.txt

I see that the keypoints in the dataset are obtained randomly and stored in the keypoint.txt file as an index. How do these indexes get?

Question about the calculation process of gt.info in getGtInfoLog.m

Hi, Andy Zeng:

In the calculation process of info matrix given by here, the q_star is the ground truth correspondence point in original Q, while i find that the Q is transformed Q in you code here. So, I am confused about that. I will appreciate if you can give me some hints.

3dmatch-toolbox/evaluation/geometric-registration/getGtInfoLog.m

Lines 50 to 67 in f6496fd

	relExt = inv(fragment1ExtCam2World) * fragment2ExtCam2World;
	fragment2Points = fragment2PointCloud.Location';
	fragment2Points = relExt(1:3,1:3) * fragment2Points + repmat(relExt(1:3,4),1,size(fragment2Points,2));
	% pcwrite(pointCloud(fragment2Points','Color',repmat(uint8([255,0,0]),size(fragment2Points,2),1)),'debug1','PLYformat','binary');
	% Find overlapping surfaces
	voxelGridSize = 0.006;
	[nnIdx,nnSqrDist] = multiQueryKNNSearchImpl(fragment1PointCloud,fragment2Points',1);
	nnDist = sqrt(nnSqrDist);
	% Check if overlapping surface is over 30% of first fragment
	alignedRatio = sum(nnDist < voxelGridSize*5)/size(fragment1PointCloud.Location,1);
	if alignedRatio < 0.3
	continue;
	end
	% Compute 5000 ground truth correspondences q* on the overlapping regions
	corresQ = fragment2Points(:,find(nnDist < voxelGridSize))';

TSDF voxel grid values

Hi,

Just for compute TDF voxel grid values given a point cloud Is it important to use Marvin ?
Is it possible to use it without Marvin (not working with windows).

Thank you in advance.

Converting 3D Data to TDF Voxel Grids

Bravo, for your good work.

I have a question. I can use just the algorithm "Converting 3D Data to TDF Voxel Grids" with Windows ?

Thank you,
sincerely.

Segmentation fault (core dumped)

when I run the ./demo command, this happened..
mo/single-depth-1.ply fragment-1
Loaded point cloud with 252179 points!
Size of TDF voxel grid: 370 x 253 x 324
Computing TDF voxel grid...
Finding random surface keypoints...
PlaceHolderDataLayer dataTest

Layers: Responses:

dataTest
data[5]={50,1,30,30,30} RF[1,1,1] GP[1,1,1] OF[0,0,0]
conv1 weight[5]={64,1,3,3,3} bias[5]={1,64,1,1,1}
conv1[5]={50,64,28,28,28} RF[3,3,3] GP[1,1,1] OF[0,0,0]
relu1
conv2 weight[5]={64,64,3,3,3} bias[5]={1,64,1,1,1}
conv2[5]={50,64,26,26,26} RF[5,5,5] GP[1,1,1] OF[0,0,0]
relu2
pool1
pool1[5]={50,64,13,13,13} RF[6,6,6] GP[2,2,2] OF[0,0,0]
conv3 weight[5]={128,64,3,3,3} bias[5]={1,128,1,1,1}
conv3[5]={50,128,11,11,11} RF[10,10,10] GP[2,2,2] OF[0,0,0]
relu3
conv4 weight[5]={128,128,3,3,3} bias[5]={1,128,1,1,1}
conv4[5]={50,128,9,9,9} RF[14,14,14] GP[2,2,2] OF[0,0,0]
relu4
conv5 weight[5]={256,128,3,3,3} bias[5]={1,256,1,1,1}
conv5[5]={50,256,7,7,7} RF[18,18,18] GP[2,2,2] OF[0,0,0]
relu5
conv6 weight[5]={256,256,3,3,3} bias[5]={1,256,1,1,1}
conv6[5]={50,256,5,5,5} RF[22,22,22] GP[2,2,2] OF[0,0,0]
relu6
conv7 weight[5]={512,256,3,3,3} bias[5]={1,512,1,1,1}
conv7[5]={50,512,3,3,3} RF[26,26,26] GP[2,2,2] OF[0,0,0]
relu7
conv8 weight[5]={512,512,3,3,3} bias[5]={1,512,1,1,1}
conv8[5]={50,512,1,1,1} RF[30,30,30] GP[2,2,2] OF[0,0,0]
relu8
feat weight[5]={512,512,1,1,1} bias[5]={1,512,1,1,1}
feat[5]={50,512,1,1,1} RF[30,30,30] GP[2,2,2] OF[0,0,0]

GPU 0: Total GPU memory: 322.529 MB

6.75 KB
3.375 KB
256 Bytes
128 Bytes
432 KB
216 KB
256 Bytes
128 Bytes
0.84375 MB
432 KB
0.5 KB
256 Bytes
1.6875 MB
0.84375 MB
Segmentation fault (core dumped)

Can this training code directly take in .ply file?

Can this training code directly take in .ply file? Instead of rgb-d file?

Num of geometric registration fragments

Hi, Thanks for your sharing. In your project website you mentioned that the ply file for geometric registration fragments Each fragment is a 3D point cloud of a surface, integrated from 50 depth frames using TSDF volumetric fusion, and saved to a .ply file. But I found the number of ply file does not equal to the rgbd images / 50. For example there are 10204 color images and depth images in sun3d-hotel_uc-scan3 dataset, but only 54 ply files is provided in the geometric registration benchmark. So do you just use part of rgbd image to generate ply file or maybe I miss some information?

Looking forward to your reply. Thanks a lot.

Intermediate files for real data benchmark has 'nan' elements.

Hello, thank you for the great work.

I found the issue on intermediate files for real data benchmark on Geometric Registration on Synthetic Data
( intermediate-files-real.zip )

The data has 'nan' elements.
For example,
fid = fopen('cloud_bin_0.desc.3dmatch.bin', 'rb')
data = fread(fid, 'single')
num_points = data(1)
num_feats = data(2)
feat = data(3:end)

any(isnan(feat))
-> ans = 1
sum(isnan(feat))
-> ans = 512

Thanks.

How to use 3dmatch code in windows?

How to get the fusion cloud and visualize it?

I use the point cloud as input and try TDF (https://github.com/andyzeng/3dmatch-toolbox/blob/master/core/demo.cu). I got the tdf.bin file but how to extract the point in Volume and visualize it.......thanks a lot....

where is makeCorresDataset

Hi, I do not find “makeCorresDataset” in 3dmatch-toolbox？
In addition， I want to use mxnet to train my net with TDF. But I don't known how to input TDF to my net？
thanks！

how to read .mat files generated from the 'makeCorresDataset.m' matlab script in python

I want to read data generated from makeCorresDataset.m matlab script using python

Some problems of dependencies

Hi, Andy. Thanks for your code.
I don't know whether your code could run in my platform which is CUDA 8.0, cuDNN 5.1.5 and OpenCV 3.2. If don't, how can I make it run properly by modifying it on this platform.
I do the Demo: Align Two Point Clouds with 3DMatch experiment, following I run ./compile.sh. Then, the errors occurred:

hoo104@amax-1080:~/3dmatch-toolbox/core$ ./compile.sh
nvcc warning : The 'compute_20', 'sm_20', and 'sm_21' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
utils.hpp(15): error: namespace "std" has no member "random_device"

utils.hpp(16): error: namespace "std" has no member "mt19937"

utils.hpp(17): error: namespace "std" has no member "uniform_real_distribution"

utils.hpp(17): error: type name is not allowed

utils.hpp(17): error: identifier "dist" is undefined

5 errors detected in the compilation of "/tmp/tmpxft_00001e7b_00000000-9_demo.cpp1.ii".

Geometric Registration Evaluation code too slow

Hi, When I try to use the code for Geometric Registration Evaluation, it seems the process is so slow, my ransacRt takes about 25 seconds or more for max trials equals 50000, and in Compute alignment percentage part it takes about 20 seconds each multiQueryKNNSearchImp. So the total time for one pair of registration will cost 1 minute or more and I have to wait many hours until all the pairs have been done. So I wonder is there any way to make the process faster? Or is there something wrong with my usage?

Thanks for the attention!

A question about how to compute *.gt (Ground Truth)?

Hi,I want to generate my own test set, but I don't know how to get the Ground Truth. Can I use Pose to calculate it?Or, can you provide the snippets from which you generated the test set ( color image and depth image )?

memory leak

Hello,how to deal with the memory leak during training?

How to transform TDF into *.off

I want to show TDF, README tall me that *.off file can be show.But How to Transform TDF into *.off
Thanks!

About the pose in data-set

Hi,
Recently i am using your frame to train with my own data-set, but i am confused about the pose in the data-set, do i need to get the pose in advance or if there are any other ways that can get the pose automatically in your frame?

Problem with Matlab script demo.m

Hi,
I tried to run the demo instructions. When I run the matlab script, it showed the error:
"Undefined function 'pcread' for input arguments of type 'char'.
Error in demo (line 16)
fragment1PointCloud = pcread(fragment1PointCloudFile); "
I am new to Matlab and using Matlab 2014a. The problem here is that function "pcread" cannot read .ply file, right?
Thank you very much

errors when run compile.sh

Hi
Many thanks for sharing this tool
I try to run compile.sh but I get 12 errors as below. Could you help me in solving these errors please.

/usr/include/c++/5/functional(78): error: class "std::thread" has no member "result_type"
detected during:
instantiation of class "std::_Maybe_get_result_type<_Functor, void> [with _Functor=std::thread]"
(86): here
instantiation of class "std::_Weak_result_type_impl<_Functor> [with _Functor=std::thread]"
(184): here
instantiation of class "std::_Weak_result_type<_Functor> [with _Functor=std::thread]"
(264): here
instantiation of class "std::_Reference_wrapper_base_impl<true, true, _Tp> [with _Tp=std::thread]"
(283): here
instantiation of class "std::_Reference_wrapper_base<_Tp> [with _Tp=std::thread]"
(399): here
instantiation of class "std::reference_wrapper<_Tp> [with _Tp=std::thread]"
/usr/include/c++/5/future(1638): here

/usr/include/c++/5/functional(266): error: class "std::thread" has no member "argument_type"
detected during:
instantiation of class "std::_Reference_wrapper_base_impl<true, true, _Tp> [with _Tp=std::thread]"
(283): here
instantiation of class "std::_Reference_wrapper_base<_Tp> [with _Tp=std::thread]"
(399): here
instantiation of class "std::reference_wrapper<_Tp> [with _Tp=std::thread]"
/usr/include/c++/5/future(1638): here

/usr/include/c++/5/functional(267): error: class "std::thread" has no member "first_argument_type"
detected during:
instantiation of class "std::_Reference_wrapper_base_impl<true, true, _Tp> [with _Tp=std::thread]"
(283): here
instantiation of class "std::_Reference_wrapper_base<_Tp> [with _Tp=std::thread]"
(399): here
instantiation of class "std::reference_wrapper<_Tp> [with _Tp=std::thread]"
/usr/include/c++/5/future(1638): here

/usr/include/c++/5/functional(268): error: class "std::thread" has no member "second_argument_type"
detected during:
instantiation of class "std::_Reference_wrapper_base_impl<true, true, _Tp> [with _Tp=std::thread]"
(283): here
instantiation of class "std::_Reference_wrapper_base<_Tp> [with _Tp=std::thread]"
(399): here
instantiation of class "std::reference_wrapper<_Tp> [with _Tp=std::thread]"
/usr/include/c++/5/future(1638): here

marvin.hpp(854): error: identifier "CUDNN_CONVOLUTION_FWD_ALGO_WINOGRAD_NONFUSED" is undefined

marvin.hpp(865): error: identifier "CUDNN_CONVOLUTION_BWD_DATA_ALGO_WINOGRAD_NONFUSED" is undefined

marvin.hpp(875): error: identifier "CUDNN_CONVOLUTION_BWD_FILTER_ALGO_WINOGRAD_NONFUSED" is undefined

/usr/include/c++/5/bits/stl_iterator_base_types.h(154): error: name followed by "::" must be a class or namespace name
detected during:
instantiation of class "std::__iterator_traits<_Iterator, void> [with _Iterator=int]"
(163): here
instantiation of class "std::iterator_traits<_Iterator> [with _Iterator=int]"
marvin.hpp(6006): here

/usr/include/c++/5/bits/stl_iterator_base_types.h(155): error: name followed by "::" must be a class or namespace name
detected during:
instantiation of class "std::__iterator_traits<_Iterator, void> [with _Iterator=int]"
(163): here
instantiation of class "std::iterator_traits<_Iterator> [with _Iterator=int]"
marvin.hpp(6006): here

/usr/include/c++/5/bits/stl_iterator_base_types.h(156): error: name followed by "::" must be a class or namespace name
detected during:
instantiation of class "std::__iterator_traits<_Iterator, void> [with _Iterator=int]"
(163): here
instantiation of class "std::iterator_traits<_Iterator> [with _Iterator=int]"
marvin.hpp(6006): here

/usr/include/c++/5/bits/stl_iterator_base_types.h(157): error: name followed by "::" must be a class or namespace name
detected during:
instantiation of class "std::__iterator_traits<_Iterator, void> [with _Iterator=int]"
(163): here
instantiation of class "std::iterator_traits<_Iterator> [with _Iterator=int]"
marvin.hpp(6006): here

/usr/include/c++/5/bits/stl_iterator_base_types.h(158): error: name followed by "::" must be a class or namespace name
detected during:
instantiation of class "std::__iterator_traits<_Iterator, void> [with _Iterator=int]"
(163): here
instantiation of class "std::iterator_traits<_Iterator> [with _Iterator=int]"
marvin.hpp(6006): here
12 errors detected in the compilation of "/tmp/tmpxft_00001958_00000000-9_demo.cpp1.ii".

Many thanks

nvcc fatal:Path to libdevice library not specified

Hello, this problem occurred when I run your program.“nvcc fatal : Path to libdevice library not specified”，please help me. I have cuda7.5 and cudnn5.1 libraries installed on my computer.

reproducing Fig 3: t-SNE embedding of 3DMatch descriptors

Dear Mr. Zeng,
I am very interesting to reproduce Fig 3 as your repored in the paper. I've tried to use the implementation of t-SNE in sklearn, I have a problem of visulizing the diemsion reduced features into different colors. Can give me some hints for reproducing the figure? Or do you mind sharing your code for this figure? Thanks in advance.

about the overlapping computation

I find there are inconsistencies in overlap computation of ground truth and prediction.
In getGtInfoLog.m, the nearst distance is computed through [nnIdx,nnSqrDist] = multiQueryKNNSearchImpl(fragment1PointCloud,fragment2Points',1);.
While in register2Fragments.m, the nearst disantce of pc1 in pc2 is computed through [nnIdx,sqrDists] = multiQueryKNNSearchImpl(pointCloud(fragment2Points'),fragment1Points',1);.
The computation in register2Fragments.m might be the correct usage of multiQueryKNNSearchImpl.

Besides, the distance thresholds in these two files are also different, which are 0.006*5=0.03 and 0.05.
These issuses might have a slight impact on the evaluation results.

using 3dmatch-toolbox on industry data

hi, I used 3dmatch-toolbox in my industry datas(Demo: Align Two Point Clouds with 3DMatch), but it didn't work very well. Actually, I rotated and translated a .ply data(single-depth-1.ply) to make another data(single-depth-2.ply), but it returned with some errors like "ransac reached the maximum number of 50000 trials " and I don't know where is the problem. I changed the voxel_size in demo.cu, but it didn't work. I had read your paper, which described the powerful descriptor you developed. I thought the descriptor is good enough to describe my data, cause they all have good geometry features. So what is your opinion?
Thank you.

showTDF.m error

I got this error when I run showTDF.m at matlab R2015b.

>> showTDF
bicycle000002
Error using pointCloud>validateAndParseInputs (line 775)
The value of 'Color' is invalid. Expected input to be one of these types:

uint8

Instead its type was double.

Error in pointCloud (line 126)
            [xyzPoints, C, nv] = validateAndParseInputs(varargin{:});

Error in showTDF (line 36)
pcshow(pointCloud(visPts,'color',colorPts));

Can this toolbox be used without matlab

Can I use the source code directly in C++?

Two questions about TSDF

Hi,
Thank you so much for your code, and it helps mu a lot. But I have two problems about the conversion from point cloud to TSDF file.

1. From the PCL tutorial http://pointclouds.org/documentation/tutorials/using_kinfu_large_scale.php and the picture from it we can see that TSDF gird has "1" for area without surface and "0" for area.
   

But from your code it computes "1" for area, like in

3dmatch-toolbox/core/demo.cu

Line 39 in f6496fd

voxel_grid_TDF[voxel_idx] = 1.0f; // on surface

So is there any significant difference between those two format?
2. The dist variable computed from

3dmatch-toolbox/core/demo.cu

Line 52 in f6496fd

float dist = sqrtf(xd * xd + yd * yd + zd * zd) / (float)search_radius;

may be larger than 1.0. Suppose xd=yd=zd=5, search_radius=5, then dist=1.73.
Is it better to change the norm term into "sqrt(3) * search_radius" to make "dist" smaller than 1.0?

About converge rate

Hi, Thanks for publishing the code!

I am trying your training procedure and trying to move your code to pytorch. But I find that the network doesn't converge after a long time. Maybe can you share your learning curve figure or could you simply describe after how many iterations could I expect it to converge.

Also I tried to add batch normalization layers after each conv layer and using ADAM instead of SGD. Do you think its helpful to accelerate the training process?

Thank you very much!

Could it work on Windows?

I want to use it on Windows,will it work?
Thank you!

memory leak

Hi, I have been using this great 3d descriptor to do some works about 3d matching. But I found that the usage of GPU's memory was growing up increasely. Have you found this phenomenon?

运行 core的demo时 程序出错错误使用 fread 文件标识符无效。使用 fopen 生成有效的文件标识符。 出错 demo (line 21) numFragment1Keypoints = fread(fid,1,'single');

错误使用 fread
文件标识符无效。使用 fopen 生成有效的文件标识符
出错 demo (line 21)
numFragment1Keypoints = fread(fid,1,'single');

Training step question

Training 3DMatch from RGB-D Reconstructions
My GPU is GTX1080, When I run training step. It has costed one night to iterate 5000 times.
Is the time cost reasonable?
By the way, can this training file (3dmatch-weights-snapshot-137000.marvin) match the scene that I captured or just for the demo your offered?