Evaluation metrics in performance boosting experiments

[zhangxy0517]

Evaluation metrics in performance boosting experiments

[Pterosaur-Yao]

The results of GeoTrans in Table 3 are same as the results in their original paper, and GeoTrans uses RMSE < 0.2m and computes scene average recall. Your registration recall are computed by pair avearge recall and use RE < 15 and TE < 30?

[Pterosaur-Yao]

Thank you for your prompt reply, but I would like to know the evaluation metrics for GeoTrans+MAC.

[zhangxy0517]

You can run the experiments on your own to verify the results.

Test files of GeoTransformer can be downloaded here

To run the test:
1、Download the latest files, replace registration.cpp with the attached file below, and compile.
registration_replace.zip
2、Put the decompressed folder GeoTrans in the same directory as the generated executable file MAC.
3、Run the command ./MAC --output_path ./test_Geo_3dmatch --input_path ./GeoTrans/3DMatch/5000 --dataset_name 3dmatch --descriptor predator --start_index 0 --no_logsto test 3DMatch 5000.

The output will be like this:

To test under different numbers of correspondences, you should change the value in line 129 of PCR.cpp and recompile:

	5000	2500	1000	500	250
3DMatch	0.999	0.995	0.99	0.95	0.9
3DLoMatch	0.99	0.99	0.95	0.9	0.9

Also, the command needs to be changed.

If the process is interrupted, you need to record the number of failed pairs up to the interruption. Assuming that Pair i is processed last when the termination happens, you need to rerun the command with --start_index i-1 to continue.

I will respond to questions as promptly as possible.

[weiya-89]

hi，when i replace the registration.cpp file and make ，there is a bug

[ 14%] Building CXX object CMakeFiles/MAC.dir/registration.cpp.o
/home/vision/work/lzw/3D-Registration-with-Maximal-Cliques-main/Linux/registration.cpp: In function ‘bool registration(const string&, std::string, std::string, const string&, const string&, const string&, const string&, const string&, double&, double&, double&, double&, double&, double&, double&, const string&, std::vector&)’:
/home/vision/work/lzw/3D-Registration-with-Maximal-Cliques-main/Linux/registration.cpp:117:10: warning: ignoring return value of ‘int fscanf(FILE*, const char*, ...)’, declared with attribute warn_unused_result [-Wunused-result]
117 | fscanf(ov, "%lf\n", &value);
| ~~~~~~^~~~~~~~~~~~~~~~~~~~~
/home/vision/work/lzw/3D-Registration-with-Maximal-Cliques-main/Linux/registration.cpp:153:11: warning: ignoring return value of ‘int fscanf(FILE*, const char*, ...)’, declared with attribute warn_unused_result [-Wunused-result]
153 | fscanf(fp, "%lf %lf %lf %lf\n", &GTmat(0, 0), &GTmat(0, 1), &GTmat(0, 2), &GTmat(0, 3));
| ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/vision/work/lzw/3D-Registration-with-Maximal-Cliques-main/Linux/registration.cpp:154:11: warning: ignoring return value of ‘int fscanf(FILE*, const char*, ...)’, declared with attribute warn_unused_result [-Wunused-result]
154 | fscanf(fp, "%lf %lf %lf %lf\n", &GTmat(1, 0), &GTmat(1, 1), &GTmat(1, 2), &GTmat(1, 3));
| ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/vision/work/lzw/3D-Registration-with-Maximal-Cliques-main/Linux/registration.cpp:155:11: warning: ignoring return value of ‘int fscanf(FILE*, const char*, ...)’, declared with attribute warn_unused_result [-Wunused-result]
155 | fscanf(fp, "%lf %lf %lf %lf\n", &GTmat(2, 0), &GTmat(2, 1), &GTmat(2, 2), &GTmat(2, 3));
| ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/vision/work/lzw/3D-Registration-with-Maximal-Cliques-main/Linux/registration.cpp:156:11: warning: ignoring return value of ‘int fscanf(FILE*, const char*, ...)’, declared with attribute warn_unused_result [-Wunused-result]
156 | fscanf(fp, "%lf %lf %lf %lf\n", &GTmat(3, 0), &GTmat(3, 1), &GTmat(3, 2), &GTmat(3, 3));
| ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/vision/work/lzw/3D-Registration-with-Maximal-Cliques-main/Linux/registration.cpp:172:19: warning: ignoring return value of ‘int fscanf(FILE*, const char*, ...)’, declared with attribute warn_unused_result [-Wunused-result]
172 | fscanf(corr, "%f %f %f %f %f %f\n", &src.x, &src.y, &src.z, &des.x, &des.y, &des.z);
| ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/vision/work/lzw/3D-Registration-with-Maximal-Cliques-main/Linux/registration.cpp:200:11: warning: ignoring return value of ‘int fscanf(FILE*, const char*, ...)’, declared with attribute warn_unused_result [-Wunused-result]
200 | fscanf(corr, "%f %f %f %f %f %f\n", &src.x, &src.y, &src.z, &des.x, &des.y, &des.z);
| ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/vision/work/lzw/3D-Registration-with-Maximal-Cliques-main/Linux/registration.cpp:227:11: warning: ignoring return value of ‘int fscanf(FILE*, const char*, ...)’, declared with attribute warn_unused_result [-Wunused-result]
227 | fscanf(corr, "%f %f %f %f %f %f\n", &src.x, &src.y, &src.z, &des.x, &des.y, &des.z);
| ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/vision/work/lzw/3D-Registration-with-Maximal-Cliques-main/Linux/registration.cpp:281:12: warning: ignoring return value of ‘int fscanf(FILE*, const char*, ...)’, declared with attribute warn_unused_result [-Wunused-result]
281 | fscanf(corr, "%f %f %f %f %f %f\n", &src.x, &src.y, &src.z, &des.x, &des.y, &des.z);
| ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/vision/work/lzw/3D-Registration-with-Maximal-Cliques-main/Linux/registration.cpp:330:12: warning: ignoring return value of ‘int fscanf(FILE*, const char*, ...)’, declared with attribute warn_unused_result [-Wunused-result]
330 | fscanf(corr, "%f %f %f %f %f %f\n", &src.x, &src.y, &src.z, &des.x, &des.y, &des.z);
| ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/vision/work/lzw/3D-Registration-with-Maximal-Cliques-main/Linux/registration.cpp:357:9: warning: ignoring return value of ‘int fscanf(FILE*, const char*, ...)’, declared with attribute warn_unused_result [-Wunused-result]
357 | fscanf(gt, "%d\n", &value);
| ~~~~~~^~~~~~~~~~~~~~~~~~~~
[ 28%] Linking CXX executable MAC
/usr/bin/ld: CMakeFiles/MAC.dir/main.cpp.o: in function analyse(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::basic_ofstream<char, std::char_traits<char> >&, int, int)': main.cpp:(.text+0x1bbf): undefined reference to registration(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, double&, double&, double&, double&, double&, double&, double&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::vector<double, std::allocator >&)'
/usr/bin/ld: CMakeFiles/MAC.dir/main.cpp.o: in function main': main.cpp:(.text.startup+0xe4e): undefined reference to registration(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, double&, double&, double&, double&, double&, double&, double&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::vector<double, std::allocator >&)'
/usr/bin/ld: main.cpp:(.text.startup+0x464b): undefined reference to registration(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, double&, double&, double&, double&, double&, double&, double&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<double, std::allocator<double> >&)' /usr/bin/ld: main.cpp:(.text.startup+0x553e): undefined reference to registration(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, double&, double&, double&, double&, double&, double&, double&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::vector<double, std::allocator >&)'
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/MAC.dir/build.make:288: MAC] Error 1
make[1]: *** [CMakeFiles/Makefile2:83: CMakeFiles/MAC.dir/all] Error 2
make: *** [Makefile:91: all] Error 2

[zhangxy0517]

@weiya-89 Thanks for your feedback. I have changed the function declaration of registration in Eva.h. Download the latest Eva.h and replace.

[weiya-89]

Thanks，that helps me a lot

[weiya-89]

hi,thanks for your reply, I would like to know how to evaluate GeoTrans+MAC.

[zhangxy0517]

Follow the instructions above. The registration recall is calculated as (total num - failed num)/total num.

[weiya-89]

i think it's a bit different from the metrics of GeoTransformer,which first computes the registration recall of each scene, then computes the mean registration recall of the 8 scenes, i would like to know whether the evluation metrics of the results reported in the paper are same as GeoTransformer.

[zhangxy0517]

All the results reported in Table 3 refer to the registration recall on all pairs.
All the methods are evaluated equally.

[chalth]

I have the same question.

[twowwj]

The results of GeoTrans in Table 3 are same as the results in their original paper, and GeoTrans uses RMSE < 0.2m and computes scene average recall. Your registration recall are computed by pair avearge recall and use RE < 15 and TE < 30?

It seems the comparison is a bit unfair, isn't it?