Questions about evaluation

[yan99033]

Questions about evaluation

[jczarnowski]

• For the ATE evaluation, we've focused on the TUM dataset as the standard benchmark and selected sequences to compare against the results published in the CNN-SLAM and DeepTAM papers. The code for these methods was not available to run. At the time, joint mapping and tracking implementation was not published for DeepTAM and CNN-SLAM is still not open source. Therefore, in the reconstruction evaluation table we've focused on reconstruction only.

• The reference to SfMLearner was to show that we follow the principle of scaling depth maps to compare them against the ground truth. We did not use the ground truth median as SfMLearner though, but for a more fair comparison we've used the 'optimal' scale estimated by the TUM script.

Comparing monocular SLAM systems is indeed hard, and there is a lot more experiments or axes we could compare on. That's why we will be releasing the full code together with the evaluation.

[yan99033]

For the ATE evaluation, we've focused on the TUM dataset as the standard benchmark and selected sequences to compare against the results published in the CNN-SLAM and DeepTAM papers. The code for these methods was not available to run. At the time, joint mapping and tracking implementation was not published for DeepTAM and CNN-SLAM is still not open source. Therefore, in the reconstruction evaluation table we've focused on reconstruction only.

What I meant is that you can have both the reconstruction accuracy and the ATE evaluations in Table II, similar to Table 1 in CNN-SLAM. I am curious to find out if the reconstruction accuracy is proportional to the camera trajectory accuracy.

Edit: The reason why this is interesting is that you are using graph-based optimisation, which supposedly should give you better mapping and tracking performance; in contrast, CNN-SLAM uses alternating optimisation (tracking and then EKF-style depth fusion), which potentially can give you good tracking but bad mapping, or vice versa (compare ICL/office2 and ICL/living1 in Table 1 of CNN-SLAM).

The reference to SfMLearner was to show that we follow the principle of scaling depth maps to compare them against the ground truth. We did not use the ground truth median as SfMLearner though, but for a more fair comparison we've used the 'optimal' scale estimated by the TUM script.

Got it. That makes sense.

Thanks again.