huangying-zhan/kitti-odom-eval

KITTI Odometry Evaluation Toolbox

KITTI Odometry Evaluation Toolbox

KITTI Odometry benchmark contains 22 stereo sequences, in which 11 sequences are provided with ground truth.

The evaluation tool is used for evaluating KITTI odometry result. We include several common metrics in evaluating visual odometry, including

• sub-sequence translation drift percentage
• sub-sequence rotation error
• absolute trajectory error
• relative pose error (translation)
• relative pose error (rotation)

Requirement

We recommend using Anaconda for installing the prerequisites.

conda env create -f requirement.yml -p kitti_eval # install prerequisites
conda activate kitti_eval  # activate the environment [kitti_eval]

Result Format

Before evaluation, estimated poses should be saved in a .txt file. Currently, there are two formats are supported.

Suppose the pose for 100-th image is

T00 T01 T02 T03
T10 T11 T12 T13
T20 T21 T22 T23
0   0   0   1

You should save the pose as

# First format: skipping frames are allowed
99 T00 T01 T02 T03 T10 T11 T12 T13 T20 T21 T22 T23 

# Second format: all poses should be included in the file
T00 T01 T02 T03 T10 T11 T12 T13 T20 T21 T22 T23

Usage

To use the tool, there are some possible options. The basic usage is

# RESULT_PATH contains the camera pose text file, 
# which should be named as 00.txt, 01.txt, ...

# Example
python eval_odom.py --result result/example_1/

The full usage is

python eval_odom.py --result RESULT_PATH --align ALIGNMENT_OPTION --seqs X X X

# Examples
python eval_odom.py --result result/example_0 --align 7dof
python eval_odom.py --result result/example_1 --align 6dof --seqs 9

X is the sequence number. If --seqs is not given, all available sequences in the folder will be evaluated.

The detailed results will be saved in RESULT_PATH

Alignment

Following prior works, certain degrees of alignment can be done in this evaluation script. Pass one of the following argument --align XXX to the script, where XXX can be,

• scale
• scale_7dof
• 6dof
• 7dof

scale

Find a scaling factor that best align the predictions to the ground truth (GT) poses

scale_7dof

Find a 6+1 DoF transformation, including translation, rotation, and scaling, that best align the predictions to the GT poses. After that, only the scaling factor is used to align the predictions to the GT for evaluation.

6dof

Find a 6 DoF transformation, including translation and rotation that best align the predictions to GT poses.

7dof

Find a 6+1 DoF transformation, including translation, rotation, and scaling, that best align the predictions to the GT poses.

Evaluation result

Here shows some evaluation result examples

Trajectory comparison

Sub-sequence error

Result summary

License

The code is released under the permissive MIT license.

If you use this toolbox, a footnote with the link to this toolbox is appreciated.

You can also cite the DF-VO which we release the toolbox in the first place.

@article{zhan2019dfvo,
  title={Visual Odometry Revisited: What Should Be Learnt?},
  author={Zhan, Huangying and Weerasekera, Chamara Saroj and Bian, Jiawang and Reid, Ian},
  journal={arXiv preprint arXiv:1909.09803},
  year={2019}
}

kitti-odom-eval's Issues

different kitti Official code

thanks a lot， can you tell me what different between your code and Official code provided by the kitti dataset?

np.linalg.inv : Singular Matrix

Thanks for the code.

I am receiving the error Singular Matrix. How should I resolve it?

Why is the translation/rotation error recalculated at every 10th frame?

I've been going through the code and I've been having a hard time understanding why the translation/rotation error recalculated at every 10th frame.

at line 191:

kitti-odom-eval/kitti_odometry.py

Line 191 in 4b850b0

for first_frame in range(0, len(poses_gt), self.step_size):

Why is first_frame resetted at every 10th frame and the whole errors recomputed for a segment (100m, 200m, etc.) We end up with multiple error values for segments of decreasing lengths. It looks like some sort of sliding window of varying length that is moved along the sequence.

Why is it computed this way and not computed over the first 100 meters, then 200 meters, etc?

Monocular Visual Odometry 6 DOF

I'm trying to evaluate KITTI odometry
first I convert pose to 6DoF using this code

def rotationMatrixToEulerAngles(self, R):
        assert (self.isRotationMatrix(R))
        sy = math.sqrt(R[0, 0] * R[0, 0] + R[1, 0] * R[1, 0])
        singular = sy < 1e-6

        if not singular:
            x = math.atan2(R[2, 1], R[2, 2])
            y = math.atan2(-R[2, 0], sy)
            z = math.atan2(R[1, 0], R[0, 0])
        else:
            x = math.atan2(-R[1, 2], R[1, 1])
            y = math.atan2(-R[2, 0], sy)
            z = 0
        return np.array([x, y, z], dtype=np.float32)

def matrix_rt(self, p):
        return np.vstack([np.reshape(p.astype(np.float32), (3, 4)), [[0., 0., 0., 1.]]])

pose1 = self.matrix_rt(self.poses[index][i])
pose2 = self.matrix_rt(self.poses[index][i + 1])
pose2wrt1 = np.dot(np.linalg.inv(pose1), pose2)
R = pose2wrt1[0:3, 0:3]
t = pose2wrt1[0:3, 3]
angles = self.rotationMatrixToEulerAngles(R)
odometries.append(np.concatenate((t, angles)))

also, model output is in the same format (6DoF)

the question is how to evaluate 6DoF results

how plot different method predict result in one image ?

like trajectory and error plot in one image

Why inverse?

Hello, in your code you apply inverse to the first frame and then multiply it with the corresponding (i-th) pose matrix to obtain the pose at the frame i. Why not just multiply the i-th pose matrix with the first pose? Why do we need inverse here?

I am talking about the lines like: poses_result[cnt] = np.linalg.inv(pred_0) @ poses_result[cnt].
I thought to obtain poses we simply need to: poses_result[cnt] = poses_result[cnt] @ pred_0.

Error Graph 800 meter limit

Hi! Thanks again for publishing your work, it has been very helpful for my project.
I have noticed that the resulting graphs, regardless of sequence, will always include up to 800 meters on the X axis.
This means that for sequences shorter than 800 meters (such as 04), the points exceeding the sequence path drop down to zero, while sequences greater than 800 meters in length (such as 00) will cut off the results at 800 meters.

Is there a way I can set parameters to resolve this on my own, or is this something that must be addressed in the core code?

why the result is always shrink to a point.?

Using Jiawang's model, I get 09.txt.
When I apply this code, the result will always shrink to a point.
Excuse me, what's the matter?

Regarding the working of the project.

Can you provide an explanation what this project actually does and how do we provide custom inputs (from the example directory) ? Your help would be appreciated.

Thanks in advance.

Speed Scene translation and rotation error ?

how to make this curve ?

understand the absolute trajectory error values

thanks to share this tool, I just wanted to understand these values, they should be the absolute trajectory error results on kitti benchmark I got them from a research paper, could you explain it please, I should produce similar values to compare, how to do that using this tool,? thanks in advance

Thanks for sharing~

It's very useful~