rust-cv/pnp

Perspective-n-Point algorithm

Rust Computer Vision

Rust CV is a project to implement computer vision algorithms in Rust.

What is computer vision

Many people are familiar with covolutional neural networks and machine learning in computer vision, but computer vision is much more than that. One of the first things that Rust CV focused on was algorithms in the domain of Multiple-View Geometry (MVG). Today, Rust now has enough MVG algorithms to perform relatively simple camera tracking and odometry tasks. Weakness still exists within image processing and machine learning domains.

Goals

Here are some of the domains of computer vision that Rust CV intends to persue along with examples of the domain (not all algorithms below live within the Rust CV organization, and some of these may exist and are unknown):

• Image processing (Wikipedia)
  • Diffusion & blur
    • Gaussian blur (Wikipedia)
    • Fast Explicit Diffusion (FED) (implementation exists within akaze crate)
  • Contrast enhancement
    • Normalization (Wikipedia)
    • Histogram equalization (Wikipedia)
  • Edge detection (Wikipedia) & gradient extraction (Wikipedia)
    • Scharr filters (exists within akaze crate)
    • Sobel filter (Wikipedia)
    • Canny edge detector (Wikipedia)
• Multiple-View Geometry (MVG)
  • Feature extraction (Wikipedia)
    • AKAZE
    • FAST (Wikipedia)
    • ORB (Wikipedia)
    • SIFT (Wikipedia)
  • Camera models and calibration (both from and to image coordinates from bearings)
    • Pinhole Camera (Wikipedia)
      • Skew, focals, and principle point
      • Kn radial distortion (Wikipedia)
        • K1 radial distortion
        • K1-K6 radial distortion
    • Fisheye Camera (Wikipedia)
      • Skew, focals, and principle point
      • K1-K4 fisheye distortion (same as OpenCV)
    • Equirectangular (Wikipedia)
  • Matching (Wikipedia)
    • Descriptor matching strategies
      • Brute force (for camera traking with binary features)
      • HNSW (for loop closure)
    • Filtering strategies
      • Symmetric matching (exists within vslam-sandbox)
      • Lowe's ratio test matching
  • Geometric verification (utilized abstractions in sample-consensus)
    • Consensus algorithms
      • ARRSAC
      • AC-RANSAC
      • RANSAC (Wikipedia)
    • Estimation algorithms
      • P3P (Wikipedia)
        • Lambda Twist
      • Motion estimation (Wikipedia)
        • Eight Point (Wikipedia)
        • Nister-Stewenius (basically done, but not packaged up)
    • Models
      • Essential matrix (Wikipedia)
        • With residual for feature matches
      • Pose of world relative to camera (Wikipedia)
        • With residual for feature to world matches
      • Relative pose of camera (Wikipedia)
        • With residual for feature matches
      • Homography matrix (Wikipedia)
        • With residual for feature matches
    • PnP (estimation, outlier filtering, and optimization) (incomplete)
  • Image registration (Wikipedia)
  • Real-time depth-map estimation (for direct visual odometry algorithms that require it) (Wikipedia)
  • Visual concept detection (used for loop closure)
    • Bag of Visual Words (BoW, see Wikpedia article)
    • Second order occurence pooling (as per the paper "Higher-order Occurrence Pooling for Bags-of-Words: Visual Concept Detection")
    • Fisher vector encoding
    • Learned place recognition (also see pattern recognition domain below)
  • Reconstruction (Wikipedia)
    • Visibility graph (Wikipedia)
    • Graph optimization
    • Loop closure (Wikipedia)
    • Exporting (point cloud Wikipedia)
      • To NVM file
      • To PLY file (Wikipedia)
  • Post-reconstruction depth-map estimation (for reconstruction post-processing) (Wikipedia)
  • Densification
    • Using more extracted features
      • Using curvature maximas (as per "VITAMIN-E: VIsual Tracking And MappINg with Extremely Dense Feature Points")
    • Using patch-match (Wikipedia) and depth-map
    • Using depth-map only with edge detection
  • Meshing (Wikipedia)
    • Delaunay triangulation
      • Filtered with NLTGV minimization (as per "VITAMIN-E: VIsual Tracking And MappINg with Extremely Dense Feature Points")
    • Poisson surface reconstruction (Wikipedia)
    • Surface refinement
    • Texturing (related Wikipedia)
• Pattern recognition
  • k-NN search
    • Brute force
    • HNSW
  • Face recognition (Wikipedia)
  • Object recognition (Wikipedia)
  • Place recognition (can assist in MVG)
  • Segmentation (Wikipedia)
    • Semantic segmentation mapping (see this blog post)

To support computer vision tooling, the following will be implemented:

• Point clouds (Wikipedia)
  • Display tool
    • LoD refinement (Wikipedia)
      • Potree file format (GitHub)
    • PLY files (Wikipedia)

pnp's Issues

Regarding vocabulary

One of the most frustrating things in CV is in my opinion swapping by mistake projection matrices and their inverse. Using a consistent vocabulary I think is thus quite important.

Usually the image formation process is modeled as x = K * T * (X) where T is the transformation due to the movement (rigid body motion of the camera) and K is intrinsics matrix. This matrix T is thus the conversion from world coordinates to camera coordinates. Therefore, in most algorithms, the objective is to find the rotation and translation composing this matrix T = [ R, t ].

Sometimes however we use the vocabulary "camera pose", which are used to describe the rotation and the translation of the camera itself in world coordinates. If we note C (for "camera") this "pose", we can note that C is actually the inverse of that camera projection matrix. C = inv(T).

In the manual test, there is usage of camera "pose" where actually it is not a pose but transformation matrix (the inverse). I just wanted to point it out since I was surprised when reading the line:

let world_points = camera_depth_points.map(|p| pose.inverse() * p);

Status of this crate?

I'm usually not the one to raise issues like this, but this is the only implementation of PnP I found in Rust, and I noticed that 1) this repo has never been published as a crate and 2) it hasn't been updated in a while to latest cv-core.

I wonder why - was a better alternative published elsewhere?