How to estimate the face pose?

[tzutalin]

How to estimate the face pose?

[EzequielAdrianM]

To estimate the face pose, you will need:

• Simulated 3D model of human face parts.
• The real landmarks of the human face we want to perform pose estimation.
• Camera Calibration parameters.

Everything was made for C++, ready to be added to Dlib.

Add the following to: detector.h

int mImageW = 0;
int mImageH = 0;

virtual inline int det(cv::Mat& image) {
    //Add this to the function.
    mImageW = image.cols;
    mImageH = image.rows;
}

//Create two new functions
int getFrameWidth() {return mImageW;}
int getFrameHeight() {return mImageH;}

And add the following to: jni_face_det.cpp

std::vector<cv::Point3f> modelPoints;
std::vector<cv::Point2f> srcImagePoints;
std::vector<double> rv(3), tv(3);
cv::Mat rvec(rv), tvec(tv);
cv::Mat camMatrix;
double rot[9] = {0};

//Load 3D Points for Face Pose Estimation
modelPoints.push_back(cv::Point3f(2.37427,110.322,21.7776));    // l eye (v 314)
modelPoints.push_back(cv::Point3f(70.0602,109.898,20.8234));    // r eye (v 0)
modelPoints.push_back(cv::Point3f(36.8301,78.3185,52.0345));    //nose (v 1879)
modelPoints.push_back(cv::Point3f(14.8498,51.0115,30.2378));    // l mouth (v 1502)
modelPoints.push_back(cv::Point3f(58.1825,51.0115,29.6224));    // r mouth (v 695)
modelPoints.push_back(cv::Point3f(-61.8886f,127.797,-89.4523f));  // l ear (v 2011)
modelPoints.push_back(cv::Point3f(127.603,126.9,-83.9129f));     // r ear (v 1138)

Then, you obtain and select only some of the necessary landmakrs:

• Left Ear
• Right Ear
• Left Eye
• Right Eye
• Nose Base
• Left Mouth Corner
• Right Mouth Corner

In the end of the for loop, you use the collected landmarks for calculating the pose.

if(j == 45 || j == 36 || j == 33 || j == 54 || j == 48 || j == 0 || j == 16) {srcImagePoints.push_back(cv::Point2f((float) x, (float) y));}
if(j+1 == shape.num_parts()) {
  cv::Mat op(modelPoints);
  cv::Mat ip(srcImagePoints);
  //Original Image Size
  int mImageW = faceDetector->getFrameWidth();
  int mImageH = faceDetector->getFrameHeight();
  int max_d = MAX(mImageW, mImageH); // Approximate the Focal Length
  camMatrix = (cv::Mat_<double>(3,3) << max_d, 0, mImageW/2, 0, max_d, mImageH/2, 0, 0, 1);
  double _distCoeffs[] = {0, 0, 0, 0};
  cv::solvePnP(op, ip, camMatrix, cv::Mat(1,4,CV_64FC1,_distCoeffs), rvec, tvec, false, CV_EPNP);
  cv::Mat rotM(3,3,CV_64FC1,rot);
  cv::Rodrigues(rvec, rotM);
  double* _r = rotM.ptr<double>();
  double _projMatrix[12] = {
    _r[0], _r[1], _r[2], tv[0],
    _r[3], _r[4], _r[5], tv[1],
    _r[6], _r[7], _r[8], tv[2]
  };
  cv::Mat tmp,tmp1,tmp2,tmp3,tmp4,tmp5;
  cv::Vec3d eulerAngleVector;
  cv::decomposeProjectionMatrix(cv::Mat(3,4,CV_64FC1,_projMatrix),tmp,tmp1,tmp2,tmp3,tmp4,tmp5,eulerAngleVector);
  int eulerX = (int) eulerAngleVector[0];
  int eulerY = (int) eulerAngleVector[1];
  int eulerZ = (int) eulerAngleVector[2];
  LOG(INFO) << "Face Rotation Angle: " << eulerX << " " << eulerY << " " << eulerZ;
}