lincolnhard/head-pose-estimation

Real-time head pose estimation built with OpenCV and dlib

head-pose-estimation's Issues

Attention Detection based on head attitude estimation

how to achive attention detection based on head attitude estimation

finding head pose looking direction

@lincolnhard thanks for your work.

I am using the below code to find the head pose looing direction. (left, right, top, bottom)

y_value = float(str("{:7.2f}".format(euler_angle[1, 0])))
               x_value = float(str("{:7.2f}".format(euler_angle[0, 0])))
               if y_value >= 20:
                   print("looking left")
               elif y_value <= -20:
                   print("looking right")

               if x_value >= 20:
                   print("looking down")
               elif x_value <= -20:
                   print("looking up")

and seems it is fine.

if anyone knows better formula please share here.

Where object_pts come from?

Where object_pts come from?
https://github.com/lincolnhard/head-pose-estimation/blob/master/video_test_shape.py#L16

Running In VS2015

Hi，have u seen this problem in VS?
Below is error code is VS :
严重性 代码 说明 项目 文件 行 禁止显示状态
错误 LNK2019 无法解析的外部符号 "public: __cdecl dlib::entropy_decoder_kernel_2::entropy_decoder_kernel_2(void)" (??0entropy_decoder_kernel_2@dlib@@qeaa@XZ)，该符号在函数 "public: void __cdecl dlib::compress_stream_kernel_1<class dlib::entropy_encoder_model_kernel_5<257,class dlib::entropy_encoder_kernel_2,200000,4>,class dlib::entropy_decoder_model_kernel_5<257,class dlib::entropy_decoder_kernel_2,200000,4>,class dlib::crc32>::decompress(class std::basic_istream<char,struct std::char_traits > &,class std::basic_ostream<char,struct std::char_traits > &)const " (?decompress@?$compress_stream_kernel_1@V?$entropy_encoder_model_kernel_5@$0BAB@Ventropy_encoder_kernel_2@dlib@@$0DANEA@$03@dlib@@v?$entropy_decoder_model_kernel_5@$0BAB@Ventropy_decoder_kernel_2@dlib@@$0DANEA@$03@2@Vcrc32@2@@dlib@@QEBAXAEAV?$basic_istream@DU?$char_traits@D@std@@@std@@aeav?$basic_ostream@DU?$char_traits@D@std@@@4@@z) 中被引用 Head_Pose_Estimation C:\Users\有对象真好\documents\visual studio 2015\Projects\Head_Pose_Estimation\Head_Pose_Estimation\Head_Pose_Estimation.obj 1

cv::Mat does not accept four-parameter constructor

There doesn’t seem to exist a constructor for a cv::Mat object which accepts the four parameters specified in your C++ code. Are you using a specific version of OpenCV which has such a constructor, or is my compiler (clang) simply misunderstanding something?

D:\path\to\face.cc(63,43): error : no matching constructor for initialization of 'cv::Mat'
          static const cv::Mat camera_intrinsics = cv::Mat(3, 3, CV_64FC1, K);
                                                   ^       ~~~~~~~~~~~~~~~~~

Increasing range of estimation.

Can we make the range of the head pose estimator go up in pitch range and others? Can someone please tell how to do so?

Need more information about glAnthropometric3DModel.cpp

Hi, thanks for this project. But I was confused about why http://aifi.isr.uc.pt/Downloads/OpenGL/glAnthropometric3DModel.cpp works! So could you pls give me some explanation or a paper or a post about this?

decomposeProjectionMatrix计算出的角度抖动的厉害

该函数计算的欧拉角在人脸静止时，数据抖动得厉害，查了下原因似乎是旋转矩阵转换为欧拉角结果不是唯一，暂时没找到解决办法，请问有什么解决方案吗

image_pts

Thank for sharing this peice on head pose extimation. pls i am getting this error while trying to replicate this: After running from the begining to that portion of the code i have that error.
def get_head_pose(shape):
image_pts = np.float32([shape[17], shape[21], shape[22], shape[26], shape[36],
shape[39], shape[42], shape[45], shape[31], shape[35],
shape[48], shape[54], shape[57], shape[8]])
_, rotation_vec, translation_vec = cv2.solvePnP(object_pts, image_pts, cam_matrix, dist_coeffs)

NameError Traceback (most recent call last)
in
----> 1 _, rotation_vec, translation_vec = cv2.solvePnP(object_pts, image_pts, cam_matrix, dist_coeffs)

NameError: name 'image_pts' is not defined

pls any solution

the running time for solvepnp

Hi:
thank you so much for the code, I have implemented it and the it works, the pose estimation result is fine. However, the runing time for the function "solvePnP" is so big, it seems about 1-second for only running this function. I just used 14 landmarks to estimate the pose(the landmarks are not obtained from Dlib, they are from some other processing output.). Have you ever met this problem or not? I use opencv3.3, and the include files for the source cpp files are:
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/calib3d/calib3d.hpp>"

if the running time for the solvepnp is so big, the real-time processing is impossible.

Thank you so much.

Final projected box is shaky

Hi, I calibrared my camera and got the correct K[] and D[]. The pose estimation seems fine.

The problem is, even though the face on the camera is still, the final landmark drawn and the projected box is shaky. Given the face is still, I expect the projected box (and the landmark points) to be still as well.

Any idea what could be the problem?

Thank you.

Video Size problem

Hello i was trying to chance the video size to:

cap.set(cv2.CAP_PROP_FRAME_WIDTH, 1280)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 720)

but when i did that the script crashes and when it works all boxes ate distorted.
How can i fix it ?

show angle result?

//show angle result
outtext << "X: " << std::setprecision(3) << euler_angle.at<double>(0);
cv::putText(temp, outtext.str(), cv::Point(50, 40), cv::FONT_HERSHEY_SIMPLEX, 0.75, cv::Scalar(0, 0, 0));
outtext.str("");
outtext << "Y: " << std::setprecision(3) << euler_angle.at<double>(1);
cv::putText(temp, outtext.str(), cv::Point(50, 60), cv::FONT_HERSHEY_SIMPLEX, 0.75, cv::Scalar(0, 0, 0));
outtext.str("");
outtext << "Z: " << std::setprecision(3) << euler_angle.at<double>(2);
cv::putText(temp, outtext.str(), cv::Point(50, 80), cv::FONT_HERSHEY_SIMPLEX, 0.75, cv::Scalar(0, 0, 0));
outtext.str("");

Seems it should be: Y-Z-X means, pitch-yaw-roll?

Where do the object_points come from and which in which units are they measured?

Hi @lincolnhard,

first - thanks for the nice project.

I would like to know where do the values for the object_points come from and in which units are they?

请问您自己试验的帧率大概是多少呢？我试验运行后速度很慢，但是demo看起来速度很快，求教。

我的运行环境是ubuntu16.04,opencv3.2,dlib是今天刚从GitHub上clone的新版，然后把从摄像头读入数据改成了读图片（我的图片是720*1280）大小的，k矩阵也相应的改了一下。
现在有两个问题：

1. 速度很慢
2. 有时候会完全检测不到关键点（基本上是正脸的时候）
   不太明白是什么原因，求教，谢谢了。

Running time between python version and C++ version

请问在Python版本与C++ 版本的帧率分别是多少？
在同一台机器上实现之后，为什么明显C++版本要比Python版本慢了几倍？（Linux下Cmake编译）是需要有什么选项的设置吗？还是dlib本身的原因？

不使用摄像头，使用标注好关键点的图片时，相机参数怎么设置？

你好，我这边使用网上找的图片计算三个角度，发现角度与实际偏差较大，是不是与相机参数K和D有关，此时相机参数应该怎么设？

HeadPose Angel (SolvePNP) (euler_angle[1, 0] giving slight wrong value)

Hello All,

I was trying to use this get_head_pose(shape) function to get the x, y and z values of the headpose.

But I was getting slight error on 'y' value whenever the head is straight towards the camera , and it gives some negative value and this value is close to zero.

Can you suggest me how I can get the 'y' value as zero whenever headpose is straight towards camera.

Couldn't run the provided cpp file !

About dlib 68_face_landmarks.dat

I am using 68_face_landmarks.dat modual in my project but its not enough stable and accurate. So the face mesh keeps vibrating.

So is there any way to make it more stable and accurate. Or any link for stable and accurate dlib module.

A question with respect to specific number within the distortions matrix.

Dear KwanHua Lee:

Sorry to bother you. I have a question with respect to your "head pose estimation" project. Could you tell me how could you get the specific number within the distortions matrix. I saw your code and you do not process any a operation.
Thanks for your help!

Best Regards
Allen

compiled on Ubuntu

Hi

How this code can be compiled on Ubuntu

5 land marks head pose estimation

Hello,
your code is very nice and works ok.

Can you please try and adapt it for newly released Dlib's 5 point landmark detection? I have tried to adapt the code by myself, but the data is way off.

Is it actually possible or are five landmarks not enough to reconstruct 3D head pose?

Thanks!

Is it fine grained ?

@lincolnhard, Thanks for such nice repo.
Is the head pose estimation done in this project, fine grained head pose estimation?
I am thinking of using it in my project, but the project needs head pose estimation to be highly precise.

Thank You

About head-pose-estimation by Python implementaiton

Hello I follow your c++ code to implementation the pose estimation,
I want to get the face pose, range from +90 to -90, like the following picture

I use the six landmark and their world coordinate to get pose

image_points = np.array([
                            (landmarks[4], landmarks[5]),     # Nose tip
                            (landmarks[10], landmarks[11]),   # Chin
                            (landmarks[0], landmarks[1]),     # Left eye left corner
                            (landmarks[2], landmarks[3]),     # Right eye right corne
                            (landmarks[6], landmarks[7]),     # Left Mouth corner
                            (landmarks[8], landmarks[9])      # Right mouth corner
                        ], dtype="double")

    # 3D model points.
    model_points = np.array([
                            (0.0, 0.0, 0.0),             # Nose tip
                            (0.0, -330.0, -65.0),        # Chin
                            (-165.0, 170.0, -135.0),     # Left eye left corner
                            (165.0, 170.0, -135.0),      # Right eye right corne
                            (-150.0, -150.0, -125.0),    # Left Mouth corner
                            (150.0, -150.0, -125.0)      # Right mouth corner                         
                        ])

(success, rotation_vector, translation_vector) = cv2.solvePnP(model_points, image_points, camera_matrix, dist_coeffs, flags=cv2.CV_ITERATIVE)
rvec_matrix = cv2.Rodrigues(rotation_vector)[0]
proj_matrix = np.hstack((rvec_matrix, translation_vector))
eulerAngles = -cv2.decomposeProjectionMatrix(proj_matrix)[6] 
yaw   = eulerAngles[1]
pitch = eulerAngles[0]
roll  = eulerAngles[2]

But I have some problem, each case(yaw, pitch, row) is correct, for example, In face roll case, roll is true, but, pitch is false. Could u give me some advise? Thanks.

Code in Matlab

Hi Sir,

Please help with Matlab implementation. It might be of great help if MEX file could be provided where calling the file will output a struct containing values of pitch yaw roll and the landmark coordinates along with any necessary information. I believe it will also save user from having to install the libraries.

Best Regards
Asad

How to get Yaw, Pitch, Roll?

Thank you for sharing your code.
Is there any way to get Yaw, Pitch, and Roll of the head?

object_pts 是不变的吗?

您好, 目前我没改 video_test_shape.py中的object_pts, 只是将K 和D 换成了自己的相机内参, 得到的结果也很近似. 所以, object_pts 是不变的吗? 如果需要改变,要怎么获取呢? 谢谢

solvePNP计算结果很离谱请教

问下，我代码运行起来后，发现solvePnP后的rotation_vec， translation_vec里面的数值都异常，都是e+30，e-11等，可能是哪里的问题，该函数输入的landmark点我看了下，都是对的，因为显示出来都没有问题。其他参数我没有改，感觉也不应该有非常大的影响。

不知道哪里的问题。。。。。，特请教，谢谢！

noisy pose estimation

solvePnP result is noisy. Especially the depth value flip frequently. For sure, there are some mistakes in your 3D model.

video_test_shape.cpp X，Y，Z 值不变？

您好，我运行了video_test_shape.cpp 后，左上角XYZ值不变，并且没有画出表示头部的立方体，请问如何解决。谢谢，期待您的回答。