saiteja-talluri/gsoc-opencv

Code written for OpenCV during GSoC 2019 related to Facial Landmark Detection

GSoC-OpenCV

Code written for OpenCV during GSoC 2019 related to Facial Landmark Detection

Facial Landmark Detector

• Description: Facial feature detection and tracking is a high value area of computer vision since humans are interested in what humans are paying attention to, feeling, and enhancing face pictures in selfies etc. At this point, we feel this should just become a standard "built in" ability that computer vision users can just call/rely on. OpenCV already has some code available on for Facial Landmark detection, see Tutorial on Facial Landmark Detector API and Tutorials for face module, but much progress has been made that we want to make available. The task is to create a Facial Landmark detector model with the following requirements:
• Expected Outcomes:
  • For mobile, make available a smaller model with lesser points ( e.g. 5 instead of 68 )
  • Run at better Accuracy than Dlib
  • Face Stabilization ( filtering and optical flow )
  • Add it to the Python API
  • Resources
  • https://docs.opencv.org/4.0.0/d5/d47/tutorial_table_of_content_facemark.html
  • https://docs.opencv.org/4.0.0/de/d27/tutorial_table_of_content_face.html
  • https://github.com/opencv/opencv_contrib/tree/master/modules/face
• Skills Required: Mastery level C++ and Python, college course work in Computer Vision that includes work on face features. Best if you have worked with deep networks before.
• Mentors: Satya Mallick
• Difficulty: Medium

Updates

• [06/04/19] Submitted the Proposal for the GSoC 2019 to OpenCV in Facial Landmark Detection.
• [07/05/19] Received the acceptance letter for the GSoC 2019 in Facial Landmark Detection project.
• [27/06/19] End of the Community Bonding Phase and Start of the Coding Phase.
• [02/06/19] Completed training the AAM model and added links to the model and the sample C++ code has been provided.
• [07/06/19] Trained the Kazemi model but the model seemes to not work as efficiently as the LBF and AAM models. May be more training is required but due to RAM limitations could not verify.
• [08/06/19] Added sample C++ code for LBF model and LBF seems to dominate the AAM model in both speed and accuracy.
• [11/06/19] Went through the implementation of the 2D-and-3D face alignment and checked if the model can be added to openCV model zoo. [Pushed to second priority as fixing the python bindings to the working LBF and AAM models is more important.]
• [15/06/19] Went through all the issues and pull requests related to the python bindings on facemark API and these are few which need to be highlighted.
  • The discussion for the issue in the python bindings started with this issue and many work arounds have been tried but many of them failed.
  • However Gusi Te proposed a method to fix the bindings and his pull request to opencv_contrib has been merged but his pull request to opencv failed as it was affecting a test file. This issue still persists as the pull to opencv failed.
  • The latest build of opencv_contrib has the same issue in the fit function and is still open to solve.
• [17/06/19] There also have been some failed attempts to expose the training code of the Facemark API to the users in python. So I am working on these two things and hope it would get completed before the first evaluation. I was able to fix the fit fucntion but drawLandmarks gave a similar error (channels() == CV_MAT_CN(dtype)) may be of because of the fix, so I am looking into it.
• [22/06/19] The python bindings for the fit function have been fixed and the changes have been merged into opencv_contrib:3.4 and soon will be merged into opencv_contrib:master as well. Exposing the training code is under process, owing to the inheritance of Kazemi from facemark and not from facemarkTrain, it is taking longer than expected time. However work is in place and proper code refactoring needs to be done before raising a PR. Expect a PR soon...

Planning and Schedule in Coding Phase

Date	Progress	Work Ahead
27th May to 3rd June, 2019	Completed training the AAM model and added links to the model and the sample C++ code has been provided.	Check the working of Kazemi Model and verify which of these models works the best, also have a look at the Adrian bulats 3D Face Alignment implementation.
4th June to 11th June, 2019	Trained the Kazemi model but the model seemes to not work as efficiently as the LBF and AAM models. May be more training is required but due to RAM limitations could not verify. Added sample C++ code for LBF model and LBF seems to dominate the AAM model in both speed and accuracy. Went through the implementation of the 2D-and-3D face alignment and checked if the model can be added to openCV model zoo. [Pushed to second priority as fixing the python bindings to the working LBF and AAM models is more important.]	Put the adrain bulats implementation as 2nd priority and start fixing the python bindings to the facemark API.
12th June to 18th June, 2019	Went through all the issues and pull requests related to the python bindings on facemark API and these are few which need to be highlighted. The discussion for the issue in the python bindings started with this issue and many work arounds have been tried but many of them failed. However Gusi Te proposed a method to fix the bindings and his pull request to opencv_contrib has been merged but his pull request to opencv failed as it was affecting a test file. This issue still persists as the pull to opencv failed. The latest build of opencv_contrib has the same issue in the fit function and is still open to solve. There also have been some failed attempts to expose the training code of the Facemark API to the users in python. So I am working on these two things and hope it would get completed before the first evaluation. I was able to fix the fit fucntion but drawLandmarks gave a similar error (channels() == CV_MAT_CN(dtype)) may be of because of the fix, so I am looking into it.	Fix the python bindings for the fit and the drawLandmarks functions in the facemark API Expose the training implementation of the Facemark API to the python users and ensure this is done before the first evaluation.

gsoc-opencv's Issues

Train on different type of custom data

Is it possible to train the LBF or AAM on a dataset (say some landmark localization in biomedical images) with different number of landmarks (say 40)? I'm getting the following error in my case.

Wrong n_landmarks, currently only 29 and 68 landmark points are supported training random forest

Also with the case of 68 landmarks, the trained model is not doing well prediction on custom objects.