How can I separate the face detection from the landmark extraction?

[gicheonkang]

How can I separate the face detection from the landmark extraction?

[gicheonkang]

Hi @ivsanro1
Hmm... That's new fresh idea.
But, you might face some issues because it is required to modify dlib's overall architecture.

In app/src/main/.../OnGetImageListener.java, there is a code like below.
results = mFaceDet.detect(mResizedBitmap);
This method contains detection, feature extraction. Now, it doesn't divided atomically.

Here is the source code of current engine. dlib
I only modified some points that can be parallel-processing.
Please check the source code if separation is possible.
Thank you for your suggestion :)

[ivsanro1]

I know, I know, but I don't think it is really about dlib. Let me explain:

As far as I know, dlib is attached to the project via shared library that also includes the compiled JNI libraries that help to comunicate the C++ code (dlib) with the java code (the app). In this project, the first call from the Java code related to the face detection is, like you said, in app/src/main/.../OnGetImageListener.java:

results = mFaceDet.detect(mResizedBitmap);

where results is an instance object from the class FaceDet, and, as you said, detect(·) is a method from that class which performs the face detection and the landmark calculation atomically.

I have seen in the dlib-android project the source files of the JNI, and I guess they are unchanged in this project, those can be found here:

https://github.com/tzutalin/dlib-android/tree/master/jni/jni_detections

and specifically, jni_face_det.cpp has the source code of the native methods that grab the code directly from dlib. This is the part that I don't understand too much, but you get the main idea: the detection and the landmark is calculated atomically (you can't calculate such things separatedly).

However, if we take a look at the dlib example face_landmark_detection_ex.cpp, that can be found in:

http://dlib.net/face_landmark_detection_ex.cpp.html

you can see that the face detector:

frontal_face_detector detector = get_frontal_face_detector();

and the landmark calculator, known as shape predictor:

shape_predictor sp; deserialize(argv[1]) >> sp;

are two separated things, and, in fact, the face detection task:

std::vector<rectangle> dets = detector(img);

and the landmark extraction task:

full_object_detection shape = sp(img, dets[j])

are being done separatedly. Therefore, despite I don't really know how to separate such code when it comes to the JNI libraries, I think that could be possible (and, in fact, it would be the best pratice) to do a JNI native method for the face detection only and another one for the landmark extractor, but separatedly. But I have a lot of problems understanding how could I achieve this, because I have no idea of JNI and I find JNI code very awful.

Thanks for your time

[gicheonkang]

@ivsanro1 Yes, Originally, Dlib process steps as you mentioned. frontal_face_detect --> feature extraction.
But, I can't fully understand your suggestion. Is this the main idea that you said?
Reduce the frequency of face-detection(per 6 frames), keep extracting feature(per 2 frames).

ex) one cycle
frame 1 Face detection, feature extraction
frame 2 -
frame 3 Feature extraction
frame 4 -
frame 5 Feature extraction

I completely know this idea can boost the speed temporarily.
But, fundamental problem of this app is detecting every N frames. N-1 frames occurs false positive.
I think changing internal algorithm has a limit. So, I'm working hard to boost the speed using external resources (GPU, FPU core, Deep learning based modeling)

Anyway, Thank you so much for your interest.
When I upgrade the app, I'll let you know first.