Does the frontalization work with most image sizes, head orientations?

[iacopomasi]

Does the frontalization work with most image sizes, head orientations?

[TalHassner]

@dshahrokhian (just saw this after responding to your question in the frontalization page)

I expect it to fail on the same images, though I may be wrong and it's worth trying. You should also try resizing the photos so that faces are roughly the same size as those in the LFW, YTF benchmarks, where this code was mostly used.

Regardless, as I mentioned there, we are working on providing a landmark-free solution.

[iacopomasi]

Hey @dshahrokhian,
good questions, thanks. The issues that you mentioned are two and are unrelated each other.

1. The first issue regards rendering (interpolating) when the source, input image has a very big size. This is a technical issue, not a research issue. I also got into this problem and the solution was to resize the image to a canonical, smaller size. (at the end also this is what the rendered will do as it won't preserve the original, input resolution). I do not think is a problem on our side. To solve this, what we should do is study the OpenCV internals of the function cv2.remap() because, at the end, that does the job. It could be an OpenCV issue. I haven't had the time to look into that.

2. The second issue regards "frontalization" when the input face undergoes large pose variants far apart from frontal. This is actually one of the main reasons why this rendered was developed, that is to avoid frontalizing a face when the face is near profile. It is better to adjust the pose locally, avoiding "big jumps" in pose angle (e.g. 50-degree face brought to frontal is likely to fail). What I have seen in my research is that makes sense if you have a good frontal face to warp it to frontal, half-profile view and even make a stretch to full-profile (~70 degrees). Although in this latter case, you might get some artifacts in the ears region, overall the quality is acceptable. What it is not recommended is to take a face which is profile and frontalize it. In the configuration file confi.ini there is an option nearView = yes that enables this. That is, deciding which view to render based on the input pose. The problem that you mentioned in the question was circumnavigated in this way, but not really solved. So, to answer your questions, if you force the code to frontalize a near-profile image is not going to work. All these also assume that you got a clean response from the landmark detector, which might not be the case; perhaps you need a confidence on the landmarks to tell you how good is landmark prediction, so to avoid hard images.

Also as @TalHassner mentioned, we are working to provide a fast solution "landmark-free" which also gives you a sort of confidence in the pose prediction.

[dshahrokhian]

Thanks for the feedback @TalHassner @iacopomasi.