toolwiz/sync-face-detection-and-replacement

Sync Face Detection And Replacement

Sync-Face-Detection-And-Replacement

Sync Face Detection And Replacement

The goal of this project is automatic detection and replacement of faces in videos. Given a test video, you will automatically replace a target face. Ideally, you should do this in the most stable way possible.

Seamlessly face replacement is a non-trivial process. For the rest of this document, we shall use the fol- lowing terminology: face(s) in the video = target face(s), face(s) used for replacement = replacement face(s). First, you will build a model of the replacement face. You can use a single, well aligned photo, or a sequence under different poses. You will have to devise a scheme to be able to detect and localize similar looking faces in video frames. One way to do this is with feature detection, matching, and voting. Once you detect a face in a test image, you must estimate the warping between the candidate replacement face(s) and the detected candidate target face(s). We could suggest extending previously explored concepts such as sim- ple affine, warp, or you can allow additional deformity using TPS. You are of course not limited to these and are encouraged to experiment with any other suitable approaches you may find. If you are using mul- tiple faces for training, you should match against all of them and use the one that requires the least morphing. Appearance matching may be necessary to compensate for shadows, lighting and skin tone. The target face will need to be cropped out, and a morph of the appropriate replacement should be inserted in its place, and the final video frame blended seamlessly. When dealing with videos, you might expect the faces in the video to move. For extra credit, you are also encouraged to explore methods for motion compensation.