Adding this repo as deepfakes/faceswap plugin

[dfaker]

Adding this repo as deepfakes/faceswap plugin

[dfaker]

@Clorr, sorry for the delay, Yes this sounds good. Were you thinking about merging in just the enlarged model, the alpha masking, or the full warping stuff?

This cross-cuts so many of the modules and at the same time is incompatible with the expected input "shapes" of the existing modes. Do you think it fits well with the current plugin system or a interface based refactor with modules broadcasting their input and output shapes makes sense?

[Clorr]

Hey no problem, glad to hear again from you!

I started the integration in deepfakes/faceswap#271 , however I did not started converting so I hope it is still doable ;-)

Reading alignments.json+cropped was a bit of a pain and should still be refactored a bit. I thought I could do a new Extract plugin with the code in your utils.py but the numpy.dstack outputs something I can't save as an image file.... so it is in a specific TrainingDatagenerator that loads images and does this preprocessing.

Maybe you can help me here, because if we can generate a savable/loadable image from that code I would prefer so that we can save time on train start and also check that data passed to trainer is correct.

For the convert I wanted to start it today, so for now i don't have a specific question. If you have some time to check my draft PR, I would highly appreciate ;-)

[dfaker]

You mean being able to read in a set of images that match the output type of load_image to speed up initialization? They could be saved as png format and loaded back in with:

cv2.imread("image.png", cv2.IMREAD_UNCHANGED)

when viewed the images would look like disembodied faces due to the alpha masking but I suppose that's accurate!

Or there's always np.save/np.load on the array but that loses the benefit of being able to visually inspect them.

[Clorr]

In one word: Awesome! I don't know if it is hard to modify extract to save pngs in the proper format, but it sounds simple... (And yes I prefer we have a visualizable image)

I'm mostly done for the 1st draft of convert. Have you noticed something else?

[dfaker]

The pngs just need to be in order BGRA which they are already coming out of your load_image.

The training and image extraction looks fine as far as I can see.

[Clorr]

Ok, if you see what's to be done, feel free to push something ;-) Otherwise I will try later on. For now I pushed a convert so you can have a look...

Result is a bit rough, but I trained for just a couple of minutes :-D

Note that your model files are still compatible if you want to try...

[dfaker]

Very cool, looks like you're not using the alpha output of the network?

I'll dig some old models out and give it a go a go when I'm back at my dev machine.

[Clorr]

Hmm, not sure, maybe I missed something. There was some dead code because transfer_avg_colorwas not used and even here in the repo i can't find a ref to it. So there is possibly a problem...
Note that I used convert_face_larger and not the code from convert_face_video

[dfaker]

Nah, transfer_avg_color is genuinely not used.

Looks like a perfectly square bounding box, the network should output two images, an RGB face and a B/W alpha channel (new_face_m then transformed into image_mask) which should restrict the area output to just the core face avoiding changing the areas outside the face.

The code in Convert_DFaker looks fine at a glance though, it could be that it's just the short training causing it.

[Clorr]

Do you have any theoritical background behind your model? Is there any related paper that I could read about?

[dfaker]

Not really, it's all trial, error and experimentation other than the core AE pattern from deepfakes and the residual block function from shaoanlu.

[dfaker]

Closing for now as the discussion is much more lively over on the PR