foivospar/arc2face

Arc2Face: A Foundation Model of Human Faces

Arc2Face: A Foundation Model of Human Faces

Foivos Paraperas Papantoniou¹   Alexandros Lattas¹   Stylianos Moschoglou¹

Jiankang Deng¹   Bernhard Kainz^1,2   Stefanos Zafeiriou¹

¹Imperial College London, UK
²FAU Erlangen-Nürnberg, Germany

This is the official implementation of Arc2Face, an ID-conditioned face model:

 ✅ that generates high-quality images of any subject given only its ArcFace embedding, within a few seconds
 ✅ trained on the large-scale WebFace42M dataset offers superior ID similarity compared to existing models
 ✅ built on top of Stable Diffusion, can be extended to different input modalities, e.g. with ControlNet

News/Updates

• [2024/04/12] 🔥 We add LCM-LoRA support for even faster inference (check the details below).
• [2024/04/11] 🔥 We release the training dataset on HuggingFace Datasets.
• [2024/03/31] 🔥 We release our demo for pose control using Arc2Face + ControlNet (see instructions below).
• [2024/03/28] 🔥 We release our Gradio demo on HuggingFace Spaces (thanks to the HF team for their free GPU support)!
• [2024/03/14] 🔥 We release Arc2Face.

Installation

conda create -n arc2face python=3.10
conda activate arc2face

# Install requirements
pip install -r requirements.txt

Download Models

1. The models can be downloaded manually from HuggingFace or using python:

from huggingface_hub import hf_hub_download

hf_hub_download(repo_id="FoivosPar/Arc2Face", filename="arc2face/config.json", local_dir="./models")
hf_hub_download(repo_id="FoivosPar/Arc2Face", filename="arc2face/diffusion_pytorch_model.safetensors", local_dir="./models")
hf_hub_download(repo_id="FoivosPar/Arc2Face", filename="encoder/config.json", local_dir="./models")
hf_hub_download(repo_id="FoivosPar/Arc2Face", filename="encoder/pytorch_model.bin", local_dir="./models")

2. For face detection and ID-embedding extraction, manually download the antelopev2 package (direct link) and place the checkpoints under models/antelopev2.

3. We use an ArcFace recognition model trained on WebFace42M. Download arcface.onnx from HuggingFace and put it in models/antelopev2 or using python:

hf_hub_download(repo_id="FoivosPar/Arc2Face", filename="arcface.onnx", local_dir="./models/antelopev2")

4. Then delete glintr100.onnx (the default backbone from insightface).

The models folder structure should finally be:

  . ── models ──┌── antelopev2
                ├── arc2face
                └── encoder

Usage

Load pipeline using diffusers:

from diffusers import (
    StableDiffusionPipeline,
    UNet2DConditionModel,
    DPMSolverMultistepScheduler,
)

from arc2face import CLIPTextModelWrapper, project_face_embs

import torch
from insightface.app import FaceAnalysis
from PIL import Image
import numpy as np

base_model = 'runwayml/stable-diffusion-v1-5'

encoder = CLIPTextModelWrapper.from_pretrained(
    'models', subfolder="encoder", torch_dtype=torch.float16
)

unet = UNet2DConditionModel.from_pretrained(
    'models', subfolder="arc2face", torch_dtype=torch.float16
)

pipeline = StableDiffusionPipeline.from_pretrained(
        base_model,
        text_encoder=encoder,
        unet=unet,
        torch_dtype=torch.float16,
        safety_checker=None
    )

You can use any SD-compatible schedulers and steps, just like with Stable Diffusion. By default, we use DPMSolverMultistepScheduler with 25 steps, which produces very good results in just a few seconds.

pipeline.scheduler = DPMSolverMultistepScheduler.from_config(pipeline.scheduler.config)
pipeline = pipeline.to('cuda')

Pick an image and extract the ID-embedding:

app = FaceAnalysis(name='antelopev2', root='./', providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])
app.prepare(ctx_id=0, det_size=(640, 640))

img = np.array(Image.open('assets/examples/joacquin.png'))[:,:,::-1]

faces = app.get(img)
faces = sorted(faces, key=lambda x:(x['bbox'][2]-x['bbox'][0])*(x['bbox'][3]-x['bbox'][1]))[-1]  # select largest face (if more than one detected)
id_emb = torch.tensor(faces['embedding'], dtype=torch.float16)[None].cuda()
id_emb = id_emb/torch.norm(id_emb, dim=1, keepdim=True)   # normalize embedding
id_emb = project_face_embs(pipeline, id_emb)    # pass through the encoder

Generate images:

num_images = 4
images = pipeline(prompt_embeds=id_emb, num_inference_steps=25, guidance_scale=3.0, num_images_per_prompt=num_images).images

LCM-LoRA acceleration

LCM-LoRA allows you to reduce the sampling steps to as few as 2-4 for super-fast inference. Just plug in the pre-trained distillation adapter for SD v1.5 and switch to LCMScheduler:

from diffusers import LCMScheduler

pipeline.load_lora_weights("latent-consistency/lcm-lora-sdv1-5")
pipeline.scheduler = LCMScheduler.from_config(pipeline.scheduler.config)

Then, you can sample with as few as 2 steps (and disable guidance_scale by using a value of 1.0, as LCM is very sensitive to it and even small values lead to oversaturation):

images = pipeline(prompt_embeds=id_emb, num_inference_steps=2, guidance_scale=1.0, num_images_per_prompt=num_images).images

Note that this technique accelerates sampling in exchange for a slight drop in quality.

Start a local gradio demo

You can start a local demo for inference by running:

python gradio_demo/app.py

Arc2Face + ControlNet (pose)

We provide a ControlNet model trained on top of Arc2Face for pose control. We use EMOCA for 3D pose extraction. To run our demo, follow the steps below:

1) Download Model

Download the ControlNet checkpoint manually from HuggingFace or using python:

from huggingface_hub import hf_hub_download

hf_hub_download(repo_id="FoivosPar/Arc2Face", filename="controlnet/config.json", local_dir="./models")
hf_hub_download(repo_id="FoivosPar/Arc2Face", filename="controlnet/diffusion_pytorch_model.safetensors", local_dir="./models")

2) Pull EMOCA

git submodule update --init external/emoca

3) Installation

This is the most tricky part. You will need PyTorch3D to run EMOCA. As its installation may cause conflicts, we suggest to follow the process below:

1. Create a new environment and start by installing PyTorch3D with GPU support first (follow the official instructions).
2. Add Arc2Face + EMOCA requirements with:

pip install -r requirements_controlnet.txt

3. Install EMOCA code:

pip install -e external/emoca

4. Finally, you need to download the EMOCA/FLAME assets. Run the following and follow the instructions in the terminal:

cd external/emoca/gdl_apps/EMOCA/demos 
bash download_assets.sh
cd ../../../../..

4) Start a local gradio demo

You can start a local ControlNet demo by running:

python gradio_demo/app_controlnet.py

Community Resources

Replicate Demo

• Demo link by @camenduru.

Pinokio

• Pinokio implementation by @cocktailpeanut (runs locally on all OS - Windows, Mac, Linux).

Acknowledgements

• Thanks to the creators of Stable Diffusion and the HuggingFace diffusers team for the awesome work ❤️.
• Thanks to the WebFace42M creators for providing such a million-scale facial dataset ❤️.
• Thanks to the HuggingFace team for their generous support through the community GPU grant for our demo ❤️.
• We also acknowledge the invaluable support of the HPC resources provided by the Erlangen National High Performance Computing Center (NHR@FAU) of the Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU), which made the training of Arc2Face possible.

Citation

If you find Arc2Face useful for your research, please consider citing us:

@misc{paraperas2024arc2face,
      title={Arc2Face: A Foundation Model of Human Faces}, 
      author={Foivos Paraperas Papantoniou and Alexandros Lattas and Stylianos Moschoglou and Jiankang Deng and Bernhard Kainz and Stefanos Zafeiriou},
      year={2024},
      eprint={2403.11641},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

arc2face's Issues

how to fix this

(venv) (base) F:\arc\Arc2Face-hf-main>python app.py
Traceback (most recent call last):
File "F:\arc\Arc2Face-hf-main\app.py", line 19, in
import spaces
File "F:\arc\Arc2Face-hf-main\venv\lib\site-packages\spaces_init_.py", line 10, in
from .zero.decorator import GPU
File "F:\arc\Arc2Face-hf-main\venv\lib\site-packages\spaces\zero\decorator.py", line 21, in
from .wrappers import regular_function_wrapper
File "F:\arc\Arc2Face-hf-main\venv\lib\site-packages\spaces\zero\wrappers.py", line 14, in
from multiprocessing.context import ForkProcess
ImportError: cannot import name 'ForkProcess' from 'multiprocessing.context' (C:\Users\ggrov\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py)

(venv) (base) F:\arc\Arc2Face-hf-main>

how use with normal prompt

like ip dapter

how to train

would you release the training code? thank you

How was the quality of the generations maintained

Given that the quality of output for portrait kind of images are not that great of stable diffusion 1.5, how does this model result into good realistic portrait quality ?

Experiments in the paper

Hello, thank you so much for the great work!
I noticed that there is not too much description of how experiments on previous works like InstantID are conducted. Would you mind sharing, were the InstantID experiments conducted using the publicly available SDXL version? Or were they based on the SD1.5 version? Were all methods compared under 512x512 resolution? Thank you.

Is it possible to train loras on this, to improve faces?

Can you confirm if other pretrained controlnets would be compatible with this model

I would assume that other pretrained controlnets would not be compatible with this model and would have to be finetuned. For example pose model would have to be finetuned.

the output is unexpected

thank you for you work.
but the output is unexpected.
I just try to run the demo code in README.md, input image: 'assets/examples/joacquin.png'

[Solved just sharing] Fixing the libcudnn_cnn_infer.so.8 Error

(I am not pro user, but someone barely maintaining the lab server from crashing. If my message messed up your server, sorry in advance!)

Hey there,
I recently ran into an issue after installing via conda and pip, and I wanted to share my solution in case anyone else encounters the same problem.

The Error
Could not load library libcudnn_cnn_infer.so.8. Error: libcuda.so: cannot open shared object file: No such file or directory

The Fix
After some research, I found that I needed to install nvidia-cudnn. Here's what worked for me:

1. Update your package list: Run sudo apt update to ensure you have the latest package information.
2. Install nvidia-cudnn: Execute sudo apt install nvidia-cudnn to install the required library.

Note on CUDA Version Compatibility
Keep in mind that CUDA version 545 might not be compatible with other dependencies. I recommend using CUDA version 535 instead.

Additional Tips

• Don't forget to reboot your system if prompted to do so after updating your drivers.
• If you encounter issues with NVML, make sure both the driver and NVML are updated to the same version (in this case, 535).

Hope this helps!

Abalation study with training with the unet frozen

Has there been any ablation study to understand the impact of freezing the unet and only training the clip encoder.

My understanding here is that if the unet is fixed this technique could be applied with other merged/lora based unets.

Error when loading insight face model

app = FaceAnalysis(name='antelopev2', root='./', providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])
When the above code runs, it produces the following error:

AssertionError Traceback (most recent call last)
in <cell line: 2>()

----> 2 app = FaceAnalysis(name='antelopev2', root='./', providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])

/usr/local/lib/python3.10/dist-packages/insightface/app/face_analysis.py in init(self, name, root, allowed_modules, **kwargs)
41 print('duplicated model task type, ignore:', onnx_file, model.taskname)
42 del model
---> 43 assert 'detection' in self.models
44 self.det_model = self.models['detection']
45

AssertionError:

How to use it in COMFYUI or WEBUI?

Does it support SD1.5 or SDXL?Now the face change tool in the AI drawing program is very troublesome to install, is this simple enough to use ?

I input the image with FFHQ1024 resolution into antelopev2 and it doesn't detect the face

any demo to use control net?

how to keep the background in generated image?

Using without models from insightface?

Hi,
is there any way to train this and without using the models from insightface, since they have a really restrictive license...

Great work! Can provide pytorch weights of arcface not only onnx?

Hi, thank you so much for the great work!
The arcface provied in [https://huggingface.co/FoivosPar/Arc2Face/tree/main] is onnx format, can you provide the corresponding pytorch weights ?