ku-cvlab/dreammatcher

Official implementation of "DreamMatcher: Appearance Matching Self-Attention for Semantically-Consistent Text-to-Image Personalization" (CVPR 2024)

DreamMatcher: Appearance Matching Self-Attention for Semantically-Consistent Text-to-Image Personalization (CVPR'24)

This is the official implementation of the paper "DreamMatcher: Appearance Matching Self-Attention for Semantically-Consistent Text-to-Image Personalization" by Jisu Nam, Heesu Kim, DongJae Lee, Siyoon Jin, Seungryong Kim†, and Seunggyu Chang†.

For more information, check out the [project page].

Environment Settings

git clone https://github.com/KU-CVLAB/DreamMatcher.git
cd DreamMatcher

conda env create -f environment.yml
conda activate dreammatcher
pip install -r requirements.txt

cd diffusers
pip install -e .

Pre-trained Weights

You can run DreamMatcher with any off-the-shelf personalized models. We provide pre-trained personalized models of Textual Inversion, DreamBooth, and CustomDiffusion on the ViCo dataset. You can find the pre-trained weights on Link.

Dataset

Image Dataset

For a fair and unbiased evaluation, we used the ViCo image and prompt dataset gathered from Textual Inversion, DreamBooth, and CustomDiffusion. The dataset comprises 16 concepts, including 5 live objects and 11 non-live objects. In the ./inputs folder, you can see 4-12 images of each of the 16 concepts.

Prompt Dataset

We provide the ViCo prompt dataset for live objects in ./inputs/prompts_live_objects.txt and for non-live objects in ./inputs/prompts_nonlive_objects.txt.

Additionally, for evaluation in more complex scenarios, we propose the challenging prompt dataset, which is available in ./inputs/prompts_live_objects_challenging.txt and ./inputs/prompts_nonlive_objects_challenging.txt.

Run DreamMatcher

To run DreamMatcher, select a personalized model from "ti", "dreambooth", or "custom_diffusion" as the baseline. Below, we provide example code using "dreambooth" as the baseline, with 8 samples. Output images for both the baseline and DreamMatcher will be saved in the result directory.

Run DreamMatcher on the ViCo prompt dataset:

  python run_dreammatcher.py --models "dreambooth" --result_dir "./results/dreambooth/test" --num_samples 8 --num_device 0 --mode "normal"

Run DreamMatcher on the proposed challenging prompt dataset:

  python run_dreammatcher.py --models "dreambooth" --result_dir "./results/dreambooth/test" --num_samples 8 --num_device 0 --mode "challenging"

Evaluation

To evaluate DreamMatcher, specify the result directory containing the result images from both the baseline and DreamMatcher. I_CLIP, I_DINO, and T_CLIP metrics will be calculated.

Evaluation on the ViCo prompt dataset :

  python evaluate.py --result_dir "./results/dreambooth/test" --mode "normal"

Evaluation on the proposed challenging prompt dataset :

  python evaluate.py --result_dir "./results/dreambooth/test" --mode "challenging"

Collect Evaluation Results

Collect evaluation results for every concept in the result directory:

  python collect_results.py --result_dir "./results/dreambooth/test"

Results

Qualitative comparision with baselines for live objects:

Qualitative comparision with baselines for non-live objects:

Qualitative comparison with previous works for live objects:

Qualitative comparison with previous works for non-live objects:

Acknowledgement

We have mainly borrowed code from the public project MasaCtrl. A huge thank you to the authors for their valuable contributions.

BibTeX

If you find this research useful, please consider citing:

@misc{nam2024dreammatcher,
      title={DreamMatcher: Appearance Matching Self-Attention for Semantically-Consistent Text-to-Image Personalization}, 
      author={Jisu Nam and Heesu Kim and DongJae Lee and Siyoon Jin and Seungryong Kim and Seunggyu Chang},
      year={2024},
      eprint={2402.09812},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

dreammatcher's Issues

Can I directly apply it on a common diffusion model, rather than a trained personalized models

It seem that DreamMatcher must be apply in a personlizing model?

If I apply it on some common model, such as original sd1.5 or sdxl, can it still work?
very look forward to your reply

About modifications diffusers version

Thank you for the great work on this project. In relation to that, I am looking to ascertain the specific commit of the diffusers library that the set of modifications is based on. My aim is to examine the crucial code changes to understand the extent of modifications and explore possibilities for simplification.

Could you share the commit hash about the base diffusers? Thank you for your time and assistance.

I want to know the reason for you to use the attn map correspond to "cat_statue" not the word "sks" for dreambooth

Question about support XL

First of all, thank you very much for conducting research on such an excellent topic.

It seems that this paper is based on training with stable diffusion 1.5.

Do you have plans to train with the diffusion XL model in the future?

I am curious about the milestones for whether this project will have additional updates in the future.

thank you!

Will the training code be released?

Love the work and would love to try training some models. Could I ask if there are plans to release the training code? Thank you!

Your code has some error.

It seem that you re-define this unet block's forward function, but you donot upload it

Could this be used for inpainting models?

Would love to see if it's possible to use this approach for inpainting areas using an image as reference

I try to generate consistent anime character using your method, but I got bad result.

I find your method will decrease image quality. first column is reference image, other column is output image according to different prompts.

Will there be scripts for getting the pretrained model weights?

Thanks for your awesome work and effort in making it open-source.
I am not familiar with this field, but I notice that there are personalized models. I guess getting them still needs some processing. It would be much easier for other researchers to follow your work if there are scripts of how to get them.

Thanks!