junyi42/sd-dino

Official Implementation of paper "A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence"

sd-dino's Issues

Object Swapping (with refinement process)

Hi Junyi,
Could you please provide the code for Object Swapping (with refinement process), current object swapping result seems blur

Establish environment

Hello, I am very interested in your work, but I encountered some difficulties when setting up the environment. I followed the steps in the README, but there seems to be some problem somewhere, and I don't know how to fix it.

AttributeError: module 'keras.backend' has no attribute 'is_tensor'

Hello,I'm sorry to bother you again. I've encountered a version issue. My TensorFlow and Keras versions are 2.13.1, and I'm getting this error. Could you please let me know the Keras version requirements for this code? I couldn't find any helpful answers online, and despite using a global search, I haven't found any occurrences of the "is_tensor" function in the code.
Thanks!

Model parameter mismatch

Hi, thanks for sharing the codes.

I found a problem when running the demo codes. I followed all the setup in readme without changing anything, but it seems the download pre-trained weights mismatch the model:

so I got the results which are very different from yours:

This problem also occurs when I run Geoaware-SC. Could you give me some advice on how to solve this?

ValueError: a must be greater than 0 unless no samples are taken

Hi, I met the problem when I run pck_spair_pascal.py
Would you mind to telling me how to fix the issue?
Thanks!

The image size for extracting the Dinov2-pretrained-model

Hi,

Thanks for this awesome work! 🤩

The image resolution is 518 in the Dinov2-pretrained-model, why you can use the 840.

looking forward your reply.

Result different from demo_vis_features.ipynb

Hello @Junyi42 , Thanks for your contribution. I ran the "demo_vis_features.ipynb on the dog that was given in the default image folder. My results are coming different than yours. Yours masked pca result was

while I am getting

Also, my clustering is

I didn't change anything in the code only dumped everything from the ipynb to .py file and I am getting these outputs in the results_vis folder in the form of png files.

Installation issues for Mask Former

Hello @Junyi42 ,
Thanks for your contribution. I am facing the an installation issue when running the "pip install -e ." command. This is giving the error as follows:

Emitting ninja build file /BS/keytr_neus/work/supplementary/sd-dino/third_party/Mask2Former/build/temp.linux-x86_64-cpython-39/build.ninja...

error: [Errno 2] No such file or directory: '/BS/keytr_neus/work/supplementary/sd-dino/third_party/Mask2Former/build/temp.linux-x86_64-cpython-39/build.ninja'

ERROR: Failed building wheel for mask2former

ERROR: Could not build wheels for mask2former, which is required to install pyproject.toml-based projects

Please help me in this

Collab Demo

Thank you for the amazing work! I am trying to visualize the feature maps for dino and SD. Do you have a collab notebook, that I can use to run it?

get_mask cannot return valid mask

Hi!
when running the demo,

src_img_path = "data/images/dog_00.jpg"
trg_img_path = "data/images/dog_59.jpg"
result = process_images(src_img_path, trg_img_path)

I found that the get_mask function cannot return a valid mask but an all-1 matrix. Is this a bug?

if DRAW_DENSE:
                if not Anno:
                    mask1 = get_mask(model, aug, img1, category[0])
                    mask2 = get_mask(model, aug, img2, category[-1])

cannot `get_mask` when I vary the cuda device

Hello Junyi, GREAT JOB! It seems that everything works well when calling get_features in extractor_sd.py using cuda:3
but the inference process failed even I change
def inference(model, aug, image, vocab, label_list):
from
demo = StableDiffusionSeg(inference_model, demo_metadata, aug)

pred = demo.predict(np.array(image))
to
demo = StableDiffusionSeg(inference_model, demo_metadata, aug)

demo.model = demo.model.to(torch.device("cuda:3"))

pred = demo.predict(np.array(image))

I guess the main problem lies in wrongly loading the decoder part of the model, but I'm not sure how to fix it.

License?

Hi,

Thanks for this awesome work! 🤩

DINO and StableDiffusion works have MIT licenses. Is your work also MIT?

Best,
Iago.

Code for PCA

https://github.com/Junyi42/sd-dino/blob/48278d9c3c1cc2386ca08438d527a35dff902c9d/extractor_sd.py#L260C1-L269C100
I guess the dimensions of tensor are [N, HW, C], I don’t know why transformed_tensor here only transforms tensor[0] and discards other samples, so that new features will only have one sample features, i.e., [1, HW, C].

how to fix the issue of 'RuntimeError: Panoptic/odise_label_coco_50e.py not available in Model Zoo!'

RuntimeError: Panoptic/odise_label_coco_50e.py not available in Model Zoo!
would you mind to telling me how to fix the issue.

Details about how to extract sd features

Hi Junyi,

I am confused about how to extract sd features. Actually the file extractor_sd.py seems to output a feature in shape of [1, 1280, 16, 16] without obvious semantic information. And it seems to use the model weights from project ODISE. Could you please provide a script to easily extract and visualize the sd features using publicly available stable diffusion model weights? Thanks a lot!

Questions about sd features

Hello, I would like to know whether the 2, 5, 8-layer features mentioned in the paper refer to the actual 2, 5, 8 layers or the layers after processing with the UpSample block. Does it mean the results obtained after the UpSample block processing? I find it a bit challenging to understand the feature extraction in the code. I hope to receive your reply. Thank you!