junyi42 / sd-dino Goto Github PK
View Code? Open in Web Editor NEWOfficial Implementation of paper "A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence"
Home Page: https://sd-complements-dino.github.io
Official Implementation of paper "A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence"
Home Page: https://sd-complements-dino.github.io
Hi Junyi,
Could you please provide the code for Object Swapping (with refinement process), current object swapping result seems blur
Hello,I'm sorry to bother you again. I've encountered a version issue. My TensorFlow and Keras versions are 2.13.1, and I'm getting this error. Could you please let me know the Keras version requirements for this code? I couldn't find any helpful answers online, and despite using a global search, I haven't found any occurrences of the "is_tensor" function in the code.
Thanks!
Hi, thanks for sharing the codes.
I found a problem when running the demo codes. I followed all the setup in readme without changing anything, but it seems the download pre-trained weights mismatch the model:
so I got the results which are very different from yours:
This problem also occurs when I run Geoaware-SC. Could you give me some advice on how to solve this?
Hi,
Thanks for this awesome work! 🤩
The image resolution is 518 in the Dinov2-pretrained-model, why you can use the 840.
looking forward your reply.
Hello @Junyi42 , Thanks for your contribution. I ran the "demo_vis_features.ipynb on the dog that was given in the default image folder. My results are coming different than yours. Yours masked pca result was
Also, my clustering is
I didn't change anything in the code only dumped everything from the ipynb to .py file and I am getting these outputs in the results_vis folder in the form of png files.
Hello @Junyi42 ,
Thanks for your contribution. I am facing the an installation issue when running the "pip install -e ." command. This is giving the error as follows:
Emitting ninja build file /BS/keytr_neus/work/supplementary/sd-dino/third_party/Mask2Former/build/temp.linux-x86_64-cpython-39/build.ninja...
error: [Errno 2] No such file or directory: '/BS/keytr_neus/work/supplementary/sd-dino/third_party/Mask2Former/build/temp.linux-x86_64-cpython-39/build.ninja'
ERROR: Failed building wheel for mask2former
ERROR: Could not build wheels for mask2former, which is required to install pyproject.toml-based projects
Please help me in this
Thank you for the amazing work! I am trying to visualize the feature maps for dino and SD. Do you have a collab notebook, that I can use to run it?
Hi!
when running the demo,
src_img_path = "data/images/dog_00.jpg"
trg_img_path = "data/images/dog_59.jpg"
result = process_images(src_img_path, trg_img_path)
I found that the get_mask function cannot return a valid mask but an all-1 matrix. Is this a bug?
if DRAW_DENSE:
if not Anno:
mask1 = get_mask(model, aug, img1, category[0])
mask2 = get_mask(model, aug, img2, category[-1])
Hello Junyi, GREAT JOB! It seems that everything works well when calling get_features
in extractor_sd.py
using cuda:3
but the inference process failed even I change
def inference(model, aug, image, vocab, label_list):
from
demo = StableDiffusionSeg(inference_model, demo_metadata, aug)
pred = demo.predict(np.array(image))
to
demo = StableDiffusionSeg(inference_model, demo_metadata, aug)
demo.model = demo.model.to(torch.device("cuda:3"))
pred = demo.predict(np.array(image))
I guess the main problem lies in wrongly loading the decoder part of the model, but I'm not sure how to fix it.
Hi,
Thanks for this awesome work! 🤩
DINO and StableDiffusion works have MIT licenses. Is your work also MIT?
Best,
Iago.
https://github.com/Junyi42/sd-dino/blob/48278d9c3c1cc2386ca08438d527a35dff902c9d/extractor_sd.py#L260C1-L269C100
I guess the dimensions of tensor
are [N, HW, C], I don’t know why transformed_tensor
here only transforms tensor[0]
and discards other samples, so that new features will only have one sample features, i.e., [1, HW, C].
RuntimeError: Panoptic/odise_label_coco_50e.py not available in Model Zoo!
would you mind to telling me how to fix the issue.
Hi Junyi,
I am confused about how to extract sd features. Actually the file extractor_sd.py seems to output a feature in shape of [1, 1280, 16, 16] without obvious semantic information. And it seems to use the model weights from project ODISE. Could you please provide a script to easily extract and visualize the sd features using publicly available stable diffusion model weights? Thanks a lot!
Hello, I would like to know whether the 2, 5, 8-layer features mentioned in the paper refer to the actual 2, 5, 8 layers or the layers after processing with the UpSample block. Does it mean the results obtained after the UpSample block processing? I find it a bit challenging to understand the feature extraction in the code. I hope to receive your reply. Thank you!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.