idea-research/dreamwaltz

[NeurIPS 2023] Official implementation of the paper "DreamWaltz: Make a Scene with Complex 3D Animatable Avatars".

dreamwaltz's Issues

how to get 512 resolution results and how to extract mesh Thanks

nice job!
1.how to get 512 resolution results
2.how to extract mesh
Thanks

[bug] Train a personalized SD1.5 model by DreamBooth Lora. Execute stage 1, but there is no personalized effect

Great job.
my code backbone：threestudio-dreamwaltz
I want to try civitai model, as you mainly mentioned make personalized avatar by load lora weight, but I found it doesn't work,

# guidance/multi_controlnet_guidance.py
# when I download or train a lora weight, my calling method refers to diffusers method as shown below
self.pipe = StableDiffusionControlNetPipeline.from_pretrained(
            self.cfg.pretrained_model_name_or_path,
            **pipe_kwargs,
        ).to(self.device)

# after
self.pipe.load_lora_weights(self.cfg.lora_weights_path",weight_name="pytorch_lora_weights.safetensors")

To verify whether lora is effective, you can test it through the following code

from diffusers import StableDiffusionControlNetPipeline, ControlNetModel, UniPCMultistepScheduler
from diffusers.utils import load_image
import numpy as np
import torch

import cv2
from PIL import Image

# download an image
image = load_image(
    "https://hf.co/datasets/huggingface/documentation-images/resolve/main/diffusers/input_image_vermeer.png"
)
image = np.array(image)

# get canny image
image = cv2.Canny(image, 100, 200)
image = image[:, :, None]
image = np.concatenate([image, image, image], axis=2)
canny_image = Image.fromarray(image)

# load control net and stable diffusion v1-5
controlnet = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny", torch_dtype=torch.float16)
pipe = StableDiffusionControlNetPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5", controlnet=controlnet, torch_dtype=torch.float16
)
pipe.load_lora_weights("pretrained_models/s15_girl_character_lora_v3_c1500",weight_name="pytorch_lora_weights.safetensors")
# speed up diffusion process with faster scheduler and memory optimization
pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)
# remove following line if xformers is not installed
pipe.enable_xformers_memory_efficient_attention()

pipe.enable_model_cpu_offload()

# generate image
generator = torch.manual_seed(0)
image = pipe(
    "a zoomed out DSLR photo of sks female anime character", num_inference_steps=200, generator=generator, image=canny_image
).images[0]
image.save("infer_test_lora.png")

The result is as follows, it is valid

pytorch_lora_weights.zip

By the way, I found it works when I didn't use lora finetune sd, but full finetune sd。

Given a particular SMPL pose and a camera location, can I get the 2d projection of it like openpose figures ?

@Yukun-Huang Also I am very new to this 3D domain so I am not sure of the terminologies and how these values are actually represented in code.

Training Time

How many gpu hours are needed to train your model on a single NVIDIA 3090?

Program stuck

When I was running DreamWaltz, the program got stuck. After debugging, I found that the code was stuck in the red box. Do you know what the reason is?

When will the code be released?

Thanks for your great job, I would like to know When will the code be released?

Could I just use DreamWaltz weights to infer, without training steps in the readme?

Hi, I think the results are amazing, but the training steps are annoying. So could I skip it and just do inference?

Where do I get the pretrained ControlNet?

Thanks for your great work! I met a problem when I want to download the pretrained model. I can not find the below models on the Hugging Face.

MODEL_CARDS = {
    'pose': "fusing/stable-diffusion-v1-5-controlnet-openpose",
    'depth': "fusing/stable-diffusion-v1-5-controlnet-depth",
    'canny': "fusing/stable-diffusion-v1-5-controlnet-canny",
    'seg': "fusing/stable-diffusion-v1-5-controlnet-seg",
    'normal': "fusing/stable-diffusion-v1-5-controlnet-normal",
}

SMPL Initialization

Why is the depth map used to initialize a NeRF of canonical SMPL? Use color map rendered from mesh will be efficient.

Supporting multiple GPU running?

Amazing work! Is this network Supports multi-GPU computing？

[misc] support High-Resolution Canonical Avatars && image-to-avatars

Hello, great job on your work! I have a couple of questions regarding it:

1. Can the current codebase support High-Resolution Canonical Avatars? What is its computational overhead compared to 64x64 avatars?

2. Can the current task accommodate image-to-avatar conversion?