[ICLR 2024] Official implementation of DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior
Home Page: https://mrtornado24.github.io/DreamCraft3D/
License: MIT License
Hi!
Thank you for the amazing work and the code release!
I tried running one of the default examples and noticed a difference in the back side of the object when comparing it against the spin from the project page:
Spin generated from code base:
Spin from the project page:
Can you provide some tips on how to match the texture and geometry quality of the spins from the project page?
Hi!
Very cool work! It would be nice to create a demo on the Hugging Face Hub!
This is a step-by-step guide explaining the process in case you're interested.
Please let us know if you would be interested and if you have any questions. ๐
Hey,
i try to setup the model on an Ubuntu 22.04 with an RTX 4090.
After running
python launch.py --config configs/dreamcraft3d-coarse-nerf.yaml --train system.prompt_processor.prompt="$prompt" data.image_path="$image_path"with following image:
i get this output resulting with this error : RuntimeError: Predictions and targets are expected to have the same shape, but got torch.Size([16384]) and torch.Size([16384, 3]).
Here is the whole log of the process
python launch.py --config configs/dreamcraft3d-coarse-nerf.yaml --train system.prompt_processor.prompt="$prompt" data.image_path="$image_path"
/root/DreamCraft3D/venv/lib/python3.10/site-packages/requests/__init__.py:102: RequestsDependencyWarning: urllib3 (1.26.13) or chardet (5.2.0)/charset_normalizer (2.0.12) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported "
/root/DreamCraft3D/venv/lib/python3.10/site-packages/controlnet_aux/mediapipe_face/mediapipe_face_common.py:7: UserWarning: The module 'mediapipe' is not installed. The package will have limited functionality. Please install it using the command: pip install 'mediapipe'
warnings.warn(
/root/DreamCraft3D/venv/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_5m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_5m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
return register_model(fn_wrapper)
/root/DreamCraft3D/venv/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_11m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_11m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
return register_model(fn_wrapper)
/root/DreamCraft3D/venv/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
return register_model(fn_wrapper)
/root/DreamCraft3D/venv/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_384 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_384. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
return register_model(fn_wrapper)
/root/DreamCraft3D/venv/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_512 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_512. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
return register_model(fn_wrapper)
WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for:
PyTorch 2.1.2+cu121 with CUDA 1201 (you have 2.1.2+cu118)
Python 3.10.13 (you have 3.10.12)
Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers)
Memory-efficient attention, SwiGLU, sparse and more won't be available.
Set XFORMERS_MORE_DETAILS=1 for more details
Seed set to 0
[INFO] Loading Deep Floyd ...
A mixture of fp16 and non-fp16 filenames will be loaded.
Loaded fp16 filenames:
[unet/diffusion_pytorch_model.fp16.safetensors, safety_checker/model.fp16.safetensors, text_encoder/model.fp16-00002-of-00002.safetensors, text_encoder/model.fp16-00001-of-00002.safetensors]
Loaded non-fp16 filenames:
[watermarker/diffusion_pytorch_model.safetensors
If this behavior is not expected, please check your folder structure.
Loading pipeline components...: 33%|โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ | 1/3 [00:00<00:00, 7.68it/s]You are using the default legacy behaviour of the <class 'transformers.models.t5.tokenization_t5.T5Tokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thouroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565
Loading pipeline components...: 100%|โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ| 3/3 [00:00<00:00, 10.42it/s]
[INFO] Loaded Deep Floyd!
[INFO] Loading Stable Zero123 ...
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.53 M params.
Keeping EMAs of 688.
making attention of type 'vanilla' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla' with 512 in_channels
/root/DreamCraft3D/inputimges/falcon.jpg
[INFO] Loaded Stable Zero123!
[INFO] Using prompt [a minimalsitic marbel stone statue of an owl] and negative prompt []
[INFO] Using view-dependent prompts [side]:[a minimalsitic marbel stone statue of an owl, side view] [front]:[a minimalsitic marbel stone statue of an owl, front view] [back]:[a minimalsitic marbel stone statue of an owl, back view] [overhead]:[a minimalsitic marbel stone statue of an owl, overhead view]
/root/DreamCraft3D/venv/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
warnings.warn(
/root/DreamCraft3D/venv/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
warnings.warn(msg)
loaded pretrained LPIPS loss from threestudio/utils/lpips/vgg.pth
[INFO] Using 16bit Automatic Mixed Precision (AMP)
[INFO] GPU available: True (cuda), used: True
[INFO] TPU available: False, using: 0 TPU cores
[INFO] IPU available: False, using: 0 IPUs
[INFO] HPU available: False, using: 0 HPUs
[INFO] You are using a CUDA device ('NVIDIA GeForce RTX 4090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
[INFO] single image dataset: load image /root/DreamCraft3D/inputimges/falcon.jpg torch.Size([1, 128, 128, 3])
[INFO] single image dataset: load depth /root/DreamCraft3D/inputimges/falcon.jpg torch.Size([1, 128, 128, 3])
[INFO] single image dataset: load image /root/DreamCraft3D/inputimges/falcon.jpg torch.Size([1, 128, 128, 3])
[INFO] single image dataset: load depth /root/DreamCraft3D/inputimges/falcon.jpg torch.Size([1, 128, 128, 3])
[INFO] LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
[INFO]
| Name | Type | Params
----------------------------------------------------
0 | geometry | ImplicitVolume | 12.6 M
1 | material | NoMaterial | 0
2 | background | SolidColorBackground | 0
3 | renderer | NeRFVolumeRenderer | 0
----------------------------------------------------
12.6 M Trainable params
0 Non-trainable params
12.6 M Total params
50.417 Total estimated model params size (MB)
[INFO] Validation results will be saved to outputs/dreamcraft3d-coarse-nerf/a_minimalsitic_marbel_stone_statue_of_an_owl@20231222-190508/save
/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:441: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=31` in the `DataLoader` to improve performance.
/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:441: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=31` in the `DataLoader` to improve performance.
Epoch 0: | | 0/? [00:00<?, ?it/s]tensor([[0.6902, 0.6902, 0.7137],
[0.6941, 0.6941, 0.7176],
[0.6941, 0.6941, 0.7176],
...,
[0.8471, 0.8549, 0.8549],
[0.8314, 0.8392, 0.8392],
[0.8275, 0.8353, 0.8353]], device='cuda:0')
tensor([0., 0., 0., ..., 0., 0., 0.], device='cuda:0',
grad_fn=<IndexBackward0>)
Traceback (most recent call last):
File "/root/DreamCraft3D/launch.py", line 252, in <module>
main(args, extras)
File "/root/DreamCraft3D/launch.py", line 195, in main
trainer.fit(system, datamodule=dm, ckpt_path=cfg.resume)
File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 544, in fit
call._call_and_handle_interrupt(
File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 44, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 580, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 989, in _run
results = self._run_stage()
File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1035, in _run_stage
self.fit_loop.run()
File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 202, in run
self.advance()
File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 359, in advance
self.epoch_loop.run(self._data_fetcher)
File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 136, in run
self.advance(data_fetcher)
File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 240, in advance
batch_output = self.automatic_optimization.run(trainer.optimizers[0], batch_idx, kwargs)
File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 187, in run
self._optimizer_step(batch_idx, closure)
File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 265, in _optimizer_step
call._call_lightning_module_hook(
File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 157, in _call_lightning_module_hook
output = fn(*args, **kwargs)
File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/core/module.py", line 1291, in optimizer_step
optimizer.step(closure=optimizer_closure)
File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/core/optimizer.py", line 151, in step
step_output = self._strategy.optimizer_step(self._optimizer, closure, **kwargs)
File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/strategies/strategy.py", line 230, in optimizer_step
return self.precision_plugin.optimizer_step(optimizer, model=model, closure=closure, **kwargs)
File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/plugins/precision/amp.py", line 77, in optimizer_step
closure_result = closure()
File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 140, in __call__
self._result = self.closure(*args, **kwargs)
File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 126, in closure
step_output = self._step_fn()
File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 315, in _training_step
training_step_output = call._call_strategy_hook(trainer, "training_step", *kwargs.values())
File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 309, in _call_strategy_hook
output = fn(*args, **kwargs)
File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/pytorch_lightning/strategies/strategy.py", line 382, in training_step
return self.lightning_module.training_step(*args, **kwargs)
File "/root/DreamCraft3D/threestudio/systems/dreamcraft3d.py", line 375, in training_step
out = self.training_substep(batch, batch_idx, guidance="ref", render_type=render_type)
File "/root/DreamCraft3D/threestudio/systems/dreamcraft3d.py", line 176, in training_substep
"depth_rel", 1 - self.pearson(valid_pred_depth, valid_gt_depth)
File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/torchmetrics/metric.py", line 296, in forward
self._forward_cache = self._forward_full_state_update(*args, **kwargs)
File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/torchmetrics/metric.py", line 311, in _forward_full_state_update
self.update(*args, **kwargs)
File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/torchmetrics/metric.py", line 470, in wrapped_func
raise err
File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/torchmetrics/metric.py", line 460, in wrapped_func
update(*args, **kwargs)
File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/torchmetrics/regression/pearson.py", line 146, in update
self.mean_x, self.mean_y, self.var_x, self.var_y, self.corr_xy, self.n_total = _pearson_corrcoef_update(
File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/torchmetrics/functional/regression/pearson.py", line 53, in _pearson_corrcoef_update
_check_same_shape(preds, target)
File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/torchmetrics/utilities/checks.py", line 42, in _check_same_shape
raise RuntimeError(
RuntimeError: Predictions and targets are expected to have the same shape, but got torch.Size([16384]) and torch.Size([16384, 3]).Hi!
Thank you for the amazing work!
Can you share more information on the compute requirements. How many GPUs did you use for the Geometry Sculpting phase and also for the Texture Boosting phase?
What was the memory size of the GPU that was required?
Thank you!
When running the first process, python launch.py --config configs/dreamcraft3d-coarse-nerf.yaml --train system.prompt_processor.prompt="$prompt" data.image_path="$image_path"
getting this error
FileNotFoundError: Text embedding file .threestudio_cache/text_embeddings/e89966d7424b11d100b42f7e2bfde05e.pt for model DeepFloyd/IF-I-XL-v1.0 and prompt [a brightly colored mushroom growing on a log] not found.
Hi
Thanks for your excellent work. I didn't find dreambooth loss in "stable-diffusion-bsd-guidance.py". If I wanna use dreambooth, do I need to follow the instructions in ReadMe to train a dreambooth model separately?
how to train a custom model?
When I wanted to run the pipeline in docker (using your Dockerfile) I was unable to run preprocess_image.py due to missing carvekit package. I tried to install it, but it downgrades several packages including torch. Does this script work in Docker?
Hi @MrTornado24 and @tonybear999!
Great work! I have been experimenting with the DreamCraft3D implementation for a while and I have seen that the lambda_depth and lambda_normal are set 0.0 for all Geometry Sculpting stage configs. lambda_depth_rel seems to be available and 0.05 for the coarse stages.
I am having trouble finding where the ground truth normal data (that is estimated by preprocess_image.py) is being utilized during optimization.
Best wishes and cheers!
Hello, thank you for your brilliant work! Your results shown on the project page are really highly detailed.
For example, your 3D CGI Pixar Lionel Messi artfully kicking paint-filled bottles has a detailed back view of Messi. But after I ran NeRF & NeuS, Geometry, and Texture boosting, I still got a result with relatively non-detailed texture. I also tried your code in the threestudio-dreamcraft3d repo, which produced a slightly better result than the former one. But it still falls behind your result on the project page.
I wonder whether I ignore some implementation details when reproducing your results (I have used .yaml files given by the repo and 8 * A100 GPUs to train all stages). If you could give me an instruction to obtain better results, I will very appreciate it.
Thanks a lot :)
The first image is produced by threestudio-dreamcraft3d.
The second image is produced by DreamCraft3D.
Hello, I am sorry to bother you again. I notice that there are some Python scripts in the threestudio/scripts folder (such as train_dreambooth.py, train_dreambooth_lora.py). Are these two scripts used to produce highly detailed texture results on your project page? I am confused about the relationships between these scripts and the main program. Should I separately train a DreamBooth, load it into bsd-guidance/vsd-guidance, then only train bsd/vsd loss and LoRA in the main program, and repeat these steps?
Meanwhile, I wonder that, in the texture refinement stage, have you used DeepFloyd instead of SD 2.1 base to achieve better results? Because I conducted experiments using SD 2.1 base, but the results fall behind yours.
Thank you very much :)
I found that it's hard for DeepFloyd to understand some synthesized concepts such as "a horse riding a man", but dalle3 does good at this. Is it possible to replace the DeepFloyd with dalle3 to generate multi view pictures so that dreamcraft would probably do better regarding these concepts? thanks!
Hi,
Would this project release code and weight that is heavily customized? Or would it rely on existing frameworks, like threestudios?
Thanks
ไฝ ๅฅฝ๏ผ่ฏป่ฎบๆ็ๆถๅ็ๅฐ่ฟๅ๏ผๆ่ง่ฟไธชๅผๅญๆฏไธๆฏๅ้ไบ๏ผ
ๅฆๅคๆณ้ฎไธdreamboothไผๅ็บน็ๅค๏ผๆฏๅฐๅๅง็ๅ่ๅพๅ ไธๅ
ถไป่ง่ง็็ๆๅพ๏ผ่ฟญไปฃไผๅstable diffusionๆจกๅๅ~
Hi, Thanks for sharing your amazing work, which config should we use for export obj?
# this uses default mesh-exporter configurations which exports obj+mtl
python launch.py --config **path/to/trial/dir/configs/parsed.yaml** --export --gpu 0 resume=path/to/trial/dir/ckpts/last.ckpt system.exporter_type=mesh-exporter
I meet this problem when export model
I have a 16GB dedicated +16GB shared capable 3080, RTX I'd like to use this project but the 20GB VRAM requirement is a bit ambiguous as to whether this needs to be dedicated or can will happily run from shared too.
Please can you clarify this requirement in the documentation?
Thanks for your excellent work. Do you plan to release the code? Maybe open inference code first
Hi
I found comment the code can speed up the training time for geometry and texture stage.
trainer:
max_steps: 5000
log_every_n_steps: 1
num_sanity_val_steps: 0
val_check_interval: 200
enable_progress_bar: true
precision: 32
#strategy: "ddp_find_unused_parameters_true"```
hi, great work!
I want to know why eikonal loss weight is set to 0, during the coarse-neus phase.
Thanks for the great work, I tried to generate a corgi in the paper using the official code, but it seems that the hyperparameters in the official code may be a little different from the paper, can you give some advice on improving the result?
The picture I use:
The prompt I use:
"A DSLR photo of a corgi wearing a beret and holding a baguette, standing up on two hind legs"
other hyperparameters are not changed.
The model I generate:
https://github.com/deepseek-ai/DreamCraft3D/assets/53119724/fc34a686-1c3b-4e9b-be6b-8f4f7c8ceb27
Hello, thank you for your brilliant work! I am running some cases based on your code. The first 3 stages (NeuS & NeRF, Geometry, and Texture) went well. But when exporting meshes, I meet the problem:
[F glutil.cpp:338] eglInitialize() failedSeems that it is related to nvdiffrast. I tried some solutions from the Internet but they did not work. In the .yaml file:
renderer_type: "nvdiff-rasterizer"
renderer:
context_type: cudaSo I have used dr.RasterizeCudaContext instead of dr.RasterizeGLContext.
Looking forward to your reply, thanks a lot :)
Hi,
I am very curious about whether this repo does the same pose optimization or any specific operation.
I wonder why pred_normal[..., 0] = 1 - pred_normal[..., 0] here
I can't seem to find the load directory and thus can't start the first step of training. Could that be added to the repo? Thanks so much! This work is great.
Setup a new clean environment.
pip installed all requirements.
git clone https://github.com/deepseek-ai/DreamCraft3D
D:\Tests\DreamCraft3D>python gradio_app.py launch
Traceback (most recent call last):
File "D:\Tests\DreamCraft3D\gradio_app.py", line 436, in <module>
launch(args.port, listen=args.listen)
File "D:\Tests\DreamCraft3D\gradio_app.py", line 296, in launch
value=load_model_config_attrs(model_selector.value)[
File "D:\Tests\DreamCraft3D\gradio_app.py", line 91, in load_model_config_attrs
config_str = load_model_config(model_name)
File "D:\Tests\DreamCraft3D\gradio_app.py", line 87, in load_model_config
return open(model_name_to_config[model_name]).read()
FileNotFoundError: [Errno 2] No such file or directory: 'configs/gradio/dreamfusion-if.yaml'
Missing file is not included with the repository?
when eta?! ;)
Hello, I run your code on the mushroom case (on A100). However, my result is quite different with the provided demo video. I wonder whether your mushroom-demo was run with the yaml file you provide in this repo. The main difference is that my "log" is not a cylinder, but either a land or a board. This is a result from neus (Stage 1)
By the way, I can often observe the pure color (e.g. the pure green in the log) in Stage 1 (and may remain even after Stage III), as you may see in the following image. Have you ever met with similar problems?
I tried everything to install this but i cant install it. I cant install tinycudann too.
Please help.
Hi authors,
Thanks for your interesting work. I found that the ICLR camera-ready running title "Published as a conference paper in ICLR 2024" is not removed๐. This is a typo since ICLR 2024 papers are still under review and I think you should fix it to avoid misleading.
Regards
Environment: Runpod.io -- runpod/pytorch:2.0.1-py3.10-cuda11.8.0-devel-ubuntu22.04
Cloned;
Followed installation instructions;
Ran Stage 1 and got this error:
python launch.py --config configs/dreamcraft3d-coarse-nerf.yaml --train system.prompt_processor.prompt="$prompt" data.image_path="$image_path"
Traceback (most recent call last):
File "/workspace/DreamCraft3D/launch.py", line 252, in <module>
main(args, extras)
File "/workspace/DreamCraft3D/launch.py", line 74, in main
import threestudio
File "/workspace/DreamCraft3D/threestudio/__init__.py", line 36, in <module>
from . import data, models, systems
File "/workspace/DreamCraft3D/threestudio/data/__init__.py", line 1, in <module>
from . import image, uncond
File "/workspace/DreamCraft3D/threestudio/data/image.py", line 15, in <module>
from threestudio.data.uncond import (
File "/workspace/DreamCraft3D/threestudio/data/uncond.py", line 14, in <module>
from threestudio.utils.base import Updateable
File "/workspace/DreamCraft3D/threestudio/utils/__init__.py", line 1, in <module>
from . import base
File "/workspace/DreamCraft3D/threestudio/utils/base.py", line 7, in <module>
from threestudio.utils.misc import get_device, load_module_weights
File "/workspace/DreamCraft3D/threestudio/utils/misc.py", line 5, in <module>
import tinycudann as tcnn
File "/usr/local/lib/python3.10/dist-packages/tinycudann/__init__.py", line 9, in <module>
from tinycudann.modules import free_temporary_memory, NetworkWithInputEncoding, Network, Encoding
File "/usr/local/lib/python3.10/dist-packages/tinycudann/modules.py", line 51, in <module>
_C = importlib.import_module(f"tinycudann_bindings._{cc}_C")
File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ImportError: libcudart.so.12: cannot open shared object file: No such file or directory
I think the subtracted error estimate e_ฯ(x_t, t, โ
) should be written as e_ฯ(x_t; โ
, t) to match the other quantities in the expression.
Can you explain why e_ฯ(x_t; y, t) is added twice in the classifier free guidance?
Hi - wonderful results. Could you please give more details of what r_{t'}(x) is in Algorithm 1. If it is the denoised version of x_{t'} , which diffusion model and which method (DDIM etc.) is used to generate r_{t'}(x)?
where is the code ?
wait
How long is the entire inference time? I found Stage I very slow on 3090 and adding gpus cannot accelerate the process
Could you include a link to one or multiple examples of exported obj+mtl combinations in the documentation?
This would allow people to inspect the generated obj+mtl end result without going through all the installation steps themselves.
Hi, I have followed the steps provided and installed threestudio, which went fine.
But then, it is unclear if I have to git clone this repo and run from inside that directory, which results in missing yaml-files, or should overwrite the files in the threestudio directory with this repo's files, because when I only get clone and then run, the load-directory is missing, so yeah. :-/
My command is:
python launch.py --config configs/dreamcraft3d-coarse-nerf.yaml --train system.prompt_processor.prompt="young woman, scarf, white marble, bust, statue, victorian" data.image_path="load/images/Z1.png"
I tried with your repo and copied the load-directory from threestudio. This resulted in this error:
Seed set to 0
[INFO] Loading Deep Floyd ...
unet\diffusion_pytorch_model.fp16.safetensors not found
A mixture of fp16 and non-fp16 filenames will be loaded.
Loaded fp16 filenames:
[text_encoder/pytorch_model.fp16-00002-of-00002.bin, safety_checker/pytorch_model.fp16.bin, text_encoder/pytorch_model.fp16-00001-of-00002.bin, unet/diffusion_pytorch_model.fp16.bin]
Loaded non-fp16 filenames:
[watermarker/diffusion_pytorch_model.bin
If this behavior is not expected, please check your folder structure.
Loading pipeline components...: 100%|โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ| 3/3 [00:07<00:00, 2.53s/it]
[INFO] Loaded Deep Floyd!
[INFO] Loading Stable Zero123 ...
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.53 M params.
Keeping EMAs of 688.
making attention of type 'vanilla' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla' with 512 in_channels
Traceback (most recent call last):
File "C:\Users\Andy\Documents\DreamCraft3D\launch.py", line 252, in <module>
main(args, extras)
File "C:\Users\Andy\Documents\DreamCraft3D\launch.py", line 120, in main
system: BaseSystem = threestudio.find(cfg.system_type)(
File "C:\Users\Andy\Documents\DreamCraft3D\threestudio\systems\base.py", line 45, in __init__
self.configure()
File "C:\Users\Andy\Documents\DreamCraft3D\threestudio\systems\dreamcraft3d.py", line 45, in configure
self.guidance_3d = threestudio.find(self.cfg.guidance_3d_type)(
File "C:\Users\Andy\Documents\DreamCraft3D\threestudio\utils\base.py", line 83, in __init__
self.configure(*args, **kwargs)
File "C:\Users\Andy\Documents\DreamCraft3D\threestudio\models\guidance\stable_zero123_guidance.py", line 137, in configure
self.prepare_embeddings(self.cfg.cond_image_path)
File "C:\Users\Andy\Anaconda3\envs\dreamcraft3d\lib\site-packages\torch\amp\autocast_mode.py", line 16, in decorate_autocast
return func(*args, **kwargs)
File "C:\Users\Andy\Documents\DreamCraft3D\threestudio\models\guidance\stable_zero123_guidance.py", line 149, in prepare_embeddings
assert os.path.exists(image_path)
AssertionError
If I go for a version in which I overwrite the original threestudio with your repo's files, I get this error instead:
Seed set to 0
[INFO] Loading Deep Floyd ...
unet\diffusion_pytorch_model.fp16.safetensors not found
A mixture of fp16 and non-fp16 filenames will be loaded.
Loaded fp16 filenames:
[text_encoder/pytorch_model.fp16-00001-of-00002.bin, safety_checker/pytorch_model.fp16.bin, text_encoder/pytorch_model.fp16-00002-of-00002.bin, unet/diffusion_pytorch_model.fp16.bin]
Loaded non-fp16 filenames:
[watermarker/diffusion_pytorch_model.bin
If this behavior is not expected, please check your folder structure.
Loading pipeline components...: 100%|โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ| 3/3 [00:07<00:00, 2.53s/it]
[INFO] Loaded Deep Floyd!
[INFO] Loading Stable Zero123 ...
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.53 M params.
Keeping EMAs of 688.
making attention of type 'vanilla' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla' with 512 in_channels
[INFO] Loaded Stable Zero123!
[INFO] Using prompt [young woman, scarf, white marble, bust, statue, victorian] and negative prompt []
[INFO] Using view-dependent prompts [side]:[young woman, scarf, white marble, bust, statue, victorian, side view] [front]:[young woman, scarf, white marble, bust, statue, victorian, front view] [back]:[young woman, scarf, white marble, bust, statue, victorian, back view] [overhead]:[young woman, scarf, white marble, bust, statue, victorian, overhead view]
C:\Users\Andy\Anaconda3\envs\dreamcraft3d\lib\site-packages\controlnet_aux\segment_anything\modeling\tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_5m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_5m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
return register_model(fn_wrapper)
C:\Users\Andy\Anaconda3\envs\dreamcraft3d\lib\site-packages\controlnet_aux\segment_anything\modeling\tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_11m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_11m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
return register_model(fn_wrapper)
C:\Users\Andy\Anaconda3\envs\dreamcraft3d\lib\site-packages\controlnet_aux\segment_anything\modeling\tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
return register_model(fn_wrapper)
C:\Users\Andy\Anaconda3\envs\dreamcraft3d\lib\site-packages\controlnet_aux\segment_anything\modeling\tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_384 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_384. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
return register_model(fn_wrapper)
C:\Users\Andy\Anaconda3\envs\dreamcraft3d\lib\site-packages\controlnet_aux\segment_anything\modeling\tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_512 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_512. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
return register_model(fn_wrapper)
A matching Triton is not available, some optimizations will not be enabled.
Error caught was: No module named 'triton'
Process SpawnProcess-1:
Traceback (most recent call last):
File "C:\Users\Andy\Anaconda3\envs\dreamcraft3d\lib\multiprocessing\process.py", line 314, in _bootstrap
self.run()
File "C:\Users\Andy\Anaconda3\envs\dreamcraft3d\lib\multiprocessing\process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\Andy\Documents\threestudio\threestudio\models\prompt_processors\deepfloyd_prompt_processor.py", line 63, in spawn_func
text_encoder = T5EncoderModel.from_pretrained(
File "C:\Users\Andy\Anaconda3\envs\dreamcraft3d\lib\site-packages\transformers\modeling_utils.py", line 2505, in from_pretrained
raise EnvironmentError(
OSError: DeepFloyd/IF-I-XL-v1.0 does not appear to have a file named pytorch_model.8bit.bin but there is a file without the variant 8bit. Use `variant=None` to load this model from those weights.
Traceback (most recent call last):
File "C:\Users\Andy\Documents\threestudio\launch.py", line 252, in <module>
main(args, extras)
File "C:\Users\Andy\Documents\threestudio\launch.py", line 120, in main
system: BaseSystem = threestudio.find(cfg.system_type)(
File "C:\Users\Andy\Documents\threestudio\threestudio\systems\base.py", line 45, in __init__
self.configure()
File "C:\Users\Andy\Documents\threestudio\threestudio\systems\dreamcraft3d.py", line 50, in configure
self.prompt_processor = threestudio.find(self.cfg.prompt_processor_type)(
File "C:\Users\Andy\Documents\threestudio\threestudio\utils\base.py", line 83, in __init__
self.configure(*args, **kwargs)
File "C:\Users\Andy\Documents\threestudio\threestudio\models\prompt_processors\base.py", line 337, in configure
self.load_text_embeddings()
File "C:\Users\Andy\Documents\threestudio\threestudio\models\prompt_processors\base.py", line 395, in load_text_embeddings
self.text_embeddings = self.load_from_cache(self.prompt)[None, ...]
File "C:\Users\Andy\Documents\threestudio\threestudio\models\prompt_processors\base.py", line 413, in load_from_cache
raise FileNotFoundError(
FileNotFoundError: Text embedding file .threestudio_cache/text_embeddings\aca2a23effdf0d18401b69d88fd03720.pt for model DeepFloyd/IF-I-XL-v1.0 and prompt [young woman, scarf, white marble, bust, statue, victorian] not found.
What is the right way? And could you add that to the installation guide please?
Hi
Thanks for your excellent work.
I am wondering the slight difference in your stable_zero123_guidance.py and zero123_guidence.py.
Could you please explain why the delta radius is different from zero123?
T = torch.stack(
[
torch.deg2rad(
(90 - elevation) - (90 - self.cfg.cond_elevation_deg)
), # Zero123 polar is 90-elevation
torch.sin(torch.deg2rad(azimuth - self.cfg.cond_azimuth_deg)),
torch.cos(torch.deg2rad(azimuth - self.cfg.cond_azimuth_deg)),
torch.deg2rad(
90 - torch.full_like(elevation, self.cfg.cond_elevation_deg)
),
],
dim=-1,
)[:, None, :].to(self.device)```ran command for stage2:
python launch.py --config custom/threestudio-dreamcraft3D/configs/dreamcraft3d-geometry.yaml --train system.prompt_processor.prompt="a cartoon boy king in robotic knight armor" data.image_path="./load/images/rey_rgba.png" system.geometry_convert_from="./outputs/dreamcraft3d-coarse-n
eus/a_cartoon_boy_king_in_robotic_knight_armor@20240302-113207/ckpts/last.ckpt"
i get this error and dont understand why . any help resolving this ?
Traceback (most recent call last):
File "/home/boltron/threestudio/launch.py", line 309, in
main(args, extras)
File "/home/boltron/threestudio/launch.py", line 252, in main
trainer.fit(system, datamodule=dm, ckpt_path=cfg.resume)
File "/home/boltron/anaconda3/envs/threestudio/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py", line 543, in fit
call._call_and_handle_interrupt(
File "/home/boltron/anaconda3/envs/threestudio/lib/python3.11/site-packages/pytorch_lightning/trainer/call.py", line 43, in _call_and_handle_interrupt
return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/boltron/anaconda3/envs/threestudio/lib/python3.11/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 105, in launch
return function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/boltron/anaconda3/envs/threestudio/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py", line 579, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "/home/boltron/anaconda3/envs/threestudio/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py", line 962, in _run
self.strategy.setup(self)
File "/home/boltron/anaconda3/envs/threestudio/lib/python3.11/site-packages/pytorch_lightning/strategies/ddp.py", line 171, in setup
self.configure_ddp()
File "/home/boltron/anaconda3/envs/threestudio/lib/python3.11/site-packages/pytorch_lightning/strategies/ddp.py", line 283, in configure_ddp
self.model = self._setup_model(self.model)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/boltron/anaconda3/envs/threestudio/lib/python3.11/site-packages/pytorch_lightning/strategies/ddp.py", line 195, in _setup_model
return DistributedDataParallel(module=model, device_ids=device_ids, **self._ddp_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/boltron/anaconda3/envs/threestudio/lib/python3.11/site-packages/torch/nn/parallel/distributed.py", line 678, in init
self._log_and_throw(
File "/home/boltron/anaconda3/envs/threestudio/lib/python3.11/site-packages/torch/nn/parallel/distributed.py", line 1037, in _log_and_throw
raise err_type(err_msg)
RuntimeError: DistributedDataParallel is not needed when a module doesn't have any parameter that requires a gradient.
HI
Thanks for your excellent work!
I've noticed your work integrates the stable-zero123. Is the coordinate system of stable-zero123 same as zero123?
Best,
Xiaobiao Du
Hello, I've just set up dreamcraft3d with WSL, but when running the first command of the first Stage (NeRF & NeuS) the process gets killed after trying to load Stable Zero123 with no traceback or errors. I've tried this with a custom image and the example mushroom_log_rgba.png
I'm running it from WSL (Ubuntu 22.04.3 LTS) with torch 2.2.0 and CUDA 12.3, GPU is an RTX 4090
(venv) bebop@DESKTOP:~/DreamCraft3D$ python launch.py --config configs/dreamcraft3d-coarse-nerf.yaml --train system.prompt_processor.prompt="$prompt" data.image_path="$image_path"
/home/bebop/DreamCraft3D/venv/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_5m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_5m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
return register_model(fn_wrapper)
/home/bebop/DreamCraft3D/venv/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_11m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_11m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
return register_model(fn_wrapper)
/home/bebop/DreamCraft3D/venv/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
return register_model(fn_wrapper)
/home/bebop/DreamCraft3D/venv/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_384 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_384. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
return register_model(fn_wrapper)
/home/bebop/DreamCraft3D/venv/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_512 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_512. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
return register_model(fn_wrapper)
Seed set to 0
[INFO] Loading Deep Floyd ...
A mixture of fp16 and non-fp16 filenames will be loaded.
Loaded fp16 filenames:
[safety_checker/model.fp16.safetensors, text_encoder/model.fp16-00002-of-00002.safetensors, unet/diffusion_pytorch_model.fp16.safetensors, text_encoder/model.fp16-00001-of-00002.safetensors]
Loaded non-fp16 filenames:
[watermarker/diffusion_pytorch_model.safetensors
If this behavior is not expected, please check your folder structure.
Loading pipeline components...: 100%|โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ| 3/3 [00:00<00:00, 13.94it/s]
[INFO] Loaded Deep Floyd!
[INFO] Loading Stable Zero123 ...
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.53 M params.
Keeping EMAs of 688.
making attention of type 'vanilla' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla' with 512 in_channels
Killed
The same issue arises whether I use stable_zero123.ckpt that I got from https://huggingface.co/stabilityai/stable-zero123 or zero123-xl.ckpt that is downloaded by the load/zero123/download.sh script
Hey, Where's the implementation code and project?
(venv) root@b755cfdb0b39:/workspace/DreamCraft3D# python launch.py --config configs/dreamcraft3d-coarse-nerf.yaml --train system.prompt_processor.prompt="$prompt" data.image_path="$image_path"
Traceback (most recent call last):
File "/workspace/DreamCraft3D/launch.py", line 252, in
main(args, extras)
File "/workspace/DreamCraft3D/launch.py", line 74, in main
import threestudio
File "/workspace/DreamCraft3D/threestudio/init.py", line 36, in
from . import data, models, systems
File "/workspace/DreamCraft3D/threestudio/models/init.py", line 1, in
from . import (
File "/workspace/DreamCraft3D/threestudio/models/exporters/init.py", line 1, in
from . import base, mesh_exporter
File "/workspace/DreamCraft3D/threestudio/models/exporters/base.py", line 6, in
from threestudio.models.materials.base import BaseMaterial
File "/workspace/DreamCraft3D/threestudio/models/materials/init.py", line 1, in
from . import (
File "/workspace/DreamCraft3D/threestudio/models/materials/pbr_material.py", line 4, in
import envlight
ModuleNotFoundError: No module named 'envlight'
Hey,
i try to run Stage 3 (Texture Refinement), After running
ckpt=outputs/dreamcraft3d-geometry/$prompt@LAST/ckpts/last.ckpt
python launch.py --config configs/dreamcraft3d-texture.yaml --train system.prompt_processor.prompt="$prompt" data.image_path="$image_path" system.geometry_convert_from="$ckpt"
i get this output resulting with this error : TypeError: AttnProcessor2_0.call() got an unexpected keyword argument 'scale'
Here is the whole log of the process
Seed set to 0
[INFO] Initializing geometry from a given checkpoint ...
[INFO] Loading Stable Diffusion ...
Loading pipeline components...: 100%|โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ| 4/4 [00:01<00:00, 2.15it/s]
Loading pipeline components...: 100%|โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ| 4/4 [00:01<00:00, 2.44it/s]
[INFO] Loaded Stable Diffusion!
[INFO] Using prompt [A girl holding a basketball wearing sportswear and a sports cap] and negative prompt []
[INFO] Using view-dependent prompts [side]:[A girl holding a basketball wearing sportswear and a sports cap, side view] [front]:[A girl holding a basketball wearing sportswear and a sports cap, front view] [back]:[A girl holding a basketball wearing sportswear and a sports cap, back view] [overhead]:[A girl holding a basketball wearing sportswear and a sports cap, overhead view]
/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
warnings.warn(
/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=VGG16_Weights.IMAGENET1K_V1`. You can also use `weights=VGG16_Weights.DEFAULT` to get the most up-to-date weights.
warnings.warn(msg)
loaded pretrained LPIPS loss from threestudio/utils/lpips/vgg.pth
[INFO] GPU available: True (cuda), used: True
[INFO] TPU available: False, using: 0 TPU cores
[INFO] IPU available: False, using: 0 IPUs
[INFO] HPU available: False, using: 0 HPUs
[rank: 0] Seed set to 0
Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/1
[INFO] ----------------------------------------------------------------------------------------------------
distributed_backend=nccl
All distributed processes registered. Starting with 1 processes
----------------------------------------------------------------------------------------------------
[INFO] You are using a CUDA device ('NVIDIA A100-PCIE-40GB') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
[INFO] single image dataset: load image load/images/girl_rgba_rgba.png torch.Size([1, 1024, 1024, 3])
[INFO] single image dataset: load image load/images/girl_rgba_rgba.png torch.Size([1, 1024, 1024, 3])
[INFO] LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
[INFO]
| Name | Type | Params
----------------------------------------------------------
0 | geometry | TetrahedraSDFGrid | 12.6 M
1 | material | NoMaterial | 0
2 | background | SolidColorBackground | 0
3 | renderer | NVDiffRasterizer | 0
4 | guidance | StableDiffusionBSDGuidance | 1.7 B
----------------------------------------------------------
1.7 B Trainable params
0 Non-trainable params
1.7 B Total params
6,984.340 Total estimated model params size (MB)
[INFO] Validation results will be saved to outputs/dreamcraft3d-texture/A_girl_holding_a_basketball_wearing_sportswear_and_a_sports_cap@20231227-003242/save
/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:441: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=79` in the `DataLoader` to improve performance.
/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:441: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=79` in the `DataLoader` to improve performance.
Epoch 0: | | 0/? [00:00<?, ?it/s]Traceback (most recent call last):
File "/root/data/lw/DreamCraft3D/launch.py", line 252, in <module>
main(args, extras)
File "/root/data/lw/DreamCraft3D/launch.py", line 195, in main
trainer.fit(system, datamodule=dm, ckpt_path=cfg.resume)
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 544, in fit
call._call_and_handle_interrupt(
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 43, in _call_and_handle_interrupt
return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 102, in launch
return function(*args, **kwargs)
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 580, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 989, in _run
results = self._run_stage()
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1035, in _run_stage
self.fit_loop.run()
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 202, in run
self.advance()
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 359, in advance
self.epoch_loop.run(self._data_fetcher)
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 136, in run
self.advance(data_fetcher)
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 240, in advance
batch_output = self.automatic_optimization.run(trainer.optimizers[0], batch_idx, kwargs)
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 187, in run
self._optimizer_step(batch_idx, closure)
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 265, in _optimizer_step
call._call_lightning_module_hook(
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 157, in _call_lightning_module_hook
output = fn(*args, **kwargs)
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/pytorch_lightning/core/module.py", line 1291, in optimizer_step
optimizer.step(closure=optimizer_closure)
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/pytorch_lightning/core/optimizer.py", line 151, in step
step_output = self._strategy.optimizer_step(self._optimizer, closure, **kwargs)
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/pytorch_lightning/strategies/ddp.py", line 265, in optimizer_step
optimizer_output = super().optimizer_step(optimizer, closure, model, **kwargs)
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/pytorch_lightning/strategies/strategy.py", line 230, in optimizer_step
return self.precision_plugin.optimizer_step(optimizer, model=model, closure=closure, **kwargs)
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/pytorch_lightning/plugins/precision/precision.py", line 117, in optimizer_step
return optimizer.step(closure=closure, **kwargs)
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/torch/optim/optimizer.py", line 280, in wrapper
out = func(*args, **kwargs)
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/torch/optim/optimizer.py", line 33, in _use_grad
ret = func(self, *args, **kwargs)
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/torch/optim/adamw.py", line 148, in step
loss = closure()
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/pytorch_lightning/plugins/precision/precision.py", line 104, in _wrap_closure
closure_result = closure()
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 140, in __call__
self._result = self.closure(*args, **kwargs)
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 126, in closure
step_output = self._step_fn()
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/pytorch_lightning/loops/optimization/automatic.py", line 315, in _training_step
training_step_output = call._call_strategy_hook(trainer, "training_step", *kwargs.values())
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 309, in _call_strategy_hook
output = fn(*args, **kwargs)
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/pytorch_lightning/strategies/strategy.py", line 381, in training_step
return self._forward_redirection(self.model, self.lightning_module, "training_step", *args, **kwargs)
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/pytorch_lightning/strategies/strategy.py", line 633, in __call__
wrapper_output = wrapper_module(*args, **kwargs)
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1156, in forward
output = self._run_ddp_forward(*inputs, **kwargs)
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1110, in _run_ddp_forward
return module_to_run(*inputs[0], **kwargs[0]) # type: ignore[index]
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/pytorch_lightning/strategies/strategy.py", line 626, in wrapped_forward
out = method(*_args, **_kwargs)
File "/root/data/lw/DreamCraft3D/threestudio/systems/dreamcraft3d.py", line 367, in training_step
out = self.training_substep(batch, batch_idx, guidance="guidance", render_type=render_type)
File "/root/data/lw/DreamCraft3D/threestudio/systems/dreamcraft3d.py", line 196, in training_substep
guidance_out = self.guidance(
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/data/lw/DreamCraft3D/threestudio/models/guidance/stable_diffusion_bsd_guidance.py", line 1086, in forward
loss_pretrain = self.train_pretrain(latents, text_embeddings_vd, camera_condition, sample_new_img=sample_new_img)
File "/root/data/lw/DreamCraft3D/threestudio/models/guidance/stable_diffusion_bsd_guidance.py", line 951, in train_pretrain
images_sample = self._sample(
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast
return func(*args, **kwargs)
File "/root/data/lw/DreamCraft3D/threestudio/models/guidance/stable_diffusion_bsd_guidance.py", line 394, in _sample
noise_pred = unet(
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/diffusers/models/unet_2d_condition.py", line 915, in forward
sample, res_samples = downsample_block(
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py", line 996, in forward
hidden_states = attn(
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/diffusers/models/transformer_2d.py", line 292, in forward
hidden_states = block(
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/diffusers/models/attention.py", line 155, in forward
attn_output = self.attn1(
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/miniconda3/envs/dreamcraft3d/lib/python3.10/site-packages/diffusers/models/attention_processor.py", line 322, in forward
return self.processor(
TypeError: AttnProcessor2_0.__call__() got an unexpected keyword argument 'scale'
Epoch 0: | | 0/? [00:03<?, ?it/s]
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google โค๏ธ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.