Hi,
Thank you for your great research.
I had a few problems when I ran Magic123 for a single example.
Whe i ran "bash scripts/magic123/run_both_priors.sh 0 nerf dmtet data/realfusion15/metal_dragon_statue 1 1 ", the mash came out well. There was no error.
But, If i used other sample( my own sample ), there was error.
---------------- cmd --------------------------
[23/08/18 15:52:01 nerf.utils]: [INFO] #Scheduler:
<torch.optim.lr_scheduler.LambdaLR object at 0x7fc3c241bdc0>
[23/08/18 15:52:01 nerf.utils]: [INFO] Loading latest checkpoint ...
[23/08/18 15:52:01 nerf.utils]: [WARN] No checkpoint found, model randomly initialized.
[23/08/18 15:52:01 nerf.utils]: ==> [2023-08-18_15-52-01] Start Training out/magic123-nerf-coarse/demo1/magic123_stitch_nerf_coarse Epoch 1/50, lr=0.010000 ...
Traceback (most recent call last):
File "----/Magic123/main.py", line 622, in
trainer.train(train_loader, valid_loader, test_loader, max_epoch)
File "----/Magic123/nerf/utils.py", line 1049, in train
self.train_one_epoch(train_loader, max_epochs)
File "----/Magic123/nerf/utils.py", line 1263, in train_one_epoch
self.model.update_extra_state()
File "----/Magic123/venv_magic123/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "----/Magic123/nerf/renderer.py", line 1483, in update_extra_state
indices = raymarching.morton3D(coords).long() # [N]
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA
to enable device-side assertions.
WARNING [23/08/18 15:54:02 main]: ckpt out/magic123-nerf-coarse/demo1/magic123_stitch_nerf_coarse/checkpoints/magic123_stitch_nerf_coarse.pth is not found
[23/08/18 15:54:02 main]: NeRFNetwork(
(dmtet): DMTetGeometry()
(encoder): GridEncoder: input_dim=3 num_levels=16 level_dim=2 resolution=16 -> 2048 per_level_scale=1.3819 params=(6098120, 2) gridtype=hash align_corners=False interpolation=smoothstep
(sigma_net): MLP(
(net): ModuleList(
(0): Linear(in_features=32, out_features=64, bias=True)
(1): Linear(in_features=64, out_features=64, bias=True)
(2): Linear(in_features=64, out_features=4, bias=True)
)
)
)
[23/08/18 15:54:03 guidance.sd_utils]: [INFO] loading stable diffusion...
Loading pipeline components...: 0%| | 0/7 [00:00<?, ?it/s]text_config_dict
is provided which will be used to initialize CLIPTextConfig
. The value text_config["id2label"]
will be overriden.
text_config_dict
is provided which will be used to initialize CLIPTextConfig
. The value text_config["bos_token_id"]
will be overriden.
text_config_dict
is provided which will be used to initialize CLIPTextConfig
. The value text_config["eos_token_id"]
will be overriden.
Loading pipeline components...: 100%|█████████████████████████████████████████████████████████████████████████████| 7/7 [00:11<00:00, 1.70s/it]
WARNING [23/08/18 15:54:16 guidance.sd_utils]: learned_embeds_path data/demo1/stitch/learned_embeds.bin does not exist!
[23/08/18 15:54:17 guidance.sd_utils]: [INFO] loaded stable diffusion!
WARNING [23/08/18 15:54:17 main]: No learned_embeds_path provided, using the folowing pure text prompt with degraded performance: A high-resolution DSLR image of stitch
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.53 M params.
Keeping EMAs of 688.
making attention of type 'vanilla' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla' with 512 in_channels
[23/08/18 15:54:28 main]: {'workspace': 'out/magic123-nerf-dmtet/demo1/magic123_stitch_nerf_dmtet', 'text': 'A high-resolution DSLR image of stitch', 'negative': '', 'dir_texts_neg': False, 'check_prompt': False, 'O': True, 'O2': False, 'test': False, 'six_views': False, 'eval_interval': 1, 'test_interval': 50, 'seed': 101, 'log_every': 20, 'use_wandb': False, 'guidance': ['SD', 'zero123'], 'guidance_scale': {'SD': 100.0, 'zero123': 5.0}, 'gudiance_spatial_weighting': False, 'save_train_every': -1, 'lambda_clip': 0, 'clip_version': 'large', 'clip_guidance': 0, 'clip_t': 0.4, 'clip_iterative': False, 'clip_image_loss': False, 'save_guidance_every': -1, 'use_shape': False, 'shape_guidance': 3, 'shape_radius': 4, 'shape_fovy': 40, 'shape_init_color': True, 'shape_rpst': 'sdf', 'image': 'data/demo1/stitch/rgba.png', 'image_config': None, 'learned_embeds_path': 'data/demo1/stitch/learned_embeds.bin', 'known_iters': 100, 'known_view_interval': 4, 'bg_color_known': None, 'known_shading': 'lambertian', 'save_mesh': True, 'mcubes_resolution': 256, 'decimate_target': 50000.0, 'dmtet': True, 'tet_mlp': False, 'base_mesh': None, 'tet_grid_size': 256, 'init_ckpt': 'out/magic123-nerf-coarse/demo1/magic123_stitch_nerf_coarse/checkpoints/magic123_stitch_nerf_coarse.pth', 'lock_geo': False, 'iters': 5000, 'lr': 0.001, 'lr_scale_nerf': 1, 'lr_scale_texture': 1, 'ckpt': 'latest', 'cuda_ray': True, 'taichi_ray': False, 'max_steps': 1024, 'num_steps': 64, 'upsample_steps': 32, 'update_extra_interval': 16, 'max_ray_batch': 4096, 'latent_iter_ratio': 0, 'normal_iter_ratio': 0, 'textureless_iter_ratio': 0, 'albedo_iter_ratio': 0, 'warmup_bg_color': None, 'bg_color': None, 'bg_color_test': 'white', 'ema_decay': 0.95, 'jitter_pose': False, 'jitter_center': 0.2, 'jitter_target': 0.2, 'jitter_up': 0.02, 'uniform_sphere_rate': 0.5, 'grad_clip': -1, 'grad_clip_rgb': -1, 'grid_levels_mask': -1, 'grid_levels_mask_iters': 3000, 'bg_radius': -1, 'density_activation': 'exp', 'density_thresh': 10, 'blob_density': 10, 'blob_radius': 0.2, 'backbone': 'grid', 'grid_type': 'hashgrid', 'hidden_dim_bg': 32, 'optim': 'adam', 'sd_version': '1.5', 'hf_key': None, 'fp16': True, 'vram_O': False, 'w': 1024, 'h': 1024, 'known_view_scale': 1, 'known_view_noise_scale': 0.001, 'noise_known_camera_annealing': False, 'dmtet_reso_scale': 8, 'rm_edge': True, 'edge_threshold': 0.1, 'edge_width': 5, 'batch_size': 1, 'bound': 1.0, 'dt_gamma': 0, 'min_near': 0.1, 'radius_range': [1.8, 1.8], 'theta_range': [45, 135], 'phi_range': [-180, 180], 'fovy_range': [40, 40], 'default_radius': 1.8, 'default_polar': 90, 'default_azimuth': 0, 'default_fovy': 40, 'progressive_view': False, 'progressive_level': False, 'angle_overhead': 30, 'angle_front': 60, 't_range': [0.02, 0.5], 'lambda_entropy': 0.001, 'lambda_opacity': 0.0, 'lambda_orient': 0.01, 'lambda_tv': 0, 'lambda_wd': 0, 'lambda_normal_smooth': 0.5, 'lambda_normal_smooth2d': 0.5, 'lambda_3d_normal_smooth': 0.0, 'lambda_guidance': {'SD': 0.001, 'zero123': 0.01}, 'lambda_rgb': 5, 'lambda_mask': 0.5, 'lambda_depth': 0, 'lambda_normal': 0, 'lambda_depth_mse': 0.0, 'normalize_depth': True, 'lambda_mesh_normal': 0.1, 'lambda_mesh_lap': 0.1, 'gui': False, 'W': 800, 'H': 800, 'radius': 1.8, 'fovy': 40, 'light_theta': 60, 'light_phi': 0, 'max_spp': 1, 'zero123_config': './pretrained/zero123/sd-objaverse-finetune-c_concat-256.yaml', 'zero123_ckpt': './pretrained/zero123/105000.ckpt', 'zero123_grad_scale': 'angle', 'dataset_size_train': 100, 'dataset_size_valid': 8, 'dataset_size_test': 100, 'images': ['data/demo1/stitch/rgba.png'], 'ref_radii': [1.8], 'ref_polars': [90], 'ref_azimuths': [0], 'zero123_ws': [1], 'default_zero123_w': 1, 'use_clip': False, 'runname': 'magic123_stitch_nerf_dmtet', 'log_path': 'out/magic123-nerf-dmtet/demo1/magic123_stitch_nerf_dmtet/log_magic123_stitch_nerf_dmtet.txt', 'ckpt_path': 'out/magic123-nerf-dmtet/demo1/magic123_stitch_nerf_dmtet/checkpoints', 'best_path': 'out/magic123-nerf-dmtet/demo1/magic123_stitch_nerf_dmtet/checkpoints/magic123_stitch_nerf_dmtet.pth', 'device': device(type='cuda')}
[INFO] dataset: load image prompt ['data/demo1/stitch/rgba.png'] torch.Size([1, 3, 1024, 1024])
[INFO] dataset: load depth prompt ['data/demo1/stitch/depth.png'] torch.Size([161997])
[23/08/18 15:54:28 nerf.utils]: [WARN] dataset: data/demo1/stitch/normal.png is not found
[23/08/18 15:54:29 nerf.utils]: [INFO] cmdline: main.py -O --text A high-resolution DSLR image of --sd_version 1.5 --image data/demo1/stitch/rgba.png --learned_embeds_path data/demo1/stitch/learned_embeds.bin --workspace out/magic123-nerf-dmtet/demo1/magic123_stitch_nerf_dmtet --dmtet --init_ckpt out/magic123-nerf-coarse/demo1/magic123_stitch_nerf_coarse/checkpoints/magic123_stitch_nerf_coarse.pth --iters 5000 --optim adam --latent_iter_ratio 0 --guidance SD zero123 --lambda_guidance 1e-3 0.01 --guidance_scale 100 5 --rm_edge --bg_radius -1 --save_mesh
[23/08/18 15:54:29 nerf.utils]: args:
{'workspace': 'out/magic123-nerf-dmtet/demo1/magic123_stitch_nerf_dmtet', 'text': 'A high-resolution DSLR image of stitch', 'negative': '', 'dir_texts_neg': False, 'check_prompt': False, 'O': True, 'O2': False, 'test': False, 'six_views': False, 'eval_interval': 1, 'test_interval': 50, 'seed': 101, 'log_every': 20, 'use_wandb': False, 'guidance': ['SD', 'zero123'], 'guidance_scale': {'SD': 100.0, 'zero123': 5.0}, 'gudiance_spatial_weighting': False, 'save_train_every': -1, 'lambda_clip': 0, 'clip_version': 'large', 'clip_guidance': 0, 'clip_t': 0.4, 'clip_iterative': False, 'clip_image_loss': False, 'save_guidance_every': -1, 'use_shape': False, 'shape_guidance': 3, 'shape_radius': 4, 'shape_fovy': 40, 'shape_init_color': True, 'shape_rpst': 'sdf', 'image': 'data/demo1/stitch/rgba.png', 'image_config': None, 'learned_embeds_path': 'data/demo1/stitch/learned_embeds.bin', 'known_iters': 100, 'known_view_interval': 4, 'bg_color_known': None, 'known_shading': 'lambertian', 'save_mesh': True, 'mcubes_resolution': 256, 'decimate_target': 50000.0, 'dmtet': True, 'tet_mlp': False, 'base_mesh': None, 'tet_grid_size': 256, 'init_ckpt': 'out/magic123-nerf-coarse/demo1/magic123_stitch_nerf_coarse/checkpoints/magic123_stitch_nerf_coarse.pth', 'lock_geo': False, 'iters': 5000, 'lr': 0.001, 'lr_scale_nerf': 1, 'lr_scale_texture': 1, 'ckpt': 'latest', 'cuda_ray': True, 'taichi_ray': False, 'max_steps': 1024, 'num_steps': 64, 'upsample_steps': 32, 'update_extra_interval': 16, 'max_ray_batch': 4096, 'latent_iter_ratio': 0, 'normal_iter_ratio': 0, 'textureless_iter_ratio': 0, 'albedo_iter_ratio': 0, 'warmup_bg_color': None, 'bg_color': None, 'bg_color_test': 'white', 'ema_decay': 0.95, 'jitter_pose': False, 'jitter_center': 0.2, 'jitter_target': 0.2, 'jitter_up': 0.02, 'uniform_sphere_rate': 0.5, 'grad_clip': -1, 'grad_clip_rgb': -1, 'grid_levels_mask': -1, 'grid_levels_mask_iters': 3000, 'bg_radius': -1, 'density_activation': 'exp', 'density_thresh': 10, 'blob_density': 10, 'blob_radius': 0.2, 'backbone': 'grid', 'grid_type': 'hashgrid', 'hidden_dim_bg': 32, 'optim': 'adam', 'sd_version': '1.5', 'hf_key': None, 'fp16': True, 'vram_O': False, 'w': 1024, 'h': 1024, 'known_view_scale': 1, 'known_view_noise_scale': 0.001, 'noise_known_camera_annealing': False, 'dmtet_reso_scale': 8, 'rm_edge': True, 'edge_threshold': 0.1, 'edge_width': 5, 'batch_size': 1, 'bound': 1.0, 'dt_gamma': 0, 'min_near': 0.1, 'radius_range': [1.8, 1.8], 'theta_range': [45, 135], 'phi_range': [-180, 180], 'fovy_range': [40, 40], 'default_radius': 1.8, 'default_polar': 90, 'default_azimuth': 0, 'default_fovy': 40, 'progressive_view': False, 'progressive_level': False, 'angle_overhead': 30, 'angle_front': 60, 't_range': [0.02, 0.5], 'lambda_entropy': 0.001, 'lambda_opacity': 0.0, 'lambda_orient': 0.01, 'lambda_tv': 0, 'lambda_wd': 0, 'lambda_normal_smooth': 0.5, 'lambda_normal_smooth2d': 0.5, 'lambda_3d_normal_smooth': 0.0, 'lambda_guidance': {'SD': 0.001, 'zero123': 0.01}, 'lambda_rgb': 5, 'lambda_mask': 0.5, 'lambda_depth': 0, 'lambda_normal': 0, 'lambda_depth_mse': 0.0, 'normalize_depth': True, 'lambda_mesh_normal': 0.1, 'lambda_mesh_lap': 0.1, 'gui': False, 'W': 800, 'H': 800, 'radius': 1.8, 'fovy': 40, 'light_theta': 60, 'light_phi': 0, 'max_spp': 1, 'zero123_config': './pretrained/zero123/sd-objaverse-finetune-c_concat-256.yaml', 'zero123_ckpt': './pretrained/zero123/105000.ckpt', 'zero123_grad_scale': 'angle', 'dataset_size_train': 100, 'dataset_size_valid': 8, 'dataset_size_test': 100, 'images': ['data/demo1/stitch/rgba.png'], 'ref_radii': [1.8], 'ref_polars': [90], 'ref_azimuths': [0], 'zero123_ws': [1], 'default_zero123_w': 1, 'use_clip': False, 'runname': 'magic123_stitch_nerf_dmtet', 'log_path': 'out/magic123-nerf-dmtet/demo1/magic123_stitch_nerf_dmtet/log_magic123_stitch_nerf_dmtet.txt', 'ckpt_path': 'out/magic123-nerf-dmtet/demo1/magic123_stitch_nerf_dmtet/checkpoints', 'best_path': 'out/magic123-nerf-dmtet/demo1/magic123_stitch_nerf_dmtet/checkpoints/magic123_stitch_nerf_dmtet.pth', 'device': device(type='cuda')}
[23/08/18 15:54:29 nerf.utils]: [INFO] Trainer: magic123_stitch_nerf_dmtet | 2023-08-18_15-54-28 | cuda | fp16 | out/magic123-nerf-dmtet/demo1/magic123_stitch_nerf_dmtet
[23/08/18 15:54:29 nerf.utils]: [INFO] #parameters: 20836380
[23/08/18 15:54:29 nerf.utils]: [INFO] #Optimizer:
Adam (
Parameter Group 0
amsgrad: False
betas: (0.9, 0.99)
capturable: False
differentiable: False
eps: 1e-15
foreach: None
fused: None
initial_lr: 0.01
lr: 0.01
maximize: False
weight_decay: 0
Parameter Group 1
amsgrad: False
betas: (0.9, 0.99)
capturable: False
differentiable: False
eps: 1e-15
foreach: None
fused: None
initial_lr: 0.001
lr: 0.001
maximize: False
weight_decay: 0
Parameter Group 2
amsgrad: False
betas: (0.9, 0.99)
capturable: False
differentiable: False
eps: 1e-15
foreach: None
fused: None
initial_lr: 0.001
lr: 0.001
maximize: False
weight_decay: 0
)
[23/08/18 15:54:29 nerf.utils]: [INFO] #Scheduler:
<torch.optim.lr_scheduler.LambdaLR object at 0x7f0f94207cd0>
[23/08/18 15:54:29 nerf.utils]: [INFO] Loading latest checkpoint ...
[23/08/18 15:54:29 nerf.utils]: [WARN] No checkpoint found, model randomly initialized.
[23/08/18 15:54:29 nerf.utils]: ==> [2023-08-18_15-54-29] Start Training out/magic123-nerf-dmtet/demo1/magic123_stitch_nerf_dmtet Epoch 1/50, lr=0.010000 ...
Traceback (most recent call last):
File "----/Magic123/main.py", line 622, in
trainer.train(train_loader, valid_loader, test_loader, max_epoch)
File "----/Magic123/nerf/utils.py", line 1049, in train
self.train_one_epoch(train_loader, max_epochs)
File "----/Magic123/nerf/utils.py", line 1278, in train_one_epoch
loss, losses_dict, outputs = self.train_step(data)
File "----/Magic123/nerf/utils.py", line 645, in train_step
outputs = self.model.render(rays_o, rays_d, mvp, H, W, staged=False, perturb=True, bg_color=bg_color, ambient_ratio=ambient_ratio, shading=shading, binarize=binarize)
File "----/Magic123/nerf/renderer.py", line 1521, in render
results = self.run_dmtet(rays_o, rays_d, mvp, h, w, **kwargs)
File "----/Magic123/nerf/renderer.py", line 1264, in run_dmtet
rast, rast_db = dr.rasterize(self.glctx, verts_clip, faces, (h, w))
File "----/Magic123/venv_magic123/lib/python3.10/site-packages/nvdiffrast/torch/ops.py", line 310, in rasterize
return _rasterize_func.apply(glctx, pos, tri, resolution, ranges, grad_db, -1)
File "----/Magic123/venv_magic123/lib/python3.10/site-packages/torch/autograd/function.py", line 506, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
File "----/Magic123/venv_magic123/lib/python3.10/site-packages/nvdiffrast/torch/ops.py", line 248, in forward
out, out_db = _get_plugin().rasterize_fwd_cuda(raster_ctx.cpp_wrapper, pos, tri, resolution, ranges, peeling_idx)
RuntimeError: instance mode - pos must have shape [>0, >0, 4]