eliphatfs/zerorf

ZeroRF: Fast Sparse View 360° Reconstruction with Zero Pretraining

zerorf's Issues

OpenIllumination data split

Dear authors, thank you for your wonderful work! In this repo, you only provided the scripts on nerf synthetic. I wonder if you can provide the OpenIllumination dataset with train/test split?

export

How to export a textured glb file

No license

Great work on this, it looks great.
However, it seems you did not include a license, which makes the code default to rather restrictive terms. Per https://choosealicense.com/no-permission/:

If you find software that doesn’t have a license, that generally means you have no permission from the creators of the software to use, modify, or share the software. Although a code host such as GitHub may allow you to view and fork the code, this does not imply that you are permitted to use, modify, or share the software for any purpose.

It would be great if you could add a license, as it stands, strictly speaking, it's not even legally allowed to test your code.

Generally speaking, a lot of influential whitepaper code is published under the MIT license, which is very open and free, allowing other projects to build upon your code, though other options are viable too. If you don't have a preference, I'd suggest going with MIT.

Question regarding files in repo

Hi, thanks for your excellent work!

I saw that OrbitCamera was imported from lib/core/ssdnerf_gui.py, but could not find any references for the SSDNeRFGUI class.
Likewise, many of the models in lib/models/architecture appear unused.
I was wondering, are they part of the workflow presented in the paper or additional files for other CV tasks?

Edit: It seems they are files left over from SSDNeRF? Correct me if I'm wrong

code and preprocessor of decoder

When choose TensorialGenerator as the self.preprocessor in decoder, in func preproc of decoder, u take code as the input of self.preprocessor(), but the actual input of the preprocessor is the inside fixed noise instead of the passed-in code. I'm confused about this.
Besides, the shape of the inside noise is (1, noise_ch, *noise_res), with batch_size=1, so as the output of the preprocessor, doesn't match the required size (num_scenes, *code_size)

Fail to reproduce OpenIllumination experiments

Thank you for your great work.

I'm tring your method on OpenIllumination dataset of 4 views but get wierd result.

dec_9999.mp4

I wonder if it's related to the world_scale param or something else. Can you provide OpenIllumination configs in detail to help me reproduce the result of your paper?

Thanks.

march_rays_train ERROR

When I run the demo: 'python zerorf.py --rep=tensorf --data-dir=path/to/nerf_synthetic --obj=hotdog --n-views=6'

the code went wrong:
get_backend().march_rays_train(rays_o, rays_d, density_bitfield, bound, contract, dt_gamma, max_steps, N, C, H, nears, fars, None, None, None, rays, step_counter, noises)

TypeError: march_rays_train(): incompatible function arguments. The following argument types are supported: 1. (arg0: at::Tensor, arg1: at::Tensor, arg2: at::Tensor, arg3: float, arg4: float, arg5: int, arg6: int, arg7: int, arg8: int, arg9: int, arg10: at::Tensor, arg11: at::Tensor, arg12: at::Tensor, arg13: at::Tensor, arg14: at::Tensor, arg15: at::Tensor, arg16: at::Tensor, arg17: at::Tensor) -> None Invoked with: tensor([[ 1.6088, -1.1919, 1.1559], [-0.2472, -2.1584, -0.7907]...

so, which parameter goes wrong?

CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling `cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &alpha, a, lda, stridea, b, ldb, strideb, &beta, c, ldc, stridec, num_batches)`

Hey, I am using wsl with ubuntu 20.04, cuda 11.8. I got the following when I tried to run: python zerorf.py --load-image=examples/ice.png

Is there any way I can try?

wandb: Currently logged in as: flandre. Use wandb login --relogin to force relogin
wandb: Tracking run with wandb version 0.16.1
wandb: Run data is saved locally in /mnt/c/Users/msz/Documents/Github_projs/zerorf/results/test/wandb/run-20231228_145824-hufokbtz
wandb: Run wandb offline to turn off syncing.
wandb: Syncing run test
wandb: ⭐️ View project at https://wandb.ai/flandre/zerorf
wandb: 🚀 View run at https://wandb.ai/flandre/zerorf/runs/hufokbtz
0%| | 0/10000 [00:00<?, ?it/s]2023-12-28 14:58:30,782 - mmgen - INFO - Initialize codes from scratch.
Shape of c2w: torch.Size([1, 6, 4, 4])
Shape of directions: torch.Size([1, 6, 320, 320, 3])
0%| | 0/10000 [00:00<?, ?it/s]
Traceback (most recent call last):
File "zerorf.py", line 227, in
lv = nerf.train_step(data_entry, optim)['log_vars']
File "/mnt/c/Users/msz/Documents/Github_projs/zerorf/lib/models/autoencoders/multiscene_nerf.py", line 207, in train_step
cond_rays_o, cond_rays_d = get_cam_rays(cond_poses, cond_intrinsics, h, w)
File "/mnt/c/Users/msz/Documents/Github_projs/zerorf/lib/core/utils/nerf_utils.py", line 65, in get_cam_rays
rays_o, rays_d = get_rays(directions, c2w, norm=True)
File "/mnt/c/Users/msz/Documents/Github_projs/zerorf/lib/core/utils/nerf_utils.py", line 56, in get_rays
rays_d = directions @ c2w[..., None, :3, :3].transpose(-1, -2) # (, h, w, 3)
RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &alpha, a, lda, stridea, b, ldb, strideb, &beta, c, ldc, stridec, num_batches)
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /mnt/c/Users/msz/Documents/Github_projs/zerorf/zerorf.py:227 in │
│ │
│ 224 best_psnr = 0.0 │
│ 225 │
│ 226 for j in prog: │
│ ❱ 227 │ lv = nerf.train_step(data_entry, optim)['log_vars'] │
│ 228 │ lr_sched.step() │
│ 229 │ lv.pop('code_rms') │
│ 230 │ lv.pop('loss') │
│ │
│ /mnt/c/Users/msz/Documents/Github_projs/zerorf/lib/models/autoencoders/multiscene_nerf.py:207 in │
│ train_step │
│ │
│ 204 │ │ │
│ 205 │ │ num_scenes, num_imgs, h, w, _ = cond_imgs.size() │
│ 206 │ │ # (num_scenes, num_imgs, h, w, 3) │
│ ❱ 207 │ │ cond_rays_o, cond_rays_d = get_cam_rays(cond_poses, cond_intrinsics, h, w) │
│ 208 │ │ dt_gamma_scale = self.train_cfg.get('dt_gamma_scale', 0.0) │
│ 209 │ │ # (num_scenes,) │
│ 210 │ │ dt_gamma = dt_gamma_scale / cond_intrinsics[..., :2].mean(dim=(-2, -1)) │
│ │
│ /mnt/c/Users/msz/Documents/Github_projs/zerorf/lib/core/utils/nerf_utils.py:65 in get_cam_rays │
│ │
│ 62 def get_cam_rays(c2w, intrinsics, h, w): │
│ 63 │ directions = get_ray_directions( │
│ 64 │ │ h, w, intrinsics, norm=False, device=intrinsics.device) # (num_scenes, num_imgs │
│ ❱ 65 │ rays_o, rays_d = get_rays(directions, c2w, norm=True) │
│ 66 │ return rays_o, rays_d │
│ 67 │
│ 68 │
│ │
│ /mnt/c/Users/msz/Documents/Github_projs/zerorf/lib/core/utils/nerf_utils.py:56 in get_rays │
│ │
│ 53 │ print("Shape of c2w:", c2w.shape) │
│ 54 │ print("Shape of directions:", directions.shape) │
│ 55 │ │
│ ❱ 56 │ rays_d = directions @ c2w[..., None, :3, :3].transpose(-1, -2) # (, h, w, 3) │
│ 57 │ rays_o = c2w[..., None, None, :3, 3].expand(rays_d.shape) # (*, h, w, 3) │
│ 58 │ if norm: │
│ 59 │ │ rays_d = F.normalize(rays_d, dim=-1) │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &alpha, a, lda, stridea, b, ldb, strideb, &beta, c, ldc, stridec, num_batches)
wandb: WARNING No program path found, not creating job artifact. See https://docs.wandb.ai/guides/launch/create-job

Question regarding the transform matrix

Hi, may I know what is the camera coordinate system used and how is the meta.json generated from rendered image dataset? How do we obtain the transform matrix when rendering a custom dataset? Thanks.

using the 7th image for reconstruction?

Hello, I noticed you used the results of zero123++ for reconstruction. However, only 6 images were used. Would it be possible to use the input image as the 7th image for reconstruction? If it's feasible, what should be the transformer JSON for this image?

TypeError: march_rays_train():

│ /data1/user/yanghaibo/eccv24/zerorf/lib/ops/raymarching/raymarching.py:287 in forward │
│ │
│ 284 │ │ # first pass: write rays, get total number of points M to render │
│ 285 │ │ rays = torch.empty(N, 2, dtype=torch.int32, device=rays_o.device) # id, offset, │
│ 286 │ │ get_backend().march_rays_train(rays_o, rays_d, density_bitfield, bound, contract │
│ ❱ 287 │ │ │ │ │ │ │ │ │ nears, fars, None, None, None, rays, step_counter │
│ 288 │ │ │
│ 289 │ │ # allocate based on M │
│ 290 │ │ M = step_counter.item() │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: march_rays_train(): incompatible function arguments. The following argument types are supported:
1. (arg0: at::Tensor, arg1: at::Tensor, arg2: at::Tensor, arg3: float, arg4: float, arg5: int, arg6: int, arg7: int, arg8: int,
arg9: int, arg10: at::Tensor, arg11: at::Tensor, arg12: at::Tensor, arg13: at::Tensor, arg14: at::Tensor, arg15: at::Tensor, arg16:
at::Tensor, arg17: at::Tensor) -> None