chikayan/d2nerf

d2nerf's Issues

Jax Installation

Hi authors, thanks for releasing the code! I am currently facing issues while running this repo on GPU (it's working on CPU but it takes forever). I think the problem is with the required version of jax and cuda support.

Can you please provide more details regarding the CUDA version and NVIDIA drivers that you are using for jax 0.2.26? TIA

The final results on camera_shadow dataset

After training for 100000 steps, the static part still has hand shadows. Does it need to be trained more?

run python eval.py

Why is there an error after running eval.py：symbol cudaGraphInstantiateWithFlags version libcudart.so.11.0 not defined in file libcudart.so.11.0 with link time reference

Artifacts forming on the border of the rendered images

Hi there, I am running your code on some custom videos of mine and for some some reason, when I render the images through the trained NeRF, I can notice some artifact mainly (but not exclusively) concentrated at the borders.
Those artifacts are like random pixels not consistent with the color of the surrounding image and a general increase in blurriness. Let me know if you have encountered the same issues with your experiments and if there is a quick solution to that.
One working theory is that those might be due to imperfect camera poses estimated with COLMAP, but in that case there would be no possible workaround I guess.

May I ask whether the dynamic Hypernerf module and the static NeRF module are optimized together, or only one optimizer is used

How can I get the same video file as the project page after training? I only have some runtime_eval png files. Thanks

Comparison to NSFF

Hi dear authors,

Thanks again for sharing this work. I am wondering have you guys tried comparing against NSFF? It also has decompositing components and seems to work reasonably. Would be a very strong baseline!

Thanks,
Hang

Thank you for your work, I have a question, why does this take so long to train, or the configuration required is so high.

May I ask that there are many zip files of songs in the data directory you provided, what do they mean? If I want to train by myself, which zip files should I download?

Generate novel views

Hi, the project is cool but I may need some help with generating novel views. Where should I modify it?

Colmap calibration.

Thanks for sharing this work.

Your paper mentioned that "we do not apply any masks when registering real-world images using COLMAP".
I wonder if your estimated camera poses are good enough for reconstruction, especially when performing SfM on a dynamic scene.

Protobuf version

If requirements.txt are installed as is, an error is raised during tensorboard import (noting that the protobuf version needs to be downgraded). Could consider explicitly adding e.g. protobuf 3.20.1 to requirements.txt? (Or whichever version was originally intended to be used.)

Adaptivity to other dynamic datasets

Dear authors,

Thank you for your excellent work.

I have tested the method on several outdoor scenes involving high ego-motion; however, the results are not entirely satisfactory. Do you believe there might be a gap in adapting to this type of data, or are there adjustments I could make to achieve better separation in such scenarios?

I also played with the hyperparameters but this is the best I can get.

Pillow model

Hi,

Thanks for publishing your synthetic dataset and including the kubrics scripts and blender files.

I'd like to do some tunings of the dataset, and I was able to find all the models from shapenet, except for the pillow. May I ask where the pillow model is from and how do you configured it in kubrics?

Thanks

May I ask why there is no static background rendering in the result of using hypernerf to render the dynamic part?

How to save rendering results as ply？

I want to save the rendered result as a ply file. Where should I modify it？

Question about configs and skewness hyperparameters

Hi! Thanks for releasing this very interesting project and sharing the code! I am wondering why there are five different configs here?

It also seems like skewness is a sensitive hyperparameters as in the checkpoints you released, all experiments have there own skewness parameter (2.0, 2.75 and 2.875). I am wondering if it's the case.

Thanks!
Hang

Error when training/rendering with 8 GPUs

Traceback (most recent call last):
  File "train.py", line 574, in <module>
    app.run(main)
  File "/home/wayve/prajwal/d2nerf/env/lib/python3.8/site-packages/absl/app.py", line 312, in run
    _run_main(main, args)
  File "/home/wayve/prajwal/d2nerf/env/lib/python3.8/site-packages/absl/app.py", line 258, in _run_main
    sys.exit(main(argv))
  File "train.py", line 495, in main
    process_iterator(tag='runtime_eval',
  File "train.py", line 528, in process_iterator
    model_out = render_fn(state, batch, rng=rng)
  File "/home/wayve/prajwal/d2nerf/hypernerf/evaluation.py", line 119, in render_image
    chunk_rays_dict = utils.shard(chunk_rays_dict, device_count)
  File "/home/wayve/prajwal/d2nerf/hypernerf/utils.py", line 289, in shard
    return jax.tree_map(lambda x: x.reshape((device_count, -1) + x.shape[1:]), xs)
  File "/home/wayve/prajwal/d2nerf/env/lib/python3.8/site-packages/jax/_src/tree_util.py", line 178, in tree_map
    return treedef.unflatten(f(*xs) for xs in zip(*all_leaves))
  File "/home/wayve/prajwal/d2nerf/env/lib/python3.8/site-packages/jax/_src/tree_util.py", line 178, in <genexpr>
    return treedef.unflatten(f(*xs) for xs in zip(*all_leaves))
  File "/home/wayve/prajwal/d2nerf/hypernerf/utils.py", line 289, in <lambda>
    return jax.tree_map(lambda x: x.reshape((device_count, -1) + x.shape[1:]), xs)
  File "/home/wayve/prajwal/d2nerf/env/lib/python3.8/site-packages/jax/_src/numpy/lax_numpy.py", line 1756, in _reshape
    newshape = _compute_newshape(a, args[0] if len(args) == 1 else args)
  File "/home/wayve/prajwal/d2nerf/env/lib/python3.8/site-packages/jax/_src/numpy/lax_numpy.py", line 1750, in _compute_newshape
    return tuple(- core.divide_shape_sizes(np.shape(a), newshape)
  File "/home/wayve/prajwal/d2nerf/env/lib/python3.8/site-packages/jax/_src/numpy/lax_numpy.py", line 1750, in <genexpr>
    return tuple(- core.divide_shape_sizes(np.shape(a), newshape)
  File "/home/wayve/prajwal/d2nerf/env/lib/python3.8/site-packages/jax/core.py", line 1438, in divide_shape_sizes
    return handler.divide_shape_sizes(ds[:len(s1)], ds[len(s1):])
  File "/home/wayve/prajwal/d2nerf/env/lib/python3.8/site-packages/jax/core.py", line 1348, in divide_shape_sizes
    raise InconclusiveDimensionOperation(f"Cannot divide evenly the sizes of shapes {tuple(s1)} and {tuple(s2)}")
jax.core.InconclusiveDimensionOperation: Cannot divide evenly the sizes of shapes (60, 3) and (8, -1, 3)

I encounter this error when training using 8 GPUs (specifically, the vrig_balloon). The error is raised while trying to render an image during eval.
Here's the corresponding PR that fixes the bug:
#8