sulvxiangxin/multinerf-pytorch

multinerf-pytorch's Issues

Some questions about "multinerf_pl/internal/coord.py"

Hello author. Thank you so much for open sourcing your work. However, I have some questions in my actual running rawnerf and would like to consult you. Hope to get a reply. When running bash scripts/train_raw.sh, I keep prompting me with the following error, which I feel very strange, I want your answer, thank you。
Found no checkpoint files in exp/raw/windowlegovary with prefix checkpoint_
Number of parameters being optimized: 615740
Begin training...
0%| | 0/500000 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/home/wangzhifeng/multinerf_pl/train.py", line 361, in
app.run(main)
File "/home/wangzhifeng/.conda/envs/multinerf/lib/python3.9/site-packages/absl/app.py", line 308, in run
_run_main(main, args)
File "/home/wangzhifeng/.conda/envs/multinerf/lib/python3.9/site-packages/absl/app.py", line 254, in _run_main
sys.exit(main(argv))
File "/home/wangzhifeng/multinerf_pl/train.py", line 142, in main
renderings, ray_history = model(
File "/home/wangzhifeng/.conda/envs/multinerf/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/wangzhifeng/multinerf_pl/internal/models.py", line 104, in forward
_, s_to_t = coord.construct_ray_warps(self.raydist_fn, batch['near'], batch['far'])
File "/home/wangzhifeng/multinerf_pl/internal/coord.py", line 289, in construct_ray_warps
fn_inv = inv_mapping[fn.name]
AttributeError: 'str' object has no attribute 'name'

A few erros when running ref-nerf（您好 我在运行时遇到了很多问题，请求您的帮助）

我运行的命令行是：
The command line I run is:
python train.py --gin_configs=configs/blender_refnerf.gin --gin_bindings="Config.data_dir = './data/helmet'" --gin_bindings="Config.exp_name = 'helmet'" --gin_bindings="Config.factor = 4"

Traceback (most recent call last): File "/home/yangtongyu/nerf/multinerf-pytorch-main/train.py", line 361, in <module> app.run(main) File "/home/yangtongyu/software/anaconda3/envs/multinerf-py/lib/python3.9/site-packages/absl/app.py", line 308, in run _run_main(main, args) File "/home/yangtongyu/software/anaconda3/envs/multinerf-py/lib/python3.9/site-packages/absl/app.py", line 254, in _run_main sys.exit(main(argv)) File "/home/yangtongyu/nerf/multinerf-pytorch-main/train.py", line 142, in main renderings, ray_history = model( File "/home/yangtongyu/software/anaconda3/envs/multinerf-py/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/home/yangtongyu/nerf/multinerf-pytorch-main/internal/models.py", line 104, in forward _, s_to_t = coord.construct_ray_warps(self.raydist_fn, batch['near'], batch['far']) File "/home/yangtongyu/nerf/multinerf-pytorch-main/internal/coord.py", line 114, in construct_ray_warps fn_inv = inv_mapping[fn.__name__] AttributeError: 'str' object has no attribute '__name__'

为了解决这个错误，这里我把参数文件models.py 中的参数 line 35 raydist_fn修改为None，但我不知道这个影响是否很大？
To solve the above probelm, I changed line 35 of model.py raydist_fn to None, but I don’t know if it has a influence?
不报错了之后，我又遇到了下面的问题：
After not reporting the error, I encountered the following problem again:

(multinerf-py) yangtongyu@amax21-1:~/nerf/multinerf-pytorch-main$ python train.py --gin_configs=configs/blender_refnerf.gin --gin_bindings="Config.data_dir = './data/helmet'" --gin_bindings="Config.exp_name = 'helmet'" --gin_bindings="Config.factor = 4" /home/yangtongyu/software/anaconda3/envs/multinerf-py/lib/python3.9/site-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.17.3 and <1.25.0 is required for this version of SciPy (detected version 1.26.0 warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}" W1109 12:21:51.020475 140295852773504 logging.py:60] Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher. Found no checkpoint files in exp/helmet with prefix checkpoint_ Number of parameters being optimized: 713230 Begin training... 1/250000: loss=0.09426, psnr=11.060, lr=2.61e-05 | data=0.08617, orie=0.00777, pred=0.00032, 40151 r/s Rendering chunk: 0%| | 0/10 [00:00<?, ?it/s] 0%| | 0/250000 [00:02<?, ?it/s] Traceback (most recent call last): File "/home/yangtongyu/nerf/multinerf-pytorch-main/train.py", line 361, in <module> app.run(main) File "/home/yangtongyu/software/anaconda3/envs/multinerf-py/lib/python3.9/site-packages/absl/app.py", line 308, in run _run_main(main, args) File "/home/yangtongyu/software/anaconda3/envs/multinerf-py/lib/python3.9/site-packages/absl/app.py", line 254, in _run_main sys.exit(main(argv)) File "/home/yangtongyu/nerf/multinerf-pytorch-main/train.py", line 291, in main rendering = models.render_image( File "/home/yangtongyu/software/anaconda3/envs/multinerf-py/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/home/yangtongyu/nerf/multinerf-pytorch-main/internal/models.py", line 661, in render_image chunk_renderings, _ = render_fn(rand, chunk_batch) File "/home/yangtongyu/nerf/multinerf-pytorch-main/train.py", line 292, in <lambda> lambda rand, x: model(rand, File "/home/yangtongyu/software/anaconda3/envs/multinerf-py/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/home/yangtongyu/nerf/multinerf-pytorch-main/internal/models.py", line 200, in forward ray_results = mlp( File "/home/yangtongyu/software/anaconda3/envs/multinerf-py/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/home/yangtongyu/nerf/multinerf-pytorch-main/internal/models.py", line 471, in forward raw_grad_density_flat = torch.autograd.grad( File "/home/yangtongyu/software/anaconda3/envs/multinerf-py/lib/python3.9/site-packages/torch/autograd/__init__.py", line 303, in grad return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

明显是如下代码产生的错误，这个错误在models.py line 475左右 ：
It is obviously an error caused by the following code. This error is around models.py line 475:
raw_grad_density_flat = torch.autograd.grad( outputs=raw_density_flat, inputs=means, grad_outputs=d_output, create_graph=True, retain_graph=True, only_inputs=True, allow_unused=True)[0]

注意，最后一行allow_unused=True是我加的，是为了不让他产生RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn这个错误，但是又会产生新的上面的错误，请问你知道改怎么修改吗？

Note that the last line allow_unused=True was added by me to prevent it from happening. But a new error above will occur. Do you know how to correct it?

ValueError: The histogram is empty, please file a bug report.

Question 1
I added (Model.raydist_fn = @torch.reciprocal) in the llff_raw.gin file，Reported the error indicated in the title
RUN
CUDA_VISIBLE_DEVICES=2 bash scripts/train_raw.sh

Setting ds_accelerator to cuda (auto detect)
Found no checkpoint files in exp/raw/nightstreet with prefix checkpoint_
Number of parameters being optimized: 615740
Begin training...
0%| | 0/500000 [00:00<?, ?it/s]W0607 15:38:49.991008 140182311453056 x2num.py:14] NaN or Inf found in input tensor.
0%| | 0/500000 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/home/zenglongjian/multinerf-pytorch/train.py", line 367, in
app.run(main)
File "/home/zenglongjian/.conda/envs/multi-pytorch/lib/python3.9/site-packages/absl/app.py", line 308, in run
_run_main(main, args)
File "/home/zenglongjian/.conda/envs/multi-pytorch/lib/python3.9/site-packages/absl/app.py", line 254, in run_main
sys.exit(main(argv))
File "/home/zenglongjian/multinerf-pytorch/train.py", line 224, in main
summary_writer.add_histogram('train' + k, v, step)
File "/home/zenglongjian/.conda/envs/multi-pytorch/lib/python3.9/site-packages/tensorboardX/writer.py", line 562, in add_histogram
histogram(tag, values, bins, max_bins=max_bins), global_step, walltime)
File "/home/zenglongjian/.conda/envs/multi-pytorch/lib/python3.9/site-packages/tensorboardX/summary.py", line 209, in histogram
hist = make_histogram(values.astype(float), bins, max_bins)
File "/home/zenglongjian/.conda/envs/multi-pytorch/lib/python3.9/site-packages/tensorboardX/summary.py", line 247, in make_histogram
raise ValueError('The histogram is empty, please file a bug report.')
ValueError: The histogram is empty, please file a bug report.

Question 2
If I don't add this statement error
CUDA_VISIBLE_DEVICES=2 bash scripts/train_raw.sh

Setting ds_accelerator to cuda (auto detect)
Setting ds_accelerator to cuda (auto detect)
Found no checkpoint files in exp/raw/nightstreet with prefix checkpoint_
Number of parameters being optimized: 615740
Begin training...
0%| | 0/500000 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/home/zenglongjian/multinerf-pytorch/train.py", line 367, in
app.run(main)
File "/home/zenglongjian/.conda/envs/multi-pytorch/lib/python3.9/site-packages/absl/app.py", line 308, in run
_run_main(main, args)
File "/home/zenglongjian/.conda/envs/multi-pytorch/lib/python3.9/site-packages/absl/app.py", line 254, in _run_main
sys.exit(main(argv))
File "/home/zenglongjian/multinerf-pytorch/train.py", line 142, in main
renderings, ray_history = model(
File "/home/zenglongjian/.conda/envs/multi-pytorch/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/zenglongjian/.conda/envs/multi-pytorch/lib/python3.9/site-packages/accelerate/utils/operations.py", line 521, in forward
return model_forward(*args, **kwargs)
File "/home/zenglongjian/.conda/envs/multi-pytorch/lib/python3.9/site-packages/accelerate/utils/operations.py", line 509, in call
return convert_to_fp32(self.model_forward(*args, **kwargs))
File "/home/zenglongjian/.conda/envs/multi-pytorch/lib/python3.9/site-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast
return func(*args, **kwargs)
File "/home/zenglongjian/multinerf-pytorch/internal/models.py", line 104, in forward
_, s_to_t = coord.construct_ray_warps(self.raydist_fn, batch['near'], batch['far'])
File "/home/zenglongjian/multinerf-pytorch/internal/coord.py", line 113, in construct_ray_warps
fn_inv = inv_mapping[fn.name]
AttributeError: 'str' object has no attribute 'name'

internal/dataset.py

hi! Appreciate your implementation!

I have a question about the dataset.py:
in class LLFF:
method _load_renderings, the following codes would cause error:Image folder images_4 does not exist.

colmap_image_dir = os.path.join(self.data_dir, 'images')
            image_dir = os.path.join(self.data_dir, 'images' + image_dir_suffix)
            for d in [image_dir, colmap_image_dir]:
                if not utils.file_exists(d):
                    raise ValueError(f'Image folder {d} does not exist.')

To run the code, I delete the image_dir in code, and the training render result is poor, does it related to the image_4?

Thanks a lot!