frozoul/4k-nerf

Official implementation of arxiv paper "4K-NeRF: High Fidelity Neural Radiance Fields at Ultra High Resolutions"

4k-nerf's Issues

Providing pre-trained scene models

Hi,

Would you consider uploading the models you've trained for the LLFF scenes for reproducibility?

Best,

Error running on Windows

Hi, I have compiled and am running the example :

D:\NERF\4K-NeRF-main\4K-NeRF-main>python run.py --config configs/llff/fern_lg_pretrain.py --render_test

And it gives me the error:

Using C:\Users\chemi\AppData\Local\torch_extensions\torch_extensions\Cache\py311_cu117 as PyTorch extensions root...
D:\Python\Lib\site-packages\torch\utils\cpp_extension.py:359: UserWarning: Error checking compiler version for cl: [WinError 2] The system cannot find the file specified
  warnings.warn(f'Error checking compiler version for {compiler}: {error}')
Detected CUDA files, patching ldflags
Emitting ninja build file C:\Users\chemi\AppData\Local\torch_extensions\torch_extensions\Cache\py311_cu117\adam_upd_cuda\build.ninja...
INFO: Could not find files for the given pattern(s).
Traceback (most recent call last):
  File "D:\NERF\4K-NeRF-main\4K-NeRF-main\run.py", line 11, in <module>
    from lib import img_encoder, utils, dvgo, dcvgo, dmpigo, sr_esrnet, sr_unetdisc, dvqgo
  File "D:\NERF\4K-NeRF-main\4K-NeRF-main\lib\utils.py", line 12, in <module>
    from .masked_adam import MaskedAdam
  File "D:\NERF\4K-NeRF-main\4K-NeRF-main\lib\masked_adam.py", line 7, in <module>
    adam_upd_cuda = load(
                    ^^^^^
  File "D:\Python\Lib\site-packages\torch\utils\cpp_extension.py", line 1284, in load
    return _jit_compile(
           ^^^^^^^^^^^^^
  File "D:\Python\Lib\site-packages\torch\utils\cpp_extension.py", line 1509, in _jit_compile
    _write_ninja_file_and_build_library(
  File "D:\Python\Lib\site-packages\torch\utils\cpp_extension.py", line 1611, in _write_ninja_file_and_build_library
    _write_ninja_file_to_build_library(
  File "D:\Python\Lib\site-packages\torch\utils\cpp_extension.py", line 2048, in _write_ninja_file_to_build_library
    _write_ninja_file(
  File "D:\Python\Lib\site-packages\torch\utils\cpp_extension.py", line 2188, in _write_ninja_file
    cl_paths = subprocess.check_output(['where',
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Python\Lib\subprocess.py", line 466, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Python\Lib\subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['where', 'cl']' returned non-zero exit status 1.

What am i doing wrong here?

LICENSE.txt?

Hi there! Incredible research, fantastic work. I'm curious to know how you intend on allowing your code to be used. Would you be willing to add a LICENSE.txt detailing your terms?

I really appreciate you and your team!

How to export mesh 3d ?

RuntimeError: set_sizes_and_strides is not allowed on a Tensor created from .data or .detach().

How to make it work with my own dataset?

I want to try to run this on custom objects, I've obtained the data in the form of the synthetic data that is mentioned in the readme file, with train/val images(frames) and transforms files for both using colmap. Any examples of config file on this type of data? Or how can i modify the config file to make it work for this data?

video_result

fern_compare_1k.mp4

horn_compare_1k.mp4

A question about "adam_upd_cuda"

Thanks for your well-organized code. When I run your code, an error is as follow:

**Using /home/qwt/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/qwt/.cache/torch_extensions/py38_cu117/adam_upd_cuda/build.ninja...
Building extension module adam_upd_cuda...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
Traceback (most recent call last):
File "/home/ls/anaconda3/envs/py38/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1893, in _run_ninja_build
subprocess.run(
File "/home/ls/anaconda3/envs/py38/lib/python3.8/subprocess.py", line 512, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/qwt/code/4K-NeRF-main/run.py", line 11, in
from lib import img_encoder, utils, dvgo, dcvgo, dmpigo, sr_esrnet, sr_unetdisc, dvqgo
File "/home/qwt/code/4K-NeRF-main/lib/utils.py", line 12, in
from .masked_adam import MaskedAdam
File "/home/qwt/code/4K-NeRF-main/lib/masked_adam.py", line 7, in
adam_upd_cuda = load(
File "/home/ls/anaconda3/envs/py38/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1284, in load
return jit_compile(
File "/home/ls/anaconda3/envs/py38/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1509, in jit_compile
write_ninja_file_and_build_library(
File "/home/ls/anaconda3/envs/py38/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1624, in write_ninja_file_and_build_library
run_ninja_build(
File "/home/ls/anaconda3/envs/py38/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1909, in run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error building extension 'adam_upd_cuda'
[1/2] /usr/local/cuda/bin/nvcc -DTORCH_EXTENSION_NAME=adam_upd_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /home/ls/anaconda3/envs/py38/lib/python3.8/site-packages/torch/include -isystem /home/ls/anaconda3/envs/py38/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/ls/anaconda3/envs/py38/lib/python3.8/site-packages/torch/include/TH -isystem /home/ls/anaconda3/envs/py38/lib/python3.8/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /home/ls/anaconda3/envs/py38/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS -D__CUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -std=c++17 -c /home/qwt/code/4K-NeRF-main/lib/cuda/adam_upd_kernel.cu -o adam_upd_kernel.cuda.o
FAILED: adam_upd_kernel.cuda.o
/usr/local/cuda/bin/nvcc -DTORCH_EXTENSION_NAME=adam_upd_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -isystem /home/ls/anaconda3/envs/py38/lib/python3.8/site-packages/torch/include -isystem /home/ls/anaconda3/envs/py38/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/ls/anaconda3/envs/py38/lib/python3.8/site-packages/torch/include/TH -isystem /home/ls/anaconda3/envs/py38/lib/python3.8/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /home/ls/anaconda3/envs/py38/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -std=c++17 -c /home/qwt/code/4K-NeRF-main/lib/cuda/adam_upd_kernel.cu -o adam_upd_kernel.cuda.o
nvcc fatal : Unsupported gpu architecture 'compute_86'
ninja: build stopped: subcommand failed.**

Could you give me some suggestions to solve it？

Out of memory during training.

Dear authors,

Thank you for your great work and amazing results. I tried to reproduce the video results that you provided in this repo. Following your instructions, I finished the encoder pretraining and then I got the 'CUDA out of memory' error when jointly training the encoder and decoder. So could you share with me the GPU memory requirement for running this code?

RuntimeError: rays_o must be a CUDA tensor

I run the example code run_sr, and I se ethe following error:

python run_sr.py --config configs/llff/fern_lg_joint_l1+gan.py --render_test --ftdv_path logs/llff/pretrain_fern_l1/fine_last.tar --ftsr_path ./pretrained/RealESRNet_x4plus.pth --test_tile 510

Downloading: "https://download.pytorch.org/models/vgg19-dcbb9e9d.pth" to C:\Users\B/.cache\torch\hub\checkpoints\vgg19-dcbb9e9d.pth
100%|███████████████████████████████████████████████████████████████████████████████| 548M/548M [00:39<00:00, 14.5MB/s]
get_training_rays: start
get_training_rays: finish (eps time: 0.9931809902191162 sec)
  0%|                                                                                       | 0/270000 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "D:\NERF\4K-NeRF-main\4K-NeRF-main\run_sr.py", line 1294, in <module>
    train(args, cfg, data_dict)
  File "D:\NERF\4K-NeRF-main\4K-NeRF-main\run_sr.py", line 1220, in train
    scene_rep_reconstruction_sr_patch(
  File "D:\NERF\4K-NeRF-main\4K-NeRF-main\run_sr.py", line 870, in scene_rep_reconstruction_sr_patch
    render_result = model(
  File "C:\Users\B\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "D:\NERF\4K-NeRF-main\4K-NeRF-main\lib\dmpigo.py", line 304, in forward
    ray_pts, ray_id, step_id, N_samples, mask_inbbox = self.sample_ray(
  File "D:\NERF\4K-NeRF-main\4K-NeRF-main\lib\dmpigo.py", line 279, in sample_ray
    ray_pts, mask_outbbox = render_utils_cuda.sample_ndc_pts_on_rays(
RuntimeError: rays_o must be a CUDA tensor

What might be causing this? Thanks!

UnboundLocalError: local variable 'optimizer_d' referenced before assignment

Traceback (most recent call last):
File "run_sr.py", line 1292, in
train(args, cfg, data_dict)
File "run_sr.py", line 1221, in train
scene_rep_reconstruction_sr_patch(
File "run_sr.py", line 1062, in scene_rep_reconstruction_sr_patch
for i_opt_sr, param_d in enumerate(optimizer_d.param_groups):
UnboundLocalError: local variable 'optimizer_d' referenced before assignment
How to solve this problem ？

RuntimeError: CUDA error: device-side assert triggered

Hi,

I find that this code (https://github.com/frozoul/4K-NeRF/blob/a8fdc8e98b/run_sr.py#L829) in run_sr.py will cause a out of bounds error:

target_4x = rgb_srgt_train[sel_b, sel_r_4x, sel_c_4x, :]

training is slow

Hi authors, thanks for posting this great repo!
I was wondering did you notice the training speed is really slow? I use single A5000 with 24GB memory, and the overall training time could up to 70h.

NULL ERRO

4K-NeRF/run_sr.py

Line 977 in ddc40ef

if cfg_train.weight_distortion > 0:

Traceback (most recent call last):
File "/root/z50023003/4K-NeRF-main/run_sr.py", line 1294, in
train(args, cfg, data_dict)
File "/root/z50023003/4K-NeRF-main/run_sr.py", line 1223, in train
scene_rep_reconstruction_sr_patch(
File "/root/z50023003/4K-NeRF-main/run_sr.py", line 991, in scene_rep_reconstruction_sr_patch
loss_distortion = flatten_eff_distloss(w, s, 1/n_max, ray_id)
File "/opt/conda/envs/mipnerf/lib/python3.9/site-packages/torch_efficient_distloss/eff_distloss.py", line 99, in forward
n_rays = ray_id.max()+1
RuntimeError: max(): Expected reduction dim to be specified for input.numel() == 0. Specify the reduction dim with the 'dim' argument.

render_result['s'] & render_result['n_max'] & render_result['weights'] is NULL

RuntimeError: CUDA error: device-side assert triggered

in https://github.com/frozoul/4K-NeRF/blob/a8fdc8e98b/run_sr.py#L829
The dimensionality of the data appears to be incorrect with the index
target_4x = rgb_srgt_train[sel_b, sel_r_4x, sel_c_4x, :]

and the rgb_srsave.shape is (12096,16128,3) weird

torch and cuda version of your work

Thank you for you great work. As far as I know you are the only one work on high resolution NeRF.

I try to run the 'traing 4K resolution with L1 loss' according to the readme.md. But get the following error .
Some times it's caused by pytorch&cuda version. So can I know the pytorch&cuda version infomation?

'
File "/opt/anaconda3/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1561, in _get_cuda_arch_flags
arch_list[-1] += '+PTX'
IndexError: list index out of range
but get "arch_list[-1] += '+PTX'" error

about Inference time

dvgo is very slow, why 4k-nerf Inference only 0.5s?

4K-NeRF on room-scale scenes

Hi,

Thanks for sharing the implementation of this great work!

I am interested in applying 4K-NeRF to real-world, room-scale scenes.

I am using the default parameters for the NeRF synthetic dataset and ensured, that the near/far plane and scene bounding box are set reasonably for my scene.
Do you have any recommendation of how to best set parameters for the room-scale setting?

I get the following test metrics and test renderings. In this dataset, the test views are pretty difficult, i.e., pretty far from the training views.

I am looking forward to your reply. Thanks a lot in advance!

	PSNR ↑	SSIM ↑	LPIPS ↓	KID ↓
Nerfacto	24.2	0.844	0.247	0.0661
4K-NeRF	21.9	0.829	0.249	0.0565

Is the pre-trained model RealESRNet_x4plus.pth trained by yourself?

May I ask if you retrained SFTNet? What dataset is used?