algohunt/streamrf

Official implementation of our NeurIPS paper "Streaming Radiance Fields for 3D Video Synthesis"

streamrf's Issues

Error in svox2.sample_grid : invalid resource handle

When training the full model on the "Discussion" scene in Meeting Room dataset, the error "Error in svox2.sample_grid : invalid resource handle" occurs. And the following frames training are all wrong and give negative psnr.

First frame model and pilot models are trained from scratch as well without a problem.

New viewpoint

First of all, I would like to thank you for your work.
I would like to ask, how can I give a new point of view when rendering?

something wrong with "grid. svox2.SparseGrid()" in "opt.py"

Thanks for your work.
When I ran the command "python opt.py -t <log_dir> <data_dir>/0000 -c configs/meetroom_init.json --scale 1.0", I got a problem.

"Morton code requires a cube grid of power-of-2 size, ignoring...
Segmentation fault (core dumped)"

It looks like there is something wrong with "grid. svox2.SparseGrid()" in "opt.py".

Do you have any idea about this error?

A problem about Initialization training on Plenoxels

Thanks for your wonderful work, could please provide the detailed training config on Plenoxels that related to the Meetroom dataset.
I'm trying to reimplement the experiments on Meetroom dataset but I got poor results on the first frame model trained by Plenoxels.

How to convert mp4s to a dataset for "Initialize the first frame model"?

Thank you for releasing the code.

I'm trying to follow README. But, I failed to train a model at the step of "Initialize the first frame model."

python opt.py -t logs dataset/MeetRoom/discussion -c configs/meetroom_init.json --scale 1.0

Detected LLFF dataset
Traceback (most recent call last):
  File "opt.py", line 271, in <module>
    dset = datasets[args.dataset_type](
  File "/StreamRF/util/dataset.py", line 13, in auto_dataset
    return LLFFDataset(root, *args, **kwargs)
  File "/StreamRF/util/llff_dataset.py", line 69, in __init__
    self.sfm = SfMData(
  File "/StreamRF/util/llff_dataset.py", line 244, in __init__
    raise Exception("Unknow dataset type")
Exception: Unknow dataset type

In llff_dataset.py, readLLFF is searching for images (dataset/MeetRoom/discussion/images) but didn't find them because the dataset contains mp4s and poses only.

tree dataset/MeetRoom
dataset/MeetRoom
├── discussion
│   ├── cam_0.mp4
│   ├── cam_1.mp4
│   ├── cam_10.mp4
│   ├── cam_11.mp4
│   ├── cam_12.mp4
│   ├── cam_2.mp4
│   ├── cam_3.mp4
│   ├── cam_4.mp4
│   ├── cam_5.mp4
│   ├── cam_6.mp4
│   ├── cam_7.mp4
│   ├── cam_8.mp4
│   ├── cam_9.mp4
│   └── poses_bounds.npy
...

I guess that you use some kind of script to make the images directory from the original dataset.
Could you upload it?

Artifacts during training the full model

Hi,
Thanks for your impressive work! I'm trying to run the code following the instructions on the MeetRoom dataset. However, I found that the final generation result is bad with clear time consistency and block artifacts. Specifically, this happens after training the full model (the pilot model seems good, see the videos below). I tried on all 3 scenes of the MeetRoom dataset, and the artifacts appear on all these scenes. Is there any bug in the "train_video_n3dv_full.py" file? like some missing input arguments or wrong hyperparameters?
Hope for your feedback!

The pilot model of scene "discussion":
https://user-images.githubusercontent.com/35017815/220124406-f7863a9c-591a-4f70-a12a-a6da89d5f3d9.mp4

The full model of scene "discussion":
https://user-images.githubusercontent.com/35017815/220124483-e0dd4b0e-a66e-48cd-a86f-72e41e342010.mp4

The pilot model of scene "vrheadset":
https://user-images.githubusercontent.com/35017815/220124677-f264adec-abdc-46dd-8eea-ea11ef7e7def.mp4

The full model of scene "vrheadset":
https://user-images.githubusercontent.com/35017815/220124801-47d75b9c-b018-4222-9e32-bae3aaec6ddb.mp4

Would you mind to share pretrained model and testing video of flame_salmon?

Hello, thank you for releasing the code and the demo video. Would you mind to share pretrained model and predicted testing video (cam00) of flame_salmon?

How do you measure the PSNR of test set for N3DV in Table 1?

Hi, thank you for your excellent work. I download the flame_salmon dataset from N3DV GitHub. After I unzipped it, it contained 40-second video for 19 cameras. I assume you use cam01~cam20 for training and cam00 for testing. Instead of measuring the PSRN every 10 frames, you measure the PSNR per frame, which is first 300 frames from cam00. Do you only report the PSNR for the first 10-second frames? Or, you split them into four chunks (each video is 10 seconds), train the model per chunk, and then measure the average PSNR between them?

Can we see the videos generated by the proposed method?

Supplementary materials of videos may be missing. I'm looking forward to watching the videos!

Train/Val split of the Meeting Room Dataset

Maybe I overlooked somewhere, but may I ask what is the train/validation split of the meeting room dataset.
Is the cam_0 used for validation?

Thank you!

Question about experiments on the MeetRoom dataset

Hello, Thanks for your great job!

I'm conducting experiments on the MeetRoom dataset, since there are three scenes in Google Drive and four scenes in ModelScope. I'd like to ask which scenes the results in Table 1 are averaged from?

Thanks in advance for any help.

the problem of dataset

I got an error like the followings, can you give me a favor?

Defaulting to extended NSVF dataset

LOAD NSVF DATA /home/llh/experiments/Downloads/MeetRoom-20221231T025252Z-001/MeetRoom/ split train

Traceback (most recent call last):

File "opt.py", line 271, in

dset = datasets[args.dataset_type](

File "/home/llh/experiments/code/StreamRF-main/util/dataset.py", line 20, in auto_dataset

return NSVFDataset(root, *args, **kwargs)

File "/home/llh/experiments/code/StreamRF-main/util/nsvf_dataset.py", line 82, in init

img_dir_name = look_for_dir(["images", "image", "rgb"])

File "/home/llh/experiments/code/StreamRF-main/util/nsvf_dataset.py", line 79, in look_for_dir

assert False, "None of " + str(cands) + " found in data directory"

AssertionError: None of ['images', 'image', 'rgb'] found in data directory

munmap_chunk(): invalid pointer

Aborted (core dumped)

Where is opt.py file as in the README.md for training?