zju3dv/object_nerf

Code for "Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering", ICCV 2021

Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering

Project Page | Video | Paper

Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering
Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, Zhaopeng Cui.
ICCV 2021

Installation

We have tested the code on pytorch 1.8.1, while a newer version of pytorch should also work.

conda create -n object_nerf python=3.8
conda activate object_nerf
conda install pytorch==1.8.1 torchvision cudatoolkit=11.1 -c pytorch -c conda-forge
pip install -r requirements.txt

Data Preparation

Please go to the data preparation.

Training

You can run train.py to train the model, and here are two examples.

# train on ScanNet 0113
python train.py dataset_config=config/scannet_base_0113_multi.yml "img_wh=[640,480]" exp_name=my_expr_scannet_0113

# train on ToyDesk 2
python train.py dataset_config=config/toy_desk_2.yml "img_wh=[640,480]" exp_name=my_expr_toydesk_2

Editable Scene Rendering

Here we provide two examples of scene editing with pre-trained models (download link).

ScanNet Object Duplicating and Moving

python test/demo_editable_render.py \
    config=test/config/edit_scannet_0113.yaml \
    ckpt_path=../object_nerf_edit_demo_models/scannet_0113/last.ckpt \
    prefix=scannet_0113_duplicating_moving

ToyDesk Object Rotating

python test/demo_editable_render.py \
    config=test/config/edit_toy_desk_2.yaml \
    ckpt_path=../object_nerf_edit_demo_models/toydesk_2/last.ckpt \
    prefix=toy_desk2_rotating

Remember to change the ckpt_path to the uncompressed model checkpoint file.

You can find the rendered image in debug/rendered_view/render_xxxxxx_scannet_0113_duplicating_moving or debug/rendered_view/render_xxxxxx_toy_desk2_rotating which should look as follows:

    

Citation

If you find this work useful, please consider citing:

@inproceedings{yang2021objectnerf,
    title={Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering},
    author={Yang, Bangbang and Zhang, Yinda and Xu, Yinghao and Li, Yijin and Zhou, Han and Bao, Hujun and Zhang, Guofeng and Cui, Zhaopeng},
    booktitle = {International Conference on Computer Vision ({ICCV})},
    month = {October},
    year = {2021},
}

Acknowledgement

In this project we use (parts of) the implementations of the following works:

• nerf_pl by kwea123.
• nerf_pytorch by Yen-Chen Lin.
• scannet by Angela Dai.

We thank the respective authors for open sourcing their methods.

object_nerf's Issues

segmentation fault when training

Hi, thank you for your great work.
I keep run into segmentation fault when trying to train on Toydesk dataset. It's confusing. Please help.

 python train.py dataset_config=config/toy_desk_2.yml "img_wh=[640,480]" exp_name=my_expr_toydesk_2
  ----------------------------------------
  exp_name: my_expr_toydesk_2
  ckpt_path: null
  dataset_name: toydesk
  img_wh:
  - 640
  - 480
  model:
    use_voxel_embedding: true
    N_freq_xyz: 10
    N_freq_dir: 4
    N_freq_voxel: 6
    D: 8
    W: 256
    skips:
    - 4
    N_scn_voxel_size: 16
    inst_D: 4
    inst_W: 128
    inst_skips:
    - 2
    N_obj_voxel_size: 8
    N_samples: 64
    N_importance: 64
    frustum_bound: -1
    use_disp: false
    perturb: 1
    noise_std: 1
    use_mask: true
    N_vocab: 1000
    N_max_objs: 64
    N_obj_code_length: 64
    N_max_voxels: 800000
  train:
    progressive_train: false
    batch_size: 2048
    chunk: 32768
    num_epochs: 30
    num_gpus: 1
    optimizer: adam
    lr: 0.001
    momentum: 0.9
    weight_decay: 0
    lr_scheduler: poly
    warmup_multiplier: 1
    warmup_epochs: 0
    decay_step:
    - 20
    decay_gamma: 0.1
    poly_exp: 2
    limit_train_batches: 0.05
  prefixes_to_ignore:
  - loss
  loss:
    color_loss_weight: 1.0
    depth_loss_weight: 0.1
    opacity_loss_weight: 10.0
    instance_color_loss_weight: 1.0
    instance_depth_loss_weight: 0.1
  dataset_extra:
    enable_observation_check: false
    max_obs_angle: 40
    max_obs_distance: 3.0
    mask_rebalance_strategy: fg_bg_reweight
    fg_weight: 1.0
    bg_weight: 0.05
    use_bbox: false
    use_bbox_only_for_test: true
    near: 0.8
    far: 24.0
    scale_factor: 16.0
    scene_center:
    - 0.2
    - 1.4
    - 7.1
    train_start_idx: 0
    train_skip_step: 1
    train_max_size: 9999
    validate_idx: 131
    split: datasets/split/our_desk_2_train_0.8
    use_instance_mask: true
    root_dir: data/toydesk/our_desk_2
    bbox_dir: datasets/desk_bbox/desk2/bbox.json
    inst_seg_tag: instance
    val_instance_id: 1
    instance_id:
    - 5
    - 4
    - 2
    - 1
    - 3
    bg_instance_id:
    - 0
    pcd_path: data/toydesk/our_desk_2/pcd_from_mesh.ply
    voxel_size: 0.3
    neighbor_marks: 3
  dataset_config: config/toy_desk_2.yml
  
  ----------------------------------------
  Start with exp_name: 220720_142351_my_expr_toydesk_2.
  Filling the voxel_occupancy...
  Voxel generated: torch.Size([214, 113, 171]) Voxel occupancy ratio: tensor(0.0426, device='cuda:0')
  Voxel used: tensor(176312, device='cuda:0')
  INFO - 2022-07-20 14:23:55,338 - trainer - GPU available: True, used: True
  INFO - 2022-07-20 14:23:55,338 - trainer - TPU available: False, using: 0 TPU cores
  INFO - 2022-07-20 14:23:55,338 - trainer - IPU available: False, using: 0 IPUs
  Training split count 121
  Train idx: 77 -> 115, skip: 1
  Read meta 00119 : 00119 instance 3
  ----------------------------------------
  Valid idx: 131
  INFO - 2022-07-20 14:24:12,671 - gpu - LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [2]
  /home/tangxiaotian/miniforge3/envs/object_nerf/lib/python3.8/site-packages/pytorch_lightning/callbacks/model_checkpoint.py:623: UserWarning: Checkpoint directory /home/tangxiaotian/object_nerf/logs/220720_142351_my_expr_toydesk_2 exists and is not empty.
    rank_zero_warn(f"Checkpoint directory {dirpath} exists and is not empty.")
  Epoch 0:   0%|                           | 4/904 [00:06<25:35,  1.71s/it, loss=1.04, v_num=0, train/psnr=10.10]段错误 (核心已转储)

Camera intrisics

Dear authors,

Thanks for your impressive work and released dataset.

I am trying to use the toydesk_data. But I wanna have the camera's intrinsic parameters.

So, could you please share it with me? Or tell me how to compute them!

Thank you very much!

assert self.embedding_space_ftr.num_embeddings >= idx_occu.shape[0]

assert self.embedding_space_ftr.num_embeddings >= idx_occu.shape[0]
AssertionError

I encounter this error when test on toy_desk dataset. Does anyone know what is causing it?
the error location is in embeding_helper.py line 197.

by the way, how can i change the transformation matrix form slam/opncv format to NDC format, is there any code exist in project that i didnt found.

The evaluation setting of the full scene rendering.

Hi, thanks for the nice paper and code!

I have one question about the evaluation setting of the full scene rendering (novel view synthesis). In Table.1 of the paper, quantitative metrics are compared. Are these metrics computed only on the test set, I mean, the frames listed in datasets/split/xxx/test.txt?

Thanks in advance!

Scene-guided occlusion identification

Hi,

Thanks for the paper and the code contribution. Nice work!

One question: could you please point out the code reference for the "scene-guided occlusion identification (scene guidance + 3D guard mask)" and how they affect the loss backpropagation? Thanks again.

About Toydesk ray sampling

Hi~
I wanna ask how do you calculate the values of near, far, scale_factor. I want to mix the your excellent project with another project. But it seems that this two project share different coordinate systems.

The content of scene branck.

Hi, Thanks for the code!

I use extract_mesh.py to extract the mesh of the scene branch and I find that the objects also exist in the scene branch. But the Figure 1 in the paper shows that the scene branch only has the background.

So, the scene branch only has "background" or "background + objects"?

How to edit single object?

Hi,

I have successfully completed the training and made a successful demo, your work is awesome. Thank you for your contribution.

However, I noticed in your paper that this algorithm should enable control of individual objects, such as rotating them, adding or removing them from the scene. I would like to experiment with this part, but it is not quite clear which parameters should be modified. Kindly could you please give me some suggestion?

All your help is very much appreciated.

remove_scene_object_by_ids

hi nice work，

Function remove_ Scene_ Object_ By_ IDS seems to be of no use as it did not delete the specified object in the object_nerf]/render_tools
/editable_renderer.py

在？代码呢？

年都过完了啊哥哥们，速度！

FileNotFoundError: [Errno 2] No such file or directory: data/scannet/scans/scene0113_00/scene0113_00.txt'

Hi,thanks for your great job
but I meet a problem that when i run "python train.py dataset_config=config/scannet_base_0113_multi.yml "img_wh=[640,480]" exp_name=my_expr_scannet_0113" it errors and shows that"FileNotFoundError: [Errno 2] No such file or directory: data/scannet/scans/scene0113_00/scene0113_00.txt'"

in the "scannet_base_0113_multi.yml" i see that " scans_dir: data/scannet/scans".However, i couldn't find a dir named "scans" in the data i download.

Removing objects from scene.

How to determine `scale_factor` and `scene_scater` in the config file.

Hi,
How to determine scale_factor and scene_scater in the config file.
The original value of scale_factor is 5.0, and the value of scene_center is [0.2, 1.4, 7.1].

How to handle the dataset.

Hi, I am a newbie to NeRF. I would like to know how to handle this dataset?

The description inData Preparation is a bit confusing, do I need to manually convert the dataset from OpenCV format to NDC format? Or is the downloadable dataset already processed?

Also, the download link for scannet_train_detection_data is no longer available, could you please upload it again?

All your help is greatly appreciated.

Code Release Date

Hello, thanks for your great work. It does push this research area forward a lot.
But I wonder when you guys will share your code? Can't wait to see it.

pre_train model and dataset link has been invalid!

please refresh the link,Thank you!

About Pose Matrixs Transformation

Hello, thanks u for awesome work.

I have some confusion about the Pose matrixs transformation among world, camera, objects. I cannot understand the meaning behind them. I wanna ask if there are some blogs i can refer to ?

Thanks in advance~

How to get bbox.json?

I want to label other scenes, how to get bbox.json?

AssertionError

Hi:
When I train toy_desk datasets, I meet the problem.

 assertError on here
 idx_occu = torch.nonzero(self.voxel_occupancy)
 assert self.embedding_space_ftr.num_embeddings >= idx_occu.shape[0]

 Can you help me solve the problem?

Warnings about the dataset

I have some error when I tran the model.Should I change the path in the code.
I have the following warnings.
UserWarning: No audio backend is available.
warnings.warn('No audio backend is available.')
UserWarning: torchaudio C++ extension is not available.
warnings.warn('torchaudio C++ extension is not available.')
RPly: Unable to open file
[Open3D WARNING] Read PLY failed: unable to open file: data/toy_desk/our_desk_2/pcd_from_mesh.ply

What a excellent work it is! And when will it be available on github?

An additional question about the code of this nice work.

Hi, I am very interested in this excellent work but I observe that the code is not available now.

May I ask another question: Did you implement your code based on someone else's code? If so, can you tell me where can I get this base code？

Thank you very much, my best wish.

question about desk1 dataset train split

Thank you for sharing the nice work!

Based on the following GitHub issue,
#13
I am trying to train my own model.

However, I noticed that the train split of desk1 data in the Github issue contains image with 95 index,
while the processed images in 'toydesk_data\processed\our_desk_1\full' does not contain "0095.instance.png" and "0095.png".
Can you make a clarification on that?

Thank you.

Object Activation Library and codes

Hi !

how to define the object activation codes for the object as per the number of objects in the scene ?
Say if I have only 2 objects in the scene what should be the activation library or say may be I have only 1 object what should be the activation library ?
So basically how are the activation codes assigned to the individual objects in the scene ?

In the code I saw that you use torch.embedding and input indices 1 to 5 , so does this mean the code library gives 5 activation codes for the 5 objects (cube,dragon, face, candy, magic cube).......... but which code belongs to which object ?
If I only wanted to render 3 objects out of the 5 objects then what should be done ?

Please help me with this concept of selecting the object codes @ybbbbt

some questions about bbox.json

Thanks for your great job！
But i have some quesions about the object_nerf/data/toy_desk/bbox/desk1/bbox.json.What's the use of the bbox,and how can i get the bbox.json if i want to use it my own data?

How to move the objects in the scene?

Hi,
I don't find the edit_type == move.
Can you show me the detailed tutorial for moveing objects?

This is an exciting work. When will you release the code?

Repetiton of the Toydesk Result in Table 1

Hi,
Thank you for the great job !
I retrain the toydesk model using the toy_desk_?.yml in confg folder. But the test result is much higher than the value in Table 1 ( psnr 25.59 for desk_top_2 and 18.73 for desk_top_1, which is much higher than 15.63 in paper) .

I also download your evaluation models here which is also about psnr 25 for the desk_top_2 scene.

There must be something wrong in my test script or the config. Could you tell me the way to repeat the result in Table 1 in ICCV21 paper? Looking forward to you reply!

Some questions about dataset details

1. 请问您的tranform_full.json文件是怎么转换得到的，Colmap直接导出的结果的格式似乎不是这样的，。
2. 请问为什么您只对C2w矩阵进行了变换（将YZ轴方向翻转，从OpenCV坐标系变换到opencl坐标系），但是使用PointCloud初始化voxel的时候对PointCloud却没有进行坐标轴的变换呢？(是不是您已经对PointCloud文件进行了这样的处理再保存的？)

ModuleNotFoundError: No module named 'torch.ao'

Hi,

I installed the conda env according to the readme file. But I got this error msg "ModuleNotFoundError: No module named 'torch.ao'" when I try to run training code on toy_desk_2 demo.

Does anyone have the same problem?

Thanks

What is the purpose of "scale_factor" parameter in config?

Hi @ybbbbt, I have a simple question: can you please explain the purpose of the parameter "scale_factor" in the ScanNet config files, such as here?

Thank you,
Yash

Dataset Format / Pointcloud

Hi,

Thanks a lot for releasing this code!

When trying to get it running on my own data, I realized that the data preparation instructions are a bit unclear. If I have a dataset in the LLFF format. Do I need to transform it into the COLMAP format?

Further, is there a way to use the voxel embeddings without knowing the point cloud of the scene a priori?

Thanks!

How can I use this code on my own images?