jackzhousz/nerd-neural-reflectance-decomposition

NeRD: Neural Reflectance Decomposition from Image Collections - ICCV 2021

NeRD: Neural Reflectance Decomposition from Image Collections

Project Page | Video | Paper | Dataset

Implementation for NeRD. A novel method which decomposes multiple images into shape, BRDF and illumination.

NeRD: Neural Reflectance Decomposition from Image Collections
Mark Boss¹, Raphael Braun¹, Varun Jampani², Jonathan T. Barron², Ce Liu², Hendrik P. A. Lensch^1
1University of Tübingen, ²Google Research

Also check our follow-up work: Neural-PIL

Setup

A conda environment is used for dependency management

conda env create -f environment.yml
conda activate nerd

Running

Replace the specific [] placeholders:

python nerd.py --datadir [DIR_TO_DATASET_FOLDER] --basedir [TRAIN_DIR] --expname [EXPERIMENT_NAME] --gpu [COMMA_SEPARATED_GPU_LIST]

Specific Arguments per Dataset

Most setup is handled by configurations files located in configs/nerd/.

Our Synthethic Scenes

--config configs/nerd/blender.txt

NeRF Synthethic Scenes

--config configs/nerd/nerf_blender.txt

Real-World

--config configs/nerd/real_world.txt 

Often objects are captured in a spherical manner and the flag --spherify should be applied for those scenes.

Datasets

All datasets are uploaded in individual git repositories. We have created a download script which automatically fetches all datasets and downloads them to a specified folder. Usage:

python download_datasets.py /path/to/dataset/root

Run Your Own Data

Mainly camera poses and segmentation masks are required. For the poses the scripts from NeRF are used to prepare the scenes. The dataset then needs to be put in the following file structure:

images/
    [IMG_NAME_1].jpg
    [IMG_NAME_2].jpg
    ...
masks/
    [IMG_NAME_1].jpg
    [IMG_NAME_2].jpg
    ...
poses_bounds.npy

The poses_bounds.npy is generated from the LLFF script.

Evaluation

The train_nerd.py can be called with a --render_only flag and the --config flag pointing to the args.txt of the experiments folder. The SGs can be converted to regular environment maps with the visualize_fit function in nn_utils/sg_rendering.py (Example in train_nerd.py).

Citation

@inproceedings{boss2021nerd,
  title         = {NeRD: Neural Reflectance Decomposition from Image Collections},
  author        = {Boss, Mark and Braun, Raphael and Jampani, Varun and Barron, Jonathan T. and Liu, Ce and Lensch, Hendrik P.A.},
  booktitle     = {IEEE International Conference on Computer Vision (ICCV)},
  year          = {2021},
}