sjtuzq/point-radiance

[SIGGRAPH Asia 2022] Differentiable Point-Based Radiance Fields for Efficient View Synthesis

point-radiance

---

This code release accompanies the following paper:

Differentiable Point-Based Radiance Fields for Efficient View Synthesis

Qiang Zhang, Seung-Hwan Baek, Szymon Rusinkiweicz, Felix Heide

Siggraph Asia, 2022

PDF | arXiv Abstract: We propose a differentiable rendering algorithm for efficient novel view synthesis. By departing from volume-based representations in favor of a learned point representation, we improve on existing methods more than an order of magnitude in memory and run- time, both in training and inference. The method begins with a uniformly-sampled random point cloud and learns per-point posi- tion and view-dependent appearance, using a differentiable splat- based renderer to train the model to reproduce a set of input train- ing images with the given pose. Our method is up to 300 × faster than NeRF in both training and inference, with only a marginal sacrifice in quality, while using less than 10 MB of memory for a static scene. For dynamic scenes, our method trains two orders of magnitude faster than STNeRF and renders at a near interactive rate, while maintaining high image quality and temporal coherence even without imposing any temporal-coherency regularizers.

Installation

We recommend using a conda environment for this codebase. The following commands will set up a new conda environment with the correct requirements:

# Create and activate new conda env
conda create -n my-conda-env python=3.9
conda activate my-conda-env

# Install pytorch and related libraries
conda install -y pytorch==1.10.1 torchvision==0.11.2 cudatoolkit=11.0 -c pytorch
conda install numpy matplotlib tqdm imageio

Then follow the official INSTALL.md to install pytorch3d.

Reproduce

You can train the model on NeRF synthetic dataset within 3 minutes. Here datadir is the dataset folder path. Dataname is the scene name. Basedir is the log folder path. Data_r is the ratio between the used point number and the initialized point number. Splatting_r is the radius for the splatting.

python main.py --datadir xxx --dataname hotdog --basedir xxx --data_r 0.012 --splatting_r 0.015

After around three minutes, you can see the following output (the example is tested on one A100 GPU):

Training time: 148.59 s
Rendering quality: 34.70 dB
Rendering speed: 120.01 fps
Model size: 7.32 MB

Citation

If you find this work useful for your research, please consider citing:

@article{zhang2022differentiable,
  title={Differentiable Point-Based Radiance Fields for Efficient View Synthesis},
  author={Zhang, Qiang and Baek, Seung-Hwan and Rusinkiewicz, Szymon and Heide, Felix},
  journal={arXiv preprint arXiv:2205.14330},
  year={2022}
}

point-radiance's Issues

regarding the onlybase flag

Hello,
thank you for your work and making the implementation open source. i had a question regarding the onlybase flag here (

point-radiance/modules/model.py

Line 92 in 501158f

if self.onlybase:

) . it seems that the renderer is able to make an image using the cameras variable which has information regarding the pose. however if onlybase is False, we also do an evaluation of the spherical harmonics. so without onlybase are we doing an approximation of the lighting, kind of like using the "lower frequency" components. and is this for stabilizing the training of the parameters at earlier stages?

About the coordinates

Hi, thanks for your great work!

I may ask a stupid question but I am really confused about the convention of these axis.

In modules/model.py , feat.shape = [1, 400, 400, 30]. Does this mean the [batch_size, h, w, c] ?
Why do you only flip the axis of h? And which space does feat use, ndc space or screen space?
feat = renderer(point_cloud).flip(1)

Thanks for your help.

——

——

Resolution of blender dataset -- reproducing issue

Hi,

At first, thanks for sharing the code!

I'm wondering about the resolution blender dataset you used in table 1? I run your code with the given cmd and realize the resolution is 400x400 whereas the resolution of other baselines use the resolution of 800x800.

Not sure if I mess something up. After I switch into the resolution of 800x800 by setting half_res = False and img_s = 800, I found the FID score for chair was decreased a lot (27 for chair vs 32.98 reported in the supplementary).

All the best,
Weiwei.

video synthesis

Hi, thanks for your great work!

I found that there is no code related to video synthesis in the repository.
I would like to know when support will be available.

Thanks for your help.

it seems the model is initialized with the ground truth point cloud xyz?

hello,
I had a doubt. In line

point-radiance/modules/model.py

Line 40 in 501158f

self.vertsparam = torch.nn.Parameter(torch.Tensor(pointcloud[:, :3]))

, it seems the the vertex params are directly initialized from the pointcloud? or am i misunderstanding this? the spherical harmonic params however are initialized at random

point-radiance/modules/model.py

Line 43 in 501158f

sh_param.append(torch.rand((pointcloud.shape[0], 3)))