hbb1/pref

PREF: Phasorial Embedding Fields for Compact Neural Representations

PREF

novel view sythesis	image regression	neural field editing

PREF is an efficient frequency-based neural scene representation that allows:

1. super fast training and rendering novel views. (fast and approximated solvers, FFT, exist.)
2. gloabl optimization behaviour. (optimizing a local point lead global change.)
3. editing optimized neural fields through frequency-domain manipulations. (convolutional kernels as Fourier multipliers.)

TL;DR

We learn frequency representations (blue) for neural signals (red).

You can also view it as a dense variant of Position Encoding or Fourier Feature Mapping by rearranging their parameters in the first linear layer into a phasor volume.

Installation

Tested on Ubuntu 20.04 + Pytorch 1.11.0 (It seems important to install Pytorh version higher than 1.11.0 for complex tensor optimization).

Using PREF

Replacing tiny PE encoder + deep MLP with PREF encoder (dense PE) + shallow MLP to accelerate neural signal reconstruction. This is a preliminary release, without being carefully tested.

Example

import torch
import torch.nn as nn
import torch.nn.functional as F
from encoder.pref_encoder import PREF

class NeuralField(nn.Module):
  def __init__(self):
    super().__init__()
    self.encoder = PREF(linear_freqs=[128]*3, reduced_freqs=[1]*3, feature_dim=16)
    
    input_dim = self.encoder.output_dim
    hidden_dim = 64
    self.mlp = torch.nn.Sequential(
      torch.nn.Linear(input_dim, hidden_dim), 
      torch.nn.ReLU(inplace=True), 
      torch.nn.Linear(hidden_dim, 1))

  def forward(self, x):
    x = self.encoder(x)
    x = self.mlp(x)
    return x

NeRF with PREF

Installation

Install environment:

conda create -n PhasoRF python=3.8
conda activate PhasoRF
pip install torch torchvision
pip install tqdm scikit-image opencv-python configargparse lpips imageio-ffmpeg kornia lpips tensorboard plyfile

Dataset

• Synthetic-NeRF
• Synthetic-NSVF
• Tanks&Temples
• Forward-facing

Quick Start

The training script is in train.py, to train a NeRF with PREF:

python train.py --config configs/hotdog.txt

You need to set --render_test 1/--render_path 1 if you want to render testing views or path after training.

More options refer to the opt.py.

We provide a few examples in the configuration folder, note that:

dataset_name, choices = ['blender', 'llff', 'nsvf', 'tankstemple'].

Notice: We have tested parameters for 'blender' dataset, and you should get about $36.4$ psnr for the hotdog scene if correctly installing the repo.

Acknowledgement

NeRF with PREF extensively uses code from TensoRF.

Citation

If you find our code or paper helpful in your research, please consider citing

@article{Huang2022PREF,
  author = {Huang, Binbin and Yan, Xinhao and Chen, Anpei and Gao,Shenghua and Yu, Jingyi},
  title = {PREF: Phasorial Embedding Fields for Compact Neural Representations},
  publisher = {arXiv},
  year = {2022},
}

Please also consider citing

@article{tensorf,
  title={TensoRF: Tensorial Radiance Fields},
  author={Chen, Anpei and Xu, Zexiang and Geiger, Andreas and Yu, Jingyi and Su, Hao},
  journal={arXiv preprint arXiv:2203.09517},
  year={2022}
}

pref's Issues

About neural field editing experiment.

Hello, thanks for sharing the nice and interesting work!

Since your feature encoder (PREF) is based on Fourier Transform, I guess that we can continuously express the multiple scales of the signal by changing the frequency bandwidth of the encoding volume or input signal. My questions are:

1. What is the clear difference between modifying linear_freqs and reduced_freqs of PREF and applying the gaussian kernel to input coordinates (mentioned in Figure 6. of the manuscript and also explained in Sec E of supplementary material)?
2. Can we apply both techniques in the training stage for progressive learning (coarse to fine)? Is this beneficial for PREF to express multi-scale signals without aliasing effect as BACON did (similar to Fig.4 of BACON's manuscript)?
3. What is the role of alpha of PREF?

Sorry for asking a lot of questions. I am looking for the possibility of whether this work is the voxel-based nerf that can continuously encode signals of multi-scales (For example, by varying sigma in Gaussian Filter). I am also curious about your personal opinion regarding this.

Thanks a lot!

Question about the paper

Hi! Thank you for such interesting work and for sharing it! May I ask you a question about the paper: is it any special meaning behind d? or it is just a typo and d=n always. Because I expected P to be of dimension k x N^n = k x N x N x ... x N