- We build the initial geometry with the 1st stage of DVGO in our implementation by default, which is
use_geo = -1in config files. - The geometry can be either initialized online (by default) or from other sources in
.txtform, which can be enabled withuse_geo = 1andpointfile = /your/file.txtin config files. - You may ignore
preprosessingfolder, which is initially for our ealy trying and not used here. - You may refer to the comments in
./configs/synthetic-nerf/default/chair.txtfor the usage of hyperparameters.
For Synthetic-NeRF dataset, we provide the initial geometry from DVGO (which is the default one in our implementation) and from MVS. Feel free to try both (e.g., use_geo = 1 and pointfile = /your/mvs_file.txt) to see the comparison.
For Scannet dataset, we use the initial geometry provided by the dataset itself. We convert the original .ply file into .txt and you may download from here.
All the codes are tested in the following environment:
- Linux 18.04+
- Python 3.6+
- PyTorch 1.10+
- CUDA 10.2+
And the layout should look like this:
Strivec
├── data
│ ├── nerf_synthetic
│ │ |──default
│ │ │ |──chair
│ │ │ │──drums
│ │ │ |──...
│ │ |──local_vm
│ │ │ |──chair
│ │ │ │──drums
│ │ │ |──...
├── scene0101_04 (scannet)
│ │ │──exported
│ │ │──scene0101_04_2d-instance-filt.zip
│ │ │──...
├── scene0241_01 (scannet)
│ │ │──exported
│ │ │──scene0241_01_2d-instance-filt.zip
│ │ │──...
├── TanksAndTemple
│ │ │──Barn
│ │ │──Caterpillar
│ │ │──...
├── 360 (Mip-NeRF360)
│ │ │──garden
│ │ │──room
│ │ │──...
We not only provide the training and evaluation code to reproduce the results in the paper, but also the code of ablation that uses local VM tensors instead of local CP tensors (results are here).
# hierachical Strivec, without rotation (grid aligned)
python train_hier.py --config ./configs/synthetic-nerf/default/chair.txt
# local VM tensors instead of local CP tensors
train_dbasis.py --config ./configs/synthetic-nerf/local_vm/chair.txt
We visualize the local tensors of different scales into ./log/your_scene/rot_tensoRF/0_lvl_k.ply, where k is the kth scale.
Here is a toy example to illustrate the TensoRF-CP (TensoRF-VM is similar) with global decomposition in (left) axis-aligned and (right) non-axis-aligned situations. The bottom shows the grid values. In axis-aligned case, only 1 component is needed to represent the scene (vector bases recover grid values by outer product). In non-axis-aligned case, however, 3 components are needed because the rank of matrix changes from 1 to 3 after scene rotation. While our design with local low-rank tensors can alleviate this issue, i.e., local tensors (2*2) are always rank-1 before and after rotation.
If you find our code or paper helps, please consider citing:
@INPROCEEDINGS{gao2023iCCV,
author = {Quankai Gao and Qiangeng Xu and Hao Su and Ulrich Neumann and Zexiang Xu},
title = {Strivec: Sparse Tri-Vector Radiance Fields},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
year = {2023}
}
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent