stevehan00/pointrix

Pointrix: a differentiable point-based rendering libraries supporting 3D Gaussian Splatting and beyond

A differentiable point-based rendering library.
Document🏠 | Paper(Comming soon)📄 | DPTR Backend🌐

Pointrix is a differentiable point-based rendering library which has following properties:

• Highly Extensible:
  • Python API
  • Modular design for both researchers and beginners
  • Implementing your own method without touching CUDA
• Powerful Backend:
  • CUDA Backend
  • Forward Anything: rendering image, depth, normal, optical flow, etc.
  • Backward Anything: optimizing even intrinsics and extrinsics.
• Rich Features:
  • 3D Reconstruction: Vanilla 3DGS (Siggraph 2023), Relightable 3DGS(WIP)
  • 4D Reconstruction: Deformable 3DGS (CVPR 2024), Gaussian-Flow (CVPR 2024, WIP)
  • 3D Generation: GSGen (CVPR 2024, WIP), DreamGaussian (ICLR 2024, TBD), LGM (arXiv 2024, TBD)
  • 4D Generation: STAG4D (arXiv 2024, WIP), DreamGaussian4D (arXiv 2023, TBD)
  • ...

Quickstart

Installation

Clone pointrix:

git clone https://github.com/pointrix-project/pointrix.git

Create a new conda environment with pytorch:

conda create -n pointrix python=3.9
conda activate pointrix
conda install pytorch==2.1.1 torchvision==0.16.1 pytorch-cuda=12.1 -c pytorch -c nvidia

Install DPTR backend and dependencies:

# note: can also switch to the original 3DGS diff-gaussian-rasterization 
git clone https://github.com/pointrix-project/dptr.git --recursive
cd dptr
pip install .

# Install simple knn
git clone https://gitlab.inria.fr/bkerbl/simple-knn.git
cd simple-knn
python setup.py install
pip install .

Install Pointrix:

cd pointrix
pip install -r requirements.txt
pip install -e .

Train Your First 3D Gaussian

NeRF-Lego (NeRF-Synthetic format dataset)

Download the lego data:

wget http://cseweb.ucsd.edu/~viscomp/projects/LF/papers/ECCV20/nerf/nerf_example_data.zip

Run the following (with adjusted data path in the config):

cd pointrix/projects/gaussian_splatting
python launch.py --config ./configs/nerf_dptr.yaml

# you can also run the following for the original 3DGS kernel
# python launch.py --config ./configs/nerf.yaml

Mip-NeRF 360 (Colmap format dataset)

Download the data and run:

cd pointrix/projects/gaussian_splatting
python launch.py --config ./configs/colmap_dptr.yaml

# you can also run the following for the original 3DGS kernel
# python launch.py --config ./configs/colmap.yaml

Advanced Approaches

Deformable 3D Gaussian

1. Download the iphone dataset and put it in your folder: https://drive.google.com/drive/folders/1cBw3CUKu2sWQfc_1LbFZGbpdQyTFzDEX

2. Run the following command to train the model (...data path in the config file...):

cd pointrix/projects/deformable_gaussian
python launch.py --config deform.yaml

Gaussian-Flow (WIP)

GSGen (WIP)

STAG4D (WIP)

Relightable 3DGS (WIP)

Release Plans

• GUI for visualization (this week).
• Implementataion of Gaussian-Flow (CVPR 2024) (this week).
• Implementataion of GSGen with MVDream SDS (this week).
• Camera optimization (this week).
• Introduction video
• Implementataion of Relightable Gaussian (arXiv 2023).

Welcome to discuss with us and submit PR on new ideas and methods.

Contributors

Made with contrib.rocks.